At 8:30AM SLT (US Pacific) all Second Life login services ceased responding.

At the time of the fault, 41,337 users were online. With users unable to log on, concurrency numbers have been falling rapidly since that time.

[UPDATE: 8:45 AM - Linden Lab acknowledges the problem, and promises further updates]

[Update: 9:05 AM - Logins are back up]

[Update: 9:15 AM - Logins remain up. Search has failed]

[Update: 10:00 AM - Search started responding to queries again about 5 minutes ago, and is hanging in there.]

At 9:30AM SLT (US Pacific) the login system began to show signs of strain, with many logins reportedly starting to fail. At 9:44AM SLT with 40,854 users online, the login system showed signs of failure. There are also reports coming in of the 'double teleport' problem, and some failures rezzing object from inventory (keep those no-copy assets in your inventory where they are safe. Don't try to take them out).

Concurrent logins are falling sharply. At present there is no word from Linden Lab.

[Update: 10:20AM - Linden Lab have acknowledged the problem and are starting to investigate]

[Update: 10:50AM - The issue is reported as resolved, without explanation. It seems likely that the cause is the same as the problems yesterday]

[Update: 10:00PM - The grid has been showing signs of instability and login problems again for the last hour. We're getting a number of reports from people who are suddenly crashing immediately after connecting to a simulator, and there's been a couple sudden drops in overall concurrency.]

Linden Lab have apparently finally started noticing the login problems and are looking into them.

Logins are closed for the duration - from 10:20PM SLT (US Pacific). If you're currently logged in to Second Life, and you've got plans this-evening, you probably want to avoid logging out. At this stage, no information is available as to the cause.

[UPDATE: 10:40PM SLT - Logins have reopened. It remains to be seen for how long. No update on the Linden blog yet]

[UPDATE: August 11, 2007 @ 12:21AM - Linden Lab reports that the problem was due to a hard-drive failure and has now been resolved]

At 1:15am SLT (US Pacific) this-morning (and presently ongoing) login servers for the Agni (main) Second Life grid stopped responding to requests from all comers. Basically login services for the main grid are down. The grid status page isn't helping much as it displaying an internal error. Doubtless something has gone wrong in the web-services backend.

No word from Linden Lab at this time. We'll keep you updated.

• [UPDATE: 1:45AM SLT - The status page is now showing that the the grid is open, instead of an error message. Logins are still failing]
• [UPDATE: 1:48AM SLT - Login servers are beginning to accept requests]
• [UPDATE: 1:49AM SLT - Linden Lab says that logins "will be" (future tense) disabled to work on some database problems. Odd that the message comes just as login services are restored.]
• [Update: 2:04AM SLT - Logins remain up. Linden Lab reports that logins are now up again. Somehow, I think the blog notices happened after all the work was actually over.]
  

A lovely roundup for your Tuesday morning on the grid. Linden Lab has finally put forward some enlightening information about the recent issues with the grid, claiming a combination of a poorly-thought-through configuration for database systems.

Surely, the prolonged attack over the weekend contributed to overall system strain, but apparently it was far from the whole story. An interesting question is why, if the problem was identified as a system configuration issue 24 hours ago, didn't someone actually say so at the time? Reconfiguration work has been in progress for a full 24 hours without any word until now.

Blue Linden reports that the Second Life grid is experiencing a presence issue. If you log out, you may not be able to log back in, because - the system hasn't apparently noticed that you're gone yet. The advice is that if you are unable to log back in, wait 30 minutes then contact Customer Support.

The presence system has had a bit of a fraught history, particularly since the day Second Life 1.9 was deployed. Formerly teleportation, sim-border-crossings, instant messages, group invitations and vote-proposals were handled through central servers. Then those servers started to get busier and busier. It was decided, in the interests of scalability, that sims should communicate with each-other in a more direct fashion.