Starting at 12:16am SLT (US Pacific) the grid began showing signs of difficulty with some residents complaining of land parcels mass ejecting them and objects being returned to them spontaneously from their land.
By 2:30am, the map, dataserver, profiles, and most teleports had ceased to function. Second Life residents are largely confined to conventional travel - except when they're being ejected from their land.
Some residents say they have filed support tickets as long ago as two hours, but as yet there is no acknowledgement from Linden Lab that they are aware of the widespread issues.
The Second Life grid had 'a bit of a flutter' from 9:30PM SLT (US Pacific) to 9:40PM SLT. During this period, people report Linden Dollar balances not loading, objects failing to rez, search failing, inventory not loading, some failed HTTP requests and outbound emails and the map and teleports unresponsive.
Over the last five minutes things have been stabilizing again. No word from Linden Lab yet that they are aware of the problem.
[UPDATE]
[10:00PM] - Linden Lab have acknowledged the problem, and identified it as packet loss on the (Level 3, presumably) network between the California and Dallas colocation facilities, and are investigating.
At 1:26am SLT (US Pacific), with a concurrent population of 24,831 inworld search and asset systems began to fail, rapidly becoming non-responsive. There area also reports about teleport failures, and Linden Dollar balances not loading, the map not working, and the inability to add or remove attachments.
As a bonus, the Second Life Support Portal doesn't seem to be working (reporting 'invalid querystring') either. There is, as yet, no word from Linden lab about the issues.
[UPDATE]
At 2:16AM, Linden Lab acknowledged the problem, indicating that the Second Life website was giving troubles from approximately 15 minutes before the inworld issues began.
While the grid and the website have apparently both recovered, Linden Lab says they are investigating the cause.
[3:00AM] The grid continues to behave in a less than stable fashion. Linden Lab says the inworld search system is going to be taken offline while the investigation continues.
[4:00AM] Linden Lab reports that everything is back to normal - reports from users seem to indicate the same thing.
Starting around 1AM SLT (US Pacific), the Second Life Wiki, the Public JIRA and the Email-to-IM gateay were restored to service, however around the same time asset issues began cropping up with all the usual symptoms you expect to see from that (objects not appearing, assorted error messages when attaching or rezzing objects, all of that). These problems persisted until just approximately 2:40AM SLT.
At about 2:50AM SLT, map region-searches, global teleports and data-server requests all began failing. Some inworld scripts complained loudly and repeatedly. The problem cleared up approximately ten minutes later, but it looks like we're in for a potentially rough day.
Around 4:30PM SLT (US Pacific) around a hundred simulators, apparently in the colocation facility in Texas went offline. These simulators remain offline though there's no sign of them being offline if you check the map - only if you try to teleport to one - so it's hard to spot.
Linden Lab has indeed spotted the problem, however, and suspects a problematic network switch, and tells us that operations is working on restoring the downed regions.
[UPDATE: 11:10PM SLT - Sims just started coming up, and Linden Lab pronounces the problem resolved.]
[UPDATE: 11:22PM SLT - The affected sims just went offline again.]
[UPDATE: 11:57PM SLT - Linden Lab acknowledges that the problem is back and now affects nearly 200 regions.]
Through the early hours of the morning, certain areas of the Second Life grid were reportedly experiencing asset, inventory and teleport problems, though the rest of the grid was operating alright at the time. A 1:05PM SLT (US Pacific time) just before peak period, the grid fluttered apparently either dropping several thousand users, or spewing false data through the statistical feeds. Reports from Second Life users suggest the former. At 3PM SLT, just after a depressed and flattened peak-concurrency it happened again.
Just about twenty minutes ago, at 8:10PM SLT, roughly 900 simulators went down, dumping approximately 3,000 Second Life users at once. Linden Lab says they're working on this. There isn't any information presently available to tie these events together, but it certainly looks like they are connected. Linden Lab still hasn't given any explanation for the recurring problems over the weekend.
At 9:30AM SLT (US Pacific) the login system began to show signs of strain, with many logins reportedly starting to fail. At 9:44AM SLT with 40,854 users online, the login system showed signs of failure. There are also reports coming in of the 'double teleport' problem, and some failures rezzing object from inventory (keep those no-copy assets in your inventory where they are safe. Don't try to take them out).
Concurrent logins are falling sharply. At present there is no word from Linden Lab.
[Update: 10:50AM - The issue is reported as resolved, without explanation. It seems likely that the cause is the same as the problems yesterday]
[Update: 10:00PM - The grid has been showing signs of instability and login problems again for the last hour. We're getting a number of reports from people who are suddenly crashing immediately after connecting to a simulator, and there's been a couple sudden drops in overall concurrency.]
At 8:00AM SLT (US Pacific) this-morning, with 37,419 users logged into the Second Life grid, something began to go wrong. Reports began coming in of teleportation problems, difficulties accessing inventory, search failures, and difficulties logging in.
At 10:10PM things started to worsen and users began logging off in large numbers. Linden Lab shut down all search facilities at that time, which suggests that there are problems with load on the database systems. Recently this sort of load has been caused by everything from misconfigured systems to hardware failures, so at this moment there's not a lot of information to go on.
[Update: 11:36AM SLT - Linden Lab reports that the problem has been found and fixed, but isn't divulging the details.]
Logins are closed for the duration - from 10:20PM SLT (US Pacific). If you're currently logged in to Second Life, and you've got plans this-evening, you probably want to avoid logging out. At this stage, no information is available as to the cause.
[UPDATE: 10:40PM SLT - Logins have reopened. It remains to be seen for how long. No update on the Linden blog yet]
[UPDATE: August 11, 2007 @ 12:21AM - Linden Lab reports that the problem was due to a hard-drive failure and has now been resolved]
We're getting numerous reports (and some direct experience, I might add) of multiple bouts of login problems with Second Life since approximately 4:30PM SLT (US Pacific). Some people are reporting it taking 20-30 minutes to get a response from the early part of the login phase, with the viewer bailing out with it's rather uninformative "Despite our best efforts..."
No word from Linden Lab on these issues yet so far. We'll keep you posted if we hear anything about them.
At 1:15am SLT (US Pacific) this-morning (and presently ongoing) login servers for the Agni (main) Second Life grid stopped responding to requests from all comers. Basically login services for the main grid are down. The grid status page isn't helping much as it displaying an internal error. Doubtless something has gone wrong in the web-services backend.
No word from Linden Lab at this time. We'll keep you updated.
[UPDATE: 1:45AM SLT - The status page is now showing that the the grid is open, instead of an error message. Logins are still failing]
[UPDATE: 1:48AM SLT - Login servers are beginning to accept requests]
[Update: 2:04AM SLT - Logins remain up. Linden Lab reports that logins are now up again. Somehow, I think the blog notices happened after all the work was actually over.]
A lovely roundup for your Tuesday morning on the grid. Linden Lab has finally put forward some enlightening information about the recent issues with the grid, claiming a combination of a poorly-thought-through configuration for database systems.
Surely, the prolonged attack over the weekend contributed to overall system strain, but apparently it was far from the whole story. An interesting question is why, if the problem was identified as a system configuration issue 24 hours ago, didn't someone actually say so at the time? Reconfiguration work has been in progress for a full 24 hours without any word until now.
By approximately noon SLT (US Pacific) today, the issues that have been plaguing the grid have been easing - though there are still abrupt simulator stoppages, and simulators not properly restarting.
Linden Lab are reportedly still working around the clock on it, so obviously they know plenty that we don't. The question is, what, exactly? Even the raw technical information would be useful as there are people who can translate that into English for the rest. Without more detailed information there seems to be little hope of Second Life residents regaining their confidence in a hurry.
Well, the issues with the grid are now quite clear. The Patriotic Nigras (or someone claiming to be them - honestly, it's all so childish that it's hard to tell - most griefer groups spend more time wailing on each-other than anything else) have used a form of a privilege escalation exploit on a number of objects around the grid to overload the communications systems.
It's unlikely that the Nigras themselves are actually responsible for the work, as most of them are 'script kiddies' (people who cannot create or modify software) and this sort of thing seems to be well above their usual displayed capacity (usually limited to screaming obscenities, using cage-guns or employing the work of others). Most likely, their eagerness to cause trouble is being exploited by someone else.
Depending on who you ask, the grid's been having what some have described as 'a cow' for about 24 hours now. It's worked in patches, I am assured. Lots of people are asking questions. The basic sorts of questions that should have answers on the Linden blog.
What, actually, is the problem?
What is being done to fix it?
Is there any estimate on how long it will take to fix? (Hours? Days?)
What is the likelyhood of it recurring?
"We don't know" and "We don't know yet" are perfectly valid answers to any of these. Not necessarily helpful, but far more transparent than simply not giving any information out. A few hundred thousand people would like to know. Many of them are not afraid of purely technical answers, and are able to translate for others.
These answers though are the sorts of things you would include in any outage or problem notice. It shows the users that you give a darn about them and all that.
How would you rate Linden Lab's transparency when it comes to grid issues, insofar as delivering the sort of basic information that you need? (What's wrong; what's being done; estimated restoration time; odds of recurring or followon problems)
Starting at 12:16am SLT (US Pacific) the grid began showing signs of difficulty with some residents complaining of land parcels mass ejecting them and objects being returned to them spontaneously from their land.
The Second Life grid had 'a bit of a flutter' from 9:30PM SLT (US Pacific) to 9:40PM SLT. During this period, people report Linden Dollar balances not loading, objects failing to rez, search failing, inventory not loading, some failed HTTP requests and outbound emails and the map and teleports unresponsive.
Around 4:30PM SLT (US Pacific) around a hundred simulators, apparently in the colocation facility in Texas went offline. These simulators remain offline though there's no sign of them being offline if you check the map - only if you try to teleport to one - so it's hard to spot.
A lovely roundup for your Tuesday morning on the grid. Linden Lab has finally put forward some enlightening information about the recent issues with the grid, claiming a combination of a poorly-thought-through configuration for database systems.
By approximately noon SLT (US Pacific) today, the 
Well, the issues with the grid are now quite clear. The Patriotic Nigras (or someone claiming to be them - honestly, it's all so childish that it's hard to tell - most griefer groups spend more time wailing on each-other than anything else) have used a form of a privilege escalation exploit on a number of objects around the grid to overload the communications systems.
Depending on who you ask, the grid's been having what some have described as 'a cow' for about 24 hours now. It's worked in patches, I am assured. Lots of people are asking questions. The basic sorts of questions that should have answers on the Linden blog.
