Just after peak load today, at decidedly non-record concurrency levels, the grid suffered some fairly severe issues for the space of half an hour. These issues have returned to us, in the last 20 minutes (commencing at approximately 9:50pm SLT). Residents are reporting object and inventory issues, asset loading problems, upload problems, failed teleportations and sim border crossings, and the failure of map search or other search functions. Additionally the population data feeds began to perform erratically, pointing towards backbone overload.The earlier incident sparked no acknowledgment from Linden Lab. Things have been problematic for only 20 minutes now, so it may be a bit too soon to expect Linden Lab to respond. User concurrency was at 27,113 at the start of the troubles, and has fallen by over 2,000 since then.
[Update. LL's Chadrick Linden responded with a blog post at 10:30pm SLT, acknowledging the problem - well, at least the search part of the problem. As of 10:55pm, search remains non-functional, and other problems including payments and account balances remain sporadic.]
[Update: 11:51 PM - It turns out fixing search is going to take more than initially thought, it will probably be down for most of the night. We apologize for the inconvenience and will have a team of crack Operations people patching it up in the morning.
I presume that means they're very good, rather than they are too busy making crack to work on the problem ;) ]
On the 8th of March we sought clarification for an ambiguous-seeming sentence at the end of a warning notice that was being distributed by Chadrick Linden. That sentence was:
Chadrick Linden shouts out that a rolling restart of all regions is about to commence in order to "fix some issues with our backend". What the issues are, exactly, aren't specified, nor is what changes have been made. When I've got issues with my own backend, I find a large warm glass of salty water does the trick. The restart should commence on or about 8PM SLT.
