At 1:26am SLT (US Pacific), with a concurrent population of 24,831 inworld search and asset systems began to fail, rapidly becoming non-responsive. There area also reports about teleport failures, and Linden Dollar balances not loading, the map not working, and the inability to add or remove attachments.

As a bonus, the Second Life Support Portal doesn't seem to be working (reporting 'invalid querystring') either. There is, as yet, no word from Linden lab about the issues.

[UPDATE]

At 2:16AM, Linden Lab acknowledged the problem, indicating that the Second Life website was giving troubles from approximately 15 minutes before the inworld issues began.

While the grid and the website have apparently both recovered, Linden Lab says they are investigating the cause.

[3:00AM] The grid continues to behave in a less than stable fashion. Linden Lab says the inworld search system is going to be taken offline while the investigation continues.

[4:00AM] Linden Lab reports that everything is back to normal - reports from users seem to indicate the same thing.

When the public issue tracker is having an issue how do you report the issue?

Right now, the public JIRA system is not responding to logins. More specifically it's rejecting credentials with the message "The credentials you provided cannot be determined to be authentic."

It isn't clear at this point whether the issue is simply that the Second Life authentication service is overloaded by everyone throwing themselves into the grid all at once, or if the Heterogenous Grid updates have changed the authentication protocol/format. A case could be made for the latter, given the problems with the Support Portal this-afternoon.

A lovely roundup for your Tuesday morning on the grid. Linden Lab has finally put forward some enlightening information about the recent issues with the grid, claiming a combination of a poorly-thought-through configuration for database systems.

Surely, the prolonged attack over the weekend contributed to overall system strain, but apparently it was far from the whole story. An interesting question is why, if the problem was identified as a system configuration issue 24 hours ago, didn't someone actually say so at the time? Reconfiguration work has been in progress for a full 24 hours without any word until now.

The Second Life grid is currently closed to logins, due to problems with electrical supply. Electrical jitters affecting Linden Lab don't affect the grid per se, so much as they do support (and indeed, the support portal is down, along with the support lines to Linden Lab. It has also hampered certain processing tasks that take place outside of Second Life, like the processing of stipends and group payments.

What it is also affecting this time is the colocation facilities in San Francisco that host many of the Second Life servers themselves. Basically things started to go awry at 2:06pm when, with 46,125 users online, some grid systems were affected by power issues. While there are still around 14,000 people logged in, the word is that things seem to be pretty shaky on the grid still, and with the electrical supply problems apparently over, Linden Lab are cleaning up the mess. At this stage, Linden Lab have not advised when they expect the grid to reopen for service.

We are told that typepad, technorati, CNet and Craigslist have also been partially or completely knocked out by the electrical woes.

[UPDATE: 5:48pm SLT (US Pacific) - The Second Life grid is provisionally back online, though still not in top shape]

The support portal is going down for maintenance again (for what is hoped is the last maintenance downtime on a weekend) from 6PM Saturday SLT (US Pacific) to 2AM Sunday SLT (7-8 July).

That's no support tickets, no support live-chat and no knowledge-base during this period, so try not to have any problems, okay? Concierge customers and estate managers who need rollbacks during this time will have to visit the Linden Estate Services office in virtual person to make their request.

Remember, Second Life will be offline tomorrow morning from 6am to noon SLT (US Pacific), while the grid is updated to 1.18.0, which implements the key pieces of Linden Lab's 14 month old "Message Liberation" project that kept coming up in so many meetings.

In a nutshell, Message Liberation changes the communications format to one that is supposed to be both faster, less buggy and more versatile. Faster is good. We like faster. Less buggy is good too. Versatile in that it allows more decoupling of the viewer and the grid, hopefully spelling an end to major update outages sometime in the not-too-far-distant future (in all but the most major cases), and allowing different sims to run different software.

The Support Portal is also going down tomorrow from 8am to 11am SLT, which I think you'll agree is a better time than Saturday night - though it's so far been nice to see all the extra time that Linden Lab has during an update to answer emails, handle support, and process abuse reports.

Last year a series of meetings were held with interested users to help shape and feed back on Linden Lab's support systems. Well, they're back again. The first of a series of meetings will be on June 28, at 11am SLT (US Pacific), and will be all about the support portal. Further meetings are apparently to be held monthly.

Only 30 people will get to attend, so you should IM Cyn Linden or Fritz Linden to register your interest in, oh, probably the next ten minutes is all it will take to fill up all the seats. If you've got an interest in improved support (and face it, who hasn't this week?) and can make the timeslot, get your IM window open and get yourself an invite.

Linden Lab's support portal (which as you've probably heard is provided by a third party vendor) recently was down two consecutive Saturday nights for unspecified (but planned) maintenance.

The support portal is also strongly linked to a bulk password reset recently, information about which Linden Lab is still not talking about, which has apparently left some users unable to restore access to their accounts, and many hesitant to use it at all.

Right now the support portal is down for reasons unknown. Linden Lab is seeking information from the vendor at present to determine cause and restoration time. Until then, it's phone support only, as all email support was closed down when the portal opened, and no inworld support is available.

[UPDATE: After approximately 4 hours of downtime, the support portal is back up, without explanation.]

Please, stop emailing me about being unable to reset your password. No, really. I understand and sympathize with your frustration but I can't really help even one of you out, let alone hundreds of you. We, here at SLI, are not associated or affiliated with Linden Lab, and are not able to provide support for your account or password difficulties. That said, we can offer a little advice.

First, anecdotal evidence suggests that when you are prompted for credentials, if you answer zero to "How much did Linden Lab last bill you for?" that your account will be locked/suspended. That's what we are hearing, anyway. If that happens, you need to get in touch with Linden Lab.

Cyn Linden reports that the third-party who provides the brand new support portal for Second Life is having a bit of downtime during the next two weekends. Specifically Saturday night (9pm to 2am SLT) 2-3 June, and the following weekend (9pm to 6am SLT) 10 June.

Technology is awesome.

We're assured that Lindens will be closely monitoring the grid for trouble, but I pity any Linden who appears in public during those period. They'll likely be flooded.