Welcome to the fifth installment of "Under The Grid", an irregular look at the mechanics underneath Second Life. This time, we're looking at the statistical data reporting from Linden Lab.

As you've probably noticed, neither Eloise nor I have been able to produce trustworthy numbers in the daily wrap reports. There's a reason for this. The numbers being published by Linden Lab on their web-servers are so much garbage.

Now, they've not said why, but there's a bit that can be inferred from the technical problems that they've had this week (plus a little practical investigation), particularly since the issue started. What looks like being a large chunk of the problem now is down to load balancing vs currency.

You see, the web-server you and I talk to over at http://secondlife.com is not just a single web server. There's more than one. They use an HTTP proxy called squid (it's appears to be version 2.6-STABLE12, but likely slightly modified) to take data from a lesser number of servers in the back room, and feed it to you. More servers handle more load, and even a little bit of content caching from squid helps distribute the load.

Now things get a little more complicated. There's more than one front-end webserver. There's more than one back-end webserver. We're not really sure how many there are of the front-end servers, but the back-end servers are easy enough to count. Some of the content changes rarely. Squid can cache that and serve it up to you time after time, and it still seems hot and fresh.

Then we've got more dynamic data, like the statistical feeds. Something, somewhere is caching the data, and feeding the same set of bunk back to the front end, time and time again. It doesn't actually look like it's squid's fault at all. Certainly not the front-end. Let's look at the back-end.

Pick an answer, any answer.

About two minutes of investigation shows the data is drawn from a back-end set of servers. Sixteen servers that are publically accessible, if a trifle slow to respond. What answers you get depends a great deal on who you ask. Mostly, they seem to be feeding back different answers, and stubbornly refusing to budge. Wherever they are getting their numbers from, they're holding on to bad data. Plus (and here is the kicker) they're not providing a correct set of cache-coherence headers. Likely the source they're getting the data from isn't either.

Cache coherence headers are transactional level meta-data used in HTTP to advise how the content should be handled. Is it fresh? Can it be stored? How long for? Can it be given to anyone? Or only to a specific person? Cache-coherence headers provide all this information. It's like an extended list of FDA information and a best-before date on your groceries.

When content is no longer fresh, the cache should throw it away, and get fresh data. If it's told not to keep the data at all, it shouldn't.

What appears to be happening here is that the second-layer webservers are not providing any cache-coherence data, giving squid no clues as to how long the data should or should not be kept. That seems to be only fair as each of those servers doesn't appear to be obtaining that information either. As a result the second-layer servers seem to be holding bad data for long periods of time and serving out conflicting answers to whichever first-layer server asks them. (Heck, one of the second-layer servers isn't answering at all). Once in a while, a number that looks plausible leaks through the system, but that isn't very often.

There may be additional grief going on behind the second-layer servers as well, but right at this moment, that seems to be the apparent cause of the silly numbers we're getting.