Why do Second Life simulators lag? How can you tell when the lag is at your end and the lag is at the server end?

Under The Grid is a new series looking at the mechanics underneath Second Life, and how it functions. This week, we're going to take a look at simulators and what we know about lag.

There are several pieces to the lag equation. Your hardware (we'll just call it your computer), your viewer (we'll call it the client), your network (this includes whole chunks of network that belong neither to you nor to Linden Lab), sims (short for simulators - those square areas on the map that are sometimes also called regions) and the servers that they run on. There are a lot of pieces there. There are even more that are involved in the running of the grid, but we'll talk about those another time, along with non-simulator causes of lag.

Firstly there's a separation of powers between the viewer (also called the client) and the grid. Some things may cause work for the client that don't cause work for the sim. Flexible prims and lighting are a good example of this. The sim doesn't care what's flexible or not, because at the sim end those prims are rigid and intangible. The client is responsible for giving them the illusion of motion. A lot of rotation scripts are the same -- the sim doesn't really pay attention to those unless it absolutely has to, and most of the time it just doesn't have to. Your client handles that too. Running scripts and physics, however, all takes place on the sim.

Like your computer, the server that the sim is running on can only do so much work. It has a finite capacity. It has an internal clock that ticks 45 times every second, and on each tick it handles all the work that needs to be done to update everything since it finished processing the last tick. These ticks are called frames, and are measured in frames-per-second, but they're not the frames-per-second that you might be used to in video games. They're not video frames, they're just slices of processing time.

A little arithmetic says that to keep up with a schedule of 45 updates per second, the amount of time spent on an update can't be more than 0.0222222 seconds. In more comfortable terms, that's 22.3ms (milliseconds) each, when rounded. The question is, what is a simulator doing? Is it keeping up? We can find out.

While you are logged into Second Life, press SHIFT CONTROL 1 and a statistics panel will pop up on the right hand side of the screen. In future updates, we'll be talking a lot more about what all this actually means to you. Some of it's of no particular use to anyone but the aggressively technically-minded, but a lot of interested people can find useful tidbits here. A lot of this information is more generally useful, and shouldn't just be limited to the deep geeks.

Much of this panel can be folded away or expanded. SHIFT CONTROL 1 will dismiss the whole panel again, and get it out of your face.

The section we're particularly interested in today is the one marked time (ms) at the bottom. It's currently folded away. Click on that line to expand it. (Tip: The statistics panel may not be very responsive if your computer is running a bit slow. If it fails to unfold, try clicking again, holding the mouse-button slightly longer)

You'll see something that looks rather like this here:


This tells us what the sim is doing, and how long it's spending doing it.

The Total Frame Time is 22.4ms. Right there, we know that the sim is lagging a little. 22.3ms means the simulator is running at capacity. As soon as it's finished processing a frame, it's time for the next one. Here, the sim is running a little overtime, using 0.1 milliseconds more time on each frame than it should. Not a recipe for disaster, but this is where it starts to lag.

If the number is 22.3ms or less, then the simulator isn't lagging, simple as that.

The numbers below that figure tell us what the sim is spending time working on. Some of these are obvious, and some have had little public information published about them.

Net Time is an obvious one. That's the amount of time the simulator spends communicating via the network to other sims and servers, and to your client and computer. Shovelling network data around, essentially. (Note: Apparently due to the way the network code is architected -- or at least how it was when this was last discussed -- network time drastically increases if there are slow clients. If there's a dialup user in your sim, performance of everyone else in the sim may be impacted. That's how it was, anyway. Perhaps this has been changed)

Sim Time (Physics) is all about the Havok 1 physics engine (remember, work on Havok 2 got suspended -- perhaps indefinitely -- earlier this year). This number deals with the amount of time the sim spends calculating real or potential collisions between objects and avatars. Avatars use a certain amount of calculation time just by being in the sim and moving around. You might not bump into anything or anyone, but the sim has to be able to figure out if you did or not. Cases where a physics-enabled object is jammed against or through something else cause this calculation time to skyrocket.

Sim Time (Other) is a bit more obscure. This involves the processing of agent movement (that's you and me), handling the weather/cloud systems, and the position of the sun and the moon. It rarely amounts to a significant portion of the Total Frame Time itself.

Agent Time is the time spent transmitting object update to agents. Prim data and positions, attachment data and positions and so forth. Also the time spent transmitting the network data for this is counted here.

Images Time is the most obscure. There's very little useful information on this figure. It seems to have a lot to do with hard disk access -- it's prone to taking huge jumps when the sim is performing a database backup. You'll also see this number climb if another sim on the same server becomes overloaded (Total Frame Time out of control). Images time is also for well...images. For processing textures and transmitting them to your client. The cost of transmitting network data related to texture transmissions is allocated here.

Script Time is the last one, and usually the largest factor. This is how much time the sim spends paying attention to scripts. Not necessarily actually running them, but managing their memory, processing potential events for them, determining if a listening script has heard something, and all of that. There's no hard and fast rules for reducing the lag in scripts. Some busy scripts with lots of timers, sensors and listens have very low impact on overall simulator lag, while some very simple scripts that take pains to avoid these can be utter lag monsters. When it comes to lag, it's more about art and experience rather than simple rules like 'no listens' or 'no sensors' or 'no timers'. Each of these systems can be used well -- or used badly.

In summary, you should now have enough information that you can tell what the sim you are in is doing, at a glance, and how badly it is performing. As a rule of thumb, even lightly overloaded, a sim still performs fairly well until the Total Frame Time exceeds about 25ms. Then simulator lag really starts to become noticeable.