Anatomy of Lag
Posted on June 22, 2009 by Gwyneth Llewelyn | 90 Comments
Hi! Ana Lutetia has kindly invited me to post on her blog, so don’t start wondering about the abrupt change of style (double-pub intended) 
Rest assured, I’m not going to pester you with any fashion tips *grins*
Instead, for this blog post, I’m trying to cover some of the usual myths related to lag, what causes it, and what most definitely won’t make it go away…
The Grid Is Misbehaving — What Shall I do?
It’s a weekend where Hair Fair ’09 is in full force, and builders are happily finishing their work for SL6B. Guess what? Nothing could make Linden Lab more happy than having half the grid down, going through “emergency maintenance”, and dealing with the painfully long list of complains that dozens of thousands of residents are submitting…
There is nothing else but to be patient (Linden Lab will finish their work when they can… they’re not exactly asleep 
), and in the mean time, what are residents doing in-world in the very crippled grid?
They accuse each other of creating lag.
This is a typical pastime of the Lag & ARC Nazis. When things start to fail, specially in very popular events or highly-crowded areas, it’s far better than to blame everybody else than to deal patiently with the issue. However, you can have fun blaming everybody else, but that won’t fix the problems. Really. Let Linden Lab do their work. Eventually things will get back to normal. Blaming everybody around you for either a super-crowded event or a failing grid will not make it happen faster. You will just get angry and get other people angry. And when we all are furious with each other, we react irrationally, and lose all the fun we have with Second Life® — trust me, that’s the last thing you wish to happen.
The Lag Myths
Let’s set the time clock a few years… six or even seven. Back then, Linden Lab had a handful of servers and the grid had just 16 sims or so. If a dozen people stayed in the same place and just chatted, sitting on the ground, the sims would not only lag, but crash — both your SL client and the sim you were on. People tried to break records — who could join the highest numbers of avatars in the same place and stay online the longest? This was usually measured in minutes, not hours or days
And of course a few would have found a trick. If you’d go in with your Linden skin, no attachments, and cleared the sim from all prims, textures, and scripts, it was quite likely they could all stay in-world for longer before the sim crashed.
Why was that so? It takes two to tango — the sim server (what Linden Lab hosts at their side) and your SL client application. Each contribute to render this wonderful virtual world. But each does quite different tasks. Each has also limitations, and that means that when you hit those limits, things start to break apart.
In the past, the sim servers behaved as if they could do just one thing at a time (which is, of course, an oversimplification). They sent each avatar prim data, and, most importantly, textures; they tracked down where on the sim your avatar was; they ran scripts; they handled physics (this does not only mean the cute physical-enabled objects — like vehicles! — but also much simpler things like knowing if your avatar was walking on the ground). The problem was that when one of those things was being overburdened for some reason, the whole sim was laggy, and this influenced all avatars on it. Similarly, your own SL client would first attempt to load all prims and all textures before showing you anything at all — even those tiny nanoprims on an earring that was 200 m away.
No wonder it was a grey world.
On those days, four or more years ago, people had only one way to deal with lag: make avatars as simple as possible; get rid of all attachments; stop scripts in the sim; keep event locations as simple as possible (cubes with blank textures); and hope for the best. The truth is, all these things actually helped. The technology was so little advanced that “keeping it simple” really improved the viewing experience. Even turning off title tags did have an effect on reducing lag!
Second Life in 2009 — Myth, Etiquette, and Anger
Well, things have changed since those days, and they have changed dramatically. Since SL pretty much looks the same — it just looks better, but not different — it’s natural that people wrongly assume that “the old methods of dealing with lag” still apply. They don’t, and they have little relevance to how SL actually works.
First and foremost, on the server side, the sims are now doing things in parallel much better, and allocate different priorities to different things. The largest impact has been on scripts. Scripts will only run when nothing else is to be done — this means they run at the lowest possible priority, and the more scripts in the sim, the slower they will run — without affecting texture download or avatar movement whatsoever. This is a crucial change. You might have noticed that sometimes your favourite AO or Dance HUD will respond slowly when you touch its buttons, on a very crowded event. That’s just the way things work now. First and foremost, you’ll get avatar data (their position), shortly followed by their shape, body textures and attachments. Then you get the sim’s prim data in your neighbourhood (it makes no sense to get you all those nanoprims from someone’s earring half a sim away, if you can’t see them anyway), as well as only the textures you can actually see from your viewpoint. And only then will the sim start to run scripts on your behalf. Having more scripts will not lag the sim: the scripts will only run slower. In fact, sometimes it might look they’re not running at all, since most of the time the sim will be happily sending you textures.
So, turning off your AO, entering “sleep mode” on MystiTool, or detaching everything with a script in it will make no difference at all. Granted, if everybody is wearing a thousand scripted attachments, they will notice that their attachments work very, very slowly, or possibly not at all. But they’re not lagging the sim.
Why do people still worry about how many scripts are running on the sim? Well, you can imagine that a slow-running vendor script is a problem: you pay to it, and have no way to know when your item is going to be delivered: either it happens instantly (on a fast sim!), or it can take minutes (on a very crowded shop). So keeping the number of running and active scripts down is important because they might be running too slowly to work at all — and when you’re dealing with shops and money transactions, this is a problem. So, although it’s usual that people host mega-parties to launch their shops (or launch a new line of products in an existing shop), this is actually not such a good idea: people will enjoy the party, but they won’t be able to shop on script-enabled vendors. A good alternative is simply to set prims to sell content (without any scripts) because these won’t get lagged; a clever shopping area designer will make sure that the place where the event takes place is not close to any shop (let them be on neighbouring sims) or that all shops in the neighbourhood don’t use script-based vendors. Not because of the lag; but because they might be simply working too slowly.
Nevertheless, technical reasons are not always acceptable to the majority of the SL users. Having in mind that it’s easier to blame others when things don’t work, than try to understand what is happening “under the hood” (because it’s just magic…), an etiquette has slowly evolved over the years. Sadly, it’s an etiquette based on superstition and magic, and not on scientific facts; in spite of that, breaking the etiquette is simply bad manners.
Suppose you invite a Jewish family to your dinner. They will be seriously offended if you don’t offer them kosher food but insist that sliced ham is perfectly reasonable and healthy to eat; it’s just their prejudice and superstition that makes them believe that pork is “impure” or “unhealthy”. In fact, in Palestine 1000 BC, conserving pork meat was hard, and it spoilt too quickly, compared to other types of meat; so it was perfectly reasonable to create a superstitious rule that told them not to eat pork “because God doesn’t want it”. In fact, it was pretty sound advice — for 1000 BC. In the 21st century, of course, we have refrigerators and vets monitoring the health of pig farms, and sterilised procedures to deal with ham manufacture. Thus, a 3000-year-old superstition (which originally was founded on a real problem!) doesn’t make any sense today, does it?
Well, no. But it’s still not polite to offend other people who believe in silly superstitions. Etiquette, or the notion that to live better in a more friendly society we ought to adopt a polite form of addressing others, and tolerating their quirks and superstitions, tell us that we would be very, very rude if we offered non-kosher food to a Jew. So we don’t. We know it’s silly. Even most of them will know it’s silly, too. But none of us violate the social norms that rule our society.
Similarly, in Second Life, a lot of etiquette rules have popped up, many of which making sense in the remote past, but not any more. Turning AO offs will not reduce lag, but people still believe they do. So, even if that’s a stupid superstition, it doesn’t mean we ought to be rude and offend them. We can be politically correct and accept that their erroneous views on how SL runs truly don’t affect us; we can live without our AOs for a few hours and be seen as polite and respectful and tolerant towards others.
Nevertheless, as time goes by, the myths self-perpetuate, and it’s a pity to see that more and more of our social conduct is based upon superstition than on cold, hard facts. Here are some of them.
Lots of prims create lag, so let’s build open-space!
It’s true that the more prims you got on a place, the more textures your SL client has to download, so that means things will take longer to rezz. Most people got that right, because it’s obvious
What they fail to understand is that nowadays the SL client uses a very aggressive method to just download what it needs. This is called occlusion — a big object in front of smaller ones will make the ones behind not visible, so it’s pointless to download textures for them. Similarly, you don’t need much detail to view objects further away, so the SL client doesn’t request much information for it either.
On the server side, a similar process also helps. The sim will keep an interest list on behalf of each avatar. This is mostly what is in the immediate neighbourhood of the avatar (other avatars and their attachments, prims, textures). So only these get sent — and it depends on the setting you have for Draw Distance on your SL client. The lower it is, the smaller the interest list. By matching all three approaches — just sending items on the interest list; just requesting objects you can actually see; not requesting much information from objects a long distance away since you won’t see more than a few pixels anyway — this will mean that the amount of content transfer between the SL client and the sim will be much, much reduced.
So what should a good shop designer do? Avoid open space! If someone drops in a sim with a Draw Distance of 256 m, it means that the interest list for that avatar will cover the whole sim (these days, with more and more advanced graphics cards, this setting is often the default — and most people don’t know, or don’t want, to change it). If there is nothing in front of the avatar, it means there will be no occlusion, and thus the SL client will request everything in sight — contributing to a huge spike in content transfer requests, and oh yes, these will lag the sim.
Now imagine 60 or 100 avatars all jumping to a hugely-packed shop at the same time. All those residents will be downloading an insane amount of content, all at the same time. That definitely lags the sim a lot!
A much better strategy is not to build open space shops. Create partitions and rooms — make them big enough to allow for easy movement around (remember the camera!) but also artificially enhance the optimisation mechanisms built in SL to take advantage of occlusion and interest lists. Thus, if none of the “rooms” in your shop is bigger than 64 m — which is large enough — you can tell people to keep their Draw Distance at that level. They will still rezz everything on sight — and when moving to a different room (or cornering a partition) they will start just to download that content, not “everything else”.
Similarly, partitions and rooms allow for avatars not to be in plain sight of each other all the time. A clever approach, if you own the sim, is to have several entry points and disallow direct teleport. That way, you can set the telehub from the Estate Tools to have multiple landing points, and avatars will pop up at different parts of your shop. This will mean that even a very crowded shop will only have a handful or so of avatars in plain sight of each other, and this will allow each SL client only to actually rezz the avatars it needs to display. You might have seen that a few designers use this approach very cleverly, and their shops, although apparently as densely-packed as others, have far less lag. Now you know the trick
Remember that for this to work you cannot use alpha’ed textures. Anything with an alpha in it will prevent occlusion to work — even if the texture looks “solid”. To make double-sure, and assuming you do your own shop’s textures, upload them as 24-bit TGA images. Only 32-bit TGA images can have transparency and alpha settings in them (they need extra information to let the SL rendering engine know which parts of it are translucent or transparent). So, a glassy partition looks classy and fashionable, but it also means that it will not only prevent occlusion from happening, but alpha textures take longer to render than the un-alpha’ed ones. So your partitions will actually make lag worse!
There are also a lot of optical illusions that you can use when planning your shop. Items on display will usually have huge textures — 512×512 at least, but often more — since people will wish to see your wares in good, close-up detail. But the rest of the shop is just “decoration” to make a better shopping experience. It’s not as “important” as the items on display, so why don’t you save prims and textures on decoration? The part that people will see most of the time is the ground — so make sure that you can get the best possible texture on it. Walls might simply get less detail, and a very good texturiser will be able to get away with 64×64 textures on walls without making them look weird. Also, use few textures. If you take a look at some of the best designs in SL — mostly by RL architects or interior designers — you will see that their builds will have a lot of prims, but actually just a handful of textures. They will play with subtle tricks of texture alignment and tinting to give the illusion that they have used more textures on the build — but it’s just illusion. That’s why their builds, even if usually far more complex than the average shop, will rezz so quickly.
A lot of problems exist that are typical of SL. For instance, imagine that you have your open space are partitioned correctly with panels — but you have items covering all the walls, from the bottom to the top. What will avatars do? They fly. And once they start flying, they will have a clear line of sight over the panels and partitions — there go all advantages of occlusion! So should you prevent people from flying in your shop? Definitely not — flying is a key element in SL, and you should adapt to it, not prevent residents from using it. So, a good shop should make things easy to find without requiring avatars to fly.
That will also mean better signage. Most amateur shop builders will use a lot of signage, usually on the walls or suspended from the ceiling, so that they don’t cover the items for sale. Well, remember what I told you that most of the time people will be looking at the floor? (It has to do with the way the standard camera works: most avatars are seen slightly from the top) So the signage, unlike RL shops, should be on the floor, not on walls or ceilings! A further advantage of placing things on the ground is that there seldom is anything behind the floor (which means that once the floor is rezzed, the SL client has nothing further to download from that direction, and finishes much faster), unless, of course, you have a multi-floored build. Nevertheless, the best shops have been placing information on the ground and colour-coding areas to make them easier to navigate.
Avatar Rendering Cost Shows That Hair is Prohibitively Laggy!
Since Linden Lab introduced Avatar Rendering Cost as an option on the Debug menu, people have been shocked at how terrible hair is! This has lead hair designers to despair — the better hair looks, the more likely it’s going to clash with the ARC Nazis, who are always eager to shout angrily “Remove your hair now! You’re lagging the sim!” So this means that as a hair designer you have a difficult trade-off to deal with: either please your customers but displease the ARC Nazis, meaning that common residents will avoid gorgeous designs (or just use it privately at home) because they fear they’ll be lagging the sim…
Nothing could be further from the truth. In fact, prim hair usually uses just one or two textures at most, and it often uses the most simplest of prims, because they’re the ones that can get flexified (when flexible sculpties become a standard feature of the Linden viewer, as they already are on some viewers, there will be a new Hair Renaissance!). They still get a very high rating on ARC, because, of course, “lots of prims” will always get a higher rating, specially if they have a lot of alpha textures on each prim’s face (which will be the case for most of the very primmy hair).
But… high ARC does not create sim lag! Seriously! A prim is just a prim, 200 prims on the ground or attached to an avatar’s head take exactly the same time to rezz (actually, prim hair will rezz first, since all attachments have priority when downloading). So what’s the problem with high-ARC prim hair (or shoes)?
To understand this, you will need to know that here is sim lag, and there is client lag. Sim lag happens on SL’s grid and affects everybody on the same sim. Client lag is what you experience on your own computer.
Typical cases of client lag are… an underpowered laptop on wi-fi. Sure, it’s fun to be chatting with your beloved one in bed, but your poor laptop on wi-fi will simply be much more laggier than a desktop computer of half the price with a wired connection to the Internet. There is little you can do about that — it’s just the way technology works. Wireless and wi-fi are bad for SL — they lose packets, and when that happens, information has to be retransmitted. Laptops overheat quickly and easily, and that means that the laptop has to start slowing things down — CPU and graphics card are the first to go, and these two are what you need for ultrafast performance in SL!
The next most common mistake is an unconfigured computer. Although these days almost everybody in the Western world owns and uses a computer daily, they’re still viewed as an appliance — like your toaster in the kitchen — that you just plug in and it works. Sadly, they’re not even close to that, even if every year they get better at predicting what kind of use you’re going to give it. Here goes a typical example. Some of the most recent Vista-based, low-end laptops have a setting to run the CPU at half speed. Why? Vista is more demanding in CPU, and this meant more power consumption, and a battery that runs out quicker. This meant that the computer manufacturers felt that their products would be evaluated against other, non-Vista computers, and people wouldn’t want such short battery life. Thus, they just switch the CPU down to half the performance, and that will make the battery last longer. It will also mean that Vista will be far slower, but since people are used to slow computers anyway, it wouldn’t make any difference at all.
How each manufacturer deals with this is different, but the point is, by just clicking a checkbox or a slider you can suddenly get twice the performance out of your laptop! There goes your SL lag away… But tweaking and experimenting takes time, requires knowledge, and patience. And usually requires an expert too!
Not all of us are experts, so SL tries to estimate what are the best performance settings for your brand and model of computer. It often guesses wrong. I always tell people to experiment with the settings under Preferences to see if they get a bit more performance out of their computers; sometimes, a trick that makes a huge difference for someone will do nothing for you — and inversely, something that everybody else tells you never to touch will make all the difference! I have a very underpowered MacBook from late 2007, and I was very disappointed with it: it has an Intel card that is not supported by Linden Lab. So I wasn’t much surprised that I just got 1-3 FPS out of it, which pretty much frustrated me.
But one day I gave it a try again, and started tweaking with the parameters. By placing all settings at the lowest setting, suddenly I got it going at 40 FPS! Granted, SL didn’t look nice, but clearly that old, unsupported Intel card was not that bad! After playing a bit with the settings I found out that the only serious limitation was with the shaders (used by Windlight to get you gorgeous sky and water). Well, I could live without water reflections — so long as I got the avatars rendering nicely, with plenty of shiny, and good enough detail on the objects. So, getting rid of water reflections turned a “useless” laptop in a quite performing machine! On a good day, I can easily get 30 FPS out of it, and SL doesn’t look that ugly, either!
So is all client-side lag caused by improper configuration? In most of the cases, yes, but the truth is, some features in SL are really meant for high-end graphics cards. If all you can afford is a US$700 laptop or a US$500 desktop, you will get a “bare bones” graphics card — good for Word and a web browser, but not for much more. You can’t expect a low-end, US$10-50 graphics card, which is what comes with the low-end computers, to be able to render things like state-of-the-art graphics cards that will cost more than your laptop!
Sadly, most people simply cannot afford a state-of-the-art graphics card, and will have to make do with what they have. I’m afraid that what this means is that you’ll always be experiencing some sort of client-side lag. Even if people around you run naked, without script attachments, and paint all walls white… you’ll always lag on a sim with 50-100 avatars on it. Your graphics card is simply not powerful enough to deal with that.
Now you know that all ARC Nazis, by definition, have low-end graphic cards I can only pity them for that, because they have a perception of the world that is just unique to them, and they yell and get angry with everybody else because they believe that the whole world is conspiring against them to make their computers more laggy. I’m sorry — but, again, this is just superstition, just a magical belief that you can transform your underpowered computer into something powerful, if only everybody else used Linden avatars.
ARC Nazis will always put the blame on someone else. If they manage to get everybody in a place turning into Ruth, and they are still lagging, they will immediately suspect that everybody is secretly running scripted attachments. So they immediately yell to people to drop their attachments, and visually confirm if someone hasn’t done so. When they’re satisfied, they will still lag… and then remember that people also have HUDs. HUDs, unfortunately, you cannot see — so the ARC Nazis, living in their RuthWorld, will still believe people are hiding their HUDs behind their screens just to create lag.
But wait — you’re thinking — I’ve actually made some measurements, and high-ARC avatars, specially a lot of them, really make my computer run slower!
To understand why this is the case, you have to forget about prims for a moment. Graphics cards know little about prims or even avatars — all they know is about polygons (or triangles, depending on the model). You can see how Second Life looks like from the perspective of your graphics card by going to the Advanced menu and looking for the “Wireframe” mode. SL will look funny that way, but you’ll see that everything is actually really made out of small triangles, all the way.
Now you’ll notice one thing: the more complex the prim (i.e. the more it’s twisted!), the more triangles it has. Avatars have a lot of triangles on their own (about 7500 at my last count). A plywood cube will just have 12.
As you can imagine, your graphics card can only render a specific amount of triangles per second. How many? That depends on the brand and model, but you can expect that higher-end cards can actually render thousand times as much as lower-end cards. Now, SL is optimised to try to feed you a stream of 25 or so frames per second (enough to give you the illusion of smooth movement). This means that in a single second, all those triangles you see in a scene in front of you — hundreds of thousands, often millions have to be rendered at least 25 times.
It’s hard to get a list comparing graphics cards — most use “synthetic tests” where the overall performance is measured using a combination of techniques, not just merely polygon rendering — but you can imagine what happens on a low-end graphics card. At some point, there are so many avatars in the same scene that the graphics card cannot simply keep up with the polygons. As a result, the frame rate drops — you get that “slideshow effect” instead of smooth, video-like motion. This is just your graphics card telling you that it has reached its limit. High-end graphics cards will have a way higher limit, and, for them, rendering hundred avatars with 10,000 ARC doesn’t make them sweat. The problem is, most of us don’t have such high-end graphics cards. So what actually happens is that most will experience a huge drop in performance when rendering too many avatars with too many attachments.
The important thing to remember is that this affects you only. You can tweak your Preferences so that avatars render with less polygons (yes, they will be ugly). You can be on a crowded sim with a hundred avatars with 25-30 FPS on a low-end graphics card (I do that every day!). But it means you have to deal with trade-offs. Typical trade-offs are using avatar impostors even at close range (avatar impostors are just a texture, they have no meshes, and thus, no polygons to render); having a very short drawing distance (keeping those avatars out of your field of vision!); turning off all shaders (bye-bye water reflections and gorgeous skies!); and so on. So… this is actually the reverse of what the ARC Nazis want: instead of lowering your settings so that you can get a good performance of your card (but probably an uglier SL!), they demand that everybody looks ugly in SL so that they can get better performance! But that’s thinking upside down: you should be the one to adapt your computer to what it can actually deliver, not the rest of the world that has to conform to your under-powered or misconfigured computer!
It’s pitiful when you look everywhere for sources of lag that don’t really exist — except on your computer. You can search and search, yell and yell, but the lag in your own computer will not go away. Unless, of course, if you kick everybody out of the sim.
Why?
Time dilation
There is, however, one thing that will definitely lag everybody on the sim. And the answer is actually paradoxically simple: more avatars will lag everybody, but not for the reason described above!
Keep in mind that your goal is to render all those polygons 25 times per second to get smooth performance. Now, to keep with that goal, the sim will need to feed you with data, at least also 25 times per second. But — here’s the catch! — it has not only to tell you where everybody else is in the sim (and what they’re doing, e.g. what they’re saying in chat, what animations their AOs have activated, etc.), but it has also to send you the textures for your SL client to render content.
So if you push up the View Statistics window, you’ll see there is a section called “Simulator”. Under that section there is a subsection called “Time”. This will tell you what the sim is actually trying to do to keep up with all the requests.
For historical reasons, the sim will try to refresh everything in it 45 times per second (not merely 25), which gives it some headroom. This means that each “sim frame” is sent every 22 ms (milliseconds). Ideally, the largest value should be on “Spare Time” – meaning mostly that the sim has plenty of time left to do whatever it needs to be doing. Net time is related to texture download (and other assets) from other sims; Agent Time is for sending data about prims to avatars; and Images Time will tell the amount spent in actually transmitting textures to avatars logged on that sim – so, if anyone in the sim is downloading things, a slice of those 22 ms will be “wasted” on downloads and not on, say, tracking where avatars are (Sim Time – Other). The example on the picture shows a very healthy sim: half a milisecond is spent every frame to track avatars down, deal with textures, and so on. But most of the time, the sim is just idling away. This means — zero lag on the sim.
What happens when suddenly a lot more avatars start entering the sim? Well, first, more time is spent to track them down, and this grows exponentially. That means that to track down 10 avatars, you have to send a hundred times more messages! Tracking a hundred avatars means ten thousand times more messages. If we wished to have a thousand avatars per sim, that would mean a million messages exchanged to keep all avatars in sync — but there is no technology sufficiently fast these days to handle that and still be able to deal with calculating each frame, 45 times per second.
But things get worse very quickly. Sim Time – Other (tracking down avatars) is actually one of the quickest things to do, in the sense that it doesn’t require much time to send updates. Texture data is another story, because a full texture requires several seconds (thousands of “frames”) to download, and while the texture is being transmitted, the sim momentaneously fails to track down where avatars actually are and what they’re doing (from your point of view, this is not very serious, because the SL client interpolates — it tries to figure out where other avatars were the last time, where they were moving to, and what animations they had loaded, so it can pretty much give you a reasonable display of what is happening until they get fresher data).
But now imagine that a hundred avatars are all requesting texture data (because they’ve just arrived to a huge event at the same time). Suddenly, all the sim is doing is sending textures everywhere. But in most cases, the textures might not be loaded on the current sim (imagine all those attachments and clothes that came from other sims!), so first the sim has to request them all — wasting time and bandwidth. At this point, the sim might simply begin to fail tracking down avatars. These will request new position updates, all at the same time, but the sim might simply not be able to send them all the required data in less than 22 ms. So here is where strange things start to happen: your avatar starts to “walk through treacle” or suddenly hiccuping and appearing a bit ahead or below of what you expected. This is just because the SL client is expecting some data that it never receives.
In the mean time, the sim starts to receive some texture data. Since all hundred avatars require the same textures, now comes the moment where one texture received from a remote server suddenly turns into hundred uploads to as many avatars. So if texture retrieval from a remote sim was already painful, now it’s a hundred times worse.
But it doesn’t stop here. Once you’re missing a lot of positioning information, you cannot feed the physical engine with accurate information on where the avatars are. Like the SL client, the physical engine can work on guesstimates — “well, this avatar was moving northwards a second ago, so it’s safe to assume it’s still going that way”. Sadly, you never know what’s in the path. Other avatars might become obstacles; the path might suddenly end; there is a stair in front of the avatar and so the physical engine has to calculate the new position based on where the “ground” now is.
When the amount of computation by the physical engine is too high, and it cannot deliver an accurate prediction on where the avatars are supposed to be, we enter “slow motion mode”. What you will see is that the value of Time Dilation, which is usually 1.0 (that means: events happen in real time) might suddenly drop. A drop to 0.5 means that the physical engine is now operating at half the speed of real time, i.e. that all actions that avatars start taking twice as long. A very overburdened sim will quickly get worse and worse — when Time Dilation is at 0.10, that means that everything happens ten times as slower, and so on.
What does this slowdown mean? Well, the SL client can compensate for the lag (what literally lag means: things are not happening in real time any more) to a degree. It doesn’t need to update the avatar position so often, for instance, because the sim tells the SL client that it can’t keep up anyway. You’ll see that for relatively high Time Dilation values (until 0.7 or so) you might not even notice the lag: sim and SL client work in tandem to compensate for the lack of available processing power on the sim side. Usually, this should not happen often: it’s designed to deal with “spikes” (when all of a sudden a lot of people drop in the sim, but quickly disperse, each going their own way).
The trouble starts with a very intense and long event (say, an hour or so) where avatars are constantly in pretty much the same spot — which is what actually happens at almost all events That’s why they lag all the time. The sim cannot keep up with so much information to transmit to all avatars; the physics engine is constantly working to keep up with the amount of information it needs to calculate everything, and never “catches” up. The event is usually laggy from the begin to the end. At the very end, as avatars start to leave, the physics engine finally catches up, the sim finally can track back all avatar positions, and at last, “slow motion” becomes real time again, sometimes all of a sudden.
So how can you prevent this? The short answer is, you cannot. You can’t tell people to stay away from events — which is the only thing that actually creates sim-server lag. The mere act of connecting to a crammed-full sim creates lag. The amount of prims on your 10,000-ARC hair is pretty much irrelevant. Just the tracking down, the physics engine, and the few textures that it’s constantly downloading from other locations and feeding to all avatars in the sim, is more than enough to bring it to its knees. What the Lag & ARC Nazis do not realise is that all this happens independently on what you’re wearing or attaching to yourself.
But doesn’t it help? After all, if you’re not wearing 200-prim-hair, you will have less information to transmit, right? So it will surely help a bit? The short answer is no — the difference is almost impossible to measure. As said, 200-prim hair is usually just two or three textures to download. If your SL client doesn’t ever get them, that’s all right, they’ll just remain grey, but that doesn’t “lag” you. Remember that a sim with Time Dilation of 0.5 is two times slower than real time — and 0.10 is ten times slower. If the overall difference of having everybody wearing 200-prim-hair is 0.01 on time dilation (probably it will be far less), that is hardly important. The sim will not recover from lag even if everybody detach their hair, shoes, and HUDs. Avatars will still be requesting data; the physics engine will still be having a hard time to calculate where avatars are. No, the only way to deal with a sim crammed full of avatars — is to get rid of them. But that’s not an option!
The conclusion?
Etiquette dictates that you ought to conform to other people’s rules when you’re a visitor, and should accept their insane superstitions, just out of politeness, even if they don’t make any rational sense. If the Lag & ARC Nazis request that you detach your things, out of politeness, you should do so. But you should also be aware, if you’re hosting your event, that it makes people angry to impose superstitions on your guests. Blaming your guests for completely the wrong reasons is as rude as ignoring your host’s requests — in fact, I would even claim that some cultures view it as much ruder. As in the example earlier, it would be far more rude for the host to offer not-kosher food to Jewish guests (and insulting them for their superstitions) or blaming them for having to be forced to serve something you dislike just because they have a silly religious rule. In almost all cultures in the world, hospitality is important, and a part of being hospitable is to be nice and welcoming to your guests, and accepting their differences and ideas. Also, in almost all cultures, guests also try to comply with their host’s reasonable demands, if their motivation is a good one. My grandfather, who was a Jew, would not engage in drama if someone would offer him a ham sandwich; he’d just smile, eat a bit not to offend the guest, and later explain that unfortunately his religion forbade him to fully enjoy the sandwich, but still thank the host for the effort in doing a nice sandwich just for him. A polite host would offer their apologies to my grandfather and remember not to use pork ham but stick to chicken ham on sandwiches next time.
Similarly, a polite host in SL would probably request guests not to use high-prim attachments and turn off their HUDs because they believe in the myth that these create lag. Insulting and yelling at your guests is very rude — specially well-informed guests that would kindly point out that this lag myth has no grounds whatsoever. On the other hand, specifically ignoring your host’s request, and wearing the highest-ARC hair you can find, attaching a thousand flexiprims on your scripted skirt, using all kinds of particle effects and sounds on your shoes and bracelets, just to provoke them, is also very, very rude. A good compromise can be met: there are less primmy hairs around which still look good enough; you don’t need to wear earrings if you have a very dense hairstyle (nobody will notice them anyway); and wearing jeans instead of a high-prim skirt is sexy and fashionable too, and a lot of prim shoes just have one sculpty and still look great! You can also turn off your avatar radar on a very crowded place, since it won’t work properly anyway (as explained, scripts will run so slow that the sensors used by the avatar radars will never really find all avatars around you).
What we call “lag” is basically split among two categories: server-side lag, which you can’t avoid in crowded areas, since it comes simply from having a lot of avatars in the same place, which require a lot of time to compute their positions and what they’re doing. Attachments will have little relevance to lower lag. The other category is client-side lag, and that one can be dealt with — both by the event hoster, who can design the environment to take advantage of the so many tricks that the SL client does to keep a good performance, but also by the visitor, who, even with an underpowered graphics card, is always able to tweak their settings to adapt to an avatar-intense event. Sure, that requires that you experiment a bit with the settings, but it’s within your power to effectively turn down your client-side lag if you’re willing to do some tweaking — instead of blaming others!
What will definitely not work is to stick to superstitions and myths, assuming that things work by “magic”, and being rude to your visitors who are better informed and are not willing to get insulted.
Hair: Amber IV – Sunset by Ingmann Creations
Outfit: Erotika by Kunglers
Second Skin by Namssor Daguerre
J’s Tight High Boots
Comments
90 Responses to “Anatomy of Lag”
Leave a Reply
-
Sponsored By
If you you are interested in having your ADvert here, contact me
-
Partners
-
Joint Ads
XStreetSL wishlist
Creative Commons License
Ana Lutetia by Ana Lutetia is licensed under a Creative Commons Attribution-Non-Commercial-No Derivative Works 2.5 Portugal Licensevisitors map
-AnaLu-- I’ve found a way to make you smile
- Impeccable logic, I wish I were Vulcan
- The SPCDH in Second Life
- Be a bad bad girl
- du4 boys
- Hair Fair 2010
- G Field :: New branch & Lucky board
- Harajukubox Closing
- All vintage, all the time! Fall collection at Dark Mouse Jewelry
- Emerald Viewer is “Dead”
- Portrait (ça faisait longtemps)
- A matter of orientation
- Treasure Hunting in Second Life
- Jessica Lyons: “the Emerald Viewer Project is dead”
- :: Stellar new build update ::
- FATPACK RIVER Cocoa
- Weekly Linden account changes (week 35, 2010)
- What the heck IS going on at Dark Mouse? New stuff, that’s what!
- Fabulous Fashion Update from Angie Mornington
- Something punny should go here
- SL – grid status
- [RESOLVED] Viewer 1.23 Search Issue
- [Resolved] Rolling Restarts for Server 1.42
- [RESOLVED]Inventory Database Maintenance Wed, Aug 25th at 6:00am PDT
- [RESOLVED] Live Chat Support Unavailable
- [RESOLVED] Parts of PJIRA down for maintenance.
- [RESOLVED] Second Life Wiki Issues
- [RESOLVED] Inworld Search Issues
- [Resolved] Xstreet Maintenance
- [RESOLVED] Rolling restart for server 1.42
- Recovering from Momentary Service Disruption
- Koinup — Second Life Places
blogity stuff
second life
fashion
June 22nd, 2009 @ 09:56
[...] friend Ana Lutetia has invited me to write something about lag on her blog. Since lag was what we mostly got this weekend, I hope you’ll appreciate an update on what I [...]
June 22nd, 2009 @ 10:31
[...] nigh on unheard of but I’m prepared to make the sacrifice, BUT having just read this rather faboosh article by Gwyneth I’m wondering if I should have [...]
June 22nd, 2009 @ 11:28
[...] works under the hood, a place I don’t look too often for fear of being swallowed up in code. Go have a read, it’s a stunning post, with great tips for [...]
June 22nd, 2009 @ 11:45
I agree with most of what you wrote (already with the 1 year old post), however one point. You claim using 64×64 textures for not-important parts are a good way to go. However recent tests I did showed that very small textures actually loaded LONGER than larger textures. The reason is most likely the JPEG2000 compression, that works much more effective on larger textures than on smaller textures.
I had particle textures at 64×16 that took forever to load, but when I switched to 512×64 they loaded instantly. Also a 256×256 texture loaded much faster, than a 64×64 texture.
Also keep in mind that each texture opens an individual download. So combining for example 4×512 textures on a single 1024 texture not only saves you 30L$ in upload fees, but also loads faster.
June 22nd, 2009 @ 13:40
Wow thats a great read
Thanks for taking the time to write this!
June 22nd, 2009 @ 13:43
Peter, you might be actually quite right on your claim! It even seems to be different depending on what SL viewer you’re using! Linden Lab’s “official” main client handles texture download in quite a different way than their new “Snowglobe” project; in turn, Kirstens Viewer S17 handles it in on a yet different way. This only points to an area in the SL client which is currently under quick development.
Some older texturisers used a different approach (learned from 3D game design): make all textures 1024×1024, but apply only a part of them to a surface, by using the settings for texture offset. This means that these textures might, indeed, take a bit longer to load, but they load only once, and when they do, all items in the scene might suddenly pop up in existence, even if they “seem” to have very different textures! This, of course, also works well for attachments (the smaller these are, the better you can use a single 1024×1024 texture to cover all parts of it using the same trick).
I’m not sure what the “best” approach is. I have seen both being very successfully employed. A typical trick is also to upload all textures on the very sim they’ll be used, since this will mean that one step will be skipped (transferring the texture from a different sim). Of course this won’t work for attachments: for these, the rule of thumb is to upload textures to sims that will never have many people in it. Put it into other words: clothes and hair designers should upload all shop textures on the sim where the shop is, but all clothes and hair textures on a sim that never has anybody in it.
June 22nd, 2009 @ 13:56
Wow, great post. I agree, Arc lies.
On the texture front, the other point I’d mention is that it depends on the texture. A white sign with 2 color lettering has huge amounts of continuous tone that is easily compressed down, while a detailed texture with slight changes throughout will not compress as effectively.
One of the places where this is felt in the latest gen skins and clothing. Back in the day when skins didn’t have so many freckles and detailed body shading and other realism factors, the skin textures loaded much more quickly because they were better compressed. Similarly, clothing textures with subtle fabric, wrinkles and shadows don’t compress as well as a relatively simple texture.
But since I love great lookin hair, fab outfits and top notch skins, I’d hardly be one to recommend everyone using flat skins and clothes.
Thinking on such things as hair fair and the like, I’m starting to think what the grid needs is a “convention center” sim configuration that steps up the server (or server cluster) to an extreme level.
That having been said, great tips, and yes, it’s common courtesy to arc down for such events (but I doubt the busier stores and clubs will only be solved with increased capacity on those specific sims – honestly – who’s gonna go dancing and intentionally dress fugly?)
June 22nd, 2009 @ 13:56
Well, the client I used was a (slightly outdated) CoolViewer based on 1.22. And actually combining as much sub-textures on a single 1024×1024 is what I learned as well – and usually it provides some good results.
I do not believe however in the upload-on-same-sim theory. This would need some testing, but my understanding is that the sim does not STORE textures, but only CACHES them.
I downloaded Snowglobe today and look forward to experiment with it.
June 22nd, 2009 @ 13:57
Fantastic article, thanks for taking the time to explain so concisely all the different aspects of lag. There are some valuable building tips as well. This should be bookmarked as required reading for all who attend large events.
June 22nd, 2009 @ 15:40
Great article! Thank you so much, Gwyn, for explained so thoroughly.
June 22nd, 2009 @ 16:07
This is such an informative post. Thank you for taking the time to write in such detail.
June 22nd, 2009 @ 16:27
Thank you so much for this! As a skeptic, I don’t necessarily believe we should honor the silly superstitions of others. So I’m going to keep this url and hand it off to any ARC nazis who bother me. Right before I get ejected from the sim.
June 22nd, 2009 @ 17:37
First of all, let me echo everyone else in thanks for this great information.
With that said, I’d like to offer up a very special technique I use to both avoid the personal experience of lag and to reduce the problem for everyone else in a laggy sim…
I leave. As quickly as possible.
There are very (very!) few times when I find it worthwhile to be in a crowded sim. When I am the event host, or someone very important to me has specifically invited me, I will crank every setting as low as possible and then find a spot to park my avatar and largely ignore everything but IM and streaming sound. OK, if I’m hosting I’ll attempt to follow the general chat, but otherwise forget it.
In any other circumstance, I’d rather be elsewhere. I’ll make my purchases on XStreet, or from a smaller, less popular store. And as for parties, I’d rather be at a small event where I can actually keep up with the conversation.
Hair Fair is a great event – a perfect time to be anywhere else, with less lag, because so many are trying to cram themselves into those few sims.
(In the spirit of full disclosure, I’ll have to admit that I did attempt to visit Hair Fair once this weekend, just to see how bad it really was. The sim was full, preventing entry – and thus my curiosity is fully sated.)
June 22nd, 2009 @ 19:05
I did write a huge reply, but the blog ate it! LL has raised the priority on 64×64 because of sculpts, and they’re much better than they used to be.
June 22nd, 2009 @ 20:47
WOW! Awesome post, thank you SO much! I hope since Ana has one of the top read blogs that this information will spread although knowing how many times she has posted about presets and not wearing face lights we still see them everywhere. I think as some have already done, us with blogs should link to this article and maybe we can educate more. I will be posting it on all 3 of my blogs. Sasara, great point! these ppl standing around at the hair fair at the sim crossings or just around afk are not only keeping others from getting into the sim they are in but adding to the problem explained above, come on people if ya get in, do it and get out. You can use the tp from home to get into the other sims instead of hogging a spot on the one you are done shopping at.
June 22nd, 2009 @ 22:23
Over-heard in one of the Hair Fair sims:
[14:22] AVATAR1 shouts: Why can’t people take off their shoes, jewelry, and hair? It would help the lag!
[14:22] AVATAR2 shouts: cause we wanna be sexy..lag = sexy av’s..just remember that
[14:23] AVATAR1 shouts: I am practically naked thanks!
[14:23] AVATAR2 shouts: doesnt make u sexy
[14:23] AVATAR3 shouts: why cant they make an expo without using overloading the sim with this building design?
[14:24] AVATAR4 shouts: My render cost is 1 you are preachin to the choir here
[14:24] AVATAR5 shouts: someone just had candy craving in rl and they built it here to satisfy it
[14:25] AVATAR6: Render cost doesn’t make a lot of difference, it’s just too many people trying to bee here at the same time
June 22nd, 2009 @ 22:46
@Sasara
Ditto!
‘Tis why I wrote this:
http://commonsensible.net/2009/06/05/vwr-13874-visibility-of-inactive-avatars-on-minimap/
June 23rd, 2009 @ 00:46
Ok, first off I loved this post. It is articulated very well. In response to the comment about lighting presets; I have been playing around with them for a while and here is my issue: I have used the various presets that I have been given, and I hate them. Some textures appear far too bright, and I can’t stand the way the sky looks. I prefer to be on the default setting where I can enjoy the times of day etc, but would also like my av to look pretty. Anyone have suggestions on how to tone it down a bit so that I don’t feel like I’m somewhere that has recently been hit by a radiation blast?
June 23rd, 2009 @ 00:47
Simple: use my presets!
(latest) WindLight presets post
June 23rd, 2009 @ 09:56
[...] you can do, and a million theories on what works and what doesn’t. You can read one great one here. All that aside, I’m going to tell you how I do it, so that my not-so-fancy laptop can get [...]
June 23rd, 2009 @ 10:12
Very very interesting article … thank you! I just got into the Hair Fair today after randomly trying for days. As it was, I was in the middle of a shoot for my blog and never expected to get in. I arrived with a heavty ARC load and before I knew it, felt like I was pounced upon by a hair designer, telling me to check my ARC. I probably over-reacted and didn’t take kindly to this person’s style of approach, but I did feel as if I was being attacked (probably a British thing … we’re over-sensitive on the politeness stakes
). I do hope that people become more informed through excellent articles like this and stop attacking one another. I have to say that I was really excited about the Fair and the incident spoilt my experience somewhat 
.
June 23rd, 2009 @ 10:33
Very interesting stuff here, great post! I do have a question though: if lag is related to the system having to grab a texture from another sim (the whole upload-where-you-use-it part), what happens if that sim goes down? Or is moved, or deleted altogether? Does this affect texture loading time in any way, or does it delete the texture altogether?
June 23rd, 2009 @ 13:44
I would like to thank you all for the very nice and enthusiastic comments so far! I should add a small update: while chatting in-world with SadieMae Jameson, she gave me a very simple and obvious tip to get rid of ALL ARC: just check the Avatar Impostors checkbox from Preferences > Graphics and set the slider for Avatar Details at the minimum. Now watch it closely: ALL avatar impostors, no matter how complex their attachment, have an ARC of *1*.
So it’s totally pointless to yell at people to detach their hair. Just let them set their sliders at the lowest setting on Avatar Details and click Avatar Impostors. As if by magic, ARC disappears.
If that’s not enough proof that attachments DO NOT lag the SIM, but only affect YOUR computer, I don’t know what to say.
June 23rd, 2009 @ 13:49
@Adeline, thank goodness that LL does have several copies spread around the grid — every sim where that texture has been used will have a local copy of it. But, in theory, if a sim is totally deleted forever and ever, and NO copy of that texture exists ANYWHERE, this texture MIGHT be broken/damaged/disappear (it will never load). Now this happens so rarely that I believe that LL has some kind of way to deal with it
June 23rd, 2009 @ 13:53
Gwyneth, with all due respect (hey, I always wanted to use this phrase), but I think you are wrong in this. A sim caches the textures used on it, but it gets stored on the asset servers! So even if a sim gets deleted and the texture only got used on that sim, the asset server still has a copy, and most likely the inventory of at least the uploader has a copy too. I assume there is some sort of garbage collection that permanently deletes assets that are neither used nor in anyones inventory. But I can’t believe a texture gets lost/inaccessible if the only place in world it is used gets removed.
June 23rd, 2009 @ 15:20
What? You mean I am no longer totally wrong on this subject? After enduring 2 years of constant abuse and ridicule from even the most notable of the bloggers the truth is out and I was not wrong after all?
Well that is nice.
but now you have really done it. LL will start ripping content to shreds by coding in limiters that just stop rendering vertices at some arbitrary limit that has no basis in reality of how SL is used. Will? heh… They already did.
http://jira.secondlife.com/browse/VWR-13868
I can say this with 100% confidence. Particles are bad. very bad. In your statistics bar what you want to look at when judging your build (or yourself) is ktris per frame. You want to keep this down. I cut half of the ktris per frame out of my store by deleting the wonderful fire torches made by the esteemed Baron Grayson (Relic). Sure the torches added an incredible level of ambiance. At great cost to viewers. So look at ktris per frame and optimize based on that. If you are on mainland I am afraid you are out of luck because there is no way you can optimize anything there. It is the nature of mainland. But on islands you can certainly optimize viewer performance.
And for fun: research how much ktris is involved in rendering windlight.
June 23rd, 2009 @ 17:01
Thanks Gwyneth! Very informative read. I plan to send this link to the SIM owners Group (Concierge Info Group), some of the worst Lag Nazis reside there!
June 23rd, 2009 @ 18:37
Wow!! I loved this article, very useful indeed. I had put to practice many of the tips discussed here and also, shame on me, was a believer of the attachment myths… But going through the advanced tab, I found an option about what I want redendered. And when I go to these major sales, or events, I simply, choose not to render any characters. Now yeah, looks dead and empty, but it sometimes, help me. Can you tell me if that has anything to do with the performance of my client?
Thanks!
June 23rd, 2009 @ 23:25
[...] [...]
June 24th, 2009 @ 00:14
Gwyneth: “But, in theory, if a sim is totally deleted forever and ever, and NO copy of that texture exists ANYWHERE, this texture MIGHT be broken/damaged/disappear (it will never load).”
I’ve only seen a texture absolutely NEVER load one time…and a friend of mine with the same item experienced the same issue. The thing just stayed grey indefinitely. I wonder if LL *does* have something to help this? lol
I can’t wait to get in world and try all these things out.
June 24th, 2009 @ 00:34
I’ve seen my 8*8 pixel black texture I use on prim sides or faces not intended to be displayed rez as a 1024*1024 semi transparent gray texture. The best possible to the worst possible And then later mysteriously correct itself.
The asset system is a pile of xml files on a fast file server cluster. Stuff happens. Sometimes a texture won’t load. Reupload it and replace it.
June 24th, 2009 @ 09:47
[...] that really opened my eyes to how sl behaves and treats lag nowadays – well worth a read on Ana Lutetia’s blog if you want to be up there with the cool kids.. posted under [...]
June 24th, 2009 @ 12:25
“clothes and hair designers should upload all shop textures on the sim where the shop is, but all clothes and hair textures on a sim that never has anybody in it”
I’m curious about the texture location situation too. I also understood that textures normally reside on the assett server and are callled from there and then cashed on the sim when needed. Hence where they are originially uploaded from is irrellevant. So are you saying that there is a distinction between textures on avatar prims (which will move from sim to sim as the avatar moves) and textures on buildings (which will stay more or less permenantly on that sim)? I think this needs a little more clarification. Otherwise its a stunning post and I’d be interested to know where you got the information from. Do the Lindens publish these technical details anywhere? If not they should as it would greatly help builders, designers and users – and hence LL itself – to utilise a much more useable environment.
June 24th, 2009 @ 13:32
@Peter, as Ann explained, the “asset server” is actually just a directory service. It doesn’t store content — instead, it points to where content is on the grid.
This approach is what allows content to be delivered from 32,000 virtual machines (on around 6000 or so servers), since delivering with 2.3 billion items taking several hundreds or so Terabytes of data on a single server (or even a server cluster; allegedly there are five, but that’s unconfirmed) to a hundred thousand simultaneous users is an impossible task for the current state-of-the-art network cards. For these scales you definitely need a distributed model of content. That’s why sims actually deliver content to the end user, not the central servers.
You can test this out by tracking down the requests made by your SL client. You’ll that after authentication, all connections are just made to the sim you’re connected to, as well as to neighbouring sims. Everything else that SL requires to run — from presence information (where your friends are in the grid), to inventory, to messaging (chats, friend requests, money transfers…), and, of course, textures and other assets, will come from the sims you’re logged in.
June 24th, 2009 @ 13:38
@Obvious, finding out all this is not a simple task. Philip and Cory have published a very early white paper on Gamasutra explaining how the grid worked… in 2003. Later, several LL developers have posted cues here and there, mostly to explain (and document) the open source code of the SL clients. They don’t reveal all secrets, of course, since a lot is going behind our backs that is proprietary and confidential.
On the other hand, you can have a rough idea on how SL works by looking at what OpenSimulator does. Granted, a lot of it will be different from how SL works. On the other hand, since you’re using the same client to log in to both LL’s grid and OpenSim-based grids, it means that at least there is a close similarity to how both technologies work. They are as similar as using, say, Apache or Microsoft IIS to deliver webpages. They’re different technologies. They’re completely different software. Nevertheless, when you use your regular browser, it will get pages from Apache or IIS in exactly the same way, so at least you know, to a degree, that what goes under the hood has to be very similar (even if not exactly the same).
Assembling the pieces together has been a task I’ve been doing for the past 4 years or so
It’s not an easy one. Unexpectedly, all my past research work got rewarded, since I can now use it for my mastership thesis on SL… which requires explaining a lot of detail on how SL works 
June 24th, 2009 @ 14:18
Ok so if the asset servers are simply directory servers and the assets themselves physically reside on the sim on which they were originally uploaded to (or created on), doesn’t this raise questions of the storage capacity of those sim servers? And for example, during a rolling restart, does that mean that various assets become temporarily unavailable whilst those sims that are storing those asstes are down? Indeed, does this mean that the assets that I see in my inventory, rather than being stored in one centralised location, are in fact scattered across the grid? feels a paradigm shift coming on!
I’d also be interested to know what the reserach question of your thesis is? Seems like you have a lot of hard work invested in it already and certainly information that all users of SL could benefit from.
June 24th, 2009 @ 15:35
Assets are stored in LLSD format XML files on a very large very fast file server cluster.
Inventory records point to the asset record. Multiple inventory records for something are just copies all pointing to the same asset record.
There are multiple inventory clusters each serving a set of residents.
The inventory databases run on mysql databases.
Textures are fetched from the asset system as needed. In the snowglobe viewer this is done over the http protocol instead of the udp protocol and in most cases is dramatically faster. If the sim is dead (full of avatars like at the hair fair) then it is slow so the request queues apparently suffer the fate of bottlenecking through the region instead of the clients requesting the textures directly from the asset system.
The textures that are “on the sim” are baked textures. I have never heard of baked textures applying to anything but textures applied to avatar meshes. If you rebake in a sim your avatar will appear to rez more quickly to people in that sim because there is no fetching across the grid to go get the bake. This is why some of us rebake on arrival in a sim. I have never seen any command to “rebake a sim”.
Kelly Linden has always been helpful and taken the time to explain things if people take the time to ask Kelly. My recommendation is to attempt a conversation with Kelly and get “the straight dope” on the topic instead of second guessing. The information I have that I trust came directly from Kelly when Kelly explained the way the asset system worked on sldev. Thus the discrepancy in respect to what is or is not baked and stored on the sim. The information only applied to avatars so I don’t know about the rest.
I spent some time last night installing occlusion panels in my store to prevent the ktris from outside impacting the shoppers. I dropped average ktris when looking south from 500 – 600 ktris per frame to 200 ktris per frame which is only ten times the ktris per frame in my pretty much stark and empty studio. A notable improvement while not removing any content. Looks like occlusion works. Then I went to the SL6B sims and marveled at the average 1400 ktris per frame. This being a LL sponsored showcasing in a world their developers are coding to simply not display more than 4096 vertices unless you manually change a debug setting. Interesting.
June 24th, 2009 @ 15:44
Per Kelly Linden (and documented on http://wiki.secondlife.com/wiki/Inventory_OS :
“There are three systems involved here that are closely tied:
* (DB) Resident Inventory lives in a database. We currently have lots of these ‘inventory’ or ‘user’ databases, each one hosting the entire inventory for a subset of our users. ex. UserA has all their inventory on db1, UserB has all their inventory on db6. The database entries are the shortcuts / links to actual assets.
* (ASC) The actual assets live in the Asset Server Cluster. Any item in resident inventory or object inventory references an asset stored in the Asset Server Cluster.
* (SIM) Items rezed and ‘in world’ do not have an asset that reflects their current state. They do have an ‘original asset’ reference, which points to the asset used to originally create the item when it was read but this won’t reflect any changes made since rez. We don’t save individual items that are currently in world to the asset server or any inventory database. We do save the entire state of a region every hour as a ‘sim state’ file – and this includes objects and their inventory items (as references to actual assets).
So, resident inventory lives in the (DB) and each item references one or more assets in (ASC). Object inventory lives in (SIM) and each item references one or more assets in (ASC). Items actually in world don’t really reference any asset in (ASC) or anything in (DB). Keeping items inside the contents of objects in world and *not* in your inventory reduces the load of (DB) and may help keep your inventory loading faster, depending on how much you do it. It doesn’t have any effect on (ASC) load though.
Attachments are the only case of an item in world that is also in your inventory, and the only case of an item in world that isn’t in the sim state saves. That makes them doubly unique, and doubly fun. They also cross region boundaries far more than any other object, which has significant implications[1]. ”
“1. When an object moves from one region to another, the region must serialize the object and the contained scripts, including each script’s state and event queues in addition to setting up the rerouting for Email, XML-RPC and HTTP. ” (btw that hints at why teleports are slow for some, fast for others not carrying a lot of attachments)
Further commentary by Kelly:
“Items that you see in world are being simulated and are not in the DB or
the asset system but live in memory in the simulator process. Everything
that is a thing (can be rezed or viewed) lives in the asset server, not
the DB. We actually save (serialize) the state of each simulator once an
hour to a file and save that file in the asset system. File type
operations on ‘simstate’ conceptually make some sense.
- Kelly”Items that you see in world are being simulated and are not in the DB or
the asset system but live in memory in the simulator process. Everything
that is a thing (can be rezed or viewed) lives in the asset server, not
the DB. We actually save (serialize) the state of each simulator once an
hour to a file and save that file in the asset system. File type
operations on ‘simstate’ conceptually make some sense.
- Kelly”
June 24th, 2009 @ 19:59
Thanks so much, Ann! Your information has been really quite useful! Note that what you call “second guessing” is pretty much the kind of information that was available publicly from around 2003-2006. Apparently a major change was done about that time by Kelly and his staff — assets apparently moved to a different storage mechanism, while inventory items still work the same way as before. There are some subtleties on Kelly’s description which I definitely will need to research further — and ask him about details — because some of his descriptions are not clear or even contradictory, at least in the way he formulated them!
Also note that although this is a side-issue apparently not directly related to lag, to be honest, if I got Kelly’s explanation right, this would mean that a high-texture-density sim, tightly packed with avatars, would have a deep impact on the whole grid (since all the textures would have to come first from the “central storage”). Hmm. That would indeed explain a lot of things, and make everything even worse than I thought!
June 24th, 2009 @ 21:18
Except the snowglobe viewer with it’s http texture pipeline was placed into production on the download page today. Ever seen the entire grid in your map viewer? You will in snowglobe. Exploring by map just became totally feasible. The amount of time spent downloading textures on my system is 1/10 of what it used to be. I’m not going back to the slow viewer. Entire island regions loading in 30 seconds is quite a change of pace. However not everyone gets this performance and busy sims seem to load more slowly. There is an opportunity for improvement there through more threads and paralleling the texture sources out more.
So the impact is expected to drop IMHO which falls directly in line with scaling up the grid concurrency which just turns around and offsets the gains rofl. I’ll take acceptable performance with ten times the online accounts any day.
Snowglobe is an example of how fast things can move. One day I was hammering it finding bugs and the next I couldn’t break it even slamming it for 16 hours straight.
Things are changing and the changes are picking up speed. I am completely re-engineering my store now to totally minimize ktris per frame. There will not be any sculpty trees inside of it rofl. Minimize ktris where business is to be done with as many avatars as possible and make places of beauty high geometry but clamp the avatars allowed count down. There is your balance.
June 25th, 2009 @ 12:30
Snowglobe definitely gives us a taste of the amazing improvements to SL architecture (as far as texture delivery performance is concerned). Last time I used map for anything else than a simple teleport tool was probably in 2004, but in Snowglobe, I can’t stop playing with it! It never felt so smooth before!
But what I think will have the largest performance impact is the move to HTTP for texture downloads. This will really give Linden Lab a lot of maneuvering space when it comes to texture caching. For example, they’ll be able to simply plug in standard and well proven web caching software (Squid, etc) between the asset storage and the client, and speed things up tremendously. Or even offload vast amounts of textures to content delivery networks, like they’re doing for the in-world map in Snowglobe – by using Amazon S3, Akamai or similar CDNs.
As a user from Europe, I’m already excited by the fact the textures could soon be delivered to me from a nearest Amazon S3 or Akamai node in Europe, instead of traveling half way around the globe.
June 26th, 2009 @ 21:50
attire
Great post and some very good and informative comments.
Based on what I read here about textures loading in a sim and to a avatars view, would it be a correct assumption that for a fashion show if all of the textures used on the outfits in the show were in view of the audience, on a sign board for example, that when the model actually put on the outfit and came in view of the \\\”viewer\\\” the textures would be \\\”rezzed\\\”? I do understand that the textures on the \\\”board\\\” would have to be the same UUID texture and not a photo of the outfit.
June 27th, 2009 @ 16:01
quote: “Turning AO offs will not reduce lag, but people still believe they do. So, even if that’s a stupid superstition, it doesn’t mean we ought to be rude and offend them. We can be politically correct and accept that their erroneous views on how SL runs truly don’t affect us; we can live without our AOs for a few hours and be seen as polite and respectful and tolerant towards others.”
Turning of AO will reduce lag significantly for my client if I can “see” your avatar. Even more so If I have local lights turned on. The AO script may or may not lag the sim. I will leave that question to the script deamons. I do know that if an avatar is moving in my view that my client is dealing with that movement. If I have local lights turned on my client is dealing with recalculating the lighting on every polygon of the avatar as it moves. If the moving avatar is wearing a face light that is visible to me, my client is recalculating the lighting on every polygon the light illuminates. Imagine what effect this is if you are at a large event like a fashion show where the interest is in the models or any other performance. If there are avatars in your view running an animated AO they are lagging your client much more than an avatar in a static pose. Multiply that by all the avatars that wont turn off their AO because they feel they are special or they don’t want to look like a noob. I go to events like a fashion show or performance to see the show, not someone with a really cool AO stagger around in a 3 meter area or constantly fidget like they had to many double espressos or need medical attention. Guest seating with animated poses is another major cause of lag in a large gathering, lagging the client of everyone that can “see” the constantly moving avatars.
I photograph events and fashion shows for SL magazines and as a business. Dumbing down my settings is not an option. I keep my camera focused on the models and performers as much as I can except when shooting audience shots or shooting a shot down the runway toward the audience. Any “guest” that is in my view in that case that didn’t stop their AO as requested, is using my client resources and creating lag.
I would love to be able to have local lights and “attached lights” turned on in my settings when I shoot events it adds drama and atmosphere to my photos. In most cases I have to give up and turn off “attached lights” in advanced or go all the way and turn off local lights alltogether. It only takes one audience member with a bright face light/s set on full radius to destroy the local lighting if I use atmospheric shaders. Removing attachments such as face lights and AO’s can a does significantly reduce client side lag for everyone that can “see” you. That is fact not myth.
June 27th, 2009 @ 19:44
If you mean the textures that were originally used by the designers to create their outfits, Sharron, yes, that would work, although they would look a bit “strange”.
Also note that apparently, from a message just sent yesterday, a Linden developer explained that right now SL clients download the full set of 21 textures for each avatar in view, instead of just the three “baked” ones (for head, torso, and legs). This was done in the past because Linden Lab was afraid that some computers would “bake” textures incorrectly (something which indeed would happen in the past), and by sending all the textures, at least other residents would be able to “bake” the many layers of their own.
Starting with SL 1.23 (and apparently SonowGlobe too), this has been abandoned: now avatars just grab the baked textures.
So, while your approach would certainly benefit all residents not using SL 1.23, but older versions, as more and more people install 1.23 and subsequent updates, there would be no point in displaying clothes’ textures in the hope of “pre-loading” them. Just recommend to your audience to install SL 1.23 instead.
Besides far less textures to download, a nice side-effect of this new approach is that “stealing” clothes from other avatars using CopyBot will be harder since your computer will not download the original textures any more, but just the baked ones
June 27th, 2009 @ 20:13
Sharron, I’m afraid that you’re confusing two issues here.
One is where you are absolutely right: if everybody in a large audience is forced to be sitting down, using a single-frame pose that cannot be overriden (i.e. it was uploaded at Priority 4), and everybody who stands up is kicked out of the sim, then yes, everybody’s SL client will lag far less — the impact on sim lag will be not noticeable, since it still tracks down that avatars are in a “sitting” position and which animation is being used; but all SL client applications will naturally just download one animation, with just one frame (since it’s a static pose), and that will mean far less client-side lag. So, yes, you’re right — forcing everybody to use the same static pose will indeed reduce client-side sim lag, and that will be quite noticeable.
Forcing people to remove AOs will make little impact if they’re allowed to roam the sim. As an avatar moves, it will trigger requests for animations — no matter if they’re overriden or not (the difference is that each animation is an extra download; animations, however, are usually small in size, and while they download, SL clients will display the default Linden anims anyway: which will continue to create client-size lag, no matter if they’re default anims or overriden ones). So the issue is not about AOs or not: the issue is to force everybody to be sitting down, not move, and use an animation that cannot be overriden. This is the only thing that actually will help reducing client-side lag.
On the issue of Facelights, Ana kindly asked me to write a new article, but in general, you’re correct. Things will even get worse with the new lighting system (not yet implemented in SL 1.23, but already under development by LL with released code; Kirstens Viewer S-17 implements it very nicely). Right now, you’re limited to up to six light sources on an area. This means that Facelights will “steal” lights from the area, but, in the worst case scenario, your SL client will just need to deal with six. The new lighting system will allow unlimited lights in a scene. Imagine what that means for a hundred avatars wearing Facelights!…
On the other hand, Facelights do not have any impact on server-side lag altogether. A badly scripted AO will have a bit of impact, but it’s negligible — scripts just run slower (without affecting the rendering whatsoever).
Your confusion comes from the fact that you’re bundling AOs, Facelights, and other attachments, all in the same bucket. Not all affect client-side or server-side lag. As a rule of thumb, however, a badly or maliciously scripted attachment can, indeed, have some impact on server-side lag — something that sadly has become popular with griefers. Attachments can impact client-side lag, but there are ways to deal with it — on your SL client though.
Besides all the technical issues, there is a probably far more important issue to discuss, which is simply politeness and etiquette. When you’re launching a fashion show, who is more important — the runway models or the audience you’re invited? From a purely politeness point of view, the question is obvious: the audience. They’re the guests. They’re what makes the event a success. If your worries are about aesthetical control when taking pictures of a model — and you place the model’s importance above the importance of your guests — just don’t invite anybody and take snapshots of the model in a controlled environment. That’s why the RL industry takes pictures of models in studios — runway shots are mostly used for the same reason as in SL: showing the audience. Even in RL, lighting conditions on a runway are quite different than on the studio, and that’s why studio pictures are used for magazine covers, catalogues, ads, or any other promotion of an outfit (or the designer), while runway shots are used for the social columns in a magazine. I have never understood why SL should be different, “forcing” the invited guests to be subjected to public humiliation through aggressive language during an event where some people wish to transform a social occasion into a studio.
The same reasoning applies to the audience as well. Just remember that people go to shows to “see and be seen” — watching the models is the pretext for doing so. Again, turning the avatar settings down and enabling avatar impostors neatly solves the issue — you can focus, if you wish, just on the current model on the runway, and forget all about the rest of the audience, which, at ARC 1 using one texture (and not being animated), will not lag your SL client at all…
June 28th, 2009 @ 03:50
Thank you Gwyenth. I am not confused. I do have quite a bit of knowledge as to the technical effects of server side and client side lag. I was writing this reply not as a fully technical piece but for more of a general audience speaking to the effect on the overall experience of a guest or participant in an event. I wrote a fairly detailed article on face lights for Runway Magazine Oct 2008 : http://en.calameo.com/read/000004379c2b12a6a31eb, page 169.
I do agree that sometimes the language of the announcer at a show can become agressive. I think this may come out of the constant frustration that many people don’t comply in the least or show any courtesy as to the resources they maybe using. I wouldn’t go so far as to say they are subject to public humiliation generally. I do wish there were a better way of doing this and that as you say above that myths on lag are not passed on. I disagree with your characterization that my intent is turning a social occasion into a studio. I fully understand the differences in SL and in RL of the use of various types of fashion photography. From what you say I assume you are not familiar with the seasonal supplements that Vogue publishes that are nothing but Runway shots, that you don’t know that for several days after I post photos of a major fashion show on my flicker account I may have over a thousand views a day. A lot of those views are by people that were at the show and “couldn’t see a any thing.”
My intent is that every guest have the best possible experience. It is for the experience of the guest that it is important that the model be able to function as intended. Not because the model is more important than the guest. That is what the designer that is paying for the show expects. The designer is paying for their outfits to be displayed in the best possible way. Considerable time and money are spent on set design, construction, lighting, rehearsals etc.to present the outfits. The model is the vehicle for the outfit. If the models can’t function there is no show. If the script performance of the sim falls below a certain level the models can not pose as intended. At one time all models ran their poses and walks manualy. No AO, no walk replacers, a sreen full of single animation ready to click for the routine and a walk to click when its time to walk. This was done because it was inevitable that scripts/huds would stop working. Some models still do this and most are told to be ready to do this if AO’s stop working. I was trained this way, but i now love having a hud loaded with poses that i can click a button or a function key to change my pose and a walk that starts when I move. The advances in script handling by the system had made this mostly a thing of the past until lately. The fast proliferation of resize scripts in clothing, hair, shoes and jewelry has significantly added to the script load in a sim where a fashion show is being held with many avatars wearing these items. I know that there are “killer” scripts now being provided to remove the scripts when the item has been sized. I don’t think many remove the script or even know why they should. This is the subject for a different rant. I have watched the stats to see the effect of this during large fashion shows, both the total number of scripts and as you show above the script processing time. I have run some tests where i can see the sim script debug screen and some of these scripted “attachments” are using considerable time, for one instance a hair that has 236 prims and a child script in each one that uses 3.6 msec of time when idle. I have a piece of jewelry that uses 7.9msec when idle. Most larger fashion shows now are structuring the show so that the audience and the models are in separate sims or that scripts are set to run only for group in the plot. I wasn’t going to get into scripts because that isn’t an area in which I have the tech knowledge to comment authoritatively. I do have enough tech savy to read the stats and understand what is lagging the sim and what lags the client.
Your attempt at advice in your two last two paragraphs is patronizing and dismissive. I know I can do those things. You miss the point that In many cases I am being paid to get good photos of the models and the audience. You miss the point that what I want is the same as the audience wants, to have the experience of the show as intended. The audience wants to see and be seen, and they want to see the show. I have done my part in this, I have invested in the hardware required and seek to educate myself on the technical aspects of SL and how to best implement what i know. I applaud your efforts to educate on the causes and myths of lag. In light of you passing advice to me, my advice to you is do what you do very well. Stick to the purely technical aspects of this issue, when you venture into opinion and interpret motivations of others I disagree with many of your characterizations and conclusions.
June 28th, 2009 @ 12:59
@Sharron – Let me help you in respect to fashion runway shows. If someone wanting to run a fashion show cannot afford 2 regions then they are out of their league and must suffer horrendous lag. Their lack of funds and high lag is not the fault of the guests. In the two (or more up to 9) region configuration The audience sits in one (or more) region(s) and the runway and models are in the other region by themselves. In this configuration it is still laggy in the audience side but everyone is sitting down and watching the show anyway so it is an acceptable trade off. On the audience side it is a good idea to have the seating easily cammable from the landing point so people don’t have to walk much. On the runway region there is only the usually high geometry decorations (less is more) and the models/announcer. Lighting is nice but people probably force mid day to see the clothes anyway. Ideally the models are in static poses on platforms that move along a pre defined track. This way they do not have to suffer the awkward issues of trying to wade through lag.
Fashion Show Problems sorted. One either has the money to do it right or not. Kind of like real life isn’t it?
Generally speaking there is a real simple formula you can use for builds (including runway stages). If you see more than about 200 to 300 ktris per frame then you experience “lag”. That rubber banding effect. You can be alone in a region and it lags because of the build. limit the prims. use occlusion barriers. I recently went and looked at a fabulous build. 2800 ktris per frame. 3 people in the region. lag city facing the fabulous build. I recently installed occlusion barriers in my store to make it so customers inside are not exposed to anything outside of the store. ktris down to 200 to 300 per frame. I am now building a completely new more spartan store design that has a certain look outside but inside it is all about the product to minimize ktris per frame. that ktris per frame is a fact of video game life people need to learn to deal with because LL is going to cripple the viewer anyway so you better cut back on vertices or your builds (as well as prim attachments on models) will be partially invisible anyway.
July 1st, 2009 @ 12:32
[...] food and pitstops not included), with all my preference settings set to minimum (even tho I know better, but I’m on WiFi, and with WiFi it’s better to “play safe”, [...]
July 1st, 2009 @ 15:41
Thanks for a really useful and interesting article (and the comments, too).
One point about highly scripted attachments (e.g. hair or boots with a resizer and a colour script in each prim) on which I’d appreciate some advice. My understanding is that that they make teleports/sim crossings considerably more likely to fail, since the sim you’re leaving has to note what state each script is in plus any relevant information, and then pass that on to the sim you’re trying to go to.
This all takes time, and the longer it takes, the more likely it is that the hand-over will time out and you’ll get an error message telling you to try again later.
So the more scripted attachments you have, the less likely you are to be able to tp to a popular event, even though your attachments wouldn’t cause much lag if you were able to get there.
Can anyone more expert in these matters than I comment on this?
July 1st, 2009 @ 17:27
Great info! A quick comment on textures… The attribute usually see mentioned is dimension (512×512, 1024×1024, 64×64, etc.). The actual telling attribute is the file size. Various pic editors work differently, but most allow setting the pg ‘quality’ or ‘compression’. This setting has a direct impact on the file size. The higher the quality/compression the larger the file will be. You must choose a setting appropriate for the image you are saving. So, to sum up… when you save a jpg look at the file size and try different compression levels to reduce the size as much as possible while still having a clear image.
July 21st, 2009 @ 23:00
I was very suprised by the amount of information I got from this artical , it taught me more than I ever thought I could find out about our lag issues . I am so pleased that now I have something to share with those that have no clue about it either , thank you so much for clearing this up for me and countless others.
July 26th, 2009 @ 17:30
i love your explanation – very thorough and writtem rather eloquently. so much so that i posted a link to it and will have as many of my residents read it as ican (12 sims)
i thought your analogy of a Jewih family was very well done. i am jewish and love bacon
outstanding write up and thank you for putting it in terms that even i can understand! =)
July 26th, 2009 @ 17:33
[...] an outstanding explanation about lag, read what Gwyneth Llewelyn’s Anotomy of Lag <– read it, it is so important to understand this Print Friendly Categories: second [...]
July 27th, 2009 @ 23:47
[...] Lag ein lesenswerter Artikel… Anatomy of Lag | .Ana Lutetia. [...]
July 29th, 2009 @ 18:48
[...] Gwyneth Llewely’s Anatomy of Lag, Ana Lutetia [...]
August 6th, 2009 @ 05:18
I have a question for Gwynn. How much effect do calling cards in your inventory have on sim lag? Is each card constantly asking for updates on whether that avatar (that you have a card for) is online or offline?
August 7th, 2009 @ 14:12
I want to thank Kurvy Rhode for recently pointing me toward this link:) this is an absolutely excellent article, as well as many of the comments and opinions posted.
I am however skeptical in regard to Sharron’s claim that local light sources contribute to the client side lag, since lighting is largely processed by the graphics hardware which should add very little, if any, overhead indeed. This is not to say that I disagree with Sharron, but perhaps the lighting issue is somewhat exaggerated. My personal observation is that turning local lights on or off in my SL preferences makes no difference at all, perhaps it is dependent on the particular hardware in use as well as the maturity of the graphics card drivers.
Faerie, I do not think that calling cards cause any slowdown since their state simply mirrors that of your friends list…I think that calling cards are simply updated as needed, that is with every notification your viewer receives about a friend logging on or off.
Once again, thank you for such an excellent read:) I will be most certain to bookmark and share this article:)
Love, Rhonda
August 7th, 2009 @ 23:28
[...] http://analutetia.com/blog/anatomy-of-lag/ [...]
August 8th, 2009 @ 00:51
Rhonda, as I understand it, the online status of your friends list is actually driven by the calling cards you have in your inventory. You have a calling card for everyone on your friends list, plus everyone who gave you their calling card, and unless you have deleted it yourself you still have one for everyone that used to be on your friends list but isnt anymore!
And how do the cards know if someone is online or offline? I also understand that each cards asks the server “what’s this person’s status?” at regular intervals, but how often? It seems to be a fairly short interval because I have never seen two people go offline at the same time which you would expect to happen sooner or later if the interval was like once a minute or more and all your cards ask for updates at the same time.
So you can see that this *could* result in a lot of data being sent back and forth if there are a reasonable number of avatars on a sim who have accumulated a large number of calling cards over their years in SL and each card is asking for updates every few seconds. I just wonder what effect that might have.
August 8th, 2009 @ 01:35
@Faerie, actually, quite a lot! You’re very right, calling cards are quite demanding on the generated traffic…
August 8th, 2009 @ 04:15
actually, it seems that those calling cards that do not have a corresponding entry in your friends list no longer reflect that person’s online status, since that would defeat your ability to hide your online status from people not on your friends list. I still think that it is not a matter of the viewer asking the grid for status updates, but the server sending updates to the viewer as they apply, which once they reach your viewer would update your friends list and calling cards at the same time. I am not certain whether calling cards are actual assets, but assume that they are not, since they really are only links to each person’s profile.
I do want to emphasize that this is simply my opinion and a logical conclusion I am drawing from what I see, if I should be completely wrong I do apologise:)
August 11th, 2009 @ 19:38
@Rhonda.
Rhonda what I wrote about lighting causing client side lag isn’t something i came up with on my own. It came from transcripts of Linden office hours on windlight and graphics and help from Lindens. I don’t know if you read the article I linked to in my comments. I sent the finished article to Torley Linden before it was published to vet any technical claims I had made. He sent it to BigPapi Linden who is one of the graphic gurus. He sent it back with a few suggestions. My statements are not based on my opinion or suppositions, but on this research.
Here is a paragraph from my article, the statement in quotation marks are bigpapi Lindens words, not mine.
—-
Client side lag is significantly affected by face lights. An avatar that is wearing face lights creates lag for all others in the environment that “see” the light no matter what the wearer’s settings are. “They are huge client-side lag generators. Each time your avie moves or changes its pose, the SL viewer has to recalculate the lighting on each polygon of every prim and avatar mesh in a 10m radius”. (The lower your radius setting, the fewer prims and avatars you are changing the lighting on). If you are a model walking in a show wearing “bad” lights you create lag for the other models if they have local lights on, even if you don’t see the lights.—-
All that said, when I am in a show I always have local lights off and now most of the time whether I am in a show or not I have attached lights turned off in advanced settings. I do agree completely with what you say about the graphics card being the key factor in the lag that any user experiences related to local and attached lights. I have a system that can deal with this very well. I wrote this mostly for those that don’t have an ultimate system, especially those on “normal” laptops. My pc is running at 3.5GHz,4gb mem, with a nvidia 260. Like you I don’t see much difference in lag if I have lights on or off. They are for me mostly annoying and if i am shooting a show I have to have them off because the model lights vary in intensity and radius so much and the lights of audience members interfere as well. Sorry I got a little off topic here and into my lighting rant. Its’ a major irritation to me.
sharron
August 11th, 2009 @ 20:39
@Hammond
I am not sure from your comment what you are getting at. What you wrote is very true for outside SL. Inside SL it is a different case. When you upload a image to SL it is converted to a jpg2000 and 1024X1024 maximum resolution no matter the uploaded format, file size or pixel depth. If I upload a 2024X2024 png of several mb once it comes into SL it will be converted to a 1024X1024 jpg2000 just as a 1024X1024 compressed jpg will be. Of course the high rez png will look the best after sl conversion because it will not have the artifacts from being compressed several times and the higher pixel depth to begin with, but once it is in SL as a 1024X1024 jpg2000 each will use the same resources to store and display (rez).
The following is from a notecard I wrote to digest the main points and effects of texures on lag from the technical data in the SL forum sticky to pass to friends or any one that cares. The technical info is from the forum.
Texturing tips for SL builds:
Check out this sticky in the texturing forum about texture resolution and how it affects lag. It begins with a discussion of file types, how SL deals with them, then into the resolution information.
http://forums.secondlife.com/showthread.php?t=150360
The main point is to use as low a resolution as you can to and still have the clarity and sharpness you want. Using a higher resolution texture than needed is a major cause of lag in a lot of builds, clothing and jewelry. Avoid using 1024X1024 textures if possible in any build that you want to be as lag free as possible. A clothing or hair designer should never use 1024X1024 textures on prims to be worn. There is no discernible difference to the eye on a garment size prim between a 1024X1024 and a 512X512 texture. The smaller the prim the lower the pixel depth required.
You can also see in this chart why alpha textures use more resources and take longer to load. It is the bit depth required.
These numbers will give you a quick example so you know how important this can be. This has nothing to do with actual file size or format of the uploaded texture, this relates to the bit depth which determines the video memory required to display the texture. (fuzzy distorted pixels use the same amount of display memory as sharp ones)
1024X1024. 32 bit with alpha = 4MB
1024X1024, 24bit = 3MB
1024X512, 32 bit with alpha = 2MB
1024X512, 24 bit = 1.5MB
512X512, 32 bit with alpha = 1MB
512X512, 24 bit = 768MB
And so on, taken from: http://www.scifigeeks.net/images/texture_memory_chart.png
What is apparent in this brief chart is that a 1024 texture is 4 times the bit depth and uses 4 time the graphics memory as a 512. Same holds true as we go down, 512X512 has 4 times bit depth and uses 4 times the graphic memory as a 256X256.
This is why a large store with many “board” displays in some cases takes so long to load. There are too many examples in stores where the texture on the board wasn’t a high resolution image to start with, uploading and using it on a 3m display at 1024X1024 is a waste.
sharron
August 11th, 2009 @ 21:57
@Sharron,
My goal is to give guidelines when preparing a file for uploading to SL, and SL can only use as good as it is sent. An image saved at a high compression level will loose information and thus details. Compression has this effect regardless of the file dimensions and it happens before SL ever has the file. Once SL has the file it will get saved as you have described, but during the file upload, SL can only read at the compression level saved.
August 11th, 2009 @ 23:13
Sharron dear, I can only say wow! I know that you are technically very savvy but it seems that I do not realise how much so:) of course I understand that the lighting issues in SL can be a real bother for photographers as they have been for me…it seems, though, that my assumption about lighting calculations in hardware versus software was somewhat misguided. Thank you for pointing me toward the articles you quoted – I had not been aware of them, and I can say that I have learned something from you yet again. *smiles:)
August 25th, 2009 @ 18:40
[...] read the entire article – it’s a long, highly instructive [...]
September 19th, 2009 @ 11:02
[...] forums for hints about those. Two excellent posts by Gwyneth Llewelyn will give you some insights: Anatomy of Lag, and Lag Myths [...]
October 3rd, 2009 @ 02:09
[...] I just recenly discovered a blog post written by Gwyneth Llewelyn. ANATOMY OF LAG will give an understandig of lag without getting very and too technical. I highly reccomend reading it HERE [...]
October 24th, 2009 @ 18:27
People…please disregard this article. It it totally bunk at all levels. Take it from someone who’s been running an island for a couple of years now. Avatars…mainly scripted hair, boots, and radars…are what lag an otherwise fast server. The server has 22.2ms every second to do everything it needs to do. If the script time goes over that the server lags. I regularly see avatars using between .6ms and 12ms of script time depending on the person and their attachments. IM Grim Bracken if you would like to discuss to FACTS about sim lag rather than this garbage article.
October 24th, 2009 @ 18:33
Dear Grim Bracken,
The article was wrote by someone who has been inSL for 5 years. As for me, in a couple of weeks I’ll be 3 years inSL.
And… somehow you are, with your two years experience, are the person with all the facts? Why? Because of your experience of running an island? Right…
October 25th, 2009 @ 07:43
Yep! Experience is the best teacher. 5 years in SL doesn’t mean you have the experience. You need to own your own island (mainland doesn’t work) to see the nitty gritty of sim resource usage.
October 25th, 2009 @ 22:45
And… who says we don’t?
You did read one of two things about Gwyn’s history inSL before making the previous comment?
I find it interesting that you’re the first to say that this article is rubbish when a lot of people that really know how SL™ works agree with the post and even quote it. However, you and the experience of running a sim say otherwise. Interestingly odd!
October 26th, 2009 @ 12:12
@Grim, perhaps the issue you have is something slightly different, and it has to do with the rather loose way we speak about “lag”, which has many meanings, depending on what you have in mind.
Before you joined SL, running a lot of scripts really did lag the server — in the sense that it started to run other things slower. What other things? Well, pushing textures downstream to the avatars, for one. Or managing the avatar’s position and current animation status. So, the more scripts were running on the sim, the less textures were been sent, and the less accurate the tracking of avatar positions was. On top of that, even public chat lagged! So running too many scripts did really affect the whole of the sim.
Things have dramatically changed since then, and I’m sorry if I wasn’t very clear on the details. Now scripts are handled separately from the rest of the sim’s duties. So if there are a plethora of scripts running, they lag each other — in the sense that all scripts start running slower, sometimes too slow to see them working at all! — but they do not affect the sim’s performance in handling the remaining tasks, e.g. texture downstreaming, avatar position tracking, chat, and all the other functions a sim has to handle.
What this means is that if you see a lot of script running, the time slice for the scripts will rise dramatically, to the point that all scripts start responding very, very slowly: AOs get out of sync (not to mention poseballs or coordinated animations!), avatar radars might not get properly updated, and, worse for many shops, script-based shop vendors might be so slow in responding that they never get the avatar the item they have just bought, driving them to frustration (and a complaint to the owner!).
However, none of the other functions in the sim are affected by that. Scripts just lag other scripts, that’s all.
Now what you experience is something different. When a lot of avatars with their dozens of scripted attachments enter a sim, they will make all the scripts in the sim run slower. But — since there are now many avatars in the sim, too! — the sim’s other functions will get slower, too: there are simply too many textures to download for each avatar. So you see two things happening when many avatars are in the same place: scripts running slower (or even stopping), but also movement becomes harder (time dilation goes up as the sim isn’t able to track all the avatars in real time any longer), and textures fail to load (or take a lot of time to do so).
This is the old problem of “correlation” and “causation”. You observe a lot of avatars with scripted attachments entering a region, and the sim starts to lag; this result is reproducible every time, e.g. you can demonstrate a correlation between “avatars running scripted attachments” and “lag” without a shadow of doubt. However, there is no causation effect — it’s not the scripted attachments that are creating the lag, but merely a lot of avatars in the same place (all of them downloading textures, all of them requiring positioning data, all of them running animations [with or without an AO], and so on). An interesting corollary is that if you get 20 bots in an area (even with scripted attachments!) they will lag less than 20 human residents with their avatars, even if all of them have ARC 1: the difference is that bots do not download textures from the sim, while the humans certainly will do!
In conclusion, yes, a lot of avatars in the same area will lag the sim, but it’s not because of their scripts. The scripts will not affect texture download lag, nor positioning lag, nor public chat lag, and so on. They will make all scripts in the sim run slower (or make some stop altogether).
Just off the record, Linden Lab is trying out a new mechanism where scripts won’t be able to consume so many resources, and this would effectively mean that “badly behaved” scripts will be penalised and not run so fast (or consume as much memory), as well written, resource-friendly scripts. A first implementation was created for the HTTP Server function in LSL. It’s expected that this will work for all the other LSL functions too, and it would mean that well-written scripts won’t get slowed down by badly behaving ones, which shall be a great improvement!
October 29th, 2009 @ 18:17
This is the most uninformed pint of drivel I have read in a long time. First off the author seems to attribute most lag to the client side of the operation. A better informed article would have stted at the beginning that there is server lag and client lag. Then there are network problems that cause lag. There seems to be a complete misunderstanding of how the process of time dilation works and what time dilation is. The article appears to suggest that a sim running 2 million scripts would not be laggy just the scripts would run slower!!! What the heck do you think lag is? the time dilation is a measurement of the difference between the physics frames per second and the sim fps. a im running 20 scripts with one person on the sim will be far less laggy than a sim running 5000 scripts with one person on the sim. Any joyeous denial of this seems at odds with intelligence and reasonable discussion. I applaud your ignorance I hope you are also in a state of pure bliss
October 30th, 2009 @ 00:22
@Jess, I’m sorry that you are so furious, but really, insulting others will not make you a happy person
You might be right that this article could have started with explaining the difference between client-side lag and server-side lag, like this one on the LSL Wiki does. But it’s a question of style really; I preferred to explain it in the middle and not at the beginning. We can surely discuss what the better approach is, and why one is better than the other.
Your issue seems mostly to be about what time dilation means, and how it is explained. The LSL Wiki explains it like this:
“Time dilation occurs when the sim can’t keep up with the processing of its tasks even after reducing the time allocated to scripts and physics. Avatars will experience this as slowed-down (slow-motion, “bullet-time”) movement.”
Tasks is an obscure definition used by Linden Lab to refer to objects rezzed on a sim (as opposed to objects on the inventory or on the asset server). You don’t need to take my word on it, ask Kelly Linden
Your alternate definition of “Time dilation is a measurement of the difference between the physics frames per second and the sim fps” is mathematically accurate, but doesn’t explain what it means. My choice was to explain what time dilation is, instead of posting a mathematical formula. Your assumption that I don’t understand maths doesn’t take into account that I always prefer to explain what’s behind the maths, not briefly stating a formula and saying, “that’s why we have lag!” — which doesn’t explain anything.
With a more technical twist, I could have said (but avoided to be too technical on Ana’s blog; yes, trust me, I address issues differently for different audiences
) that scripts that move/change/manipulate rezzed objects (“tasks”) or affect avatars will require more processing from the physical engine (Havok 4, in SL’s case). If the burden on the physical engine is too high, it has not enough time slices left to continue to update the scene in real time for all the users — time dilation is thus a measurement of how quickly the sim is able to proces all the information related to tasks in the sim, compared to ‘real time’ (in practice, 45 FPS is almost twice ‘real time’, since we humans process around 20 frames per second — thus, time dilation of 0.7 is bad, but not terrible; everything below 0.5 is really too much).
Now how do scripts fit into this? Obviously, a script that is placing a heavy burden on the physics engine (like, say, a vehicle or a gun shooting a projectile) will increase the time dilation. That’s unavoidable. Scripts rezzing prims or moving them around, specially if they’re set to physical, will increase the burden on the physics engine, and this will increase time dilation if there are too many of those around.
And your extreme case is also correct, but for a different reason. 2 million scripts in a sim will take a huge amount of memory — even if they aren’t doing anything. But that memory is needed for far more things beyond scripts: for instance, all rezzed objects on a sim are on the sim’s memory. If there is no (physical) memory left, this will mean that the sim might require virtual (disk-based) memory instead, and, of course, this is about 100-1000 times slower. So obviously 2 million active scripts (or even ‘only’ 5000) are far worse than just a handful of scripts — even if they’re doing ‘nothing’. (Note: LL is addressing the issue of ‘memory abuse’ on a future server version; it’s being actively implemented as I write this.)
However, the biggest burden placed on the physical engine, for most of the regions in SL, is not scripts updating objects — it’s avatars. Avatars are unavoidable lag bombs since they all require interaction with the physical engine to track their constant movement around a sim
So, to recap — scripts don’t directly affect time dilation unless they are manipulating objects (or avatars). Thousands or tens of thousands of running scripts will affect a sim because all that memory has to come from somewhere — but that’s not related to time dilation itself, just the way any computer works with a finite amount of memory
The problem here is that the usage of the word “lag” is too widespread, when actually it should really just be used for network delays (not covered in my article at all — deliberately so). We tend to employ the word “lag” to say “it’s slow”. Linden Lab prefers to call server-side lag “time dilation”, which is measurable and more accurate. Client-side lag is by far more common than server-side lag, but there are good ways to deal with it, by using clever building techniques.
And you’re right, I do live in bliss, but that’s because I don’t get upset with anyone, even if they insult me in public
November 14th, 2009 @ 06:11
[...] Here is some educational material on the subject. Be sure to read all the comments. http://analutetia.com/2009/06/22/anatomy-of-lag/ [...]
November 15th, 2009 @ 09:23
[...] good resources are Anatomy of Lag and Internet [...]
December 13th, 2009 @ 19:38
We run a heavily textured replica of the city of Munich, Germany. All the involved builders who contributed textures over the years went through a
learning process. And the discussion about the best texturing strategy started on day one (see
http://www.echt-muenchen.de/blog/2007/05/lag-prims-vs-texturen-lag-ist-unser.html). Today the sim is one of the most realistic sims in Second Life. But
we have severe (client) lag problems. People with slow computers can barely walk through the streets of Munich in SL. That’s why we started to hunt for
disproportionate large textures. I know there are tools for analysis builtin the client (e.g. Texture Area (sqrt(A))) but I cannot find any documentation.
Can anybody here give me a hint?
December 20th, 2009 @ 09:42
[...] myths people have about Lag. For those of you that don’t have time to read the full article here. I will summarise it for you! Basically one massive cause or even way that lag occurs isn’t just [...]
December 22nd, 2009 @ 04:19
It’s amazing how many people jump out of the woodwork as soon as something like this is posted and say “IT’S NOT MY FAULT, IT’S YOURS!”
Typically in a whiny, childish voice. (Or can easily be read in one.)
Honestly, I knew something was up when I could, on various low-end computers (Windows XP, Vista, and Mac OSX) get petter performance than people on allegedly better computers than me. This goes for SL and WoW.
Why is this? I’ve set both programs to run what my computer can HANDLE. If I have lag, I know it is usually my own fault, simply based on the fact my computer isn’t that powerful.
I link this post everywhere. It’s been very helpful and informative, and I just wish people would own up to their mistakes instead of always blaming everyone else. The world, not just the grid, would be a better place.
December 30th, 2009 @ 13:32
[...] I discussed the issues with lag related to the avatar we looked at Gwyneth Llewelyns article at Analutea to gain a clear understanding of lag and all that is involved. One of the previous mentioned areas [...]
February 24th, 2010 @ 17:41
[...] Anatomy of Lag [...]
[WORDPRESS HASHCASH] The comment’s server IP (66.135.48.209) doesn’t match the comment’s URL host IP (76.74.254.123) and so is spam.
March 12th, 2010 @ 13:30
[...] http://analutetia.com/2009/06/22/anatomy-of-lag/ [...]
March 13th, 2010 @ 06:04
[...] If you want more information about reducing lag, read Taturo Nino’s article for Massively, Gwenyth Llewellyn’s blog for Ana Lutetia, and Joel Foner’s [...]
March 22nd, 2010 @ 23:56
Here’s a link for everyone that should show you that you shouldn’t believe anything in this blog post. To enlightenment…
https://blogs.secondlife.com/community/technology/blog/2009/12/15/script-limits
March 23rd, 2010 @ 04:25
This blogpost does bother you a lot, doesn’t it…?
I really can’t see the connection between the link you posted and this article but I am sure that you will enlighten me someday… or not.
March 23rd, 2010 @ 06:23
@Grim, thanks for the link! I’m glad that Babbage went public in December, explaining with a bit more detail what I did allude twice on the comments: that Linden Lab is addressing two current problems with scripts, namely, too many scripts consuming all memory in the server, forcing it to use disk space (which is up to a thousand times slower than memory) instead; and the situation where “suddenly” (i.e. in a very short time span) a sim has to deal with a huge increase of scripts due to, say, dozens of avatars, running thousands of scripts each, all teleport to the same sim at the same time.
This is a situation I’ve alluded on comment #73:
Just off the record, Linden Lab is trying out a new mechanism where scripts won’t be able to consume so many resources, and this would effectively mean that “badly behaved” scripts will be penalised and not run so fast (or consume as much memory), as well written, resource-friendly scripts. A first implementation was created for the HTTP Server function in LSL. It’s expected that this will work for all the other LSL functions too, and it would mean that well-written scripts won’t get slowed down by badly behaving ones, which shall be a great improvement!
and #75:
And your extreme case is also correct, but for a different reason. 2 million scripts in a sim will take a huge amount of memory — even if they aren’t doing anything. But that memory is needed for far more things beyond scripts: for instance, all rezzed objects on a sim are on the sim’s memory. If there is no (physical) memory left, this will mean that the sim might require virtual (disk-based) memory instead, and, of course, this is about 100-1000 times slower. So obviously 2 million active scripts (or even ‘only’ 5000) are far worse than just a handful of scripts — even if they’re doing ‘nothing’. (Note: LL is addressing the issue of ‘memory abuse’ on a future server version; it’s being actively implemented as I write this.)
Note the timestamp of my two comments. They were written before Babbage’s article in December and pretty much explain what Babbage later announced publicly.
This is what Jess Parker meant when noting that a sim running 2 million scripts would lag a sim far more than a sim running 20 or even 5,000 scrips. Jess is right (and I agreed!). In fact, if a server has 4 GBytes of RAM, and runs four regions, this would effectively limit the number of running scripts to a theoretical maximum of 16.7 million scripts (just do the maths); in practice, that number has to be way lower, as the simulator software takes memory for other tasks, too. Jess’s arbitrary number of “2 million scripts” might be quite closer to reality.
The surprise for me was understanding that this situation — running hundreds of thousands or millions of scripts — might be more common that I thought. I thought it was an extreme, hypothetical value. But apparently it’s not so hypothetical at all. Apparently some avatars, thanks to resizing scripts and extremely complex HUDs, can easily run a thousand or two scripts at the same time; hundred such avatars would be running 100-200 thousand scripts, and that doesn’t include similar amounts of scripts running inside the sim’s prims themselves. So possibly there are many areas in SL with that level of script density — enough to warrant Babbage and his Mono developer team to implement limits for those.
And as a side-note, things like resizing scripts on hair can nowadays be done with different functions (freshly introduced) that allow a single script to control all hair prims (instead of having one script per hair prim). But it will take some months (or years!) for all hair designers to change that…
April 20th, 2010 @ 08:07
[...] http://analutetia.com/2009/06/22/anatomy-of-lag/ [...]
May 10th, 2010 @ 17:30
While I agree with alot of what you said I found your writing style to be longwinded and patronizing. One of your major points was “use smaller and fewer textures” its a simple concept that requires little in the way of explanation..a paragraph at the most. Patronization is the definition of condescension and people pick up on it pretty quick. In short you could have expressed your ideas in fewer then half the words and had a much more interesting and educational article.
May 19th, 2010 @ 18:25
Patronising? o.O
I for one appreciate the detail to which Gwyneth has gone to explain the technical side of things, all in order to explain *why* and *how* certain things cause lag.
No one is being patronising here, no one forces you to read all of it if you do not want to. There are plenty of other articles about lag, including many that are quite lacking in information. Perhaps you would prefer to read them, but kindly do not be condescending to those who seek the level of information provided here.