Efforts towards fixing the lag

[ShireDweller]

It takes SO LONG to log in, like at least 5 minutes on my 75 lm. Lower level characters are better but still take a few minutes. Maybe it's because they're outside the great river area, idk, I can test this later. But when I'm logging, the little glowy circle above the 'loading...' screen will freeze and if I click the window it will say 'not responding' but then recover after a few seconds.

I get this too on my level 71 character, long loading time plus "client not responding" a few times till I get in. I also get the "lotroclient not responding" message every time I exit the game since Update 6. Windows looks for a solution and then tells me to just close it. I've got the in-game problems as well.

Sapience, thanks for telling us about the "halo" effect - I can appreciate the difficulty of pinning down the cause(s) of the problem.

[auximenes]

A stutter is essentially the same as a freeze, it's just that those with slow setups experience a freeze, and those with fast setups experience a stutter. The stutter lasts as about long as the glitch, the freeze lasts longer as your setup works frantically to catch up. Anyway, that's my guess so far.

That sounds about right.

I notice this too. When i walk around a town, or ride around any area, the game stutters, or hitches, as some call it. This never happened to me pre-U6. Now it happens no matter where I go. I even reduced my GFX settiing in an vain attempt to fix it. Apparently it has nothing to do with settings. It simply hitches and stutters for no apparent reason when you travel.

I tried the same thing. My stuttering/freezing does not appear to be a graphics issue. I did have a similar problem before U6, maybe at RoI launch, but it went away I think after the first follow-up patch. The problem resurfaced with the Great River patch.

The other thing someone mentioned was they play in windowed mode not full screen. I too play in windowed mode and have very few problems with lag comparatively to what is described in this thread.

I normally play windowed, and I've tried full screen as well. Both ways are equally affected.

[ChromiteSwiftpaw]

I'm starting to believe that a major trigger for lag is proximity to other players, as has been hinted at before.

I don't believe so. I was harvesting Dwarf-iron ore by myself in the deserted Trollshaws the other night and I was still getting the lag. So there goes your theory

[Jaedor]

Just before U6, I got a brand new gaming rig:
AMD Phenom II 3.8 GHz
16 GB Ram
GeForce GTX 560 Ti
120 GB SSD + 1TB Data drive
850W PS
Win 7 Pro 64Bit

For a couple weeks after I got the new pc, the game would load my level 75 in about 30 seconds, lower toons in about 15 seconds. I was getting ridiculously high FPS in static settings like my house, averaging 150 FPS on the landscape, ranging 60-100 in Galtrev. Latency steady at between 79-83.

But after U6 went live, I noticed a hitch or odd kind of rubber-banding. The effect was like moving my character a half step to the side. At first I thought it might have to do with the new dismount animations, but then I noticed it happened every time I came to a stop - whether riding, running or walking. The effect is like snapping my character to another location on the grid.

I've only had the stable mount freeze issue once - on an alt in Moria, going from 21st Hall to Orc Watch. The freeze lasted about 10 seconds, long enough to really see this issue other people were talking about, mouse over the latency alert panel and figure I was going to lose the connection. But it didn't crash and the mount kept going until it reached Orc Watch. Latency didn't change during the freeze and neither did FPS.

These days it takes several minutes to load any of my characters.

[Osgril]

I don't believe so. I was harvesting Dwarf-iron ore by myself in the deserted Trollshaws the other night and I was still getting the lag. So there goes your theory

Yes, I had the same problem as I said. Just meant that proximity to other players seem to make it happen more frequently. If I'm reading Sapience right it seems they're on that track too (the halo effect he mentioned).

[Sapience]

Yes, I had the same problem as I said. Just meant that proximity to other players seem to make it happen more frequently. If I'm reading Sapience right it seems they're on that track too (the halo effect he mentioned).

Actually halo effect is more like the butteryfly effect. A butterfly in chicago flaps its wings and a Tsunami takes out Charlie Sheen's beach house in Malibu. Seemingly unrelated things that impact each other in almost mystical ways. That's a halo effect.

[Armitas]

Actually halo effect is more like the butteryfly effect. A butterfly in chicago flaps its wings and a Tsunami takes out Charlie Sheen's beach house in Malibu. Seemingly unrelated things that impact each other in almost mystical ways. That's a halo effect.

So Ashton Kutcher was in the movie butterfly effect. I'm just saying...I smell conspiracy....

[FormulaTroll]

Actually halo effect is more like the butteryfly effect. A butterfly in chicago flaps its wings and a Tsunami takes out Charlie Sheen's beach house in Malibu. Seemingly unrelated things that impact each other in almost mystical ways. That's a halo effect.

I don't think it means what you think it means.

[Armitas]

I don't think it means what you think it means.

Inconceivable!!



...but I digress...

[Sapience]

I don't think it means what you think it means.

Nope. Sorry, you're wrong.

And we now return you to the proper context.

[Evenprimes]

I don't think it means what you think it means.

Agreed. The usage with which I'm familiar refers to bias (and cause and effect only within that limited context), not general cause and effect.

[dauthiatull]

I have seen on another game ( cant remember the name) where the number of spawns ( items the player could interact with or event triggers not just mobs) on the land block would cause lag.
when a large number of them would spawn or despawn it would lag. also when crossing land blocks you would lag as the client would have to swap out the items for new ones.

it seems sometimes the lag is bad and sometimes I have none. I've been trying to keep track of it by taking note of the spawn density around me and the amount of lag.

is there some sort of trigger or event involving spawns that is on a timer that is supposed to be synced to all clients in proximity(same land block)?

[Nostake]

I apologize if you were offended. Let me try to explain what "client side' means. It certainly does not mean 'your fault' as it seems you took it to me. It means the game client itself seems to have some sort of issue. Because this is an MMO and grouping requires cooridination of clients to some extent, grouping will actually make the issue more pronounced, causing all clients in the group to react the same way. So in fact, if you were not seeing a whole group suffer the same issue at the same time then the more likley culprit would be something unique to those player suffering the issue and not the client, or even possibly the server.







In short, correct.





That's pretty close, and why tracking it down is hard. Lag means specific things. Unfortunately anything impacting game play is usually listed as lag or 'server issues' by player, even when it's something else. You'd be surprised how many times we get 'lag' reports from people who are running 7 year old machines on HIGH and have massive hitching. Turning down the graphics settings to medium immediately makes it go away but they report the 'lag' is back as soon as they put the game back to HIGH.

Clientside networking issues (especially packet handling) can cause hitching locally as well with no apparent cause or impact anywhere else in the connection (thus why we are looking at servers that are performing as expecting with no issues, but users are experiencing the issue).




The client side networking code is part of what we're investigating. There seem to be several changes that together are impacting performance in very unexpected ways. What we call "halo" effect.

Halo effect can be pretty crazy sometimes. Imagine someone down the road from you plants an apple tree and your toilet backs up. The two things shouldn't be related, but somehow they are. That's halo. trying to find what connects the two things is not an easy task.

With how the lag is so greatly affecting MOST of you paying customers i sincerly hope that turbines tech's are working weekends to get this issue resolved asap.......i know many people who are going to be dissapointed to the point of not loggin in anymore when 6.1 does not fix the lag

[Khafar]

These things can be hideously difficult to track down, and it's not uncommon that efforts to add more logging or to instrument the client will alter the timing sufficiently to make the problem disappear. Or even manifest in a totally different way. Every experienced developer usually has a few war stories along those lines, and one I was involved with some years back wound up taking approximately 500 hours of engineering time to track down. Real time, it sucked about 3 60-hour weeks out of my life. Ouch.

The following from Asheron's Call was actually a relatively simple problem, yet because it was so difficult to quantify... it was also difficult to track down to a root cause (and fix). Took a couple of years, actually. Sorry for the length, but it's illustrative of just how hard it can be to get to a solution sometimes:

For those of you who don't know what the Wi Flag means, a little history. From the beginning of AC, some players have complained about unbelievably bad luck. When the swarm of Lugians spawn in the citadel, they will go after certain players--every time. The player's level doesn't seem to matter, nor does the number of other players in the room. What does seem to matter is that this player is cursed with that most unfortunate of distinctions: the Wi Flag.

For some players, the flag came and went. For others, it was a perpetual nightmare, present in nearly every monster experience. To live a Wi-Flagged life meant to be hunted at every turn. Perhaps other adventurers could know peace in a BSD or a Citadel, but there was no rest nor respite for one under Wi.

Our developers at Turbine initially answered these complaints by saying that they could find no such bug. Occasionally, a senior Turbine engineer could be found who would admit that perhaps there was something “not quite right” with the system, but they still could not identify a cause, if one even existed. Easy culprits, such as a malfunctioning random-number generator, were eventually dismissed.

But our search went on. For there were people even at Turbine convinced that the Wi Flag existed, and that they had it in spades.

And then one day, long after most people had learned to either forget or ignore the Wi Flag, the answer was found. Here in the report from Sandra Powers, AC Live's Lead Engineer, on the nature of the Wi Flag. We warn you in advance that it is a very technical explanation, but we hope it is of some interest to those of you who have long been afflicted with this terrible burden.

“Many of our players have complained for a long time that their character are `Wi-Flagged'--that is, that creatures attack them a much greater proportion of the time than random chance or distance should dictate. After looking at the code in depth, I believe I have found out why this might happen.

Generally, a creature chooses whom to attack based on who it was last attacking, who attacked it last, or who caused it damage last. When players first enter the creature's detection radius, however, none of these things are useful yet, so the creature chooses a target randomly, weighted by distance. Players within the creature's detection sphere are weighted by how close they are to the creature -- the closer you are, the more chance you have to be selected to be attacked.

The actual algorithm for selection looks like this: We roll a random number within a certain range--say between 0 and 1. Each player is given a portion of the range based on how close they are to the creature. The closer you are, the larger a portion you get. The player who owns the portion into which the random number falls is selected to be attacked.

This algorithm is sound. The problem comes up when we are assigning portions of the range to various players. If we wanted distance from the creature to be proportional to your chance to be selected--that is, if the closer you are the *less* chance you have of being attacked--then we would assign this range by taking your distance from the creature over the total distance--the distances of everybody under consideration added together. But we really want the inverse of this ratio--so that the closer you are, the *more* chance you have of being selected. So we invert this ratio by subtracting it from 1 to assign you the size of your portion.

An example:
A is 5 meters from the creature.
B is 2 meters from the creature.
C is 3 meters from the creature.
D is 10 meters from the creature.
Total distance is 20.
The size of A's portion is 1 - 5/20, or 0.75.
The size of B's portion is 1 - 2/20, or 0.90.
The size of C's portion is 1 - 3/20, or 0.85.
The size of D's portion is 1 - 10/20, or 0.50.

So we assign these people these portions of the total range:
A has between 0.00 and 0.75.
B has between 0.75 and 1.65.
C has between 1.65 and 2.50.
D has between 2.50 and 3.00.

Notice, however, that while the original ratios added to 1 (.25 + .1 + .15 + .5 = 1.0) that the inverted ratios -- and thus the total range from which we should have rolled the random number -- no longer add to 1. Instead, they add to 3. (Some algebra will convince you that the assigned portions always add to n-1, where n is the number of people under consideration.) So in order to randomly select some portion of this total range, we should roll a number between 0 and 3.

But in the existing AC code, we always roll a number between 0 and 1.

You can easily see in this example that if the random number is always been 0 and 1, only A and B have any chance at all of being selected, and A has the majority of the chance. And the reason that these two have all the chance is simply because they are first in the list, and so were assigned the low parts of the range. Normally--if we had rolled between 0 and 3 in the example--your order in the list should have no effect on how likely you are to be chosen. But because we only rolled between 0 and 1, the earlier you appear in the list, the more skewed your chance of selection is. And as it happens, in AC code, your position in this list is determined by the InstanceID of your character, which is assigned when you create the character and never changed. (Note that the InstanceID is hashed--mutated by the system into another number--to determine position. So it's not a simple relationship like the older the character, the earlier in the list they will be. It is, however, a static relationship--an ID that hashes to an early position will always hash to an early position, although it's exact position will depend on what other ID's are also under consideration.)

So what does this mean? The way this random targeting algorithm is implemented right now, if you happen to have an InstanceID that hashes to an early position, you will tend to be attacked more than your fair share when the creature is using random targeting, regardless of your distance from the creature. In other words, you are Wi-Flagged.”

We're glad we were finally able to fix this bug. With the July 2002 Event, may you know peace in the fast-spawning dungeon of your choice!

Khafar

[shann81]

I work as programmer and it's clear that Turbine developers broke The First Rule of our Order.. :-)

Don't change source code that is already working however bad that code looks or how good is your idea to improve it.. :-)

[qetuop]

I work as programmer and it's clear that Turbine developers broke The First Rule of our Order.. :-)

Don't change source code that is already working however bad that code looks or how good is your idea to improve it.. :-)

I also do programming. The rule I have to follow is: Do what your managers tell you even after you warn them that changes will break stuff.

[Nitten]

I mainly notice the "rubberband" effekt, when I have my shared storage open and will drag items into it or over to my own vault from my bags.

a nother thing, i also noticed that sometimes the game thinks that i doubled click on my mouse, when i try to drag things to the vault.

It is primery in Galtrev, and when deconning in "new world" when i approch a old vault keapers i dont have the problem.

ohh so much food has been eaten og potions consumed because of this

[PerinStone]

With how the lag is so greatly affecting MOST of you paying customers i sincerly hope that turbines tech's are working weekends to get this issue resolved asap.......i know many people who are going to be dissapointed to the point of not loggin in anymore when 6.1 does not fix the lag

Since they haven't found the problem and the patch notes don't mention the lag, why would anyone expect it to be fixed when 6.1 is released?

[Mark_J]

Since they haven't found the problem and the patch notes don't mention the lag, why would anyone expect it to be fixed when 6.1 is released?

The more important question is will they still release 6.1 knowing they have a huge existing problem that they can't figure out how to fix?

[Gylve]

Does the halo effect mean that 6.1 will be an Xbox360 exclusive?

Looking forward to my True Heroics bubble being permanent and recharging when I don't take damage for a few seconds. Since I already dual-wield, I'm excited that I'll soon be able to quadruple-wield. Also excited about being able to play a brute in the 'moors. My one concern is that I might not use the word "bro" enough in voice chat, so I guess I'll have to work on that in order to be prepared for 6.1's release.

[Khafar]

The more important question is will they still release 6.1 knowing they have a huge existing problem that they can't figure out how to fix?

The answer to that is almost certainly "yes". Not everyone is impacted by this (or at least impacted by it much), and it's just foolish to suspend all other work because one problem is proving intractable. "All hands on deck" to solve some problem is rarely useful, and if they do have a client packet handling issue... content developers, artists, game systems designers, etc can't do a thing about it.

I actually have a problem like this going on right now where I work. Myself and one other engineer have been assigned to work on an intermittent resource contention problem that's manifesting in a variety of ways and is proving quite difficult to track down to a root cause. We've been doing nothing else for a week now, but the rest of our team is moving along on their tasks, making progress toward a release coming up this summer. That's as it should be.

Khafar

[Mark_J]

The answer to that is almost certainly "yes". Not everyone is impacted by this (or at least impacted by it much), and it's just foolish to suspend all other work because one problem is proving intractable. "All hands on deck" to solve some problem is rarely useful, and if they do have a client packet handling issue... content developers, artists, game systems designers, etc can't do a thing about it.

<snip>

Khafar

I wasn't talking about the development of 6.1 but the release of 6.1. Of course they should continue developing it but they should NOT release it until they are convinced it won't make the current problem worse.

As for the issue not impacting players 'much', I strongly disagree. I've been in several instances where it made the game completely unplayable and caused fellowship wipes. If the 100+ combined pages related to the issue along with the non-stop GLFF chatter and Turbine's acknowledgement that they are working on locating the source of the problem are any indicator at all, it's a widespread issue that negatively impacts large numbers of players.

[Khafar]

As for the issue not impacting players 'much', I strongly disagree.

What I meant was that it isn't having much impact for some of their players, which is undeniably true. The problem I'm working on doesn't have much or any impact for some of our customers either - yet it's quite important to the ones seeing it most. Push come to shove, though, we'll do our next release whether we have a fix for this or not (although I'm going to be one miserable engineer if we don't!).

Khafar

[Depher]

They won't delay any other patches or stuff because as the person above says they can't stop everything they are doing to deal with the one issue, but i wonder how long people will feel comfortable paying for a broken game. Im right to the end game content and i can't play right now because the lag is that bad. I know and am aware that they are trying to fix it but all the effort in the world doesn't change the fact that many of us can't play. I might be alone in feeling that if i pay for a product / service that i should at least be able to use it but somehow i don't think so.

[Sapience]

Since they haven't found the problem and the patch notes don't mention the lag, why would anyone expect it to be fixed when 6.1 is released?

One thing to always remember about patch notes on Bullroarer is that they are NEVER final. Just because it doesn't show up in Bullroarer patch notes doesn't mean anything. It may mean that whatever was being worked on wasn't part of that specific build.

The more important question is will they still release 6.1 knowing they have a huge existing problem that they can't figure out how to fix?

We believe we've figured it out. A great deal of effort was put towards understanding it in time to make a fix part of 6.1.

As for the issue not impacting players 'much', I strongly disagree. I've been in several instances where it made the game completely unplayable and caused fellowship wipes. If the 100+ combined pages related to the issue along with the non-stop GLFF chatter and Turbine's acknowledgement that they are working on locating the source of the problem are any indicator at all, it's a widespread issue that negatively impacts large numbers of players.

So let's be clear. There's a difference between pages and posters. In terms of actual posters, there are less than 300 in this thread (unique posters). We've had a tiny handful (less than 50) actual reports to CS/Tech support. So in terms of factually measurable volume of complaints, this is a very small number of players (as related to the overall population). Now, that's not to say that those being impacted aren't being impacted in a significant way because it's clear they are. To put it another way, Mark_J's posts account for nearly a page alone, as do mine and our posts are half of what the most frequent posters to this thread have contributed.


With all that said, we do believe we've found the issue behind both the Horse travel route problems and the hitching/stuttering on the client. The fix (it's one issue that caused both problems, and it was within the client) has been tested and confirmed and will be included in Update 6.1.