Efforts towards fixing the lag

[enginekid]

What it means is i have had very limited playtime since the reinstall and i have not had any lag since.

If you have had "very limited playtime", maybe wait until you have had a reasonable amount of playtime, before telling Turbine there is no U6 lag.

[Darmokk]

I play on Arkenstone, one of the lowest population servers available, and I experience repeated spikes along with literally every person in my kin and, by the looks of glff rage, most of the server population.

It would be great to copy'n'paste such chat channel outbursts. I think there are people here who just don't take this serious enough and when people talk about whole fellowships wiping out they think it's a case of "this is important to me so I just make up that everybody else has the same problem, too".

Also, the exact timestamp of this happening is probably helpful, in particular on the large bursts that affect much of a server's population.

The most interesting bit would be to know whether two or more server experience such an event at the same time. Because if they do that strongly points to a networking issue.

[Fudoshin]

The thing that is most frustrating, is I keep hearing it's not the server, it might be ISP (or not) it might be client (or not)....basically from my perspetive sitting at my keyboard I can say with 100% certainty one thing and one thing only. IT IS DEFINITLY A GAME ISSUE. I'm not saying Turbine isn't trying to fix it and having little knowledge of such stuff I am not sayng it's a simple fix. But as many have said, before U6 I had no issues, now I have issues, and only with this game..nothing else.

[LeRaginAsian]

Decided to pop in and contribute to the conversation.

I built a new PC a few months back with specs that far exceed lotro's requirements. Any and all problems I had previously with my 6+ year old machine were gone, even in raid situations. People talked about lag and I just laughed quietly to myself and knew it was their inferior machines that weren't delivering.

Until this week.

It started tanking a standard Roots of Fangorn run. My health started getting uncomfortably low at points because my healer's skills weren't firing. I just assumed it was either the Ensnared attack duration debuff or her own lag; either way, we were still getting the job done so we moved on. In subsequent pulls, however, I began to have some issues myself. A Fray the Edge on a venomous spider sat queued for what felt like hours before I realized it and went sprinting across the room to rectify the situation. Again, I assumed it was the Ensnared debuff, but then it happened again and there was no debuff. And then again. And again. Somehow we made it through, but it was one of the more painful runs I've ever struggled through, but I decided (albeit not fairly) to attribute it to our champ not hitting his interrupts and hoped that would be the end of it.

And then I started leveling my warden.

Not only was the skill lag magnified by my gambits, but I started to encounter a new problem: undead targets. After killing one mob in a large pull and moving to attack the next, I would get the message that my target was already dead, despite the fact that it clearly was not. Any skills I attempted to execute would result in the same message until I de-selected and re-targeted it. It was a pain, to say the least, to have my rotation stalling because of the combined issues. Luckily this was often off-set by the fact that I would sometimes run right through an aggressive mob but they wouldn't aggro. I would have bugged it but.... well.... I've been doing that for years and frankly haven't been given any reason to believe that would accomplish anything worthwhile.

Finally I decided leveling wasn't worthwhile with these combat issues so I took to crafting instead. That's when the freezes started happening. While processing mass amounts of wood, I would inexplicably stop and be unable to do anything as if my client were going to disconnect. Only I wouldn't. After a few minutes of waiting for the disconnect, I'd be able to move again and continue my crafting efforts. This happened at least five times over the course of the next hour. I could still communicate with friends through the chat window and never got the message that I'd been disconnected from the chat server. It seemed like I just locked up for no good reason other than to inconvenience my progression.

I want to stress that nothing on my side has changed. I don't have any additional software or upgraded hardware that could be causing the issues, my Fios internet is as stable and fast as it always has been, and none of my other programs have been affected. So if I may use the same logic you Devs have previously, if lotro is the only problematic process on my PC, then the problem is not with my PC at all but lotro.

Note that this was not a gradual degradation of performance for me, it was simply not there until it was.

[enginekid]

We need to remind ourselves about the past in order to make sense of the present.

Great quote of wisdom:

Those who cannot remember the past are condemned to repeat it. -- George Santayana

Remember about one year ago? The game started getting absurd amounts of constant lag. Everyone was experiencing across all servers. There were copious threads about it with hundreds of posts.

In those threads there were players who said there is no lag, or, they are not experiencing any...

Then what happened?

One forum goer and lotro player discovered through his own investigations something that documented the lag. The NPC's that roam would stop in one place, walking on the spot, as if stuck. This would occur with clock-like precision every 31 seconds.

Up until that persons discovery Turbine was continually saying the same kind of things they say in this thread. And people were continually saying they have no lag.

But this discovery was overwhelming evidence of a problem. And Turbine began acting on it, and even said well it looks like there is indeed a problem. Later they implemented a fix, and BINGO! All the heinous lag was gone! Then a couple weeks later the entire lotro game servers crashed! And the servers were down for a looong time.

Finally the game was back up and live, and Turbine said they had to replace faulty hardware.

That is history folks. What can we learn? People will deny a problem exists for strange reasons when it clearly does exist. No one will do the needed investigations to find the problem until someone does the work for them.

History my friends is the best teacher.

[Rawlor]

Our network operations team continues to monitor the situation and still sees no evidence that this is on our servers.

NetOps does not equal SysOps.

Your systems team needs to monitor actual servers. The network can be running at light speed but the servers themselves can have issues.

The fact still remains: This started with U6. Everyone experiences it at the same time. From being in systems administration for about 10 years, that sure feels like an app issue.

[Southpa]

Some kind of real-time analysis has to happen. If monitoring the servers for several weeks turns-up nothing, then frankly, you should start looking elsewhere - as others have said, the game code.

So many things were (re)broken with U6. Look at the code, Turbine. That's where your answers are.
I would be curious to know if Turbine employees use commercial installs of the game or something else. Do they update by connecting to the same servers we do? Maybe a dumb question, but dumb events such as this awful lag sometimes mandate such things.

And to reiterate, there are people right in Boston who have the same lag that those of us in the Midwest, West Coast, Canada and New Zealand have. It is not a proximity issue.

This weekend was awful. The game would literally freeze for several seconds at a time on a regular basis. The whole game. Not just me, not just my comrades, not just the mobs. Everything. I even created an aural meme for when it occurs so my friends and I could feel some solidarity rather than the typical pangs of disgust.

[enginekid]

This weekend was awful. The game would literally freeze for several seconds at a time on a regular basis. The whole game. Not just me, not just my comrades, not just the mobs. Everything. I even created an aural meme for when it occurs so my friends and I could feel some solidarity rather than the typical pangs of disgust.

This happens to me and everyone in my fellowships everyday now. Thanks for sharing the truth.

[Saelyth]

Someone said in a raid yesterday that CombatAnalysis plugin makes his FPS drop a lot, something even lag.

Guys... are you using that?

[ChromiteSwiftpaw]

Someone said in a raid yesterday that CombatAnalysis plugin makes his FPS drop a lot, something even lag.

Guys... are you using that?

It's not that. I load up CombatAnalysis on demand when I want to use it. (Yay for BootStrap!) I lag without it loaded. You can't tell me that 100% of the people I play with and 100% of the people in GLFF are using CombatAnalysis... Also, my friend who I duo with gets the lag and doesn't run CombatAnalysis at all.

Stop grasping at straws and just face the facts: We know it's a server issue. It may not be a NETWORKING related server issue but it is a server issue.

[Gilarianen]

Instead of QQing about it on here, does anyone actually have a contact email address for anyone high enough up at Turbine to make a formal complaint to?

[Mark_J]

Someone said in a raid yesterday that CombatAnalysis plugin makes his FPS drop a lot, something even lag.

Guys... are you using that?

Sometimes it can take a long time to load plugins, way too long in fact. That was another gaff from U6 and it broke a lot of plugins as well. That said I've personally tried running the game with and without all plugins and it makes no difference. The lag hits no matter what I've got running...

[Gillianrial]

Someone said in a raid yesterday that CombatAnalysis plugin makes his FPS drop a lot, something even lag.

Guys... are you using that?

No.

And it's not about FPS dropping. It's about the game ceasing to repond for short periods. It's all been described in the thread.

[Osgril]

If the problem isn't evident in Turbine's office and you want to see it with your own eyes, what you guys should do is simply send off a +Name+ to one of the volunteering raid teams I've seen in this thread. It doesn't matter if you yourself (i.e. the Turbine employee) will experience lag or not - you will clearly see an entire team of 12 people moonwalking intermittently. Of this, I'm absolutely certain.

As for the people who say they are not experiencing any lags: are there any of you who routinely run the Isengard 6-mans or the Orthanc raid? I've yet to hear from anybody running that content that has not experienced these issues frequently. Personally I'm on the Eldar server and I experience the same issues as most other people in this thread, on a daily basis. It does appear to be much worse during peak hours, which leads me to believe there is a bottleneck problem somewhere.

The problem at hand, sounds to me like intermittently there is a delay in either the acceptance of or the acknowledging of packets to and from the game servers, causing a queue to be formed at some point from us to Turbine. Packets get stored up for a second or more, and then they all go through in a burst. Think of the classical bottleneck/hourglass - it gets crowded and suddenly you're pushed through! Now I'm no network techie, but that's about how it feels to me. Any other ideas I can think of should've been easily detected by Turbines techies already (database issues, memory shortage / paging / I/O bandwidth issues and so on).

One thing is absolutely certain however: the root cause is not and cannot be either the client PC or the ISP it's attached to. We've already heard multiple reports from players around the globe, with nothing in common with regards to PC setups and ISPs, who despite this experience simultaneous lag. This is proof enough that the issue either resides in the game client itself, somewhere along the path to Turbines servers (including Turbines own ISP), or on the actual game server itself. I would assume that this much is a given by now.

If you allow me to speculate a little - code-wise the primary suspects would have to be either the Instance Finder or the new Wallet. Both are new pieces of code, with probably pretty serious database access going on. Might there be something in there that can cause a "queue" to build up and cause simultaneous delays for players? Perhaps simply the increased number of db connections due to the Wallet is causing something unexpected to happen? I'm thinking about trivial things like 'max number of concurrent connections to db' and similar things. I don't know obviously. Might be worth pursuing anyway.

Finally just throwing out an idea to Turbine: if the problem is as difficult to diagnoze as it appears, perhaps one idea would be to prepare a 'debug build' of the client and send that out to a select few players and see what data you get back from them and the game server's response also. Wouldn't have to save log files to the client as that would be sensitive data, but the client could log a bunch of timings and network stuff and ship it back to you 'behind the scenes' so to speak? I'm quite sure you would find volunteers for such an endeavor. This is of course all assuming you are unable to reproduce the lag at your offices or locally from employee's homes.

I wish you all the luck in solving this problem!

[Khazneh1]

I tried reinstalling and found something that may be worth investigating.
Back towards the end of January 2012 I install from the high res installation set freshly downloaded all I was experiencing was the network/server style lag that many of us have been reporting (simultaneous lag experienced by large numbers of players).
Today I reinstalled from a high res installation set I downloaded in July 2011 and patched it up. After this, in addition to the simultaneous lag I was also getting horrible graphics lag independent of others (which matches the description given by some in this thread), it appears that the launcher patching has not properly patched this older distribution.
I am currently restoring from a backup itook prior to this reinstall and expect to return to the highly annoying simultaneously experienced lag effects.
It appears from my experience that there may be more than one problem, one related to the patching of older installation kits an one related to this simultaneous network/server lag effect.

[probitas]

Instead of QQing about it on here, does anyone actually have a contact email address for anyone high enough up at Turbine to make a formal complaint to?

Posting that info will likely result in an immediate thread lock. DON'T DO IT.

[Gilarianen]

The trouble is that we don't seem to be getting any info others than "we're looking into it...but its not our servers...." and its getting frustrating, like being fobbed off constantly.

I only ask about a contact for complaints as if as customers for anything else that was broken you would want to complain, and if nothing seemed to be done then the next thing is to try and be heard at a higher level.

[IngeniousMrToad]

I have a suggestion for everyone that is fed up with the lack of ownership of this problem from Turbine, and the level of disrespect they have shown their customers by nonchalantly dismissing all these voices... Just wait a little while, and when the right time comes, do a fresh install of GW2.

[Griffy]

Ah, found that vid clip again, finally.

You say the comment about it not happening to a specific person isn't helping, but I think it can help. We have had some players who experience the lag post the information requested by Turbine earlier in this thread. Maybe people who don't have any problem should post their information as well.

If I can find the info asked for by Mirthgar, I shall come back and fill that in. (I don't have the time atm. I'm already trying to rush through this post.)

Until I see a comment from someone who runs end game fellowship or raid content and claims not to suffer lag I don't believe that it isn't an issue that affects everyone.
*snipped*
When we have someone come forward we need them to offer their time to be taken through a couple of instances with a large group so that we can see if their views of no lag change when grouped up with people that have problems. Ideally this person will be a trusted member of one of the larger kins, can communicate via voice with the group and is willing to update this forum with their experiences.

I'm one of those folks who has not seen much of this lag, and I've been online at Elendilmir Prime Times 7pm EST - until the wee hours of 5am EST, where it starts to become Aussie Prime Time. I run in skirm raids with full groups. (Even got my first Earring of the Mathom Tender.) I recently was invited to LM for an Orthanc raid. And I saw no acknowledgeable lagged out skills, no skippy/hitching movement of mobs. (and yes, I remember the lag back a year or so, where npcs stopped, so I do have things to compare against.) I was also on their ventrillo server during ToO, and I don't recall a single lag spike, that would have sent that entire group of people on vent into a (rib-splitting funny) rage, IF it had happened. (There were other things they were grumping about, but it wasn't lag related.)

I am looking for the lags. And so far the only ones I can acknowlege are:

• the crafting (I'm going to call it, because that's what it looks like) animation frameloss. Where you run to a crafting station, then you stop, but your character skips a frame forward. I wouldn't have noticed this if not for your video enginekid. Now it's a pesky nuisance.
• Unstable stables - I've only had 1 freeze up entirely (so far, knock on quartz). It was my small warden alt. She relogged, got stuck on loading screen. (I alt f4'd out, restarted the client, and that kicked her stable back in.) My other stable incident was a pretty much perfect stable ride from 21st to Orc Watch. I only noticed one bump in the road where the goat ran it's nose into a pillar, but it actually knocked itself back on the road. I was a bit worried it might drop me off the bridge, but the goat made it the whole trip.
• Invalid Target - My hunter getting this one fairly often, if she tries to power through a bunch of nearby mobs really fast. (This isn't a new lag bug, it's just rearing it's ugly head again.)
• One awkward border transition. Thinglad to Wailing Hills? You can watch the golden trees fade in or out in chunks.
• The occasional call outs on glff of "lagspike!" The two of these I heard, my hunter was running around with no issues in Water Works, except for the usual 'invalid target' issue.

So I guess I'm not 'immune' to this lag, but I have not had at anytime wanted to fling my keyboard at the monitor. Sure, I do have soloing around as 65% of my time in this game, but that last third of my time in game, I group up into the larger groups. Randomly pug into any of the 6-man Moria instances on occasion. Actually I think I solo a lot less than I used to, so I am soloing around much less than I used to, but the range is still close'ish to my estimate.

PS. There's another case of lag, that I can show off everytime I log on. Map, Completed Quests List, Vault & Shared Storage, and The Supplier vendor can lock me up for a chunk of time. But those lagspikes have been in game FORever, and aren't anything new, and are just server database access lag. Normal.

PSS. I'd gladly come along on a ToO run and see if I can get even a marginal lagspike. I love running deep in Ancient Master.

PSSS. When raids are such a gamble, maybe Turbine shouldn't make them require such precision, that unpredictable lagspikes and losses of movement feeds a whole group to the boss. Just caught up on the post from jwbarry about ToO stuff. Between fixes for random bad luck, it sounds like there's some other very reasonable changes coming. Coolbeans!

PSPSPS. Gosh that was much more time than I wanted to spend replying. I'll send any left over cookies. And look for Mirthgar's info post later.

edit, found a followup post that I gotta agree with really quick:

If you allow me to speculate a little - code-wise the primary suspects would have to be either the Instance Finder or the new Wallet. Both are new pieces of code, with probably pretty serious database access going on. Might there be something in there that can cause a "queue" to build up and cause simultaneous delays for players? Perhaps simply the increased number of db connections due to the Wallet is causing something unexpected to happen? I'm thinking about trivial things like 'max number of concurrent connections to db' and similar things. I don't know obviously. Might be worth pursuing anyway.

[Majic]

Lagging Indicators

The problem continues, and while I can't say where it lies (client, server, something in between, or some combination of these), I do know:

1. It doesn't involve client crashes. I can't remember the last time my client crashed, yet I still see this problem. I think players dragging client crashes into this are muddying the waters with other problems. Not that they aren't important, but they're just not this problem.

2. It doesn't involve graphics performance. I've been monitoring my framerate constantly on all characters I play since this became a problem. There is NO correlation between framerate and the lag. It occurs at 70fps and at lower fps in more intensive areas without any apparent difference in lag intensity.

3. It doesn't appear to involve network performance. After disabling it last summer, thankfully GLS.LOTRO.COM can be pinged now, and I've been pinging it a lot as I've played, trying to determine if the problem is due to network latency or packet loss. Even with a solid average ping of ~100ms and ZERO packet loss (as in not just 0%, but not a single packet lost or any deviation from the normal latency spread), the problem is evident. I can't say whether that rules out a network problem per se (perhaps there's another hop or two between gls.lotro.com, or some other confounding factor), but it does point away from that.

4. It is not clear if it relates to server loads. The problem does sometimes seem to get worse when Multiple Dynamic Layers are active, or the "blue volcano", or during normal peak log-in periods, but the problem is also evident in the absence of these things. That doesn't mean server loads in a given zone aren't a factor, but I'm also seeing significant problems in quiet areas as well. Since loads are probably distributed across multiple machines (I would be shocked if they weren't), a "quiet" zone may still share a physical machine with one that's getting slammed, but that's not something I can determine as a player.

Anecdotally, one of my lowbies took a long horse ride from Trestlebridge to Esteldin last night and wound up stuck just northwest of Gatson's farm in the North Downs. Behind me, about 30m away, was another player who was AFK and also apparently stuck. Interesting. I was AFK myself when my toon got stuck (it's a long ride), but the fact that they both stalled so close together may be a clue.

In fairness to Turbine, they are getting a LOT of conflicting information from us and I wouldn't even want to try to imagine how difficult it is for them to parse out meaningful feedback from noise.

But here's hoping they can find a solution to this mystery, because the problem is real, and it clearly won't get better on its own.

[Sapience]

It appears my earlier comment may have been misunderstood. Our network operations team is not solely monitoring the switches and network hardware. They are monitoring the physical servers them selves at every level from memory and CPU usage, to connectivity, to instance performance and usage. That's inaddition to the actual connectivity to and from the data cetner. To date, this monitoring has shown no evidence of an issue on our side. In fact, to date the only 'solution' anyone has reported (and we have confirmed it here on at least one machine) is that a completely clean, brand new download and install (with 0 carry over data from a previous install or installer) seems to fix the lag issue. However, we've also seen at least one case where it did nothing.

For those who feel we are doing nothing, I apologize, but I cannot go into details on what we are doing, who we are in contact with, or what steps we're taking with those individuals to test on both ends. The reasons why should be obvious.

[jcmanda]

Our network operations team continues to monitor the situation and still sees no evidence that this is on our servers.

Quick question - does your network operations team include coders, out of curiosity, or does it include only skill sets that the title implies? I ask because if the problem is in the software, and how the server environment kicks off events in the client, which then calls certain "bits of code" (very simply put), which then cause the symptoms, then networking folks aren't going to see much of anything.

Just wondering if you all have considered that scenario: server-side environment causes calls on the client software, which cause The Lag Spikes(tm). The environment causes could be varied - everything from sprite loads, layer-switching/loads, proximity to "event-generating-item-X", you name it. The call on the client software, though, would be pretty specific if it's so widely experienced.

Edit: Cross-posting FTW!

[Darmokk]

We need to remind ourselves about the past in order to make sense of the present.



Remember about one year ago? The game started getting absurd amounts of constant lag. Everyone was experiencing across all servers. There were copious threads about it with hundreds of posts.

In those threads there were players who said there is no lag, or, they are not experiencing any...

Then what happened?

One forum goer and lotro player discovered through his own investigations something that documented the lag. The NPC's that roam would stop in one place, walking on the spot, as if stuck. This would occur with clock-like precision every 31 seconds.

Up until that persons discovery Turbine was continually saying the same kind of things they say in this thread. And people were continually saying they have no lag.

But this discovery was overwhelming evidence of a problem. And Turbine began acting on it, and even said well it looks like there is indeed a problem. Later they implemented a fix, and BINGO! All the heinous lag was gone! Then a couple weeks later the entire lotro game servers crashed! And the servers were down for a looong time.

Finally the game was back up and live, and Turbine said they had to replace faulty hardware.

That is history folks. What can we learn? People will deny a problem exists for strange reasons when it clearly does exist. No one will do the needed investigations to find the problem until someone does the work for them.

History my friends is the best teacher.

Any links to those threads? Or were they nuked?

To help Sapience in his "one person" assumption: if you experience lag after a complete reinstall of the game, can you please raise your hand again? My memory is different from Sapience's, I remember that more than one person said a reinstall of the game didn't help.

[jcmanda]

It appears from my experience that there may be more than one problem, one related to the patching of older installation kits...

Man... I wish I hadn't re-installed on my old machine now. I would love to have a comparison of files at this point....

[Majic]

Word From Afar

For those who feel we are doing nothing, I apologize, but I cannot go into details on what we are doing, who we are in contact with, or what steps we're taking with those individuals to test on both ends. The reasons why should be obvious.

I think a lot of us were concerned that the problem was being brushed off. It's very reassuring to know that you're taking this seriously and are investigating it vigorously.

Thanks for keeping us informed despite the onslaught. It can't be fun being caught in the middle like this, but that's why you're making the big bucks.

For my part, I'll be trying to get a clearer understanding of the symptoms I'm seeing on my end, and I'm sure I'm not alone in saying that my prayers are with you.