Efforts towards fixing the lag

[ColorSpecs]

We've been investigating since my lest post and have been closely monitoring server and netowork performance on our end and all of the indications show that our servers are working as expected and there are no issues on the server side or within our datacenter.

With that said, we'll continue to look at other possible causes, including the reports from others that a number of non-tubine titles began seeing reports of lag within their communities at the same time.

As stride points out, if it were 100% on our end then 100% of players would be experiencing it. The fact that they are not makes this more likely to be some other cause. Regardless we'll continue to investigate and monitor our services to insure we didn't miss something over the past couple of days.

Have you guys investigated the Pando Media Booster software you use? Could that be an issue?

[Thane9]

We've been investigating since my lest post and have been closely monitoring server and netowork performance on our end and all of the indications show that our servers are working as expected and there are no issues on the server side or within our datacenter.

With that said, we'll continue to look at other possible causes, including the reports from others that a number of non-tubine titles began seeing reports of lag within their communities at the same time.

As stride points out, if it were 100% on our end then 100% of players would be experiencing it. The fact that they are not makes this more likely to be some other cause. Regardless we'll continue to investigate and monitor our services to insure we didn't miss something over the past couple of days.

Have you logged into the game and played for any length of time since U6?

Quite literally 100% of the players I have spoken to are experiencing this. People from 3 continents, people near and far, all across the US. Everyone.

It's easily most apparent in the ettenmoors, but also very observable in several other PvE locations, the end game instances etc...

[Sapience]

Have you guys investigated the Pando Media Booster software you use? Could that be an issue?

Since Pando is only required to download the original installers you can turn it off or uninstall it once you're done. Any lag that was being generated by Pando would be local to your machine and wouldn't impact our servers at all.

Have you logged into the game and played for any length of time since U6?

Quite literally 100% of the players I have spoken to are experiencing this. People from 3 continents, people near and far, all across the US. Everyone.

It's easily most apparent in the ettenmoors, but also very observable in several other PvE locations, the end game instances etc...

Yes I have. I've logged in both from here at work (my desktop here isn't a screaming game rig, but it's decent and out connection while good is shared with several hundred people), at home on 2 different desktop PCs hardwired to my home LAN and two different notebooks on that same network connected via WiFi ( I have comcast cable). I've even player with 4 machines at home all logged in and playing together (my family plays). All machine are of varying stats from very good to "I may have spent a lot more money than I needed to". I didn't experience any lag on any of the machines either during Solo play nor while grouped.

As others posting here have also stated that they are not experiencing lag and at least one has posted a wireshark showing he isn't seeing anything it's pretty clear this is not a 100% of the player base issue.

Given that fact, and the fact other games seem to be seeing a similar spike in lag reports, it's possible that we may not be able to 'fix' this as it's not something we have control of.

[danno8]

As others posting here have also stated that they are not experiencing lag and at least one has posted a wireshark showing he isn't seeing anything it's pretty clear this is not a 100% of the player base issue.

Sorry Sapience just to be clear that I believe you are referencing my post about my Wireshark results (post 29). If you were, then to reiterate that the "lag" problem was there, just that Wireshark showed a clean transmission during the "lag" spike.

I am using quotes around the word "lag" because I believe this is not a networking issue, or at least that since Wireshark shows no packet delays on my network to Turbine servers, it is not a simple network issue.

I am now in the camp that thinks this is a coding/driver issue. That some bit of information is throwing our client side software/graphics driver for a loop. I don't know what else to think seeing how Wireshark shows no issue. :/

I will continue to try and recreate the issue. The first pull for RoF had several of us all freezing for about 6-7 seconds. If I get a chance I will try the area again, to see if I can at least narrow down...something, lol.

Edit: And just to add that I noticed the "lag" while testing U6 on BR before U6 went live, and the live servers did not have this issue at that time. At that time I just assumed it was because BR was very busy.

[Milithion]

here my experience from today on Morthond at 5.45 pm german time.

situation in moores: raid(12) vs raid(12)

each start of a fight we got an initial lag of about 1-4 seconds. but not right at the beginning it was a bit later.
no network-spikes while or after the lag, no brakedown in frames while or after the lag. no increases cpu or gpu usage during that time.

it feels like the server or whatever is waiting for some data / waiting till synchron

if you want to i could write down all times when i got a lag like this and sent it to you, so it might be easier to see when it happens.

[Ozthorn]

Lag (or at least it feels like lag) can be horrible in the Moors. I'm on an i7 using Comcast, and the Moors often feel laggy. I also get high CPU usage warnings, which I don't get in more recent MMOs with higher spec requirements.

[SHADOW83]

As others posting here have also stated that they are not experiencing lag and at least one has posted a wireshark showing he isn't seeing anything it's pretty clear this is not a 100% of the player base issue.

Given that fact, and the fact other games seem to be seeing a similar spike in lag reports, it's possible that we may not be able to 'fix' this as it's not something we have control of.

One thing to take into account is that almost everyone here that is saying they arent experiencing lag has stated that they solo.Most of us that are getting lag are only getting in group content and raids.the patch the other week did fix most of the complaints and even what I was getting solo,but in raids myself and a good portion of our raids still get spikes.

[Paragnosis]

Since Pando is only required to download the original installers you can turn it off or uninstall it once you're done. Any lag that was being generated by Pando would be local to your machine and wouldn't impact our servers at all.



Yes I have. I've logged in both from here at work (my desktop here isn't a screaming game rig, but it's decent and out connection while good is shared with several hundred people), at home on 2 different desktop PCs hardwired to my home LAN and two different notebooks on that same network connected via WiFi ( I have comcast cable). I've even player with 4 machines at home all logged in and playing together (my family plays). All machine are of varying stats from very good to "I may have spent a lot more money than I needed to". I didn't experience any lag on any of the machines either during Solo play nor while grouped.

As others posting here have also stated that they are not experiencing lag and at least one has posted a wireshark showing he isn't seeing anything it's pretty clear this is not a 100% of the player base issue.

Given that fact, and the fact other games seem to be seeing a similar spike in lag reports, it's possible that we may not be able to 'fix' this as it's not something we have control of.

Anecdotal, but I don't recall experiencing the lag we're talking about until after U6, at which time it became quite noticeable. Things have improved since the last update, but there are still issues. If it is accurate that the lag issues suddenly became widely noticeable immediately after a software update then to suggest that somehow networks around the world suddenly started creating lag issues that were not noticeable before the update beggars belief. Again, if it is accurate that the problem only become such an issue immediately after a software update then most likely the problem is with something in the software change. It is also possible that there was always an underlying issue with how LOTRO uses the network but the problem was only recently exposed with the change. Based on my personal experience as a user of LOTRO and the network I don't think that it was the network that degraded overnight. It would be interesting if it were possible to rollback a test server to pre-update 6 and have some pre-update 6 clients bang away at it, ideally from a location that is experiencing lag problems post-update 6. That might help further isolate the problem.

I will also say this. I spent 36 years in the software business and have been responsible for overseeing the architecture, design, development, implementation, and maintenance of some large server based applications used by large corporate customers. If even one of those customers called up and said that after a software update they were suddenly experienceing significant performance issues, and I told them it was a network issue and there was nothing we could do about it I would probably be out of a job. The network is the given, and it is the responsibility of the software team to use it effectively.

But, we probably need more than anecdotal evidence to determine when and if lags became such an issue.

[Griffy]

Comcastic user here as well and I've not encountered any lag worth complaining about. (Other than the normal things that always lag me out, Supplier & Vault.) A day or two after release, I had some slow transitions in Thinglad to wailing hills area, but it didn't bother me, and was actually peaceful.... if you can say that about lag. My first day on was in Rushgore, after I followed the river, and I soaked that area in. (Somethin' about it...) and was on all day there.

I've been in multiple skirmish raids since U6 with no issues. A DN run just the other night, and the only one who seemed to go LD multiple times was our only cappy, whom I think gave up on his last LD.

Lotro has never been more stabler for me than now. (Enedwaith was iffy to quest in. RoI's Dunland is the worst, and the area still manages to crash me and my online buddies, Always once or twice. But the new area has been delightful performance-wise.) I didn't want to be that person who says "I'm not experiencing lag." But since Sapience beat me to it, I'm not that person now. *points at Sapience, then stealths away*

[Fipiara]

While it probably doesn't have an impact I'm curious - if/how all the extra barter items that our characters are now carrying in the wallet impact loading/performance. With players having the possibility of having up to at least a 100+ items and more in the wallet that were not tracked previously with a character, does that add to the load times and/or times to transfer when travelling zone to zone? I'd imagine the amount of data tracked with the barter wallet to be smaller compared tracking bags full of LIs, but I'm always curious.

There are a number of items going automatically to the premium wallet that may have previously been vendored/destroyed or thrown in the vault - items such as Ivar's Banner from GA, Angmar coins/armour barter, Naruhel's Dress, taxidermy barter, etc. are the things that will probably clutter up barter wallets over time. Unintentional items gained while doing other things (such as GA runs for Eglain rep, Uru/CD runs for quest items and/or deeds, etc.) that in the past were vendored/destroyed or what-not will probably just sit in the wallet until the player pro-actively travels to a specific location to use/barter.

[The_Legacy]

When Soloing I get one kind of Lag; The game runs smooth until suddenly at all freezes for around 1sec not more and then speeds up to the point where I was suposed to be with a little feeling of "rubberbanding".
These lags comes irregular and not very often.

When in Moors I get two kinds of Lag; (Low Pop server: Gilrain) The game runs smooth until I am near a raid or raids I don't have to se it or iether my graphics doesn't even have to load... I usually say, Creeps are near when I get this lag and I am... well... never wrong.
The lag makes all players accept me freeze at spot with running animation 1-2sec later everyone poofs up ahead of me, when this lag happens everyone says 'Lag spikes' so it can't be just my PC, everyone feels it.
When freep and creep raids clash togather lag is unbearable for many of us players, this time I get 2 kinds of Lag and it's the same one I just mentioned everyone freezes accept for me and then suddenly it all returns to normal but it happens more often like, each 10-60sec.
And then the other Lag is, my game freezes all skills locks I try to execute skills but they are all stacking on que and my character runs on the same spot suddeny she jumps a few steps and back again and... ow it's unplayable... --->THE LAST LAG<--- Is the Lag I can do something about on my PC, what I do is I go in to Options -> UI settings -> Special effects and turn them all off.

Though I wanna point out that I can't remember having to do that on my new PC before U6 in EM even when we where 24+ vs 24+ in Tol Ascarnen... So for some kind of reason I think Lotro (EM-play atleast) requires alittle more RAM than before U6.
I am totally rookie on computers I am more like plug and play so I don't know... just wanna share my experience and thoughs.

EDIT: Oh, my new PC has been playing Lotro maxed without problems before... what I can recall and I am on a FiberCable.

[Nyrion]

While coming here just to state that I'm not having lag isn't exactly helpful, I think it's important get as many experiences as possible.

Since the downtime for 6.0.1 I have done much questing, in the new zone, in the North Downs and Evendim. I have also taken part in a Roots of Fangorn instance and ran around in the Moors some.

In all that time I have had exactly no problems with lag at all(even when I used Lightning Storm, probably the most graphic intensive skill in the game). My kinnies and the people I grouped with haven't reported any lag either.

[Lothran]

This is my first post to this thread, and I haven't seen any of my kin/raid mates posting here either, so this is new, additional data.

I note only 'regular' lag when soloing, stuff that's a minor annoyance but not a killer. Long load screens once in a while, sudden 1-3 second lag spikes once every couple of hours or so (not at all regular or predictable), but nothing killer.

When we PvE raid in Orthanc, the entire kin gets hit with lag spikes at the same time. They last 1s to 10s, can come as frequently as several times in a half-hour, and can be absolutely killer if they come in the wrong places. No idea if it is at Turbine's end (Sapience and his techs haven't seen it, but they haven't raided with us either, heh) ... or in key nodes on the webz in between (my raid mates are spread out all over the US from all kinds of different ISPs) ... or client-side but triggered by some shared event. But whatever it is, it is definitely affecting GROUPS of Turbine clients at once. This last fact means that it remains important for Turbine to find and fix (if it's in their code or hardware), or find a way to minimize impact (if it's purely network issue beyond their control).

Be happy to provide more info if Turbine needs it.

Best,
Loth

p.s. we've been avoiding acid wing of Orthanc, because the lag spikes come frequently enough that there's a significant chance we'll be hit during the flood and effectively be one-shotted as a group. Experiencing that over and over is not the kind of fun we log on for.

p.p.s. Oh, another element of information that may help you narrow it down. Frequently (I won't say always, but more often than not), the NPCs we're fighting in the raid are also frozen in time with us. So it's not purely client-side with the game servers continuing on their merry way while we wait to unfreeze--it's EVERYTHING frozen in the fight, us, the NPC enemy, etc. That doesn't sound like a purely client-side or intermediate network node issue to me, but what do I know, I'm not a computer geek! Good luck figuring this one out, it's affecting a lot of us.

[Macbeth]

When Soloing I get one kind of Lag; The game runs smooth until suddenly at all freezes for around 1sec not more and then speeds up to the point where I was suposed to be with a little feeling of "rubberbanding".
These lags comes irregular and not very often.

When in Moors I get two kinds of Lag; (Low Pop server: Gilrain) The game runs smooth until I am near a raid or raids I don't have to se it or iether my graphics doesn't even have to load... I usually say, Creeps are near when I get this lag and I am... well... never wrong.
The lag makes all players accept me freeze at spot with running animation 1-2sec later everyone poofs up ahead of me, when this lag happens everyone says 'Lag spikes' so it can't be just my PC, everyone feels it.
When freep and creep raids clash togather lag is unbearable for many of us players, this time I get 2 kinds of Lag and it's the same one I just mentioned everyone freezes accept for me and then suddenly it all returns to normal but it happens more often like, each 10-60sec.
And then the other Lag is, my game freezes all skills locks I try to execute skills but they are all stacking on que and my character runs on the same spot suddeny she jumps a few steps and back again and... ow it's unplayable... --->THE LAST LAG<--- Is the Lag I can do something about on my PC, what I do is I go in to Options -> UI settings -> Special effects and turn them all off.

Though I wanna point out that I can't remember having to do that on my new PC before U6 in EM even when we where 24+ vs 24+ in Tol Ascarnen... So for some kind of reason I think Lotro (EM-play atleast) requires alittle more RAM than before U6.
I am totally rookie on computers I am more like plug and play so I don't know... just wanna share my experience and thoughs.

EDIT: Oh, my new PC has been playing Lotro maxed without problems before... what I can recall and I am on a FiberCable.

For what it's worth, the kind of hitching and lagging in PvMP that you're talking about has always been there to a greater or lesser extent. When I used to run in full raids Creepside on Ark (I'm talking 2008, 2009 now) we called it our "Freep Radar" because of the way the server would behave when we'd get near a freep raid (even if we couldn't see them yet).

The other kind, the rubberbanding and skill lag, that's something different, though; for me, that's been part of this recent rash of performance issues, and I'm mainly seeing it in instances runnning either a 6 man or 12 man. But I (and my kin) ARE still seeing it. Last night there were several lag-deaths in Roots of Fangorn because of it.

[Galaxiana]

I have been in 6-man fellowships with my kin doing the LG daily, with people from 3 continents, (often with 1 or 2 other fellowships in the vicinity competing for pulls in the late afternoon, Eastern time) and all of 6 of us are in Vent, and all 6 of us will freeze stiff for several seconds, all of 6 of us will comment about the lag.

It is highly unlikely that this sort of thing is a network issue, unless perhaps it is Turbine's own networks (which you insist you've checked and are monitoring and are A-OK.)

I'm with the guys who think it's a database issue of some sort, where Turbine's data retrieval and/or transmission system is somehow freezing or lagging or whatever.

It is definitely more noticeable to me in:

- The new areas when lots of people are fighting the mobs
- The new Roots instance, especially afternoons and evenings
- Any 6-man or 12-man RoI instance during prime time (on my server, ~8:30p to ~11:30p Eastern)

I don't play in the Moors much, so I can't speak to that.

I can say, though, that I have had both types of lag at much increased levels since U6 - the personal kind that affects me only (nobody else in Vent is affected, or maybe only 1 or 2 others are who live near me), and the kind that affects us all, all at once, at the same exact time, no matter where we live in the world or who our home ISP is.

Thank you for your continuing efforts to pin this down. On a couple of occasions, we've had to abandon our t2 efforts to complete Roots after numerous wipes entirely because of the lag, and I know this isn't the kind of quality player experience Turbine wants to be giving its customers. (We simply avoid running it during prime time now.) :\

[Valerian_Moonfire]

Lags in Draigoch and ToO for the whole raid. No lag in Teamspeak however

[Darlgon]

Yes I have. I've logged in both from here at work (my desktop here isn't a screaming game rig, but it's decent and out connection while good is shared with several hundred people), at home on 2 different desktop PCs hardwired to my home LAN and two different notebooks on that same network connected via WiFi ( I have comcast cable). I've even player with 4 machines at home all logged in and playing together (my family plays). All machine are of varying stats from very good to "I may have spent a lot more money than I needed to". I didn't experience any lag on any of the machines either during Solo play nor while grouped.

Wow.. Sapience not only takes his work home with him, he also has his whole family beta test stuff. YOU DA MAN.

Thank them for all the work.

Lags in Draigoch and ToO for the whole raid. No lag in Teamspeak however

Interestingly enough, when I play without Teamspeak loaded, I have much less lag than when I am in TS. However, when I have TS up, and am doing intense content like Draigoth while he is on the floor, I sometimes have my character go non-responsive, with rings of dashes circling my gambits, then hear TS time out the connection after say a minute to two minutes, all the while, I see people moving past me, doing stuff, see them chatting but my responses do not show up, then loose chat with the Lotro servers.. and lock up. If I am fast enough seeing the lag and lack of them seeing my game chat, sometimes I can unplug and replug in my cable modem with Charter, wait about 20 seconds and then see everything that should have been in chat appear at 10X normal speed. Sometimes, my group has moved on to another room or area, by the time I catch up and can move. They actually tell me "Welcome Back", like I logged out and back in. Unfortunately, this part is not new, as it has been happening since before RoI came out, and I have been blaming my ISP.

[Lothran]

Wow.. Sapience not only takes his work home with him, he also has his whole family beta test stuff. YOU DA MAN.

Thank them for all the work.

If they're not raiding Orthanc as a family, they need to step up their game.

[Jamesm429]

I am just going to come in here on this one too. I am one of those people that plays at home with my wife, we are both hard wired into the router and we raid 3 to 4 times a week. We have never had lag in game UNTIL U6 and it is only in raids and does not matter if we are raiding together ot not. Infact when it first started I yelled at her, asking what she was downloading, and she yelled back NOTHING. So thanks for getting me yelled at. But anyways, we did the first 3 bosses in OT last night T2, saw more lag in Lighting than Acid. So again, never had lag in raids untill after U6. And oh, when it happens for one of us in the raid, we all get it at the same time, so yes, it is something on your guys end. Maybe not the network, but something. Infact, I will fraps our run tonight and try and posted something and ask clearly did everone get what I just got.

[Ezekiel90]

As stride points out, if it were 100% on our end then 100% of players would be experiencing it.......

At least on snowbourn server it is everyone. Not a single kin is not experiencing this problem and raiding is very tough and a game of luck whether a lag spike will hit the whole group.

[Singalras]

Iv noticed the lag since the update for a wile now iv not been to worrid since most updates get increased activaty but to be fair not many mates have come back for this update & its now been going on to long to be normal.

Last night in dragon was a funny bit that most of us jumped to say rdy & most people reported us all staying in mid air for some time, I saw this also my end. Im getting little lag spot almost any place, last night after the dragon I was stopping very oftern wile walking along.

I also put this as server lag with the auto horse travel issue, I was on a auto horse/goat in moria & I lagged & it broke the goats pathing, lag my end connot effect a server side goat, that has to be turbines side but I think that was pre latest fix.

Sort it out pls & if not turbine severs do some trace routes & find who has the issue & ask them to sort it its not our issue to sort.

[Gylve]

I've been getting the big lag spikes especially in raid settings. Everyone in the raid sees them at the same time and starts commenting simultaneously in vent. Raids include people from all over the US and Canada. I personally haven't seen the bugged horses yet, but have been noticing more general world-lag doing things like accessing vendors, deed completions, LI's leveling, etc.

I have noticed the "game didn't freeze but skills don't fire" thing quite a bit, too. It happens throughout Roots of Fangorn but especially right at the start of the first spider pull, when the extra adds show up. My FPS doesn't drop, my latency indicator shows no issues and vent works fine, but I just stand around like an idiot with full fervour pips mashing AoE attacks and watching nothing happen.

[GoldenusG]

I've noticed no lag since the last update, but I've only played briefly and then it was in low population areas - I haven't done any group content or wandered out to the Great River yet.

[OptimusPrimeRib]

Update 6 lag is still around and makes playing frustrating at times. Especially in the Great River area and instances there are noticable lag spikes. We're using vent and people from all over say lag at the same moment. Its a shame that the games performance goes to garbage when good content is released. Talking to friends(former players) about the update all you can say honestly is that it's great content but the servers are terribly laggy now. I play several other online games and haven't noticed any increased lag except for lotro.

[Yosoff]

if it were 100% on our end then 100% of players would be experiencing it.

100% of the players in our ToO raid last night (multiple timezones/countries/ISPs) had lag at the same time, by you own logic doesn't that mean it's on your end?

You guys closed two posts basically saying; "it's not us, it's you." Every post you've made since then has basically said; "we're looking at it *wink* *wink* but we still think it's not us, but maybe it's not you, maybe it's somebody else entirely."

You've even gone so far as to basically say; "we heard some other game has lag too, so see, it's not us."

Well, here's my response to all of that finger-pointing nonsense; "I KNOW it's not me. The lag is making it frustrating and nearly impossible to play high-end content. I don't care if it's not your fault. YOU figure it out and fix it. Stop making excuses."

Why aren't you yelling at your ISP as loudly as we're yelling at you? Why aren't you yelling at your IT guys as loudly as we're yelling at you? Why do we seem to care more about the lag issues with your company's game than you do?