Efforts towards fixing the lag

[Galaxiana]

In an effort to try to pin down and document the cause of the lag we're suffering, I let an auto-traceroute program (called Ping Plotter) run a ping every 15 seconds from 9:37pm CDT yesterday until 1:13p today, and the 2 routers (internap-gw.ip4.tinet.net,77.67.77.54 and border1.te8-1-bbnet2.bsn003.pnap.net,63.251. 128.107) just outside the Turbine domain gateway (turbine-7.border1.bsn003.pnap.net,64.9 5.76.202) are showing an ongoing and consistent 30% packet loss. This loss frequently happens in chunks that returns 0 packets at all (noted in Ping Plotter as ERR or *).

Here is an example from my data of some of these ping returns:

(Note: This first block of text is the listing of hosts, in order, that traceroutes are showing. The 2 routers outside Turbine showing the consistent and ongoing 30% packet loss are the ones 4 and 5 hops from the right in the data below the host listing.)

Host Information
1, -------------- , (hidden for privacy reasons)
2, -------------- , (hidden for privacy reasons)
3,te-6-4-ur07.elmhurst.il.chicago.comca st.net,68.86.116.1
4,te-8-4-ur08.elmhurst.il.chicago.comca st.net,68.87.210.30
5,te-2-3-0-0-ar01.area4.il.chicago.comcast. net,68.87.231.61
6,pos-3-9-0-0-cr01.350ecermak.il.ibone.comca st.net,68.86.90.45
7,pos-1-3-0-0-pe01.350ecermak.il.ibone.comca st.net,68.86.86.158
8,xe-9-0-0.chi10.ip4.tinet.net,77.67.71 .145
9,xe-2-0-0.bos11.ip4.tinet.net,89.149.1 82.170
10,internap-gw.ip4.tinet.net,77.67.77.54
11,border1.te8-1-bbnet2.bsn003.pnap.net,63.251. 128.107
12,turbine-7.border1.bsn003.pnap.net,64.9 5.76.202
13, -------------- ,74.201.102.154
14, -------------- ,74.201.102.42

Some data samples showing ongoing loss (* indicates the router returned ZERO packets!):

"4/9/2012 10:42:23 PM",0,38,10,10,12,16,14,11,57, *,39,39,38,38
"4/9/2012 10:42:38 PM",0,30,13,10,16,15,15,11,39, 41,50,*,40,38
"4/9/2012 10:42:53 PM",0,37,11,9,14,15,12,12,37,4 0,40,40,40,40
"4/9/2012 10:43:08 PM",1,30,9,10,14,26,13,12,42,* ,40,38,40,39
"4/9/2012 10:43:23 PM",0,31,9,10,14,14,13,11,41,* ,41,*,41,41
"4/9/2012 10:43:38 PM",0,36,8,9,11,15,12,12,44,*, 40,40,*,39
"4/9/2012 10:43:53 PM",0,28,9,10,13,13,13,39,41,* ,40,*,42,38
"4/9/2012 10:44:08 PM",1,33,11,13,16,11,13,28,41, *,41,41,*,40

What I think this shows is that despite enjoying a nice speedy 40ms ping, I'm experiencing intermittent freezing-type lag because packets are simply disappearing in chunks when this router decides it doesn't want to respond.

4/9/2012 9:43:36 PM",1,30,9,10,14,12,24,10,46,* ,38,39,40,40
"4/9/2012 9:43:51 PM",0,38,9,23,12,15,20,11,42,5 8,*,39,44,39
"4/9/2012 9:44:06 PM",0,23,9,10,15,18,14,12,37,3 9,39,38,41,39
"4/9/2012 9:44:21 PM",0,37,14,9,12,15,18,28,43,4 1,*,39,39,39
"4/9/2012 9:44:36 PM",1,39,9,9,25,11,12,22,37,39 ,40,38,41,43
"4/9/2012 9:44:51 PM",0,25,11,11,14,14,15,10,40, 40,39,39,40,51
"4/9/2012 9:45:06 PM",0,33,9,26,12,11,12,12,40,4 0,40,38,39,40
"4/9/2012 9:45:21 PM",0,32,11,12,12,12,12,76,55, *,41,43,*,46
"4/9/2012 9:45:36 PM",0,35,9,10,13,15,30,40,45,* ,41,43,*,40
"4/9/2012 9:45:51 PM",1,21,9,10,13,16,12,20,39,4 0,38,39,41,41


4/9/2012 10:26:08 PM",0,32,*,10,13,15,18,13,46,* ,40,*,41,42
"4/9/2012 10:26:23 PM",1,29,10,8,11,15,14,11,48,* ,39,40,38,39
"4/9/2012 10:26:38 PM",0,25,9,9,20,11,12,12,44,43 ,*,38,39,40
"4/9/2012 10:26:53 PM",0,34,10,8,12,14,12,12,44,* ,39,41,*,39
"4/9/2012 10:27:08 PM",1,28,12,9,14,14,13,12,49,* ,41,*,39,40
"4/9/2012 10:27:23 PM",1,39,8,9,15,13,12,12,49,*, 69,*,39,39
"4/9/2012 10:27:38 PM",0,*,15,9,12,13,12,10,37,39 ,40,39,39,40
"4/9/2012 10:27:53 PM",0,28,14,10,12,15,23,13,43, *,38,39,42,38
"4/9/2012 10:28:08 PM",0,39,8,11,13,25,12,13,42,* ,39,40,41,42
"4/9/2012 10:28:23 PM",0,31,10,12,12,17,11,11,47, *,39,39,38,40
"4/9/2012 10:28:38 PM",0,32,9,10,12,12,12,12,42,* ,40,38,39,40
"4/9/2012 10:28:53 PM",0,38,11,25,13,13,11,11,43, *,39,40,38,38
"4/9/2012 10:29:08 PM",0,20,9,9,11,13,12,11,49,*, 39,41,41,39


"4/10/2012 12:16:11 AM",0,36,10,8,14,15,13,11,38,4 0,*,39,38,38
"4/10/2012 12:16:26 AM",1,29,10,8,12,14,11,12,39,4 1,*,42,41,39
"4/10/2012 12:16:41 AM",0,29,9,9,12,16,12,10,75,*, 37,39,39,38
"4/10/2012 12:16:56 AM",0,23,9,11,11,18,12,10,37,4 0,38,38,40,38
"4/10/2012 12:17:11 AM",0,32,10,9,15,11,12,12,41,* ,40,38,51,40
"4/10/2012 12:17:26 AM",0,27,20,9,11,13,12,12,37,1 10,39,*,39,39
"4/10/2012 12:17:41 AM",0,30,9,10,12,14,13,10,37,3 9,37,38,39,38
"4/10/2012 12:17:56 AM",0,27,10,9,14,14,13,12,37,4 0,*,41,*,40
"4/10/2012 12:18:11 AM",0,34,8,9,15,12,12,12,39,39 ,40,40,41,39
"4/10/2012 12:18:26 AM",1,41,*,9,13,13,12,12,38,39 ,40,38,39,40
"4/10/2012 12:18:41 AM",0,*,25,9,16,13,12,12,37,43 ,*,38,42,39
"4/10/2012 12:18:56 AM",0,23,9,10,15,11,12,42,*,44 ,*,38,40,40
"4/10/2012 12:19:11 AM",0,36,10,8,12,15,11,12,39,3 8,39,39,41,39
"4/10/2012 12:19:26 AM",0,38,10,9,15,14,12,12,37,4 3,*,37,38,39


4/10/2012 2:57:00 AM",1,33,9,8,13,14,13,11,38,41 ,*,40,38,38
"4/10/2012 2:57:15 AM",1,22,11,9,13,14,12,12,37,3 9,39,38,39,40
"4/10/2012 2:57:30 AM",0,39,8,9,14,19,12,17,40,39 ,39,38,38,39
"4/10/2012 2:57:45 AM",1,36,10,11,12,14,11,12,39, 38,41,*,38,39
"4/10/2012 2:58:00 AM",1,39,8,9,14,13,11,12,37,23 1,39,38,38,39
"4/10/2012 2:58:15 AM",1,22,10,10,11,14,11,11,38, 40,*,39,38,38
"4/10/2012 2:58:30 AM",1,23,9,10,16,16,13,11,38,4 0,*,39,40,40
"4/10/2012 2:58:45 AM",1,31,10,10,13,12,14,13,77, 42,*,39,41,40
"4/10/2012 2:59:00 AM",0,32,10,9,13,13,12,12,49,* ,39,37,38,39

And so on for the whole ~16 hours I tracked (and it's still running.) Some periods show much less loss than indicated above. Occasionally other routers are briefly implicated.

Now I'll be the first one to admit that I don't really understand much about this stuff, but I do know that 30% lost packets overall means lots of lag because packets have to keep being resent over and over until they get through.

Sapience or whomever: If my full data set would be of use to Turbine, I'll be happy to send it, and/or post instructions so others can also run these traces to submit them to you if they're useful at all. Ping Plotter is shareware with a 30-day free trial.

Galaxiana

[JiiPee]

Man... I wish I hadn't re-installed on my old machine now. I would love to have a comparison of files at this point....

I did it. I used http://www.md5summer.org/ to generate sha1 hashes from every single file from lotro installation folder. Then I did uninstall, deleted lotro folder and rest of the files what uninstaller didn't clean up. I downloaded fresh copy of highres client and installed it. Patched it to up-to-date and then made sha1 compare. Result is this:

client_anim.dat ERROR: Checksum did not match.
client_cell_1.dat ERROR: Checksum did not match.
client_cell_2.dat ERROR: Checksum did not match.
client_gamelogic.dat ERROR: Checksum did not match.
client_general.dat ERROR: Checksum did not match.
client_highres.dat ERROR: Checksum did not match.
client_highres_aux_1.datx ERROR: Checksum did not match.
client_local_English.dat ERROR: Checksum did not match.
client_map_1.dat ERROR: Checksum did not match.
client_map_2.dat ERROR: Checksum did not match.
client_mesh.dat ERROR: Checksum did not match.
client_sound.dat ERROR: Checksum did not match.
client_surface.dat ERROR: Checksum did not match.
dx9c.exe ERROR: G:\Games\The Lord of the Rings Online\dx9c.exe does not exists.
mce.lnk ERROR: Checksum did not match.
patchcache.bin ERROR: Checksum did not match.
unins000.dat ERROR: Checksum did not match.

I have not include some trash like cache folder etc. Also please remember that *.dat files are actually file containers, including lot of files, compressed and uncompressed. They are kinda .zip files. So even that many (maybe all) .dat files checksum didn't match, it won't mean that I had corrupted files. It just means that my files was patched many times and some files might been in different location in container.
There is also some lotro defrag tool out there what should optimize .dat files, but I haven't used it ever.

If there was mismatch of .exe or .dll files, I would have been worried.

Oh and no, fresh install didn't help anything. Well maybe some loading times are faster now. But "lag" is still there.

[Kirvay]

I still recommend changing the name of the Document folder "Lord of the Rings Online" to "Old Lord of the Rings Online" even though it apparently didn't do much for me. It has worked wonders for my husband who is experiencing very little lag now.

To add to the conversation, I have noticed that I seem to have 1 skill that delays going off about 40% to 50% of the time outside of the "lag" issue. Is anyone else noticing anything like this?

On my burg, my Improved Subtle Stab (which has no induction) will sometimes "hang" before going off, and other times everything is smooth sailing. Any other burgs noticing this - or having issues with a different skill doing the same thing? What about other classes?

[Ardineck]

No a complete reinstall of my OS , game and driver update had no impact at all.

To be clear, we're working with a small group of players directly. This thread has reverted back to something other than a useful feedback thread so we selected a small group and are working with them. My reference to one person was correct as it referenced this small group.

It does seem that a client issue is possible, but we have not ruled out something else going on - whether server side or elsewhere. Just because or tracking and monitoring hasn't turned up anything doesn't mean we've stopped investigating possibilities.[/QUOTE]

I am still looking for an answer. How can I ensure a complete clean uninstall/install? Will the uninstaller provided remove ALL files or do I need to hunt down specific files to delete? What about plugins? Do those need to be removed? Folders? How can I make sure that I am fulfilling that part of the suggestion, so that any problems encountered afterward can honestly be addressed without fradulently claiming I have done everything asked?

[Sapience]

Please do not reinstall. Maybe I did not make things clear enough. We have seen that it helps in some cases and not in others. That means it is not the root cause. If it were, then it would resolve the issue 100% of the time. It doesn't.

What we do know, for a fact, at this point, based on all our research, is:

• It is not server side.
• It is not in our datacenter.
• It is client based.

We have some theories we're testing and should know more once we run them all down.

[Bootstwaddle]

On my burg, my Improved Subtle Stab (which has no induction) will sometimes "hang" before going off, and other times everything is smooth sailing. Any other burgs noticing this - or having issues with a different skill doing the same thing? What about other classes?

In short, yes. It's not noticeable or bothersome so much most of the time.
But I most definitely noticed the lag and hanging of skills soloing in Limlight (I'm not a particularly well geared Burg, so I need a little room for error/lag/hanging).

I'll take a closer look next time I'm on to see which specific skills are hanging,
but I've definitely noticed a slow down in Burglar skill activation.
Whether that has to do with the lag or not, I have no clue.

[auximenes]

It would be great to copy'n'paste such chat channel outbursts.

/chatlog on|off
/chatlog capture

[Fudoshin]

Please do not reinstall. Maybe I did not make things clear enough. We have seen that it helps in some cases and not in others. That means it is not the root cause. If it were, then it would resolve the issue 100% of the time. It doesn't.

What we do know, for a fact, at this point, based on all our research, is:

• It is not server side.
• It is not in our datacenter.
• It is client based.

We have some theories we're testing and should know more once we run them all down.

That's really good to hear, unfortunately I had got so fed up I just downloaded and did a full re-install anyway, 7hrs later, it hasn't fixed the problem but seemingly the problem is less than it was. There was definitly something funky with my files (undoubtedly user created) as I went to remove programs, it still said shadows of mirkwood! I removed it and was still able to play the game after the fact, which I found very odd but the main files were still there. I then deleted everything...and I mean everything, cleared out my whole game drive (partition) short of formatting it and also went into the documents to make sure I didn't have any plugins key stuff etc.

I still stop on horses periodically, I still have some skill lag (click wait +1-5 sec to fire), but (using baseless statistics) 50-60 percent of the troubles I had been having are gone. I downloaded entire high rez, I'm not sure how the client had updated previously but I wonder if there is some confusion with settings regarding which files are being expected and the actual files that are being found/sent. Anyhow I won't try to sound like I understand that stuff because I don't, great to know you are getting there with nailing it down.

[JiiPee]

On my burg, my Improved Subtle Stab (which has no induction) will sometimes "hang" before going off, and other times everything is smooth sailing. Any other burgs noticing this - or having issues with a different skill doing the same thing? What about other classes?

That is exactly "lag" issue what we are talking about. With my warden that "hang"/"freeze"/"lag" will mess up gambit every time it happens. I don't experience anything like 10sec freeze. It's just 1-2sec.

[Tainted_Black]

I'm leaving until this lag is fixed. Th eonly thing left in the game for me atm is Raiding and that Lag makes that impossible. Tonight we failed epic style in the acid wing of ToO due to lag spikes killing us in the jumping phase. After a few attempts we gave up and walked out with 800s+ repair bills.

We've been saying for weeks that it's been a problem with update 6 as it only started getting this bad since it was released. Turbine just figuring that out just shows you that they don't listen to their customer base..

*claps sarcastically*

[Cadennza]

Please do not reinstall. Maybe I did not make things clear enough. We have seen that it helps in some cases and not in others. That means it is not the root cause. If it were, then it would resolve the issue 100% of the time. It doesn't.

What we do know, for a fact, at this point, based on all our research, is:

• It is not server side.
• It is not in our datacenter.
• It is client based.

We have some theories we're testing and should know more once we run them all down.

I find this incredibly simplistic and frankly, insulting.

How can it be client based when entire raids lag AT THE SAME TIME? I am on one side of the country, we have someone from South America, Alaska, west coast and points in between -- and we all lag SIMULTANEOUSLY. I admit to not knowing much about the technical side of things, but I cannot accept that this is client based.

Please don't insult us further with such explanations.

[damnedangel1]

Don't get all bent out of shape just yet people.

The fact that they have admitted there is a problem at all is a step forward. The fact that it has been stated it is a client side does not necessairly mean that the problem is your computer/internet connection. What I take from this statement now is that there is a problem with the GAME client they were unaware of. While they still don't know exactly what it is, they are looking into it.

[Areonium]

I find this incredibly simplistic and frankly, insulting.

How can it be client based when entire raids lag AT THE SAME TIME? I am on one side of the country, we have someone from South America, Alaska, west coast and points in between -- and we all lag SIMULTANEOUSLY. I admit to not knowing much about the technical side of things, but I cannot accept that this is client based.

Please don't insult us further with such explanations.

You should read Sapience post here: http://forums.lotro.com/showthread.p...08#post6100508

He explains what client based means.

[bastiat1]

How can it be client based when entire raids lag AT THE SAME TIME? I am on one side of the country, we have someone from South America, Alaska, west coast and points in between -- and we all lag SIMULTANEOUSLY. I admit to not knowing much about the technical side of things, but I cannot accept that this is client based.

When Sap says "client" he means the client-side game code. The client of everyone in the raid is responding simultaneously to certain events in the game so it isn't inconceivable that some call during those events is causing the lag. A lot of us concluded a bit ago that it was a coding issue released with Update 6- which was buggy as @#$% regardless of the lag.

[Mark_J]

I find this incredibly simplistic and frankly, insulting.

How can it be client based when entire raids lag AT THE SAME TIME? I am on one side of the country, we have someone from South America, Alaska, west coast and points in between -- and we all lag SIMULTANEOUSLY. I admit to not knowing much about the technical side of things, but I cannot accept that this is client based.

Please don't insult us further with such explanations.

I think you're confusing network issues with software issues. This is a communication link between server and client (not to be confused with network routes) that is creating a delay of some sort, it certainly IS possible that everyone who has the same client could lag at the same time.

Take it easy on the only information source we've been getting. I find this to be good news that they think they have the problem isolated to client side scripting...

[naliia]

I find this incredibly simplistic and frankly, insulting.

How can it be client based when entire raids lag AT THE SAME TIME? I am on one side of the country, we have someone from South America, Alaska, west coast and points in between -- and we all lag SIMULTANEOUSLY. I admit to not knowing much about the technical side of things, but I cannot accept that this is client based.

Please don't insult us further with such explanations.

Possibly because one character is setting off the client issue, and because all of your clients must also register what that the one character is doing, it somehow sets off the issue in everyone's client at approximately the same time. I don't know. You don't know. And Turbine doesn't know (yet).

Still, while there are lots of people saying that they're all lagging at the same time, just saying so isn't helping. The thing is, some people are experiencing little or no lag, while others are experiencing enough lag to affect their character skills, and others, like yourself, are experiencing a lag spike that causes an instance failure.

My characters are all on the same server, being played on the same computer by the same person (obviously), and not a single one of them has experienced a stable horse stalling. They've experienced lag in high population areas. One character experienced 2 odd occurrences, once being dismounted for no clear reason and a second failing a timed quest that wasn't even close to being done, in a single session that I can almost positively attribute to a router freakout on my end. I regularly play with my boyfriend (who's in the same room) and at least one other person who is located half the country away and on the opposite coast as the LOTRO servers. As far as I know, we've never experienced simultaneous lag.

Being all indignant at the "insult" is just plain silly because it's not an insult. And it's not helpful. It would be far more helpful if each of the people in your raid could describe EXACTLY what their char was doing at the time you all lagged. If there is a common thread, it might help Turbine figure out what's broken.

[LeRaginAsian]

Given all the accidental rollbacks on fixes from pre-U6, I'm not at all surprised. I hope rolling out 6.1 changes won't further obscure the issue.

[Darmokk]

One client side issue causes these two?

• All members of a raid in the same boss fight having for the same number of seconds at the same time
• You getting unhorsed on a stable horse

Sounds far-fetched. The only possible thing would be if the client-side networking code has been screwed up (e.g. it now is broken when handling some network conditions).

Non-networking client side issues also offer little explanation for Turbine's employees doing better. Unless of course they didn't really test the things that were lagging for the users.

[revoked]

I think this may be getting lost in all the posts. Client-side doesn't mean it's an issue with your computer.

Yes. Because as Nuth points out a client issue doesn't always mean your PC, it can mean the code of the game itself, it could also mean the data is fragmented (and not just on your drive), or that the client/server handshaking (which wouldn't show up as any kind of measurable issue on the server or network side of things) is wonky.

[Kerin_Eldar]

True enough. But the problem seems more and more to be pointing at client code.

Given Turbine have failed to see the problem on their own systems, which clearly means CLIENTS (aka. 'cient code' as you put it) as well as servers, then your assertion seems without foundation.

[Petsoukos]

In other words we are failing to be helpful here! Rather than fill this thread with a bunch of nonsense, I told you so, your doing it wrong, and hilarious pictures, though who ever put that lol cat up is awesome and amazing, lets put in some useful info. Lets make this thread helpful again.

Post a bit on your computer or your windows rating if you don't know.
Graphics =
Memory =
Processor =
Operating system =
DSL, Cable. =

Then tell your story, be 100% accurate.
"I lag when X, But not when X"
"I tried this and it didn't fix it"
"The lag started after X"
"It never lags X"

^^The faster you do that, the faster it gets fixed.

Graphics = AMD 4870 1GB (stock clocks)
vsync'd always at 60FPS playing windowed mode because full screen isn't an option, native resolution is to high

System Memory = 6GB 1333Mhz Tripple Channel

CPU = i7 920 2.66Ghz (stock clocks)

OS = Windoze 7 64bit (latest updates, drivers etc)

Internet = 2048/512 downstream/upstream DSL

Other noteworthy hardware: SSD with a 280MB/s read and 230MB/s write capability. So lag due to game loading isn't a hardware issue.

Lotro High Res Client

Lag Spikes came with the great river update. (u6 you call it?) Followup updates didn't improve anything for me.

What am I doing? Just playing normally as before. Switched from my main (75 LM) to my Warden. Gambits, when there's lag spikes, are getting messed up because I set skill 1 to execute and queue skill 2. When lag strikes, skill 1 has the dashed line animation going around for 1-2s but doesn't fire then poof the -cancel smoke animation- on the skill icon and skill 2 gets fired. Gambit messed up. To compensate when I see lag spikes I reset skill 1 and just wait. First skill 1 get's canceled but then since skill 1 is in queue it fires normally then continue to build up the gambit.

Playing from Greece on a US server (Arkenstone) with about 120-140ms latency (160ms today) The freezing happens less when I'm playing after launch (2pm-4pm which is when US is sleeping) but gets worse in the evening when US is playing. All the above apply to my kin mates from Greece. With different hardware/software configurations.

Hope this info helps.

offtopic: I'm a freelance web dev working at home and actually switched my working hours around so I can play from 12am to 4pm and work later 5pm to 5am. I hope they will fix it soon so I can play at normal hours again.

[jcmanda]

Except that I disagree with every word of what you said there.

Congratulations. Would you like a cookie?

[Mirarian]

I play a fair bit daily on several realms on several character. All level 30 or less.

When I first log in I frequently have a bit more stuttering movement kind of lag than I did pre the update. That usually MOSTLY dissipates inside half an hour.

I have had ONE stable horse stallout. That was a complete I am never going to move may as well hop off and ride. Which is what I did after letting it sit there for 20 minutes.

I was helping a guy with an instanced group quest the other day. It was one of those that you have to talk tot he NPC you are escorting multiple times during it. Huge problems with the NPC getting stuck and not being able to talk to him for the next part.

I can't speak for real instances or dungeons. I have yet to do one.

Intel(R)core(TM)i7 CPU 860@2.80GHz and 2.80GHz
Installed memory(RAM) 8.0 GB
64 bit WIn 7
ATI Radeon HD 5850 1GB Vram
Cable Internet I do not honestly remember the speed but its the fastest we can get here.

[jcmanda]

I did it. I used http://www.md5summer.org/ to generate sha1 hashes from every single file from lotro installation folder. Then I did uninstall, deleted lotro folder and rest of the files what uninstaller didn't clean up. I downloaded fresh copy of highres client and installed it. Patched it to up-to-date and then made sha1 compare. Result is this:
(snip hash)

I have not include some trash like cache folder etc. Also please remember that *.dat files are actually file containers, including lot of files, compressed and uncompressed. They are kinda .zip files. So even that many (maybe all) .dat files checksum didn't match, it won't mean that I had corrupted files. It just means that my files was patched many times and some files might been in different location in container.
There is also some lotro defrag tool out there what should optimize .dat files, but I haven't used it ever.

If there was mismatch of .exe or .dll files, I would have been worried.

Oh and no, fresh install didn't help anything. Well maybe some loading times are faster now. But "lag" is still there.

Thank you very much!

[jcmanda]

Given Turbine have failed to see the problem on their own systems, which clearly means CLIENTS (aka. 'cient code' as you put it) as well as servers, then your assertion seems without foundation.

Your LOTRO client talks to the LOTRO server.

If your LOTRO client doesn't know how to talk to the LOTRO server properly, because it's having a bad day, or it's sad because its dog just died, or because one of its files got corrupted or something, then you might notice a problem.

Now, if there was a problem in one of the LOTRO client files that got PUSHED to every user's machine in a big update, and that problem only manifested itself in a certain (common) environment (network card type X, video card type X, etc), and furthermore only when certain SERVER-side events triggered that bit of CLIENT-side code to load....

See? More stone than sand.