Why....You screwed up.

[Khafar]

Unfortunately we'll never know if the issues are server side because we cannot view the activity on the servers...

If it were on the servers, I suspect they'd have solved it by now. Those are under their direct control, are far easier for them to instrument/debug, and are fully characterized, fixed-specification environments. It makes far more sense that a persistent problem that's been so hard to track down is in the client, and it's great to hear they've added more instrumentation to those. Sounds like they need even more.

Anytime the number of people in an area increases, so does the consumption of bandwidth, on both ends.

It's far worse on the server end due to the N-squared nature of the communications they have to do for players who are near each other. But like I said, that's well-characterized... they know what the servers can handle. LOTRO's architecture used to permit large, open worlds means that unless they're using layering tech in an area, they cannot spread the load across multiple servers - at least within a single 160x160 meter landblock. That's why this game will never have 100 players in a small area battling each other. N^2 is getting quite large at that point. Games like GW2 were architected differently. For example, if you're going to an event with 200 other players it can actually handle that. Sort of. What it does is to shrink the radius at which characters can get data about each other from something pretty typical for MMOs (like 100 feet) down to something truly tiny. I'm talking 20 feet here. Past that radius, the hordes of players just "vanish" if there are too many of them. Pretty weird, actually, but it does keep the servers from falling over.

Anyway, clients are far trickier to debug. The developer doesn't control their environment (which varies immensely), has fewer tools to work with out there, and often cannot easily reproduce the problems seen by customers in real time. When I've had to deal with stuff like that, I've done what they seem to be doing... carpet bombing the critical sections with performance logging, shared resource logging, etc, all in the hopes of getting enough data to figure out plausible causes for whatever multi-threaded nightmare of a problem I'm dealing with.

Khafar

[DutchEZmoder]

As Angadan pointed out, what I said was the client. Which is the software that actually runs on your computer. So what you describe is perfectly in keeping with what I said. Network synchronization plays a huge role here and so does the code controlling it and the tolerances involved. So yes it can be 100% "on your pc" without it being your PC.

Ah right, when i think client side, i always think it's the users problem. Didn't think of it being the actual client itself.

[Schinderhannes]

I smell salami...

Ok, better then nothing.
This small pieces of information on lags let it sound like that the client has a "long" patience before he gets in timeout and communicates "nothing" to all the other clients or just the server.
If this happens alot at the same time, all clients in the "area" become asynchronous and clientside lag infects the server/other clients.
Am I right? ... Propably not.

[Cadd_EU]

Oh, I always thought of the "client" as the computer (my computer, any player's computer). I guess I thought that because it mirrored "server" and "client" better. Both being hardware, an actual physical box, you know. But it appears that in Turbine terminology (which may be common to most/all other computer gaming companies too?), "client" is shrink-wrapped to refer specifically and only to the software of the game that resides on players' computers. That's very interesting, and clears up a lot of miscommunication between players and Turbine Team over the past couple of years.

I think that the full clarification would be:
Client: The Software Turbine designed that resides on your computer, specifically even excluding the Launcher.
Clientside: Everything on your end of the internet cable. This includes your hardware, your OS, other applications running simultaneously, and the client mentioned above.

So if Turbine says "the problem is clientside", it could mean it's your computer/OS whatever, but it could also mean it's the client, or the somewhat unpredictable interaction between the two.
If however Turbine says "the issue is with the client" that means there's really not much you can do, as it's with the software they've produced, just not actually serverside.

[Milithion]

A few things i want to say:
-Lag in Ettenmoors. You say it's client side, but if once every 5-10 minutes everyone in the raid says on the same moment "Oh damn, lag spike" or "****ing lag again" etc, how is this client side? Personally i have a great pc, and lotro isn't that demanding for pc's at all.

there could be some code in the client that causes lags on the pc. if the server sends a command to the client that action A should be done, all clients that receive this command will now have the lag simultaneously.
if its the case you could have a 10.000$ pc and would still have the same lags a a person with a 500$ pc.

a good way to see this: you still get damage from the NPCs during most of the lags.

i could imagine about 10-15 things that could have such behaviours and are caused on client and not sever side.

[ColorSpecs]

I completely believe this. Me, being a computer noob, had no idea about picking up a computer and bought a PC. I lag in lots of instances (not always the newest ones), in mounted combat and in general in the Rohan areas, running in medium settings, with a lot of effects disabled (reflections, shadows, lightning, etc). I blamed the lag (of course, I think there are some internet/server side issues, but not all of the issues

You blame the lag when if you are surrounded by 50 other players and all of them are lagged at the exact same time as you.

You blame your computer when you are the only one out of the 50 lagged.

It's an important distinction.

[Sapience]

I think that the full clarification would be:
Client: The Software Turbine designed that resides on your computer, specifically even excluding the Launcher.
Clientside: Everything on your end of the internet cable. This includes your hardware, your OS, other applications running simultaneously, and the client mentioned above.

So if Turbine says "the problem is clientside", it could mean it's your computer/OS whatever, but it could also mean it's the client, or the somewhat unpredictable interaction between the two.
If however Turbine says "the issue is with the client" that means there's really not much you can do, as it's with the software they've produced, just not actually serverside.

That's reasonably accurate.

Client would absolutely mean the LOTRO client. The piece of software that connects to our services. So "In the client" would be the LOTRO software on your machine (and thus technically everyone's).

Client-side or 'on the user's end' would be pretty much anything on your side of your connection to the ISP that isn't ours (software). it might be your PC hardware, your network/WiFi in your house, your router, your NIC, anything.

I know people always want to scream about it being our fault or that we're blaming the player, but if you look at those two distinctions the odds are greater it's on your end (due to the larger number of possible points of failure/issue) than in the client. In this specific case we know there is something in the client for most poeople and for some it's very likely both the client and somewhat client-side (so both our software and your hardware/network). We're working to isolate the issue in the client, but the other issue are not something we can address. But at least we'll be able to fix what we can. Reviewing the minimum and recommended specs will also give us a chance to understand if people aren't simply running on hardware that can't handle the changes made over the past 6+ years (yes it does happen).

[Sapience]

I smell salami...

Ok, better then nothing.
This small pieces of information on lags let it sound like that the client has a "long" patience before he gets in timeout and communicates "nothing" to all the other clients or just the server.
If this happens alot at the same time, all clients in the "area" become asynchronous and clientside lag infects the server/other clients.
Am I right? ... Propably not.

You're more right than wrong. it's an oversimplification, but not inaccurate. The client can perform perfectly well with no meaningful impact to your game play even at pings as high as 300-350ms. Once you get to 400 things do start to go south quickly, but prior to that there's little difference.

There is synchronization code that allows all of these different ping times to play nice together. If that sync gets off, then bad things happen. We believe the client/server latency resilience in much more robost (in truth the data shows it's more robust than we thought) than the sync resilience. That's where we believe the issues players report exists.

[Cazthelm]

You blame the lag when if you are surrounded by 50 other players and all of them are lagged at the exact same time as you.

You blame your computer when you are the only one out of the 50 lagged.

It's an important distinction.

Of course it is. But not an easy thing to see some times.

In my case, as I mentioned, I "lagged" a lot on the PC. In the ettenmoors it was unplayable... "lag" everywhere. Of course I was thinking it was lag because I could only see that. Me freezing, everybody else freezing (thanks again, to my awful PC), and some kinnies saying something about the lag.

Then I got on the laptop again. I can play in the ettenmoors with no freezes at all, even when full raids and effects are present. Kinnies stay complain about lag, but as far as I'm concerned, I am not affected by it.

[ColorSpecs]

Of course it is. But not an easy thing to see some times.

Its actually an easy thing to see when you experience it. Basically, its 50 players running around one another and nothing is happening. It becomes ridiculously obvious when grouped and everyone is cussing and swearing because they can't get a skill off. If you're saying you don't experience it on a laptop, then you haven't been under the conditions that it is affecting so many players.

I just wonder why they can't duplicate this in a stress test. Or maybe they have. If not, it would seem to be a worthwhile endeavor.

[daneyul]

There is synchronization code that allows all of these different ping times to play nice together. If that sync gets off, then bad things happen. We believe the client/server latency resilience in much more robost (in truth the data shows it's more robust than we thought) than the sync resilience. That's where we believe the issues players report exists.

That would explain the moors lag, but do you feel that lag is the same as the other oft-complained about lag: War steeds rubber banding and speed fluctuations?

Are the new abilities to chart lag showing us any hope for that problem, which honestly likely affects many more players.

[Angadan]

That would explain the moors lag, but do you feel that lag is the same as the other oft-complained about lag: War steeds rubber banding and speed fluctuations?

Are the new abilities to chart lag showing us any hope for that problem, which honestly likely affects many more players.

it certainly seems like that is also a synchronization problem: your war steed galloping along just fine, all of a sudden it syncs up with the server and boom you're 20 meters back and 5 meters to the left. Maybe Sapience can answer whether it seems like it's the same sync problem.

Also, I recently noticed that the war steed sync problem tends to occur, repeatably and predictably, at certain places. Didn't notice this until I started doing the Forlaw - Survivor repeatable quests, particularly the Yst-alphabet one, where you race through 9 blue gates and 12 red gates. For me, every time I get to Red 7 (I think that's the one, it involves a hard left turn around a tree just before you run to it), my war steed is always shifted a good 5-10 meters left of the path I'm running. So predictable that I came to the point of stopping a few seconds just at that gate, giving the game time to sync up with my client.

If it's that predictable at specific spots, maybe there's also a geography interplay component to this issue. Maybe be more complicated than just ONE thing....

[Elaida]

That would explain the moors lag, but do you feel that lag is the same as the other oft-complained about lag: War steeds rubber banding and speed fluctuations?

This reminds me of the good old battle; how do you describe lag? Perhaps Sapience could answer this question as well. I'm kinda curious about the answer.

My own description of actual lag is very simple; lag is strictly packet loss. Symptoms are typically rubberbanding and other kind of problems with movement, or perhaps skills not being fired off at all when you press the button, due to some packets being literally lost along the way.

High latency is not lag in my books; no packets are being lost there, they are simply delayed. Synchronization issues are not lag either in my books, as they are just that; player being in a different position on his/her own screen than he/she is on the server side. That can cause similar rubberbanding with packet loss when the game synchronizes the player's location properly, so it's sometimes hard to figure out whether it is caused by lag (by my description) or if it's "just a sync issue". Game client and/or computer performance can cause very lag-like symptoms as well, including "rubberbanding" where you seem to stop regularly, but that's not lag either; it's just poor performance.

In-game connection status gives a good clue though. Packet loss should stay at zero - period. If it's showing even a fraction of a percent of packetloss for me, things aren't working smoothly everywhere, so that's the first thing I'd check and blame if I'm rubberbanding. Doing several traceroutes usually points out the flaw and shows where the packets are actually getting lost, and it's very rarely at Turbine's end. For me, the most common culprit is the overseas (or more like underseas) connection between Europe and USA, with the cables going from Finland towards central Europe being at the second place.

You only need to lose a few packets here and there to make things go downhill. If there's no packet loss, then it's not really lag, but something else. And yes, I've experienced my warsteed going nuts both with and without packet loss being shown. The difference is that when there is packet loss, it's not just riding with warsteeds that gets me rubberbanding, I get it even on foot. When there isn't packet loss, it's only the warsteed at it's full speed going nuts.

So, all in all, I think the most important thing in solving the problem is to be more specific in what kind of issues there really are. Categorizing everything I've mentioned in this post under the generic term "lag" doesn't really help anyone, so everyone needs to take a good look at the situation and try to pinpoint the actual cause of those symptoms. For example, if you stutter and rubberband, try turning your graphics to minimum; if it stops, it's poor performance. If the stuttering and rubberbanding continues, it's something else - as in, synchronization issues or even actual packetloss.

That's nothing but basic troubleshooting: If you're not certain about the actual cause, you have to try every imaginable thing to rule out possibilities one by one. It's awkward and slow, but quite often, when we're dealing with computers and software, it's the only way to find the ultimate solution.

[Angadan]

This reminds me of the good old battle; how do you describe lag? ... [nice info about packet loss v. latency v. computer performance] ... That's nothing but basic troubleshooting: If you're not certain about the actual cause, you have to try every imaginable thing to rule out possibilities one by one.

I appreciate your insights. My personal definition of lag is anything that causes the game to slow down, stutter, or stop, from MY perspective. I know that's probably the most all-inclusive possible definition, and so the least meaningful to those trying to isolate problems, but it is my definition.

See, I'm not one of those who can easily tell the difference between latency, packet loss, performance, etc. There is nothing on my screen that shows packet loss or latency rates, nor do I want to add these gauges. I'm here to play, to goof off, and while for some people watching those indicators is enjoyable, for me it would not be, it would take from the gaming experience and immersion. Now, I would certainly be WILLING to watch for these different potential sources of lag, for some limited time period, if Turbine specifically thought that I could help them fix the problems. Not me and 250 others, that just makes me a weak link (because probably 235 to 249 of the others would understand and be able to watch/describe/relate what they're seeing better than I). But if _I_ personally could help make a difference, I'd do it. For a limited time. Not likely Turbine would need that, though.

Which leaves me back where I started: (1) happy to be ignorant of all the possible sources and causes of lag, (2) just wanting it to go away, and (3) very appreciative that smart folks like Turbine and you are on the case on behalf of all of us.

[ChromiteSwiftpaw]

Also, I recently noticed that the war steed sync problem tends to occur, repeatably and predictably, at certain places. Didn't notice this until I started doing the Forlaw - Survivor repeatable quests, particularly the Yst-alphabet one, where you race through 9 blue gates and 12 red gates. For me, every time I get to Red 7 (I think that's the one, it involves a hard left turn around a tree just before you run to it), my war steed is always shifted a good 5-10 meters left of the path I'm running. So predictable that I came to the point of stopping a few seconds just at that gate, giving the game time to sync up with my client.

If it's that predictable at specific spots, maybe there's also a geography interplay component to this issue. Maybe be more complicated than just ONE thing....

Actually, I believe that's built into the instance or quest. If you'll notice, right when that shift/bump to the side happens you'll hear the sound effects of gusting wind. I think they put that there to make that gate-race a little more tricky.

[Elderban]

If it were on the servers, I suspect they'd have solved it by now. Those are under their direct control, are far easier for them to instrument/debug, and are fully characterized, fixed-specification environments. It makes far more sense that a persistent problem that's been so hard to track down is in the client, and it's great to hear they've added more instrumentation to those. Sounds like they need even more.

Not entirely true. Certain amounts of lag are "allowed" to happen in some parts of Middle Earth to free up resources and bandwidth in other parts of Middle Earth. In the case of the Ettenmoors, that zone is the red-headed step child, and they will most likely use more and more resources from it as the game grows to alleviate issues in other parts of Middle Earth.

It's pretty evident that as the game grows, so do the issues in the Ettenmoors.

[daneyul]

For the War Steed issue--for me, it's not rubber banding. At least not very often, and usually just when I'm in town and "seem" to jump a fence only to get whipped back a few seconds later. It's not a biggie at all for me.


For me, the problem is horrible speed fluctuations. Speed up/slow down/speed up/slow down. Happens anywhere on a war steed--even in the Shire or Bree. Done every conceivable trouble-shooting (throttle fps, engine speed, turn down pretty much everything--no change). It might go away if I turn off DX 10/11. Maybe. But that's about it. Doesn't seem to be limited to busy times, and my connection speed is 20/5 fios, which is way ample. System specs should be fine (ssd, 6950 video, i7). And it ONLY occurs when I get on a war steed. I've read about, and talked to quite a few people experience it too.

Is it lag? Or something else? Not sure. Just wish it were fixed! That's why I was hoping Sapience might give a word on if they have any leads on this particular issue.

[ColorSpecs]

For the War Steed issue--for me, it's not rubber banding. At least not very often, and usually just when I'm in town and "seem" to jump a fence only to get whipped back a few seconds later. It's not a biggie at all for me.


For me, the problem is horrible speed fluctuations. Speed up/slow down/speed up/slow down.

It wouldn't surprise me if that was a form of rubberbanding.

[Angadan]

Actually, I believe that's built into the instance or quest. If you'll notice, right when that shift/bump to the side happens you'll hear the sound effects of gusting wind. I think they put that there to make that gate-race a little more tricky.

Ah really cool! I confess, I don't have my fx sounds high enough, just enough to catch them when I'm listening for them. Thanks for letting me know about this!

[damnedangel1]

Has anyone thought to look into the data that is being sent to other players in an area as part of the problem? I have noticed that I load quicker and lag less when logging in and playing a low level toon with no crafting, empty bags and only a few quests completed or in progress. When loading this low level toon, login takes about 5 seconds. With a level cap toon with crafting maxed, all of the recipies available for the professions, hundreds of quests completed and bags full of stuff, login takes about 30 seconds to a minute. This is with the game loading from a high end SSD on a i5 with 8GB of ram. My thinking is, if we are sending all of this data to each player that can see us, that would probably explain a lot of the lag when large groups of level cap toons come together in ways such as large raid vs raid battles in the ettens or player run events such as weatherstock. If my assumption is correct, simply (yea I know, its not actual simple on the back end) not sending the data regarding what we can craft or what's in our bags and only sending data on what is currently equipped would have a pretty big impact decreasing sync issues caused by the client trying to receive all that character data for multiple toons in the area.

[Aldeld]

It is understood. Just as clearly as it is understood that there are players with top end machines who report very rare or no lag in the same situations. That's why we believe it to be both of the issues I mentioned.

I run in a config where I get anywhere from 70-120 FPS in LOTRO. My network bandwidth isn't anything to sneeze at, either, and my latency is usual very low (like 40ms or so). My machine runs other newer games at very high settings and still gets around 40FPS (FarCry 3, etc.) That said - LOTRO has laggy gameplay in the moors, and I often find myself inexplicably stuck in one spot, unable to move. I still see others around me moving, see text chat, hear voice chat, but my toon just sits there stupidly stuck while I'm seeing my framerate rock a steady 90FPS or so. Whatever issues you're chalking this up as, it's not due to shoddy hardware in my case at least. It's 100% either client or server issues, or a combination of the two.

[Sapience]

This is off topic a little bit, but I found it interesting.

There's a TV show that films all sorts of crazy things in slow mo so you can see how things actually react to outside forces. Like a stick hitting a cymbal and the forces transmitted through the metal.

One experiment they did was to see how long the brain "lagged" behind the body. By filming with high speed cameras at known frame rates them could show how long it took a pain stimulus to reach the brain and a reaction to occur. I thought their times were interesting...

Leg pain - 100ms
Shoulder - 80ms
Ear - 40ms

So your body is kind of laggy at the extremities.

[PulseDiver2]

when the lotro client will be 64-bit ?

why lotroclient with recommended system requirements of 2009 year slow down a high\top pc 2013 in etten raids (24vs24, 48vs48 and more)?

when Windows 7\8 and DX11 wil be in official system requirements of lotro client?

[Reximus]

Your comment about the leaderboards is only half true. People may not have been using it on the LOTRO site but they were on the Black Appendage site because they cut and displayed the data in a more useful format. If you want a truer idea of the amount of people who were actually viewing total leaderboard data, I am sure the Black Appendage people can provide you with their usage information.

It's my understanding that Leaderboards were near real time, in which case any third party site providing the information would have to be pulling that information when requested by a user, which means the Turbine servers would still have been recording hits as if the user was using their boards directly.

[Yula_the_Mighty]

when the lotro client will be 64-bit ?

Probably never because Turbine will not want to:
1) Provide four clients instead of two.
2) Convert the existing two to 64-Bit clients thereby giving up all the customers that have 32-bit OSes.

when Windows 7\8 and DX11 wil be in official system requirements of lotro client?

Probably never because Turbine will not give up customers that have machines that do not have Windows 7 or 8. And or do not have machines capable of running DX 11.

I do not think there are a lot of current games that require Windows 7+ and DX11.

Even a much newer game such as Swtor only requires 2 gigabytes of core memory, Windows XP and Direct X 9.

Turbine made this kind of mistake with their first two games Asheron's Call and Asheron's Call 2. Turbine changed the game software so that you needed a much newer and more powerful machine to play the game. The result was a big loss in revenue from current customers who could no longer play the game on their PC. Upgrading their PC was not an option. It has been a long time - I believe for Asheron's Call Turbine changed the game software from needing a video card that fully supported DirectX 5 to requiring a hardware compliant DirectX 9 video adapter.