Current server lag state

[Rogdush]

I would like to ask SSG what is the current state of servers stability.
We all know lag is just common think in Lotro. Its not connection lag, its just server lag. I dont play PvMP just because its impossible to do anything there, lags for 30s and just wait till every packet get spitted to u at once so u can see ur death. The same is happening slowly in pve. Earlier there were lag spikes, once every half hour u had to have 10s lag or its not lotro u playing. Now its not lag spike its just lag all the time and sometimes u get 30s spikes when u cant do anything. The same happen every time when new zone is realesed. Gundabad.. khazad dum everything.
Its just server code fault, or hardware? Its definitely not connection altho this is also debatable while playing from europe...
Anyway, since festival started game is just unplayable. Skill execution 2-3s is best u can get most of the time.
I wrote ticket about it and i just got templated response "we're working on it". Well i play this game for a long time and how long u want to work on it? 15 years were not enough ?
I still remember the same lags in pelennor area, where skill execution took 5s every single time. That area is cursed since long time.
The paradox of this game is that you want more ppl to play, but when more ppl play u cant play. Everyone is praying for this festival to end now. Thats what u want SSG ?
Why dont you give us some update what u doing about lag? Its the worst thing in this game. You made dynamic very fast instances/raids where u need to be just fast or die. Now you just watch bleeds tick 200k on u and u cant do nothing because skill wont execute.
I just think this should be top priority and at least once a month should give us an update will this be possible to fix it or its just deep in server code and you need like another year for fix ?
Should it not be the top priority?

[MourneBlade]

Guess you didn't see the other 99 threads about lag already, we really didn't need another one...

[Winzhi]

Guess you didn't see the other 99 threads about lag already, we really didn't need another one...

Well apparently we do since the lag is still going on and SSG is very quiet about it eh.

They could litteraly, FINALLY, announce: "We know especially Evernight is lagging as hell, we looked into it and it will cost us too much to get a better hosting location so we just hope the server population will drop again" and the lag threads would go down by 90%.

[maartena]

Well apparently we do since the lag is still going on and SSG is very quiet about it eh.

This is my beef with SSG at the moment.

Yes, we all know THEY know about the lag.
Yes, we all know they are supposedly "working on it".

But.... there have been major lag problems for the better part of 6 years now, and they have been promising fixes for about that long, without any meaningful result. Or at least, nothing tangible that can be seen or measured from the client side. The fact that they seem to completely ignore the lag threads on the forum, and that they have now resorted to removing skills from Loremaster to reduce the lag leads me to believe they currently have no idea where the lag comes from and how to fix it.

We're talking about a game with 17+ years of code (they started building this well before the launch in 2007), and unfortunately a lot of people with expertise of the old code have been let go and aren't coming back. On top of that, they keep releasing new content, and every time new content gets released, new things get added to the database, which will grow ever larger, and thus cause more performance issues.

Now, what IS the solution? I do not know..... but I do know that SSG just ignoring the lag threads as if all is well with the game is an insult to the players.

So yes, please..... Ravinian, Cordovan? Please give us a status update as to what is next in combatting the lag.

64-bit servers?
More servers on bare metal hardware vs virtualization?
More resources for servers? New hardware in general?

Where are we at with all of this?

[DoRonRon]

All the eggs are in the 64-bit server code basket, like they were for the 64-client basket before.

Buying in some hardware pales in comparison to savings on licences for a database though. We have sayings for a reason; "no such thing as a free lunch".
But Daybreak were masters of extracting all they can out of games other publishers gave up on. Which means less overheads and a set loyal customer base and drawing $52 rather than $15 a month from them.

This last month has been like sending invites out for a huge party but no one thought to ring the caterers. Strobe light show works, but it wasn't supposed to strobe.

[Thentix]

I don't see how it is still a problem with how much money they charge for their absurdly over priced expansions and packs. I know there is a lot of people buying the highest bundles because they shame you into buying it cause if you don't, you aren't the "ultimate fan" of lord of the rings so shame on you. Overpay us for $40 content or be shamed. It's severely messed up.

[WorrBinpike]

The reason why SSG says nothing about the lags is, that they just dont have a clue how to solve it. Every now and then there comes a comment that they found some bad code and there is a fix, but cant tell how good this will work about lags.

Dont expect any technical reason to explain the lags from SSG.

[PatasdoLeao]

The reason why SSG says nothing about the lags is, that they just dont have a clue how to solve it. Every now and then there comes a comment that they found some bad code and there is a fix, but cant tell how good this will work about lags.

Dont expect any technical reason to explain the lags from SSG.

If this is true, maybe the community could give an hand with suggestions.
In the other topics I heard many possibly solutions like: Spend an effort to test all commands, work on the compatibility of the 64bits,...

---------------------------------------
edit: Thanks for the reply of the community team.

[Winzhi]

If this is true, maybe the community could give an hand with suggestions.
In the other topics I heard many possibly solutions like: Spend an effort to test all commands, work on the compatibility of the 64bits,...

I mean the servers are hosted by daybreak.

EQ1, EQ2 and PS2 have had lag issues ever since they moved the servers to a cloudservice many years ago.

The only server that was not moved to this cloud service is the EQ2 EU server Thurgadin which is hosted in the EU and surprise surprise, its the only one without lag.

[Rogdush]

Guess you didn't see the other 99 threads about lag already, we really didn't need another one...

Thats exactly why we need another topic, because there is no serious take on the issue from SSG.
No one really knows why we having lags even.
Is it the hardware issue? You need new xeon with 32 cpu in to make it happen? Is server code even multithreaded? Do SSG even try to resolve it or just pray it will be good? Do server code need to be seriously refactored to make it happen?
I think we need some good info what is really happening. I am just seeing it every time whenever more ppl log or new xp goes live. Every time its just huge lag and u cant even enjoy new area. You know you loose so many players because of it ? I still remember ppl leaving to wow just because they had enough of lag spikes. You just invest everything in this lag issue #### for month or two and get it resolved. I bet more ppl would play and more money gonna come.
I dont believe SSG dont know what is the problem, its just money issue for sure. Either time for code refactor or hardware. You think software enginieer cant figure out 17 years old server code? Its not the issue.
Would be just so good to get some statement from SSG what is happening and how/when they can resolve it.

[Raninia]

Lag continues to be a problem, and we continue to work on it. I understand it's frustrating that it continues to happen, but I can honestly say, I don't enjoy us having to constantly deal with this problem any more than y'all enjoy experiencing it.

Here's a quick update as to where we are now:
Overall, performance is in a better place than when Gundabad released, despite larger player concurrencies and overall player population. "Better" doesn't mean good, but it's nice to see that the months we spent working on various improvements over the past six months have born some fruit.
With Update 33, we saw a huge influx of players, the likes of which we haven't seen in at least 6 years. This has led to worsening performance across all worlds. We even thought we'd lost a Landroval blade - someone's gonna use this to try and argue we are or aren't virtualizing, we still are, but VMs still gotta live somewhere - but it went down temporarily and was successfully revived. We also re-rolled out a fix we'd introduced for 32 and then disabled for 32.0.2 because of issues related to the Spring Festival. It's gone really well and has further reduced some of the issues we're seeing.

We have in the works, and in various stages, several initiatives, some of which I've spoken about before:
64-bit servers are closer, but they have been just one can of worms after another. A lot of the game is built on assumptions about underlying architecture, so it's led to a lot of digging and reworking, but it's making progress. I don't have an ETA yet, but I've been told "Soon." This is a constant conversation I have with the folks involved, and I'll be getting more updates later today.
In part based on data we've been provided, we've sussed out previous issues, and are building better tooling to help us identify problems and their solutions faster. There are several things happening within here, but most of them won't have any immediate player-facing impact, since they'll just make it easier for us as we're solving the problems, not solve any itself. Nonetheless, this is important, and the next one should be ready as soon as this week.
Our deprecation of the 32-bit client is related to this work, as some of the improvements we're looking to make require 64-bit. Those next set of changes have been in testing now for a few weeks, and will hopefully get stabilized soon. Further improvements to the 64-bit client are coming, again based in part on player reports and data that have been provided, so thanks again for sending that stuff to us. Keep it coming
Later this week, we're also hoping to have several sets of new features ready for internal testing, that will help with some of the sources of lag around communication with the game databases. That's another avenue we're looking at to chip away at the problem.

There's a bunch of other stuff that's also down the line, but isn't as heavily focused atm, so don't want to speak about it too much yet. And this doesn't include changes we're making to our creation processes for new updates that will reduce the stress those areas place on clients and the game server, some of which we've seen benefit what we've released with the Angle and Yondershire, as well as future stuff to come.

We're still looking at this, we're still working on it, and we will continue to. A significant portion of the engineering effort that is being done and planned for the coming year is focused on continuing to improve the performance of the game for everyone. Thanks for your patience, I know it doesn't always come easy.

[DoRonRon]

I don't see how it is still a problem with how much money they charge for their absurdly over priced expansions and packs. I know there is a lot of people buying the highest bundles because they shame you into buying it cause if you don't, you aren't the "ultimate fan" of lord of the rings so shame on you. Overpay us for $40 content or be shamed. It's severely messed up.

If you can't hack getting to grips with all but a few aspects of the game's various systems how else are you to prove your worth as a player? It's beyond me how some continue to look down on others from such a lowly position. Where else do bottom feeders find their prey? All getting their hands on their LP bonus packs now to feed their obsessions and make up for their inability to function without them. But that's Daybreak's pricing for all it's games.

The recent "gifts" just expose how much the compilers of these bundles are concern for the game; throwing the inept into the late game without the knowledge and resources to function with valars and carry-all functionality often unfit for purpose to meet needs. How many different essence, tracery and enhancement items are in the game now? Thousands... And we can't open up space in packs because AS capping stops us flaming all the obsolete stuff.

While some I'm sure will be pressed to finish the 100 tracker by close (see above) we have others reaching the 1000 at max already (FIVE AS Caps!). Without the option for purple+ 131+ enhancement barter in the archive all the AS in Middle-earth is worthless.

You need to play it to know it!

[maartena]

Lag continues to be a problem, and we continue to work on it. I understand it's frustrating that it continues to happen, but I can honestly say, I don't enjoy us having to constantly deal with this problem any more than y'all enjoy experiencing it.

(cut most text)

We're still looking at this, we're still working on it, and we will continue to. A significant portion of the engineering effort that is being done and planned for the coming year is focused on continuing to improve the performance of the game for everyone. Thanks for your patience, I know it doesn't always come easy.

Thank you Raninia. This is the kind of communication I expect from SSG, and I want you to know it is very much appreciated. Thanks for giving us a few more details and insights in the process, and I hope you get things sorted out soon.

[Thorebane]

Lag continues to be a problem, and we continue to work on it. I understand it's frustrating that it continues to happen, but I can honestly say, I don't enjoy us having to constantly deal with this problem any more than y'all enjoy experiencing it.

Here's a quick update as to where we are now:
Overall, performance is in a better place than when Gundabad released, despite larger player concurrencies and overall player population. "Better" doesn't mean good, but it's nice to see that the months we spent working on various improvements over the past six months have born some fruit.
With Update 33, we saw a huge influx of players, the likes of which we haven't seen in at least 6 years. This has led to worsening performance across all worlds. We even thought we'd lost a Landroval blade - someone's gonna use this to try and argue we are or aren't virtualizing, we still are, but VMs still gotta live somewhere - but it went down temporarily and was successfully revived. We also re-rolled out a fix we'd introduced for 32 and then disabled for 32.0.2 because of issues related to the Spring Festival. It's gone really well and has further reduced some of the issues we're seeing.

We have in the works, and in various stages, several initiatives, some of which I've spoken about before:
64-bit servers are closer, but they have been just one can of worms after another. A lot of the game is built on assumptions about underlying architecture, so it's led to a lot of digging and reworking, but it's making progress. I don't have an ETA yet, but I've been told "Soon." This is a constant conversation I have with the folks involved, and I'll be getting more updates later today.
In part based on data we've been provided, we've sussed out previous issues, and are building better tooling to help us identify problems and their solutions faster. There are several things happening within here, but most of them won't have any immediate player-facing impact, since they'll just make it easier for us as we're solving the problems, not solve any itself. Nonetheless, this is important, and the next one should be ready as soon as this week.
Our deprecation of the 32-bit client is related to this work, as some of the improvements we're looking to make require 64-bit. Those next set of changes have been in testing now for a few weeks, and will hopefully get stabilized soon. Further improvements to the 64-bit client are coming, again based in part on player reports and data that have been provided, so thanks again for sending that stuff to us. Keep it coming
Later this week, we're also hoping to have several sets of new features ready for internal testing, that will help with some of the sources of lag around communication with the game databases. That's another avenue we're looking at to chip away at the problem.

There's a bunch of other stuff that's also down the line, but isn't as heavily focused atm, so don't want to speak about it too much yet. And this doesn't include changes we're making to our creation processes for new updates that will reduce the stress those areas place on clients and the game server, some of which we've seen benefit what we've released with the Angle and Yondershire, as well as future stuff to come.

We're still looking at this, we're still working on it, and we will continue to. A significant portion of the engineering effort that is being done and planned for the coming year is focused on continuing to improve the performance of the game for everyone. Thanks for your patience, I know it doesn't always come easy.

I'm just going to say in general, thank you for some kind of notice/update.
Alot of people here have complained for years and years regarding well.. just general management/upper tier of staffing communication, so to actually have some kind of official reply is nice.
Even if it's doesn't have all the answers or fixes we are hoping for, just more constant updates goes a long way.

[alwaysbroke]

Lag continues to be a problem ...

You say all that that like you didn't expect additional server load. You say all that like it's hard to simulate server load using a version of beta server, watch it burn, then make adjustments.

You say all that like you're actually making an effort, but you had the tools, you had the opportunity, and you still let the servers burn during anniversary. How am I supposed to recommend you without embarrassment? Oh I know, I'll just tell everyone you'll give away a bunch a free stuff to make up for your lack of effort.

Thanks for the gifts though.

[CaerArianrhod]

Lag continues to be a problem, and we continue to work on it. I understand it's frustrating that it continues to happen, but I can honestly say, I don't enjoy us having to constantly deal with this problem any more than y'all enjoy experiencing it.
.

Why is this here and not at a much more prominent place in the forum?

[DoRonRon]

Why is this here and not at a much more prominent place in the forum?

In three threads already though... out of dozens and...

[Trixie_Me]

Lag continues to be a problem, and we continue to work on it. I understand it's frustrating that it continues to happen, but I can honestly say, I don't enjoy us having to constantly deal with this problem any more than y'all enjoy experiencing it.

Thank you, Raninia. The communication is very much appreciated. FWIW, I left 7 years ago because the lag was so bad, the game was quite literally unplayable for me. It is better than it was (or I would not have stayed), but there is a long way to go to.

[Thentix]

What kind of servers do they use? More importantly, why do games that have more players than them with the same amount of revenue or less, not have these kind of problems? I know they have money, see above. Why can't they just get better servers? I guess I don't get it.

I would love to play warden but, I just can't because of this lag. Skills fire off too late and I died faster than a RK in melee. I get a better connection living in california playing mmos overseas than I do playing lotro.

[Rogdush]

There's a bunch of other stuff that's also down the line, but isn't as heavily focused atm, so don't want to speak about it too much yet. And this doesn't include changes we're making to our creation processes for new updates that will reduce the stress those areas place on clients and the game server, some of which we've seen benefit what we've released with the Angle and Yondershire, as well as future stuff to come.

We're still looking at this, we're still working on it, and we will continue to. A significant portion of the engineering effort that is being done and planned for the coming year is focused on continuing to improve the performance of the game for everyone. Thanks for your patience, I know it doesn't always come easy.

Thanks for the update Raninia. Appreciated.
Basically what you say its that the hardware is the issue as the code also. Good you try to work on it from several angles and I hope it will be better finally.
I would advice to write about this in update log or some producer letter, because I think its the most important thing right now to deal with. Lag is very constant from begining of anniversary and i bet most ppl get annoyed a lot lately.

[Toddlepip]

Thank you Raninia. This is the kind of communication I expect from SSG, and I want you to know it is very much appreciated. Thanks for giving us a few more details and insights in the process, and I hope you get things sorted out soon.

Just to add my support for these sentiments.

Even when things are particularly tough right now with a huge influx of players drawn by the Anniversary gifts and events, that we as players are provided with good communication which may not offer immediate solutions, keeps us informed of progress. Knowing that issues are not simply avoided but being actively addressed and improvements (on all levels) planned out for the future.

[DoRonRon]

What kind of servers do they use? More importantly, why do games that have more players than them with the same amount of revenue or less, not have these kind of problems? I know they have money, see above. Why can't they just get better servers? I guess I don't get it.

I would love to play warden but, I just can't because of this lag. Skills fire off too late and I died faster than a RK in melee. I get a better connection living in california playing mmos overseas than I do playing lotro.

Essentially it comes down to a choice made in the infancy of the game's development. The choice was to make the server the sole arbiter of where any client was in the game space given the various inputs received. Games like WOW and later ESO trusted the client to inform the server where it was. The second method presumably then requires a lot of early game costs to whack-a-mole the various cheats; porting to adjacent "perches" farming rewarding drops from open world bosses in WOW and node farming bots under the ground surface and inaccessible from harm/interruption from mobs in ESO. I have to assume they close down these loop holes at some stage and all the issues are behind them, hurrah!

For Lotro there were none of the same issues and while the servers can "correct" position in a timely fashion we don't notice so much. However the "server" has a bunch of other things to do and to an extent the order they do things might have been tweaked in recent times in the chase for performance gains.
But each new class that's added comes with a bunch of buffs and debuffs to keep track of. Every character in a fellow will have stats change on every tick as the multiplicity of effects come and go and it's the server that has to resolve and distribute that data. Your FPS won't have anywhere near the variability our servers have to deal with.


Turbine's Asherons's Call wasn't immune to the cheats with "Life" mage levelling bots draining the life out of mobs stuck behind a wall with the floor littered in loot from the decayed mobs. If you want to know bad lag, it was there, on dial up trying to pick up a few choice drops (named for dropping on floor I guess). It might have taken several minutes just to move your character to the drop and another wait for it to appear in your pack. But not too much because the game had encumbrance and loaded too much and you couldn't move in open clear landscape let alone a lag fest. AC2 had it's lag fests when the whole server pretty much loaded into the same dungeon (carefully, as step forward would "pull" the room) space once a week.

With Turbine and then SSG putting off or ignoring the impact additions have had on performance it's no surprise we are where we are. A new class can generate this much income and cost so much to implement is as far as those calculation have likely gone. Who's going to speak up or even be heard with all the usual hype that surrounds each new thing?

Would trusting the client's view of where it was cause more problems? Yes it more than likely would but how about lending it some measure of autonomy when not in combat or when not on auto attack and no skill in the queue to devote resources to those that are.

[LagunaD2]

I'm just going to say in general, thank you for some kind of notice/update.
Alot of people here have complained for years and years regarding well.. just general management/upper tier of staffing communication, so to actually have some kind of official reply is nice.
Even if it's doesn't have all the answers or fixes we are hoping for, just more constant updates goes a long way.

Basically, instead of provisioning adequate hardware resources, or hiring people to fix all the problems with the game, they hired someone to constantly make excuses and/or apologize for them.

[IcarusLSC]

I rather see improvements in lag then any new content by a mile.

[Baritone]

This is a recent post from Everquest(also daybreak)...apparently its a common issue with dated games like EQ and LOTRO. Programmers come and go and the new hires just can't figure out the code, and in LOTROs case the game has run horribly since post-Moria. If they had a blank check they could write any time like Blizzard I'm sure things would happen faster


For those of you upset and frustrated by lag, we understand and hear you. It frustrates us, too. This post is meant to provide some insight into what causes these issues, how we got to where we are today, and what is being done to improve performance now and in the future.

Game performance is something we work on continuously and is very challenging to solve completely. EQ has a lot of complex systems, and at least 25 years of code. More than 50 different programmers have come and gone over the years. One of the many challenges of working on EQ is understanding what someone was trying to do when they wrote the code we now are modifying, and doing so correctly without breaking existing things. It is common that we do break things (despite thorough testing), and we fix them as soon as we can.

Unfortunately, in the past, EQ hasn't always reliably measured what impact changes to the game have made to performance over time. As more and more things were added to the game, performance declined. The game's current state is that we need to fix numerous issues that can be quite difficult to fix (a minor improvement may take one programmer a week to complete), and see only a small improvement which is imperceptible to players.

Over the past few years, we have made a lot of changes to alleviate performance issues, but it is still a significant problem, and is well known and cared about by the developers. However, it is understandable that these changes feel as though nothing has been done. It is as frustrating for you as it is for us to have poor server performance. We want nothing more than to allow more players to be on at the same time, for zones to run smoothly in raids, and for playing the game to be a good experience. This year, the top issue for the engineering team to tackle is raid zone performance. It is important to us to improve things, and we know very well the impact it has on your enjoyment of and ability to play the game.