Evernight is unplayable due to lag... again

[Cmanc]

Can't sign in due to lag stuck on loading screen

[DoRonRon]

A while back, Cord asked folks to send specific data to help diagnose the issue. I think he got plenty of response from that -- although I confess I haven't personally sent anything -- but so far, no status update on what they're doing with that data, other than a rate-limiting/-throttling change that was made a few months ago, and seems to have had negligible effect. I think that was from a code review, though, not necessarily from user-submitted data.

With the very greatest respect - I say this earnestly - you MUST send the info and you must encourage everyone you know to send info too. A bit like RL voting, unless they get a good sample, how can they identify the problem or even take it seriously if people don't respond. Sorry for butting in but I feel very strongly about this both with LotRO and with RL voting/info gathering. If you don't add your info/vote, it's then difficult to complain about how schite things are.

My evaluation was that that call was missing the point entirely and little more than Cord enabling trolling of us. There was too much evidence of blanket server wide onset of lag. We only got the client traffic increase because Rarinia caught a particular player post mentioning it. Don't get me wrong, all the specific locales that are worse than elsewhere have remained the same (least no one has listed the fixes to common black spots) but it's a bigger issue than /loc data can expose. Asking for /loc data in such circumstances could only have the majority of reports coming from spurious locations as a wave of lag hit the server, asinine.

His boss would have told Cord to get his #### on Evernight in 2016 if anyone had cared. Resides on Landroval instead. MIA streaming there this past month too.

[BlitzKr1eg]

SSG would be really stupid if they didn't put huge resources into fixing the performance issue before the new Amazon shows premiers. They've already modernized their F2P experience, now they need to fix the performance...They can't miss this opportunity for an insane growth that this new billion dollar series will give them.

[Raninia]

Lag continues to be a problem, and we continue to work on it. I understand it's frustrating that it continues to happen, but I can honestly say, I don't enjoy us having to constantly deal with this problem any more than y'all enjoy experiencing it.

Here's a quick update as to where we are now:
Overall, performance is in a better place than when Gundabad released, despite larger player concurrencies and overall player population. "Better" doesn't mean good, but it's nice to see that the months we spent working on various improvements over the past six months have born some fruit.
With Update 33, we saw a huge influx of players, the likes of which we haven't seen in at least 6 years. This has led to worsening performance across all worlds. We even thought we'd lost a Landroval blade - someone's gonna use this to try and argue we are or aren't virtualizing, we still are, but VMs still gotta live somewhere - but it went down temporarily and was successfully revived. We also re-rolled out a fix we'd introduced for 32 and then disabled for 32.0.2 because of issues related to the Spring Festival. It's gone really well and has further reduced some of the issues we're seeing.

We have in the works, and in various stages, several initiatives, some of which I've spoken about before:
64-bit servers are closer, but they have been just one can of worms after another. A lot of the game is built on assumptions about underlying architecture, so it's led to a lot of digging and reworking, but it's making progress. I don't have an ETA yet, but I've been told "Soon." This is a constant conversation I have with the folks involved, and I'll be getting more updates later today.
In part based on data we've been provided, we've sussed out previous issues, and are building better tooling to help us identify problems and their solutions faster. There are several things happening within here, but most of them won't have any immediate player-facing impact, since they'll just make it easier for us as we're solving the problems, not solve any itself. Nonetheless, this is important, and the next one should be ready as soon as this week.
Our deprecation of the 32-bit client is related to this work, as some of the improvements we're looking to make require 64-bit. Those next set of changes have been in testing now for a few weeks, and will hopefully get stabilized soon. Further improvements to the 64-bit client are coming, again based in part on player reports and data that have been provided, so thanks again for sending that stuff to us. Keep it coming
Later this week, we're also hoping to have several sets of new features ready for internal testing, that will help with some of the sources of lag around communication with the game databases. That's another avenue we're looking at to chip away at the problem.

There's a bunch of other stuff that's also down the line, but isn't as heavily focused atm, so don't want to speak about it too much yet. And this doesn't include changes we're making to our creation processes for new updates that will reduce the stress those areas place on clients and the game server, some of which we've seen benefit what we've released with the Angle and Yondershire, as well as future stuff to come.

We're still looking at this, we're still working on it, and we will continue to. A significant portion of the engineering effort that is being done and planned for the coming year is focused on continuing to improve the performance of the game for everyone. Thanks for your patience, I know it doesn't always come easy.

[Arantha]

Lag continues to be a problem, and we continue to work on it. I understand it's frustrating that it continues to happen, but I can honestly say, I don't enjoy us having to constantly deal with this problem any more than y'all enjoy experiencing it.

Here's a quick update as to where we are now:
Overall, performance is in a better place than when Gundabad released, despite larger player concurrencies and overall player population. "Better" doesn't mean good, but it's nice to see that the months we spent working on various improvements over the past six months have born some fruit.
With Update 33, we saw a huge influx of players, the likes of which we haven't seen in at least 6 years. This has led to worsening performance across all worlds. We even thought we'd lost a Landroval blade - someone's gonna use this to try and argue we are or aren't virtualizing, we still are, but VMs still gotta live somewhere - but it went down temporarily and was successfully revived. We also re-rolled out a fix we'd introduced for 32 and then disabled for 32.0.2 because of issues related to the Spring Festival. It's gone really well and has further reduced some of the issues we're seeing.

We have in the works, and in various stages, several initiatives, some of which I've spoken about before:
64-bit servers are closer, but they have been just one can of worms after another. A lot of the game is built on assumptions about underlying architecture, so it's led to a lot of digging and reworking, but it's making progress. I don't have an ETA yet, but I've been told "Soon." This is a constant conversation I have with the folks involved, and I'll be getting more updates later today.
In part based on data we've been provided, we've sussed out previous issues, and are building better tooling to help us identify problems and their solutions faster. There are several things happening within here, but most of them won't have any immediate player-facing impact, since they'll just make it easier for us as we're solving the problems, not solve any itself. Nonetheless, this is important, and the next one should be ready as soon as this week.
Our deprecation of the 32-bit client is related to this work, as some of the improvements we're looking to make require 64-bit. Those next set of changes have been in testing now for a few weeks, and will hopefully get stabilized soon. Further improvements to the 64-bit client are coming, again based in part on player reports and data that have been provided, so thanks again for sending that stuff to us. Keep it coming
Later this week, we're also hoping to have several sets of new features ready for internal testing, that will help with some of the sources of lag around communication with the game databases. That's another avenue we're looking at to chip away at the problem.

There's a bunch of other stuff that's also down the line, but isn't as heavily focused atm, so don't want to speak about it too much yet. And this doesn't include changes we're making to our creation processes for new updates that will reduce the stress those areas place on clients and the game server, some of which we've seen benefit what we've released with the Angle and Yondershire, as well as future stuff to come.

We're still looking at this, we're still working on it, and we will continue to. A significant portion of the engineering effort that is being done and planned for the coming year is focused on continuing to improve the performance of the game for everyone. Thanks for your patience, I know it doesn't always come easy.

Thanks a lot for this answer. Hope, that the answer is also worth for all other servers. Because the situation is there similar and not satisfying anymore.

[Fadil]

Lag continues to be a problem, and we continue to work on it. I understand it's frustrating that it continues to happen, but I can honestly say, I don't enjoy us having to constantly deal with this problem any more than y'all enjoy experiencing it.

Here's a quick update as to where we are now:

<snip>

Thank you for your reply and for all the efforts to fix the problems.
I must say that yesterday was a bit better for me on Evernight, I also noticed far less people around the party tree in the Shire, maybe the worst festival lag is behind us.

[BlitzKr1eg]

Thank you for your reply and for all the efforts to fix the problems.
I must say that yesterday was a bit better for me on Evernight, I also noticed far less people around the party tree in the Shire, maybe the worst festival lag is behind us.

Personally I had the worst experience on Evernight last night, especially in ettens. Things aren't improving, quite the opposite actually.

I get my hopes up until there some major change announced instead of small incremental "fixes".

[JH15547]

Lag continues to be a problem, and we continue to work on it. I understand it's frustrating that it continues to happen, but I can honestly say, I don't enjoy us having to constantly deal with this problem any more than y'all enjoy experiencing it.

Here's a quick update as to where we are now:
Overall, performance is in a better place than when Gundabad released, despite larger player concurrencies and overall player population. "Better" doesn't mean good, but it's nice to see that the months we spent working on various improvements over the past six months have born some fruit.
With Update 33, we saw a huge influx of players, the likes of which we haven't seen in at least 6 years. This has led to worsening performance across all worlds. We even thought we'd lost a Landroval blade - someone's gonna use this to try and argue we are or aren't virtualizing, we still are, but VMs still gotta live somewhere - but it went down temporarily and was successfully revived. We also re-rolled out a fix we'd introduced for 32 and then disabled for 32.0.2 because of issues related to the Spring Festival. It's gone really well and has further reduced some of the issues we're seeing.

We have in the works, and in various stages, several initiatives, some of which I've spoken about before:
64-bit servers are closer, but they have been just one can of worms after another. A lot of the game is built on assumptions about underlying architecture, so it's led to a lot of digging and reworking, but it's making progress. I don't have an ETA yet, but I've been told "Soon." This is a constant conversation I have with the folks involved, and I'll be getting more updates later today.
In part based on data we've been provided, we've sussed out previous issues, and are building better tooling to help us identify problems and their solutions faster. There are several things happening within here, but most of them won't have any immediate player-facing impact, since they'll just make it easier for us as we're solving the problems, not solve any itself. Nonetheless, this is important, and the next one should be ready as soon as this week.
Our deprecation of the 32-bit client is related to this work, as some of the improvements we're looking to make require 64-bit. Those next set of changes have been in testing now for a few weeks, and will hopefully get stabilized soon. Further improvements to the 64-bit client are coming, again based in part on player reports and data that have been provided, so thanks again for sending that stuff to us. Keep it coming
Later this week, we're also hoping to have several sets of new features ready for internal testing, that will help with some of the sources of lag around communication with the game databases. That's another avenue we're looking at to chip away at the problem.

There's a bunch of other stuff that's also down the line, but isn't as heavily focused atm, so don't want to speak about it too much yet. And this doesn't include changes we're making to our creation processes for new updates that will reduce the stress those areas place on clients and the game server, some of which we've seen benefit what we've released with the Angle and Yondershire, as well as future stuff to come.

We're still looking at this, we're still working on it, and we will continue to. A significant portion of the engineering effort that is being done and planned for the coming year is focused on continuing to improve the performance of the game for everyone. Thanks for your patience, I know it doesn't always come easy.

Appreciate the transparency. Know it's not easy to explain these things to a relatively novice audience. And "known unknowns" are just a fact of life in this type of work, though few people like to hear that (and fewer still accept what that implies).

[Shazzbot]

Lag continues to be a problem, and we continue to work on it. I understand it's frustrating that it continues to happen, but I can honestly say, I don't enjoy us having to constantly deal with this problem any more than y'all enjoy experiencing it.

Here's a quick update as to where we are now:
Overall, performance is in a better place than when Gundabad released, despite larger player concurrencies and overall player population. "Better" doesn't mean good, but it's nice to see that the months we spent working on various improvements over the past six months have born some fruit.
With Update 33, we saw a huge influx of players, the likes of which we haven't seen in at least 6 years. This has led to worsening performance across all worlds. We even thought we'd lost a Landroval blade - someone's gonna use this to try and argue we are or aren't virtualizing, we still are, but VMs still gotta live somewhere - but it went down temporarily and was successfully revived. We also re-rolled out a fix we'd introduced for 32 and then disabled for 32.0.2 because of issues related to the Spring Festival. It's gone really well and has further reduced some of the issues we're seeing.

We have in the works, and in various stages, several initiatives, some of which I've spoken about before:
64-bit servers are closer, but they have been just one can of worms after another. A lot of the game is built on assumptions about underlying architecture, so it's led to a lot of digging and reworking, but it's making progress. I don't have an ETA yet, but I've been told "Soon." This is a constant conversation I have with the folks involved, and I'll be getting more updates later today.
In part based on data we've been provided, we've sussed out previous issues, and are building better tooling to help us identify problems and their solutions faster. There are several things happening within here, but most of them won't have any immediate player-facing impact, since they'll just make it easier for us as we're solving the problems, not solve any itself. Nonetheless, this is important, and the next one should be ready as soon as this week.
Our deprecation of the 32-bit client is related to this work, as some of the improvements we're looking to make require 64-bit. Those next set of changes have been in testing now for a few weeks, and will hopefully get stabilized soon. Further improvements to the 64-bit client are coming, again based in part on player reports and data that have been provided, so thanks again for sending that stuff to us. Keep it coming
Later this week, we're also hoping to have several sets of new features ready for internal testing, that will help with some of the sources of lag around communication with the game databases. That's another avenue we're looking at to chip away at the problem.

There's a bunch of other stuff that's also down the line, but isn't as heavily focused atm, so don't want to speak about it too much yet. And this doesn't include changes we're making to our creation processes for new updates that will reduce the stress those areas place on clients and the game server, some of which we've seen benefit what we've released with the Angle and Yondershire, as well as future stuff to come.

We're still looking at this, we're still working on it, and we will continue to. A significant portion of the engineering effort that is being done and planned for the coming year is focused on continuing to improve the performance of the game for everyone. Thanks for your patience, I know it doesn't always come easy.

Awesome, thank you for the reply, sir. Meanwhile I'm playing on Anor because it's lag free right now and I have to feed my lotro obsession lol

[IcarusLSC]

I rather see improvements in lag then any new content by a mile...

[1Asy1]

An average day on Evernight.



I was also in the new raid last night at about 1 am doing a T1, it was literally unplayable - ~500 nonanon online.

./loc is in the video towards the end.

[Dave1892]

Is Evernight down again ?

Been stuck on the loading screen for ages after being stuck in a queue

scratch that, finally loaded

[Foxmaiden14]

Laurelin keeps getting afflicted by connection issues too. Got stuck in the loading screen twice today. So y'all are not a lone during this lag plague.

[DoRonRon]

Laurelin keeps getting afflicted by connection issues too. Got stuck in the loading screen twice today. So y'all are not a lone during this lag plague.

Maybe didn't get the reset that Evernight, Sirannon and Shadowfax got?

I got 4 of 5 chars that ended up naked on Laurelin char screen during that period. I can wait 'til Wednesday to sort gifts out. I'm about to see if I'll face a queue or not right now though.

They are building the tools to see the issues though, happy 15th.

[Dave1892]

Complete nightmare yet again due to massive amounts of lag

I really do not see the benefit of posting an alert about reduced awareness, frankly I think we are very much aware when it gets bad or really just worse than normal and they seem to be a semi-permanent thing at the minute (and for a long time past as well). I'm starting to think it would be better to post an alert when it isn't lagging but I think we will wait a long time to see that.

Getting very frustrated with this and the continued failure to address it despite being told it is a priority and changes have been made, I'd planned to finish Mordor Besieged this afternoon but had to come out before the pc ended up being smashed to pieces in frustration.

[Nothrandir]

16:45 UK time and can't get any of my chars to load on Evernight. Every one of them hangs and I have to alt+tab out, force LOTRO down, relaunch, and try again.

Tried all the usual things - logging different chars, logging into different worlds (chars load fine on Crick btw) - nothing works.

Have run into this issue before, but never on every single char (5 in total).

Evernight seems to be unhealthy.

On a related topic, it would be great to have a 'Quit' or 'Cancel' button that appeared while the char load is being attempted so you could dump out of the process if it hangs. Probably not possible, but it would be so much cleaner than having to force down the LOTRO client.

[Mezoti]

I've given up on playing Lotro for the time being. A friend and I want to play from about 17:00 UK time on weekdays and more on weekends but it has been unplayable after that time for both of us. We are in different parts of the UK and on different ISP's and are both IT literate and can see the difference between network lag and server lag. It has been bad for us since we started replaying the game in September last year and has got progressively worse. When the Anniversary event hit with the influx of extra players and the F2P bonuses drawing in more, we were unable to enjoy the game due to lack of movement and combat response, loading hangs, login queues etc. Another friend of ours based in Canada was experiencing the same issues at the same time we were so location and ISP aren't the sole factors and he was saying that he was seeing the same lag issues as us.

Therefore, 3 V.I.P subs have been discontinued as there is no benefit in paying for this service.
We want to play this game but are unable to do so because of these ongoing issues and until they are resolved, will not be handing any more money over to the company.

[DoRonRon]

I've given up on playing Lotro for the time being. A friend and I want to play from about 17:00 UK time on weekdays and more on weekends but it has been unplayable after that time for both of us. We are in different parts of the UK and on different ISP's and are both IT literate and can see the difference between network lag and server lag. It has been bad for us since we started replaying the game in September last year and has got progressively worse. When the Anniversary event hit with the influx of extra players and the F2P bonuses drawing in more, we were unable to enjoy the game due to lack of movement and combat response, loading hangs, login queues etc. Another friend of ours based in Canada was experiencing the same issues at the same time we were so location and ISP aren't the sole factors and he was saying that he was seeing the same lag issues as us.

Therefore, 3 V.I.P subs have been discontinued as there is no benefit in paying for this service.
We want to play this game but are unable to do so because of these ongoing issues and until they are resolved, will not be handing any more money over to the company.

Plenty of IT illiterate to pick up the slack and play out their fantasy "proving" their loyalty along the way. But how committed is anyone going to be "working hard" to improve performance when the elephant in the room has been staring at them for seven years. It's just shovelling poop. If you have that in mind while playing, at no cost to yourselves, it can be more barrable. Humour goes a long way; treat it as the joke it's become and be glad the mobs are equally lagged, else it would be unplayable (but that would be client side ofc).

[Emaldiom]

Lag continues to be a major problem for end game raiders who are trying to do t4\t5 raiding
There is a time gated title tied into clearing both t4 and t5 of new raid, but atm there is so much lag and annoyance that it's literally impossible to do the highest content.

You spend time getting what's supposed to be a good "server" and then everything just goes down the drain after 1-2 pulls on t5b2 (It's already insane bad even the first 1-2 pulls, but after this you get constant 5-30 second lag spikes where nothing goes off)
Is it the amount of adds \ particles \ puddles \ mechanics etc that is causing the instance server to just lag out completely ?

The longer you spend in there the worse it gets

I know you are saying you are on this and fixing it, but let's be real. It's already been years
It is only in recent time there has actually been some feedback, but all we see is empty words of "It's getting better", but it's really not.. It gets worse and worse

Never have i seen so many people complain about lag, so it seems like it effects the whole server not just instances anymore

[Arantha]

also on Belegaer, this morning, about 150 online, have to relogg three times because couldn't move anymore

[Emaldiom]

You said 20 days ago that you are working on it
You said 3 months ago that you were working on it
You said 2 years ago that your working on it

It's getting WORSE and WORSE and nothing is improving at all
Atm it is actually impossible to play the game at all in raids / instances

Are you actually looking at your servers at all? It should take you 10 minutes to see the lag on Evernight

[DiorEluchil]

You said 20 days ago that you are working on it
You said 3 months ago that you were working on it
You said 2 years ago that your working on it

It's getting WORSE and WORSE and nothing is improving at all
Atm it is actually impossible to play the game at all in raids / instances

Are you actually looking at your servers at all? It should take you 10 minutes to see the lag on Evernight

You still play, emaldiom? Good to see an old timer.

and yes, I dont even play on evernight but the lag is terrrible on every server so I can only assume its much worse on the bigger one haha

[Burzdolion]

No time to fix the lag, gotta work on the next £100 mini expansion or something.

[Emaldiom]

You still play, emaldiom? Good to see an old timer.

and yes, I dont even play on evernight but the lag is terrrible on every server so I can only assume its much worse on the bigger one haha

Yeah i still try to raid as we are trying to progress on t5 atm, but its impossible now
Even t3 was impossible today as we had constant 10\20 sec skill delay during Hrmiril fight (This lasted 20min over 2 runs)

Imagine t4\t5 being time gated and servers are like this..
And 0 feedback from devs outside of the "We are working on it", but NOTHING is happening

Even tho you say you are doing stuff under the hood we PLAYERS do not see ANY improvement at all
its worse than ever before and this had been said for ages already

[Polymachos]

I just downloaded a tool called WinMTR (My TraceRouter for Windows, free from Sourceforge) and ran it against the IP number 198.252.160.104 which is supposed to be Evernight.

|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| fritz.box - 0 | 201 | 201 | 0 | 1 | 5 | 1 |
|p3e9bf6e8.dip0.t-ipconnect.de - 0 | 201 | 201 | 8 | 11 | 94 | 11 |
|pd900c92a.dip0.t-ipconnect.de - 0 | 199 | 199 | 11 | 59 | 2155 | 13 |
| 193.158.39.70 - 98 | 42 | 1 | 0 | 18 | 18 | 18 |
| 4.69.214.37 - 15 | 127 | 108 | 108 | 108 | 123 | 108 |
| 4.31.43.210 - 0 | 201 | 201 | 111 | 113 | 127 | 112 |
...the rest got completely dropped...
|_____________________________ ___________________|______|___ ___|______|______|______|_____ _|
WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

It looks like the hop with the IP number 4.69.214.37 (a Level 3 hop) is losing packets, which the Lotro servers cannot cope with...

I have no idea how to resolve this.


Greetings, Polymachos