Evernight is unplayable due to lag... again

[DiorEluchil]

Yeah i still try to raid as we are trying to progress on t5 atm, but its impossible now
Even t3 was impossible today as we had constant 10\20 sec skill delay during Hrmiril fight (This lasted 20min over 2 runs)

Imagine t4\t5 being time gated and servers are like this..
And 0 feedback from devs outside of the "We are working on it", but NOTHING is happening

Even tho you say you are doing stuff under the hood we PLAYERS do not see ANY improvement at all
its worse than ever before and this had been said for ages already

Its like they dont want more players tbh...

[Nubja]

its really cool watching the last add wave of hrimil spawn 1 mob at a time

@raninia time for another update mate. Servers are utter dog. While it's clear your direction as producer is to deprioritise group content (helms deeps 2 here we goooooooooo) even festivals are unplayable. Anniversary was so cool when i have to wait 20 seconds to start picking up an item.

[Nakah]

SSG would be really stupid if they didn't put huge resources into fixing the performance issue before the new Amazon shows premiers. They've already modernized their F2P experience, now they need to fix the performance...They can't miss this opportunity for an insane growth that this new billion dollar series will give them.

As a VIP who started playing LOTRO from day one, I recall that this matter has often been raised about taking opportunities for LOTRO when new cinema or such releases are made in the LOTR franchise. They are missed opportunities, but I think it is more to do with licensing and legalities rather than an unwillingness to do so from those who develop this game. A number of fingers are in the LOTR pie now and and it is not as simple as just developing new content, but I do agree it is a great opportunity to promote LOTRO and legalities aside let's hope this happens.

[maartena]

It looks like the hop with the IP number 4.69.214.37 (a Level 3 hop) is losing packets, which the Lotro servers cannot cope with...

I have no idea how to resolve this.

No one is going to resolve this.

Losing packets on a hop between two servers actually means absolutely nothing. If it is a major traffic hub, it is common practice to place ICMP traffic on a very low priority, which means that traceroutes and pings to the host are often showing packet loss, where in reality that hop is working perfectly fine and passing ALL traffic to the next hop.

The reason this is done is to "harden" these traffic hubs against DDoS type attacks, which typically use a flood of ICMP traffic to try and bring down a host by simply overflooding it.

This is why traceroutes are notoriously unreliable in diagnosing network problems. It will show you what route the packets are taking, but packet loss on any hop between A and B does not mean that there is a problem, and in 99% of cases it is actually a deliberate priority lowering for ICMP traffic.

What matters is whether you packets get from A to B. Now, many game servers disable ICMP traffic on their border router as well, and for the same reason: They do not want a disgruntled player doing DDoS attacks on the servers. We've actually had that issue a few years ago, so they shut down ICMP traffic, and that is the reason you can't ping gls.lotro.com anymore. (gls = global logon server)

No IP address in the entire range of 198.252.160.0/24 will accept any ICMP traffic, so pings and traceroutes as tools to determine network issues..... are pretty much completely useless.

I do not believe that the lag is caused by internet/routing issues. If there are issues, they might be with whatever peering agreements the local provider has in place with Level3 (now Lumen) and whether those peers are overloaded. But that would be a discussion between the player and their ISP. Instead, I believe that the lag is caused by the servers themselves. We are talking 32-bit servers here, so they can only use 4 GB of ram to host the worlds. Obviously there are many instances in the server cluster, so its not like Evernight is running on ONE server or ONE VM, there are likely dozens each doing their part.

But at the end of the day:

- The servers are old.
- Each server in the cluster runs a 32-bit server so is limited to 4 GB.
- They are running code from 2007 through 2022, and that gets messy.
- Each time a new zone is released, more lag happens.
- Each time more players log on, more lag happens.
- Sometimes new features cause lag.

So SSG has a lot of work to do, but quite frankly...... they should have done that work already. Their primary focus always is MORE CONTENT, must roll out more content. And it is true that as a whole, players start to get bored and leave the game after about 3-4 months of no new content, so they have to make sure they always have content for the players..... and as such the project to solve the lag gets put aside. I know SSG has limited resources, and limited amounts of money, but I think it is time to ask EG7 for some help, get some temporary resources hired, and at least try to knock out the lag for the most part over the next three months.

Yes, once again, I am asking SSG to put new content on hold for a while, and make Update 34 the "Let's fix some old bugs and tackle the lag" update. What is the point of stuffing more content on servers if they currently can't handle the players?

[egarthur]

No one is going to resolve this.

Losing packets on a hop between two servers actually means absolutely nothing. If it is a major traffic hub, it is common practice to place ICMP traffic on a very low priority, which means that traceroutes and pings to the host are often showing packet loss, where in reality that hop is working perfectly fine and passing ALL traffic to the next hop.

The reason this is done is to "harden" these traffic hubs against DDoS type attacks, which typically use a flood of ICMP traffic to try and bring down a host by simply overflooding it.

This is why traceroutes are notoriously unreliable in diagnosing network problems. It will show you what route the packets are taking, but packet loss on any hop between A and B does not mean that there is a problem, and in 99% of cases it is actually a deliberate priority lowering for ICMP traffic.

What matters is whether you packets get from A to B. Now, many game servers disable ICMP traffic on their border router as well, and for the same reason: They do not want a disgruntled player doing DDoS attacks on the servers. We've actually had that issue a few years ago, so they shut down ICMP traffic, and that is the reason you can't ping gls.lotro.com anymore. (gls = global logon server)

No IP address in the entire range of 198.252.160.0/24 will accept any ICMP traffic, so pings and traceroutes as tools to determine network issues..... are pretty much completely useless.

I do not believe that the lag is caused by internet/routing issues. If there are issues, they might be with whatever peering agreements the local provider has in place with Level3 (now Lumen) and whether those peers are overloaded. But that would be a discussion between the player and their ISP

Lies, damned lies, and tracert results :-)

But seriously, some variations of traceroute use UDP rather than ICMP, but that -- or the "TTL exceeded" ICMP responses -- is often blocked too. Every tracert/traceroute result needs to be interpreted with a grain of salt, preferably by a networking professional.

I'm not even sure that there is a static mapping between servers, e.g. Evernight, and public-facing IP addresses. There's most likely a load-balancer in front, possibly the servers are arranged in a cluster, etc. The "diagnosis" of "I tracert'ed this IP, which I assume to be Evernight, and it stopped at XX.XX.XX.XX, therefore that must be where the problem is!" makes all sorts of assumptions about infrastructure, and frankly, is superficial/amateurish.

[Polymachos]

Just for clarification: 4.69.214.37 is the next-to-last hop, not the last one.

I said this 'supposed to be Evernight' because I am aware that IP numbers may be swapped around, and also do not represent a monolithic server, as the landscape is getting provided on the fly, maybe even by virtual servers, piece by piece, and the move from one to the next can be felt each time the handover doesn't sychronize well (rubber bands, stalls, and the like).

Taking this into accound, it is clear that there has to be not only a load balancer, but something that provides said landscape pieces, and distributes them across the servers as they are available, and this something is of course put up between the players and the actual gaming services. There is no reason to show these landscape servers in the public internet; it would get very convoluted within minutes, with the pieces being loaded and unloaded, and loaded for the next player maybe on a totally different virtual server.

Which still leaves some issues.

a. If some packets get dropped (in the long run, it is about 5%) they have to be sent again. To a realtime game. How much does this affect the communication?
b. It is 4.69.214.37 which drops the packets. Your description makes it sound as if it were putting us on the backburner. 4.31.43.210 otoh just forwards them. Does it rely on the packet drops that 4.69.214.37 already did, or why does it behave so differently?
c. Is net neutrality still working?
d. why does the game stall even in mid fight? I haven't seen this before 2022, and I have been playing Lotro for ten years.


Curious, amateurish Polymachos

[DoRonRon]

I don't know if virtualisation was necessarily good or bad for Lotro and DDO.

It could be that in just accepting they had done enough to have it working was all they set out to do. But it hasn't delivered from a customer perspective. Those involved in the change over are surplus to requirements, at that point.

You could argue we suffer from arbitrary cessation constantly as the budget is wasted on unreasoned timeframes. Get into any debt cycle and you just go deeper. Performance debt like any other.

[Shazzbot]

Back in the day when I used to work for netgames.uk (counterstrike server admin) I used to email the router IP (you can sometimes find out the email address for the particular ip router using a traceroute program) and I would tell them myself their server was showing high latency.
Failing that, email your ISP and ask them to route you a different way or to notify the IP there is a problem. Hopefully SSG can look into it and do this also.

I just downloaded a tool called WinMTR (My TraceRouter for Windows, free from Sourceforge) and ran it against the IP number 198.252.160.104 which is supposed to be Evernight.

|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| fritz.box - 0 | 201 | 201 | 0 | 1 | 5 | 1 |
|p3e9bf6e8.dip0.t-ipconnect.de - 0 | 201 | 201 | 8 | 11 | 94 | 11 |
|pd900c92a.dip0.t-ipconnect.de - 0 | 199 | 199 | 11 | 59 | 2155 | 13 |
| 193.158.39.70 - 98 | 42 | 1 | 0 | 18 | 18 | 18 |
| 4.69.214.37 - 15 | 127 | 108 | 108 | 108 | 123 | 108 |
| 4.31.43.210 - 0 | 201 | 201 | 111 | 113 | 127 | 112 |
...the rest got completely dropped...
|_____________________________ ___________________|______|___ ___|______|______|______|_____ _|
WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

It looks like the hop with the IP number 4.69.214.37 (a Level 3 hop) is losing packets, which the Lotro servers cannot cope with...

I have no idea how to resolve this.


Greetings, Polymachos

[Burzdolion]

Games a ####ing joke at the moment

[Dave1892]

currently in Elderslade and the lag I'm experiencing is industrial strength, absolutely unplayable, constant freezes and unbelieveable waits for everything.

SSG would you please actually do something to address this

[Once_of_Bree]

Evernight back to being unplayable.

Got the loading error, you know where game hangs on load, have to restart and on restart character naked.
Disconnected from chat server
Reduced aware ness
Horrible unplayable lag
Handing in a quest takes minutes.

etc. etc.

[Brinswith]

It has been terrible for the last few hours. Steepset is grinding to a halt with it taking 3 minutes to reach the stable master from the doors to the hall. Also very difficult to log out. Had to force quit the game as I tried logging out twice with no luck.

[DoRonRon]

It has been terrible for the last few hours. Steepset is grinding to a halt with it taking 3 minutes to reach the stable master from the doors to the hall. Also very difficult to log out. Had to force quit the game as I tried logging out twice with no luck.

You could join this Friday's stream and ask what specifically is different to the way this space has been created? Better that the architects be held to account. That's not likely to get an acknowledgement. The theme of these streams is to show an unpopulated Middle-earth after all. Yeah I know it's for the coming content.

A prime time Evernight stream would be a hoot though.

[kerbelettos]

again its dia show ...well lets hope they are working on it ..like its been last 14 years ..what i hear ...