lag lag lag and now servers are all down

[Sherush]

servers are all down no one can log in

[Dinara]

Yup, I try to restart the game and it does some kind of teleporting check but the game never starts.
I was on Crick.

[Sherush]

yup all servers are down

[Elmagor]

All servers crashed

[Dinara]

I wonder how long it will take to get a response from SSG...

[DroboTheDwarf]

"The Hamsters have died due to misadventure."

[Elmagor]

We able to get back, but all instances start from fresh

[Tralfazz]

This domain, lotro.com, was also briefly unreachable. So it's unlikely the servers crashed, most likely the server host suffered a failure which affected everything hosted at that site. Could have been a connection interruption, less likely but the symptoms fit a power failure, too.

Everything seems to be back up now, less than 20 minutes later.

[Cordovan]

The game worlds went down briefly a bit ago, but everything is back up now, and people seem to be able to get back in without issue.

[maartena]

This domain, lotro.com, was also briefly unreachable. So it's unlikely the servers crashed, most likely the server host suffered a failure which affected everything hosted at that site. Could have been a connection interruption, less likely but the symptoms fit a power failure, too.

Everything seems to be back up now, less than 20 minutes later.

I'm going to guess it was a DNS change, and the TTL (Time To Live) for DNS records varies. Usually worldwide a change is seen by ISP DNS servers within about 15 minutes - sometimes as quick as 5 minutes, but if you have a poopy, slow as molasses internet provider, it may take an hour or longer.

[YamydeAragon]

I blame the weather for that short interrupt. Lets not forget that right now a huge winterstorm covering all USA. https://weather.com/forecast/nationa...nal-forecast-0

[maartena]

I blame the weather for that short interrupt. Lets not forget that right now a huge winterstorm covering all USA. https://weather.com/forecast/nationa...nal-forecast-0

Could very well be indeed! Wouldn't be the first time bad weather causes an ISP oopsie. And even if you have a backup provider, it could take a solid 5 minutes before the backup really takes over.

[Polymachos]

I'm going to guess it was a DNS change, and the TTL (Time To Live) for DNS records varies.

Why should the Lotro servers suddenly change their IP addresses? It isn't even Wednesday. If anything went wrong DNS-wise, this would have happened on a different level, be it the provider, or even an internet router. There have been odd events when a country wanted to remove a certain internet service from their territory, by announcing a spoofed DNS entry, which got broadcasted not only within said country, but world wide, causing a global downtime. But I cannot see what would make any institution announce a fake DNS entry, that would affect Lotro.

There is still the possibility that a(n internal) DNS service got broken because of some hardware failure, or a router ran amuck.


Greetings, Polymachos

[Cordovan]

This was related to some behind the scenes maintenance that was not intended to impact the live game worlds, but accidentally did. The issue was quickly spotted and corrected, and we apologize for the issue it caused.

[auximenes]

This was related to some behind the scenes maintenance that was not intended to impact the live game worlds, but accidentally did. The issue was quickly spotted and corrected, and we apologize for the issue it caused.

Thanks for the rapid response and notification.

[Flaxie]

This was related to some behind the scenes maintenance that was not intended to impact the live game worlds, but accidentally did. The issue was quickly spotted and corrected, and we apologize for the issue it caused.

Thank you for the explanation. I noticed that the lag seemed better as well, hope that was intended and will continue! :P

[Elmagor]

This was related to some behind the scenes maintenance that was not intended to impact the live game worlds, but accidentally did. The issue was quickly spotted and corrected, and we apologize for the issue it caused.

In other words: cleaning service woman vs lotro server room - 10:0.

I'm happy what this issue was so quickly spotted and fixed, great job!

[Malcoriant]

In other words: cleaning service woman vs lotro server room - 10:0.

I'm happy what this issue was so quickly spotted and fixed, great job!

"No. No. I clean now."

[egarthur]

I'm going to guess it was a DNS change, and the TTL (Time To Live) for DNS records varies. Usually worldwide a change is seen by ISP DNS servers within about 15 minutes - sometimes as quick as 5 minutes, but if you have a poopy, slow as molasses internet provider, it may take an hour or longer.

As someone who was the manager for DNS for a very large company, for decades, I really wish folks would be more precise in their problem descriptions.

This domain, lotro.com, was also briefly unreachable

Could mean "lotro.com" was unresolvable, which would definitely point towards DNS. But it could also mean "I can't reach any of the servers with names in the lotro.com domain" (e.g. gls.lotro.com, forums.lotro.com and so on), which is mostly likely not a DNS problem. Even if the lotro.com domain is unresolvable, that could be symptomatic of an underlying network problem that is not specific to DNS.

So, feel free, to speculate about TTLs and such. Feel free to clear your resolver caches, reboot your clients, reboot your routers. But it may not have been a DNS problem in the first place.

Then again, I've been humbled a few times, in my career, by this meme:


EDIT: Another piece of info I'll point out, is that DNS rarely plays a role in an established connection. When the servers went down, I was running SV with some friends, on (non-LOTRO) voice chat, and everyone froze at the same time. Our connections broke. DNS lookups are made when starting a connection. The chance of DNS being involved when an existing connection breaks, is minimal. And that's exactly what happened here.

[maartena]

EDIT: Another piece of info I'll point out, is that DNS rarely plays a role in an established connection. When the servers went down, I was running SV with some friends, on (non-LOTRO) voice chat, and everyone froze at the same time. Our connections broke. DNS lookups are made when starting a connection. The chance of DNS being involved when an existing connection breaks, is minimal. And that's exactly what happened here.

Yep! This made me think someone made a back-end change to a switch or firewall that had a wee bit of a unexpected effect...... And was quickly reverted.

And yes, me too have been burned by that DNS meme!

[LabadalofDorlomin]

can't see an update about why it is still down.

ANY NEWS?

[Dinara]

can't see an update about why it is still down.

ANY NEWS?

Downtime has been extended to 2:00 PM, so that's an extra hour.