Offical Discussion : Service issues Sept 1- Sept 2

[Sapience]

First, as we didn't need 78 unique threads on the issue, most saying some version of "fix it", they've been removed and this thread will be used going forward.

Second I'll try to fill in what I can about what happened.

Right off the bat, no this was in no way related to Pando. I know some were trying to tie the two together but pando is unrelated to this issue and their announcement has had 0 impact on us in any way.

This issue is one we've seen before with a slight twist. This was again a routing issue with a very large internet backbone provider and how it routes traffic in and out of our data center provider (so you're talking 2 layers removed from our direct ability to 'fix it'). The issue also created a cascade issue when the routes were reset that didn't play well with some of the switches and routing hardware in the data center. This is why we've been working through the night with our vendors (provider, hardware, consultants, and others) to make sure each step of the connection from our service through various vendors services and products was checked, doubled checked, and confirmed to be either working correctly or needed to be addressed. We continue to do so, though we believe we are much closer to a resolution than at any point in the past several hours and we feel we are very close to restoring services.

As we've mentioned before, there are circumstances that can create a situation where the launcher and our forums are not accessible to us (myself and the rest of the community team). This is why we try very hard to convince you guys to follow us on twitter and like us on Facebook, because during times like this these are the only methods of commutation we have with you. Even if you can still post here. That is why this is the first post you're seeing from me on the forums, but I've made a large number on our social channels throughout the night.

Hopefully this tones down all the crazy conspiracy theories we've seen recently (pando, hacking, shut downs, contract issues. It's a pretty impressive list!). We appreciate those who tried to share back the information we were posting elsewhere.

We appreciate you've all been frustrated, we also appreciate that this is a major Holiday weekend in the US. Many of us have spent the bulk of it working around the clock to address these issues, so believe me when I say we fully understand and appreciate you had plans. Most importantly we appreciate the passion you all clearly displayed for our game.

[totallylegitaccount]

Thanks for the info bro. I hope we have a giant bag of loot in our mail boxes once we log in! A girl can dream!

[Amberjack]

Thank you very much for this post. The removal of other posts was looking a bit sinister before this popped up; I'm glad that's not the case.

[C-Hound]

(Deleted - forums were wonky, misread old thread as relevant to current issue - nm)

[Egilric]

First, as we didn't need 78 unique threads on the issue, most saying some version of "fix it", they've been removed and this thread will be used going forward.

Wouldn't it be far more logical to have this posted in Announcements?

So that a nice Red New Post alert is there next to where we should be getting all the important info on the game, the Announcements section.

All The Best

[TinDragon]

Sapience Posted...
Sapience's Avatar
Sapience
Sapience is offline Community Manager & Harbinger of Soon

Join Date
Aug 2008
Posts
7,496

As the issue the thread was created for has been resolved and we're now discussing something completely unrelated, it's time to close this down.

Ummm... it's not resolved. At least, not for this one user. Rebooted computer, rebooted router, still stuck at log-in screen w/ "Failed to read authentication response..."

Maybe one clear, clean post about exactly HOW to log back in, if in fact it has been resolved?

Thanks!

(Oh - and while I'm sure you're tired, and I KNOW people are lazy $#!^$ for posting OT in your official thread, maybe a bit hasty in closing that. Perhaps - just a thought.)

Since you didn't post what thread that's actually from, I can only guess that the thread was originally created for a different issue than the one that's preventing everyone from currently logging on.

[Ben-Wah]

Unofficial Lotro Downtime Faq

Q: What major outages have there been in the past month?

* Aug 3rd-5th (40 hours)
https://www.lotro.com/forums/showthr...empt-20-of-20)

* Aug 8th (7 hours)
https://www.lotro.com/forums/showthr...EASE-POST-HERE

* Aug 20th (20 hours)
https://www.lotro.com/forums/showthr...to-a-character

* Sept 1st (24+ hours)
https://www.lotro.com/forums/showthr...ke-the-servers
(deleted 38 page thread)


Q: Is it true that Lotro relocated their servers to Bangladesh as a cost saving measure, so WB execs can afford more cigars and better ski trips?
A: No, but thanks for the great idea it will be considered ASAP

Q: Where can i find out the latest statement from lotro on these service issues?
A: Historically been twitter, facebook, then forum last.

Q: Whose fault is it?
A: Lotro says it is out of their control and their service providers -- just general internet outages that have to be routed around.

Q: Hey wait, then why haven't my other internet applications had a similar amount of outages this month?
A: If you like your health care plan, you can keep your health care plan.
Hey look over there, chocolate!

[Susurrus]

Since you didn't post what thread that's actually from, I can only guess that the thread was originally created for a different issue than the one that's preventing everyone from currently logging on.

It's all the same issue, I believe. The errors are just changing around as they're continuing to work on the problem.

[Yazuli]

Quote ...Ummm... it's not resolved. At least, not for this one user. Rebooted computer, rebooted router, still stuck at log-in screen w/ "Failed to read authentication response..." End Quote

Not resolved for this user either (assuming there was some post or announcement that the "Failed to read authentication..." was supposedly solved).

[Gandalf116]

Thank you Sapience, this is the kind of communication that helps to quiet the howling mob. Please keep us informed.

[Gildoriel]

Sapience Posted...
Sapience's Avatar
Sapience
Sapience is offline Community Manager & Harbinger of Soon

Join Date
Aug 2008
Posts
7,496

As the issue the thread was created for has been resolved and we're now discussing something completely unrelated, it's time to close this down.

Ummm... it's not resolved. At least, not for this one user. Rebooted computer, rebooted router, still stuck at log-in screen w/ "Failed to read authentication response..."

Maybe one clear, clean post about exactly HOW to log back in, if in fact it has been resolved?

Thanks!

(Oh - and while I'm sure you're tired, and I KNOW people are lazy $#!^$ for posting OT in your official thread, maybe a bit hasty in closing that. Perhaps - just a thought.)

Er - Maybe you read another post by Sapience than I did, but in his post in this thread here he never says the issue is resolved. He clearly stated they continue to work on the problem.

[clark1980]

Thank you very much for the update. I can't speak for everyone but I greatly appreciate the information. I await the restoration patiently.

[ContraMundum]

As we've mentioned before, there are circumstances that can create a situation where the launcher and our forums are not accessible to us (myself and the rest of the community team)...

It was like a prison riot. Ah good times, good times.

[RBrown]

So is this issue the reason that I could not redeem my points card that I purchased?

[robbie1435]

First, as we didn't need 78 unique threads on the issue, most saying some version of "fix it", they've been removed and this thread will be used going forward.

Second I'll try to fill in what I can about what happened.

Right off the bat, no this was in no way related to Pando. I know some were trying to tie the two together but pando is unrelated to this issue and their announcement has had 0 impact on us in any way.

This issue is one we've seen before with a slight twist. This was again a routing issue with a very large internet backbone provider and how it routes traffic in and out of our data center provider (so you're talking 2 layers removed from our direct ability to 'fix it'). The issue also created a cascade issue when the routes were reset that didn't play well with some of the switches and routing hardware in the data center. This is why we've been working through the night with our vendors (provider, hardware, consultants, and others) to make sure each step of the connection from our service through various vendors services and products was checked, doubled checked, and confirmed to be either working correctly or needed to be addressed. We continue to do so, though we believe we are much closer to a resolution than at any point in the past several hours and we feel we are very close to restoring services.

As we've mentioned before, there are circumstances that can create a situation where the launcher and our forums are not accessible to us (myself and the rest of the community team). This is why we try very hard to convince you guys to follow us on twitter and like us on Facebook, because during times like this these are the only methods of commutation we have with you. Even if you can still post here. That is why this is the first post you're seeing from me on the forums, but I've made a large number on our social channels throughout the night.

Hopefully this tones down all the crazy conspiracy theories we've seen recently (pando, hacking, shut downs, contract issues. It's a pretty impressive list!). We appreciate those who tried to share back the information we were posting elsewhere.

We appreciate you've all been frustrated, we also appreciate that this is a major Holiday weekend in the US. Many of us have spent the bulk of it working around the clock to address these issues, so believe me when I say we fully understand and appreciate you had plans. Most importantly we appreciate the passion you all clearly displayed for our game.

Thanks Sapience, I'll be sure to ship you some awesome goodies for your hard work! (How do you feel about receiving some more coffee =D)

[Crell_1]

Could you please switch the authentication servers back on so those of us who finally have functioning routes can enjoy the game?

[Helnoth]

As much as I might be annoyed that the game is unavailable to play my inconvenience pales compared to being called in to help resolve this issue through the weekend. I hope at least some plans are salvageable by the time the issue is resolved.

[ferdinanda]

Thank you for the updates, Sapience, what a way for you to spend the holiday weekend, ugh!

I found this site that tracks outages, http://www.outageanalyzer.com/ and there's one listed that's almost 22 hours old, pd.ed10.net. I wonder if the problem is something like this.

[Lanari]

Er - Maybe you read another post by Sapience than I did, but in his post in this thread here he never says the issue is resolved. He clearly stated they continue to work on the problem.

i imagine sapience went back and reworded his post? and we are seeing the fixed version unlike those who saw it first?

[aircorian]

thx sapience, lovely sence of humor u have

just a wild gues (probertly a no bhut heck) any eta ?

[Ploo]

Well, it seems no matter what I do, I can't seem to even get my launcher to show up on screen.

[TheVasar]

I suggest everyone to start checking their Twitter at times like this, I've found this always fastest way to get any information and they did answer a lot of people there.

[Garamell]

Thanks for the information!

[Jelerian]

So, I am curious as to how many people replying on this thread have any experience with large data center routing? I asked my brother (who does) about the issues last night and his response sounded very similar to what Sapience just posted. I just called him and read this to him and he told me sounds pretty legit.

Sapience, cheers to working through what was probably supposed to be a holiday weekend for you (if not you some other person who is probably still sitting there after roughly 40 hours staring at something that they may or may not actually have the power to fix). I appreciate you keeping us up to date on Facebook and Twitter. I haven't been sharing or retweeting but I will now hoping that it helps you guys out.

Have a beer on me.

[Coesite]

Sorry you lost your holiday, we at least had other options available to us. Thanks for the long hours spent.