Offical Discussion : Service issues Sept 1- Sept 2

[Wolflord75]

Cheers for the fix! In the downtime I made like a Hobbit and...had a nap. It was great! Thanks again for your hard work and for putting up with "some" of the people here. It's (insert your favourite beverage here) time!!!

[IAmWillowing]

Thanks for the info, Sapience, and for keeping us updated through Twitter and Facebook. I appreciate all the hard work you guys put into getting us back up again! Woohoo!!

[ragesangel]

Thanks for the update
Relax, take your time and get all the pieces duct taped back together as best you can. Life is all good , plenty of Other things to do during the down time

[JuicedMoose]

"soon" ahh

[louisdee]

Thank you for the post.

[Starina]

It is great to see in these posts how many actually appreciate the hard work that must be going on behind the scenes right now. I can only add my thank-yous here to theirs, and sorries for ruined holiday plans of those who try get the servers back on.
But among all the complaing and panicking and crying.. I guess the good thing that's clear is that Lotro is still majorly loved and definitely missed the moment it suffers an outage ^___^

[Anniah]

Thanks!

[QueenieZee]

Hi, I'm new to all of this. I just wanted to say thanks for the good hard work all of you did to get us back in our game today. I was worried because I couldn't find anything telling me why I couldn't log-in. So, I thought it was my fault and not your servers until I clicked the "support" link on the log-in page. I copied the error message and pasted it in the search engine of your support sight. It gave me the link I needed to bring me right here where I needed to be. Good job! Thanks again and have a great day of what's left of it! XO

[NeoBasilisk]

The servers are back up, in case anyone's interested.

[ziggy29]

Thanks for the hard work you all put into this. I know it sucked a lot of a holiday weekend away. I hope there's a postmortem with "lessons learned" to reduce the likelihood of this again wherever the culprit was located.

[camillionaire]

Thanks for the update Sapience! I can't wait to get back in game, thank you for working on the weekend for us!

[Ceremony]

At least the game is back up, it's not like we wait for a Sapience's Soon

[Bragoin]

I love the game and I hope you get this issue properly resolved. I know and appreciate you're doing the best you can under the circumstances. I just hope my VIP time is credited back to my account. No complains, just fairness. Thanks and good luck.

[ShireDweller]

Thanks to everyone who was involved in restoring the game servers.

[PhoenixCWS12345]

So, I am curious as to how many people replying on this thread have any experience with large data center routing? I asked my brother (who does) about the issues last night and his response sounded very similar to what Sapience just posted. I just called him and read this to him and he told me sounds pretty legit.

Sapience, cheers to working through what was probably supposed to be a holiday weekend for you (if not you some other person who is probably still sitting there after roughly 40 hours staring at something that they may or may not actually have the power to fix). I appreciate you keeping us up to date on Facebook and Twitter. I haven't been sharing or retweeting but I will now hoping that it helps you guys out.

Have a beer on me.

Completely believable that this could happen to a data center. Extremely odd that it should happen many times for extended periods in the same quarter. A simple architecture design choice that keeps your service uptime looking good despite craapy data center issues is to not put all your servers in the same physical location. So while the routing issue is definitely not lotro/turbine's fault, they're clearly not running with a very fault tolerant architecture.

Theres a ton of ways they can fix this if they don't want to carry the risk going forward... especially since this particular risk has materialized multiple times in a short period of time... at least for their official forums if not their login servers (where they could get the outage info to the customers using the same means the customer uses to communicate to the down service) if its not financially possible to split the game servers.

So its not their fault, but its within their power to handle it a lot better... kinda like if you've been dependent on a buddy to pick you up from the airport 4 times in the last two months and hes always hours late because his cell didnt ring... well hes in craapy coverage, didnt check the emails you sent him and didnt go online to check to see if the flight was in, but when you get down to it, hes right its not his fault that the call didnt go through, hes paying for the service and should have received it on time and gotten to the airport...

[britian777]

Great work!! Now time to play!!!!

[jjaxew]

Thank you, Sapience, and you all, guys, for your hard work.

[beastybunny]

Big thank you to Sapience and all the Turbine staff who had to give up part of their holiday weekend to help get the game backup and running. Also appreciate all the status updates on Twitter and Facebook. If you guys go to PAX East next year, I will buy you a drink!

[Wildwind]

Just wanted to say thanks to all the devs and techs that spent their holiday weekend fixing this issue. Just know that your hard work is greatly appreciated!

./hatsofftoya!

[DutchEZmoder]

Will there be any sort of compensation for the downtime? Like all accounts +bounder tokens or something?

[bigsix66]

I think it would be nice at some point to see Turbine actually say, "We're sorry."

Love means never having to say you're sorry.

[jorlan]

My sincere thanks to all at Turbine and their vendors for working diligently thru THEIR holiday as well to get us back into Middlearth.

Jorlan

[Sapience]

Sapience,

I almost never post, but I feel the situation warrants a response from Turbine's side. There is one particular thing I find worrying, and sadly, it has been the trend with many companies these days. Every passing day, more and more companies seem to heavily rely on Facebook and Twitter for support, announcements and regular message-sharing with their community, which I think is completely flawed. Facebook no longer allows you to freely browse company Facebook profiles anymore (Don't believe me? try looking at a corporate or business profile without logging in, it's restricted now, although not fully). Twitter doesn't have this issue. The thing is, I don't have any of these accounts and we shouldn't have to open one to get information about the current state of the game.

Your community, first and foremost, lies within these forums, with social networks being a secondary means of communication. So why are these forums ignored? As community manager I don't think this is what you want for the community you manage. I'm sorely disappointed about this whole ordeal.

Now, to address the bit about you not being able to reach these forums from within Turbine's network: I know a bit about networking, and I believe you 100% that you or anyone else within Turbine's network couldn't log into the forums to post because of the network issue. I believe it, but in today's day an age, where every device out there has an Internet connection and there are so many ways to access the net, why wouldn't you take advantage of this? Maybe there was an employee at home at that time that could've log into the forums and make an announcement, maybe a forum moderator that works from home, heck, even anyone (yourself included) within Turbine's offices with a mobile phone with a data plan logging in directly from his phone, or using it as a mobile hotspot, tethered connection, anything that could use their data plan to log into the game's official forums and post an update. Someone could've even gone to the Starbucks you get your coffee from and post there, but no. I don't think we as a community deserve this, and I sure won't open an account with Twitter or Facebook just to get LOTRO updates.

Just my $0.02. I appreciate your hard work and hope to get back in-game soon.

I have been at home the entire time (which has taught me that I really need a new desk chair). Employees, for a number of reasons, do not access things the same way players do. I appreciate your comments and thank you for your feedback, but your suggestions would have had no impact during this issue.

As for opening an account on Twitter or Facebook, you don't have to. Simply bookmark these two pages and read them.
http://www.twitter.com/lotro
http://www.facebook.com/lotro

No account is required, no sign in needed. You can read every word we post and you really only need to pay attention to one or the other as we mirror major announcements and updates across both services. Actually, I'd recommend Twitter as you can just read our page and you see only our tweets (no comments, misquotes, or random info) in the order they went out with a relative time stamp (2h ago, etc.).

[SatinKnight]

I was just able to log in

[Ellyllon]

Thank you for the update. Also, for relieving me of the innate paranoia that terrorist-hackers were after my gaming time.