Was there a big outage or was it just me who couldn't connect?

[Jewl_of_the_lake]

I captured a video when the crash was happening, it looked like normal freezes for about 2-3 minutes. ~15 minutes later some people reported in glff the store was down, 5 minutes later everything went down.

Some people got kicked out of the server, I didn't and played normally.

Imladris last count was 14 in glff, 23 global

I was in a ToO and we wiped from the freeze but the game was still running until i logged out

[Iktomi]

Some one might think about calling the community manager at home. The phone list is electronic and its on a dead server. Or all they can find is the office number which does not help much.

At which point I'd say they either need to implement, or seriosly update, a/their disaster response plan that includes local hard copies of the relevent important contacts. That was something we updated and verified quarterly with our office, home, company cell, and personal cell contact numbers.

Most of the time the folks at the tip of the spear are so busy trying to fix the problem. They have no capacity or interest in distributing news.

The should at least put some effort into contacting someone upstream who can make those other decisions.

[maxjenius]

Wow, more customers slobbering all over themselves thanking a vendor for doing its job. Amazing.

[Fortinobrand]

How big would the chance be that during an outage in the middle of the night, Boston time Turbine's community reps would be sitting at their offices?

In this day and age companies that offer a 24/7 product/service have the tools to do just this, without having to actually be at the office. Given the ease of such notifications it really has become an expectation of the customer base, and failing to understand that is yet one more bit of evidence that Turbine is very out of touch with it's customer base.

For many parts of the world, this outage did not happen in the middle of the night -- it was in the afternoon, or the evening. If you're willing to accept money from those customers, you have an obligation to them.

[Frisco]

Not really. If they'd just said "we're aware of a problem with our servers" or something similar, I wouldn't have started smacking my computer with a hammer and blaming my stupid isp again.

Here is a simple solution: try to go to any one of the millions of sites on the interwebs. If Google.com is down, it may be your ISP. If it works, assume it's Turbine servers.

[Apparently]

On behalf of anyone who has ever had to carry a pager and be on-call to be awakened in the middle of the night to deal with some frantic emergency...

on behalf of anyone who has ever had to work in a customer service profession and deal with abuse from customers over a situation for which you were not responsible, given no information about and were powerless to prevent or solve...

on behalf of anyone who has ever been in the deeply unenviable position of having to be the one to wake up a Big Boss to be the bearer of bad news...

I would like to thank those within Turbine who may have found themselves in these positions over the last several hours, and apologize for the attitudes and entitlement of those who have never stood in your shoes, and feel no need to empathize with your pain. And I hope your day gets much better from here.

That is all.

[BrumrocAlebuck]

Of course the outage happened at the exact moment I attacked a mob.

Of course I also ended up dying. So much for getting the Undying title on my new character. Time to re-roll... Hours of my life I'll never get back.

[Sapience]

We're still investigating the root cause for the outage, but we do know that this was a sort of 'pefect storm' that impacted both primary and secondary sytems.

There are a lot of assumptions being made in this thread and honestly most of them are wrong. The nature of last night's issues are entirely based in hardware and connectivity failures. The way they unfolded caused a cascade of issues that lead to communication issues both internal and external (with our International GM and CS teams).

For the record, the entire community team is on call 24/7. Community Managers like myself and Tolero (DDO) are on call 24/7/365 including vacations and Holidays with very rare exceptions. No one ever hesitates to call us at 2am if need be. It's the job and we're happy to do it.

That said, the nature of last nights issues was such, as noted above, that communications were impacted.

We'll continue to investigate the root cause. We'll address the issues and failures we discover, and will adjust policies where we discover they are needed as a result of those investigations.

We'll share what information we can when we can and if we can. It's probably worth noting that investigations like this usually are not done in a day, but over several days.

[valderk]

thx for working hard to keep our gaming up and runing. us late night gamers dont always think of the poor guy in IT that has to get out of bed at O dark thirty when our games not working but thx for the hard work keeping LOTRO up and runing

[Crell_1]

We're still investigating the root cause for the outage, but we do know that this was a sort of 'pefect storm' that impacted both primary and secondary sytems..... The way they unfolded caused a cascade of issues that lead to communication issues both internal and external (with our International GM and CS teams).

Obviously, I dont know how your internal communication systems are set up. That said, I did get some reports of hitching this morning and what looked like a drop in incoming traffic shortly after the festival went live on #riddermark (recieve data dropped by a factor of 10, but without packet loss) . Does the team anticipate any lingering issue that may require a full shutdown and restart of the gameworld infrastructure?

[Fredelas]

We're still investigating the root cause for the outage, but we do know that this was a sort of 'pefect storm' that impacted both primary and secondary sytems.

Turbine also experienced a "perfect storm" 18 months ago (on my birthday), although that time it was the power systems that were to blame. Today's outage was corrected more than twice as quickly. Even though the communication was missing today, I think this is still an improvement.

Nine months ago, the forum database was compromised. After figuring out what had happened, Turbine eventually did a good job of informing its all its customers.

If this pattern holds, we should expect the next perfect storm about 9 months from now. Does anyone want to make any bets on which systems will be involved next time? We've already done power systems, databases, and apparently networking hardware.

I'm guessing that a catastrophic pie failure in the office might render the Shire unplayable. Fat hobbits will become skinny, and skinny hobbits will become short elves.

[Erebthoron]

I really think that Sapience is right. I see it in some other companies that the chain of communication break down especially if something serious happen. Just someone miss a call and all the other people think its done and they now can fix the problem. Its never nice when it happen but calm down, it is a game. And yes I'm also one of those who pay and could not play in Europe for 5 hours but there is a thing called life

[Saniko]

This was horrendous as my game wasnt working and I thought I had screwed something up and was working for hours trying to see what was wrong. I ended up reinstalling the game. No forums were up and I went to twitter and facebook to look there and no sign of there being a major outage just made things worse.

I think you should reverse your troubleshooting process. :P There wasn't anything official about the outage on https://twitter.com/lotro when I checked ~10 hours ago, but I searched for "#lotro" and Twitter came up with several other tweets about not being able to log on. Twitter is now down.

My family was at home and it was very scarey for a good few hours. My thoughts are that we should hope all the people caught up in the storm are safe. Sure it is too bad the the game went down but I think we should remember that we have no control over the weather and it is not Turbine's fault that the weather made the lights go out and Turbine could not report it to us because of this.

This.

http://www.boston.com/news/local/mas...om_mass_storm/
In a situation like that, I'd be more concerned about people's safety and attending to critical systems first ("Crews were working to restore electricity to hospitals..."), NOT a bunch of game servers, regardless of the company's size.

Thank you for the update, Sapience! Drive safely!

[nubcat]

Well, now Twitter are experiencing issues, and Pottermore was down all morning. Maybe it's all related!

[Nickysmom]

Turbine also experienced a "perfect storm" 18 months ago (on my birthday), although that time it was the power systems that were to blame. Today's outage was corrected more than twice as quickly. Even though the communication was missing today, I think this is still an improvement.

Nine months ago, the forum database was compromised. After figuring out what had happened, Turbine eventually did a good job of informing its all its customers.

If this pattern holds, we should expect the next perfect storm about 9 months from now. Does anyone want to make any bets on which systems will be involved next time? We've already done power systems, databases, and apparently networking hardware.

I'm guessing that a catastrophic pie failure in the office might render the Shire unplayable. Fat hobbits will become skinny, and skinny hobbits will become short elves.

Oh please, no, not the Shire; after over three years in this game, it still is my favorite starter zone. And as for a skinny hobbit, well the world might just as well end, that's all I've got to say.

And my thanks to all at Turbine who had a part in getting us up and running again.

[djheydt]

Has anybody even considered the possibility that whatever caused the crash also crashed ALL links whereby the techs could talk to us??????? In other words, maybe none of them have power or maybe even the cell towers aren't working to post on Twitter???

After the crash I looked at Twitter, which had nothing.

So I suspect you may be right.

We'll stay tuned....

[djheydt]

On behalf of anyone who has ever had to carry a pager and be on-call to be awakened in the middle of the night to deal with some frantic emergency...

on behalf of anyone who has ever had to work in a customer service profession and deal with abuse from customers over a situation for which you were not responsible, given no information about and were powerless to prevent or solve...

on behalf of anyone who has ever been in the deeply unenviable position of having to be the one to wake up a Big Boss to be the bearer of bad news...

I would like to thank those within Turbine who may have found themselves in these positions over the last several hours, and apologize for the attitudes and entitlement of those who have never stood in your shoes, and feel no need to empathize with your pain. And I hope your day gets much better from here.

That is all.

+ rep on behalf of my husband and daughter, who have had to stand in those shoes.

[nastyhobbits]

"Doodle Jump" is a game. "Angry Birds" is a game. "LOTRO" is a massively multiplaying online role-playing... All right, technically it's a game.

[djheydt]

This is a link to a Boston-area weather site. I have it bookmarked so I can check up on whether Turbine is having ten-foot snowdrifts or monsoons or whatever else might take down the game/endanger or inconvenience the Turbinites.

http://www1.whdh.com/weather/

Scroll down through most of the page to see video/screenshot reports on the storm that went through last night and the damage it left behind.

I forget which scientist it was who said, "We live on a wild planet, not a tame one."

[Iktomi]

On behalf of anyone who has ever had to carry a pager and be on-call to be awakened in the middle of the night to deal with some frantic emergency...

Did that for 8 years.

on behalf of anyone who has ever had to work in a customer service profession and deal with abuse from customers over a situation for which you were not responsible, given no information about and were powerless to prevent or solve...

Did that for 2 years, and to add to that, be nice and polite back to the customer despite the abuse.

on behalf of anyone who has ever been in the deeply unenviable position of having to be the one to wake up a Big Boss to be the bearer of bad news...

Did that for about 5 or 6 years.

I would like to thank those within Turbine who may have found themselves in these positions over the last several hours, and apologize for the attitudes and entitlement of those who have never stood in your shoes, and feel no need to empathize with your pain. And I hope your day gets much better from here.

That is all.

And to quote Super Chicken, "We knew the job was dangerous when we took it". (Now I really am dating myself; I should get me some flowers )

I did have assumptions that Sapience has corrected. I still do feel there was a breakdown in communications procedures, but he says it's all being looked at in that area as well as the technical side so all is good. I do empathize with them even while being a tad critical.

[whheydt]

With all the trouble coming from this outage, I assume it is safe to say that the Farmers fair, which was supposed to start today, will not happen...again. According to the calender, it was supposed to start on the 19th then was pushed back to the 26th. seems if things keep going the way they are, we will see the summer fest in December.

Wrong! The Farmers Faire is up and running this morning....as per the revised schedule.

--W. H. Heydt

Old Used Programmer

[Teslana]

Wow, more customers slobbering all over themselves thanking a vendor for doing its job. Amazing.

I think some people were thanking a man. You know, a human being. That's a 'he', rather than an 'it'. Sapience is not Turbine; he just works for them. Unless he actually is some sort of mighty, mobile, marauding megacorpbot, in which case HE MUST BE DESTROYED! DooOOOoOom!!1!! Hunters, take aim! Minstels, ready your war drums! Loremasters, summon!, etc.

[Adder]

We're still investigating the root cause for the outage, but we do know that this was a sort of 'pefect storm' that impacted both primary and secondary sytems.

There are a lot of assumptions being made in this thread and honestly most of them are wrong. The nature of last night's issues are entirely based in hardware and connectivity failures. The way they unfolded caused a cascade of issues that lead to communication issues both internal and external (with our International GM and CS teams).

For the record, the entire community team is on call 24/7. Community Managers like myself and Tolero (DDO) are on call 24/7/365 including vacations and Holidays with very rare exceptions. No one ever hesitates to call us at 2am if need be. It's the job and we're happy to do it.

That said, the nature of last nights issues was such, as noted above, that communications were impacted.

We'll continue to investigate the root cause. We'll address the issues and failures we discover, and will adjust policies where we discover they are needed as a result of those investigations.

We'll share what information we can when we can and if we can. It's probably worth noting that investigations like this usually are not done in a day, but over several days.

Thanks for the update Sapience.

---Deleted---

I give up. I get positive rep for flaming against some things Turbine does, but I stand behind them on an issue and get hit with negative rep.

Anyway, it'll be interesting to hear the results of the investigation.

[Sapience]

he actually is some sort of mighty, mobile, marauding megacorpbot

I was totally behind this description! Until we got to the 'must be destroyed' part. Kinda lost some of its luster then.

[SmellyCheese]

And to quote Super Chicken, "We knew the job was dangerous when we took it". (Now I really am dating myself; I should get me some flowers )

Sorry, I have nothing of value to add except...

bok bok bok bok!