Game Services and Helm's Deep Update

[HoldumDown]

Something you impatient types can check out while you wait for your pre-paid expansion: http://elderscrollsonline.com

[Glumposneak]

It's kind of amazing how many people assume that bad weather caused the power loss at Turbine's data center yesterday afternoon/evening. Turbine didn't say that. Turbine didn't describe the cause of the power loss in any way. Could it have been human error? Sure. By a Turbine employee or contractor? Sure again. Maybe not, though. We have no way of knowing. Neither in a rush to condemn, NOR in a rush to defend, should we rush to judgment. Some are also assuming the data center in question is not directly owned by Turbine. That may also be a bad assumption. Would be interesting to know what caused the power outage, but we'll likely never be told. Would be interesting to know why the data center's backup power source(s) (alternate grid, generator(s), battery farm(s), whatever) did not kick in as they should, but we'll likely never be told that, either. So, for both the blind fanbois, AND the blind ragers, sit back and stop with the assumptions. Power went down at datacenter, backup power didn't kick in, servers crashed hard, Turbine still cleaning up that mess, so Expac is delayed a couple of days. That's what we know on this end, and all we really need to know.

Knowing many many IT bods from many different industries, while one or two may be genuinely appreciative of the thanks and praise given to them on facebook etc, I bet the majority are in hysterics at it, shaking their heads in disbelief and jokingly discussing amongst themselves how bad a service they could get away with and still get many Lotro players praise. And I'm being serious.

[floydisme]

Even if it wasnt in there control I expected the expansion to be late some how so no biggie. Tbh when they set the release date I moved it up on my calendar to Nov. 28th. So many people having trouble logging on is disheartening though..

[kidnova78]

I happen to live in the area where the Data Centers are. Yes we had severe weather warnings. Yes we had flooding and hail as well as several tornadoes touch down. There were several areas that were affected with power outages.

Where would that be exactly? Most seem to believe the data center in somewhere near Boston as that's where Turbine's offices are located. If that's true then the weather timeline doesn't match up since the servers went down in the afternoon and the severe weather didn't hit that region until later in the evening. If they're located closer to the Midwest, well maybe the weather theory has some merit. IMO, if the weather were the cause I think Turbine would have come out and said it by now.

[Veria]

Guys please, no name calling (in either direction!). People are upset, we understand that. There was an issue, we did everything we could to get all of our services restored. Not just LOTRO, but all of our services across all of our titles. Now that they're all up and available, we considered some of the things we heard frequently during the downtime.

• You wanted more time to get in your last minute per-orders
• You wanted more time to turn in your Bounder's Tokens.
• You wanted more time to get your characters to 85 with the 100% XP boost.

Now you have the opportunity to do all of that and you'll still get Helm's Deep this week.

Thanks, I'm actually quite happy about delay although not for the reasons you mentioned. I'm glad because:

• My kin was supposed to have the last pvp night before class changes and moors becoming mess for a while. Now we can have 1, maybe 2 and probably slightly less laggy since it's not weekend day anymore.
• 2 days should be enough to get me at least 5 more renown scrolls or at least 25 morale pots for the future.
• Really hate changes coming to one of my 85s (champ, thankfully not my main) so 2 more days to enjoy it in all its glory. Last 2 days to enjoy great versatility of my main (minstrel), too.

[Angadan]

I happen to live in the area where the Data Centers are. Yes we had severe weather warnings.

Congratulations on living near them, I'm sure your interaction rates with the servers should be MUCH better than mine. Nevertheless, you don't know why the power failed to that data center any more than I do. Do you?

[addict82]

Doesnt have to be a blue name. Players on those specific servers are reporting issues. Non access to characters and disconnected. just because a blue name has not made it official does not mean problems still dont exist and being reported.

update: im 1 of the players reporting a problem. Server Meneldor. Searching for Authentication server, Connection successful, Then it locks up for 5 minutes like its going to disconnect.

So what you're saying then is.... its possible that HD simply wasn't delayed because of tokens, xp, preorders?

So what you're sayin then is... its possible Sapience lied or gave a half-truth? There are more issues, and Helm's Deep isn't ready for other only-related-to-Turbine reasons?

GUIZE, HALP! Found Gagmuk over here CONSPIRACY THEORIZING!
Defend Sapience!

[Unomisan]

Does it really matter why the power loss happened? There are a lot of things that can cause the power to go out in an area. My power company can't keep the power on for more than a few weeks at a time (though typically very brief, the outages last just long enough to shut down all my electronics). Sure, the skies were clear in Boston. All the tornadoes were hundreds of miles away. But that doesn't mean the power can't go out for other reasons. I checked the national grid status online when the game went down and there were reports of outages in the Boston area.

[BINKLEY1]

All LOTRO Game Worlds and Websites are now available. Thank you for your patience while we worked to resolve issues that resulted from a power outage at our data center. In order to provide the best possible experience for players, we will be delaying the release of Helm’s Deep until Wednesday November 20th.

Thank you for taking a cautious approach.

In my work managing risk is essential. Decisions like this are painful but necessary.

Regards,
Gloimli @ Windfola

[kidnova78]

Does it really matter why the power loss happened? There are a lot of things that can cause the power to go out in an area. My power company can't keep the power on for more than a few weeks at a time (though typically very brief, the outages last just long enough to shut down all my electronics). Sure, the skies were clear in Boston. All the tornadoes were hundreds of miles away. But that doesn't mean the power can't go out for other reasons. I checked the national grid status online when the game went down and there were reports of outages in the Boston area.

To be clear, I 100% believe that a power loss occurred, just not sure why everybody is jumping to the conclusion that the weather was the cause.

[Darlgon]

Knowing many many IT bods from many different industries, while one or two may be genuinely appreciative of the thanks and praise given to them on facebook etc, I bet the majority are in hysterics at it, shaking their heads in disbelief and jokingly discussing amongst themselves how bad a service they could get away with and still get many Lotro players praise. And I'm being serious.

I know after 15 years in IT, I am laughing at how people are reacting with praise at poor output. But, then again, I liked doing business with those same nice people. They gave me bonuses for fixing their screw-ups.

[Ithrien]

Does it really matter why the power loss happened?

Not in the slightest, nor where or how long or causing how much destruction... since Sapience clearly posted that all services are restored, up, and available.

Officially.

So now we can all stop using the weather to fanboi.

[Angadan]

Does it really matter why the power loss happened? There are a lot of things that can cause the power to go out in an area. My power company can't keep the power on for more than a few weeks at a time (though typically very brief, the outages last just long enough to shut down all my electronics). Sure, the skies were clear in Boston. All the tornadoes were hundreds of miles away. But that doesn't mean the power can't go out for other reasons. I checked the national grid status online when the game went down and there were reports of outages in the Boston area.

Of course it matters! It matters for the people providing power so that they can repair the damage (presumably already done) and take steps to avoid future failures from same source. It matters to Turbine, hugely as a tech entertainment company, so that they can gauge the reliability and look for trends in their power provider. And it matters to us players because we're likewise trying to gauge the reliability of our on-line entertainment provider. The DEGREE to which it matters drops a tier at each of those nodes, but we all care. Most specifically to us players, the big thing that matters is why the BACKUP power source didn't kick in. That's much more directly attributable to Turbine (probably, no assumptions here), and has more direct impact on the future reliability of the game we play. Now, in the grand scheme of our lives, is any of this more important than frog lipstick? Not at all, this is just a game. But see, sitting around wondering what happened IS entertainment for those of us bored with the existing LOTRO content, waiting for the new stuff to arrive on Wednesday. We're meta-playing LOTRO in the forums by correcting each other's assumptions about storm patterns over Needham, Mass (or wherever that data center sits), see?

[Gagmuk]

So what you're saying then is.... its possible that HD simply wasn't delayed because of tokens, xp, preorders?

So what you're sayin then is... its possible Sapience lied or gave a half-truth? There are more issues, and Helm's Deep isn't ready for other only-related-to-Turbine reasons?

GUIZE, HALP! Found Gagmuk over here CONSPIRACY THEORIZING!
Defend Sapience!

Defend Sapience, ROFLMAO.. get in line. not 1 of my posts since this event occurred has blamed Sapience or Turbine. Plus, Sapience did not lie. the servers are back up but their are still some minor issues. hence the possible reason for the delay until Wed..

just takes common sense to come to the same conclusion. oh wait, what am i thinking this is the internet where common sense is no longer needed

[trcanberra]

Stay up unexpectedly while having to think under stress over night Sunday, then try to catch up a bit of your sleep on Monday, you will be in no fit state Tuesday. Wednesday at the earliest makes sense to give them time to properly recover.

They also need the servers to be up and running stable for a while so that anything unexpected that happens at upgrade, they know for sure the upgrade caused it and not the power outage.

Sorry - I worked in IT hardware support for years and pulled plenty of 24-hour shifts, so the sleep thing just doesn't cut it.

I don't mind the delay if they need it for IT reasons (which seems to be the actual reason); I just didn't like spin they put on it about the tokens etc. etc. - trying to make it look like it was all about pleasing the players. Then we get posts saying everything is up and running when that doesn't seem to be the case. Bad PR just annoys me.

[Mortybenson]

Oh, boy. If you think our power grid is bad, you don't even want to hear about our internet infrastructure. And you really, really don't want to hear about the state of our road bridges...

Hehe, if you think YOUR internet infrastructure is bad, you should see the state of affairs in Canada... We don't let competitors in, so the companies that own the monopoly aren't forced to lower prices OR make improvements. Our road bridges are also pretty bad - but at least they are working on fixing them. The issue with our power situation is an odd one... we sell power to the U.S. grid at a loss because our law says x% of the power in our grid needs to be "green" power - which is exponentially more expensive (i.e. we pay much more than we really should be)

[addict82]

Sorry - I worked in IT hardware support for years and pulled plenty of 24-hour shifts, so the sleep thing just doesn't cut it.

I don't mind the delay if they need it for IT reasons (which seems to be the actual reason); I just didn't like spin they put on it about the tokens etc. etc. - trying to make it look like it was all about pleasing the players. Then we get posts saying everything is up and running when that doesn't seem to be the case. Bad PR just annoys me.

Exactly.

Defend Sapience, ROFLMAO.. get in line. not 1 of my posts since this event occurred has blamed Sapience or Turbine. Plus, Sapience did not lie. the servers are back up but their are still some minor issues. hence the possible reason for the delay until Wed..

just takes common sense to come to the same conclusion. oh wait, what am i thinking this is the internet where common sense is no longer needed

I was being sarcastic if you hadn't picked that up, to showcase how common-senseless it is to believe HD is actually being delayed because of a power outage/the weather or because of the playerbase's vocal minority.

[Caelth]

Thanks for the update, I know you all are working very hard to get things up and running and done correctly. I know Turbine couldn't prevent the severe weather. The same series of storms that knocked out power killed 6 people. For heavens sake I can wait till Wed. for the expansion.

[Almagnus1]

Eh... the real reason why the servers went down is cause they got depeessed from enduring the constant QQ and decided to end it all.

Fortunately for us, there were backups.

[Angadan]

I know Turbine couldn't prevent the severe weather. The same series of storms that knocked out power ...

Jumps up and down on the manhole cover. 58! 58! 58!

[addict82]

Thanks for the update, I know you all are working very hard to get things up and running and done correctly. I know Turbine couldn't prevent the severe weather. The same series of storms that knocked out power killed 6 people. For heavens sake I can wait till Wed. for the expansion.

That storm only killed the minority, who weren't necessary for the continued existence of the game.
Turbine doesn't have time to consider single-digit-percentages of the playerbase.

[Pausekey]

Honestly they should take the servers down for a month and overhaul the game. Right before an xpac is the perfect time. Give some free TP to shut up the Turbine fan club and all is good. Change as much as possible and put out a decent product instead of the tired old usual. They keep trying to put their donkey in the Kentucky Derby, and at some point they need to realize that it will never win.

[JBomber]

Are people really comparing Minecraft with LORD OF THE FRIGGING RINGS? SERIOUSLY? The fact that Lotro servers are on the same servers Minecraft is on should really be troubling to a lot of people (everyone).

One is the greatest IP ever (sorry Star Wars geeks) that is now under the umbrella of a multi billion dollar company. One some guy built in his basement and is browser based.

It would be funny if it werent so sad.

I did not compare minecraft to LOTRO (and as far as I can see I was the only one who posted about the minecraft servers), I was simply pointing out that it was more than just turbines fault. according to the akilz twitter feed they use some data center called Internap or something like that which if their comment about LOTRO and DDO sharing there data centers is to be believed means that they both hire the same company for their data center needs. THough once again most of this is conjecture and me making inferences based on some twitter posts, take that as you will.

oh, and just to point this out, Minecraft, while having a browser option is not really browser based (unless I fail to understand what you mean by browser based) but that is neither here not there since it does not matter to this discussion.

[Mar-Evayave]

All LOTRO Game Worlds and Websites are now available. Thank you for your patience while we worked to resolve issues that resulted from a power outage at our data center. In order to provide the best possible experience for players, we will be delaying the release of Helm’s Deep until Wednesday November 20th.

As a result, we will be extending our Helm’s Deep pre-purchase offers as well as the 100% XP boost and Bounder’s Bounty events until the 20th as well.

Thank-you, thank-you, thank-you! As a person who was eagerly awaiting the xpac and am currently without power (one of the half a million in Michigan without any power whatsoever) I'm so blissfully gleeful that the release date has been pushed back a few days. Estimates in my area range anywhere from 3 days to almost a week before power will be restored, and while Wednesday is sooner than either of those estimations, I'm grateful for the extra time anyway!

Thank goodness for power generators, too... I'm hoping I'll be able to get a teensy bit of playtime during the few hours we can afford to run it...

Of the four things here, the launch of the expansion is far and away the most important to the majority... isn't that who Turbine is here to cater for?

Turbine/WB/Sapience and the Devs are the ones who wanted/need "more time", not the playerbase.
Obviously, we wanted no issues, to get our #### done last night (or be able to finish it up after launch) and to have HD launch hours ago.

THAT is what we wanted.

You do not speak for me.

Almost half a million residents of Michigan (only one state in some 6-8ish states effected by the recent storm, a storm that has killed several people and dealt serious damage to a lot of places in a lot of states) are without power. I am one of them and may have the rest of the week without power. I'm GLAD that the HD xpac is delayed. That means I am not missing out (and as leader of a kin, I feel it is important that I stay as up to date on systems and gameplay and quest progression, etc. as much as possible) while the rest of my kin moves on ahead of me. I'm able to get online at my workplace very briefly, and it's my hope that our power generator at home is up to the challenge of giving me internet for brief periods of time, but for the rest of the week that may be the best I will get.

[JBomber]

their are still a few problems happening on a few servers so they need time to finish tracking those down and fixing them. then they will probably want to allow some time to make sure everything is back to normal.

1 day (today) to finish finding and repairing the data centers.
2nd day (tomorrow/Tuesday) to monitor the data center and make sure it is all fixed.



As I stated in another thread earlier,

you dont need storms to have or produce a power outage. Just about anything can cause 1. Here are a few examples of what can cause a power outage.

#1 I worked at a steel metal plant that manufactured the racks that go into the Frigidaire refrigerator freezers. That plant had its own power sub station due to all the welders and power draw. Beautiful day, no bad storms at all. i was working just inside in the dept closest to the the power station. Heard this buzzing noise then what sounded like a short circuit. power blinked and then BANG the largest power transformer blew up and we lost power for 2 days. had to wait for the power station to get repaired. no other facilities in the industrial park were effected.

#2 Was working a job at a injection molding facility another nice night. then the power goes out. maint couldn't figure out why. 2:30 am consumers guys found the problem. A squirrel had shorted the power to the building and caused it to blow out. We all went out with our flashlights to take a look at it and sure enough a squirrel attached to the power line and burnt to a crisp. no other businesses or houses were effected.

#3 worked at a CNC shop and was waiting for 2nd shift to start. the rectifier board or control panel suddenly melted. had 3 days off waiting for parts to restore power. no storms this day either and only effected the 1 plant.

plus sometimes when a power company adjusts the amount of electricity going down the lines will cause problems. If a piece of equipment cant handle the cycle changes, increased amps this will cause older out dated building power lines and power control panels to blow out. New equipment can sometimes fail as well if corners were cut with manufacturing materials and assembly lines were not inspected properly.

just wanted to say that their doesn't need to be a storm to have power issues.

as fun side note to add to your story of strange power outages, a few years back where my parents live, a turkey got into the power grid for the town and fried itself inside one of those machines and a good part of town lost power, twas hilarious.