Game Services and Helm's Deep Update

[Jackel3]

Thank you for the information, and keep up the great work.

[Glumposneak]

Give Turbine a break here... geez. They do their absolute best to give us 100% up-time, but it's impossible to do. You can't control mother nature and until they make servers that don't need a power source things like this will happen from time to time. They do a great job at communicating and they usually throw in an extra perk to VIP's for downtime. How many other companies do either of those things, much less both?

It[s not the first time a power outage has affected them and it's not the first time the backup system failed to kick in.

Yes there were terible storms in parts of the US yesterday but from what I've read (and I could off course be wrong) the devastation wasn't in Boston.

Turbine choose to use that datacentre, if the data centre isn't up to the job, it's down to Turbine to find another one that is.

From what I've read, mother nature was not the cause of the problem, inadequate backup systems were.

As I've said elsewhere,. there's many people that play Lotro that work in IT and if those in charge tried to fob their customers off that it was the datacentres fault, they would be hauled up infront of their bosses and their bosses would want to know why it happened, and what they are going to do to make sure it never happens again.

If it happened a second time, heads would roll.

I think this is the third time?

What usually happens is we get told it's all the data centres fault and beyond Turbines control, but when it's fixed, miraculously Turbine take all the credit for fixing it, when they should be taking the blame for not having an adequate system in place in the first place, and it is their responsibility, they choose to continue to use that data centre.

They (Turbine, not individual workers who have no say in what data centre is used etc) don't deserve a break to be honest as any experianced IT worker will tell you, they deserve the exact opposite. It is really inexcusable. You run a 24/7 365 operation, you should have the correct and tested backup procedures in place. Again most people that have worked in IT will know that if they let this happen once, there would be hell to play and their bosses would want written details of what's in place to make sure it can never happen again, twice and heads would probably roll. Happing a 3rd time, well I wouldn't like to work for any of the companies I worked for if something like this happened twice and presuming I got away with it twice (highly unlikely) I wouldn't still be employed if it happened a 3rd time, if it's a 3rd party we use, someone decided they were up to the job and that person is responsible..

Obviously theres the odd thing you cant account for, acts of nature/terrorism that break the main internet pipes etc, but that did not happen to the best of my knowledge in Boston last night, when the forums were down, I was online on various Boston websites looking for news about the storm etc and from what I read, it really wasn't that bad in that particular area. It seems just the data centre had problems and yet again their backup system wasn't up to scratch.

As a side note, I don't think overall Turbine do a great job in general at communicating, sometimes they do but quite often they don't. For example, these forums were back up for something like 6 hrs before anyone at Turbine posted an update here and before someone mentions twitter, these are the official forums, if a turbine staff member can take the time to post on twitter, then it's common curtsey for someone to take the time to post in the official place too.

As another poster put in another thread, it's like giving thanks and praise to the binman when he knocks your trashcan over.

[Xnr]

I know you guys are working hard but I have only managed to log in 1 out of 5 times today. This has now been an issue for a day. Are Turbine going to extend the VIP of current subscribers because, as it is, I cannot play the game I am paying to play? I know answer may not be available immediately but I can't justify paying for something which I might or might NOT get.

Additionally, I am amazed Turbine can use such a poor (in terms of reliability) data centre. Most big users demand 99.9% availability and just in the 4 months I've played, Turbine have had 3-4 days offline which translated into % of days in 2013 is 0.82% to 1.1%. That is just unacceptable. Turbine should seek compensation and then compensate their customers but I just can't see myself staying with such an unreliable game.

[gixxer1025]

Thanks Sap, for the update and thank you to everyone else for the hard work and dedication to the game that I love.

[Grimdoin_EU]

More time? Seriously? We've had months to deal with the bloody tokens... Again the whiners win

[Elrohiras]

I have an idea.. Bump the preorder price of HD up to $199 and then... Buy a UPS!! W00t - genius isn't it!

Seriously though been in the game for 6 years, there has never been a trouble free release (code masters then turbine). Around these times it's time to take those little breathers from the game..

[suzelger]

my riddermark character that was in bree when my client crashed will not load. the one in lothlorian will load

[Fantoma]

Really, Turbine?

I really can't trust you guys. Don't strange if so many players stop playing. EVERY expansion you guys let something like that happen. I bought HD Premium Pack, pay my VIP every 3 months, but it's starting to get no sense. Can't play. HD delayed. 1 day without the whole game and now 3 characters bugged.

Will be 1 day that you guys will provide the game without these interruptions? I love the game, I love your support. But it's time to reconsider somethings around.

Next expansion, you better HOPE AND PRAY that Turbine can END TORNADOS!!!
Seriously, you're blaming them for the weather now???

[Ilhsul]

I cannot help but giggle a little. Seriously? An extra two days? You gave us enough time already. You knew people wanted the HD xpac badly, more than bounders tokens, and im willing to bet once people played the HD content they'd forget bounders tokens in a whole despite my past opinions. I've been waiting for this xpac since i started playing the game, ill wait another two days. Really though, get it together guys. I love lotro, i love turbine, but theres always a delay, and understandably, this wasnt completely your fault. Hopefully we get more goodies than extra time for bounders tokens/xp boost.
Excuse anything taken offensively though..as other people are also outraged im a little upset myself, but atleast we get it this week not next week or month or year.

In Turbine's defence - there was a WHOLE lot of whining about bounders tokens on Facebook, they've taken the right decision. You simply can't please everyone.

[Conancrush]

Guys please, no name calling (in either direction!). People are upset, we understand that. There was an issue, we did everything we could to get all of our services restored. Not just LOTRO, but all of our services across all of our titles. Now that they're all up and available, we considered some of the things we heard frequently during the downtime.

• You wanted more time to get in your last minute per-orders
• You wanted more time to turn in your Bounder's Tokens.
• You wanted more time to get your characters to 85 with the 100% XP boost.

Now you have the opportunity to do all of that and you'll still get Helm's Deep this week.

So the vocal minority told you they waited till the last second to do those things, and you decided that instead of releasing the xpac that people have been waiting months for instead you will delay it another day so they can catch up??????

Whaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaa?????? ?

[k3nn3th]

I cannot help but giggle a little. Seriously? An extra two days? You gave us enough time already. You knew people wanted the HD xpac badly, more than bounders tokens, and im willing to bet once people played the HD content they'd forget bounders tokens in a whole despite my past opinions. I've been waiting for this xpac since i started playing the game, ill wait another two days. Really though, get it together guys. I love lotro, i love turbine, but theres always a delay, and understandably, this wasnt completely your fault. Hopefully we get more goodies than extra time for bounders tokens/xp boost.
Excuse anything taken offensively though..as other people are also outraged im a little upset myself, but atleast we get it this week not next week or month or year.

Not their fault it happened, and when. Not entirely their fault how long it took them to fix it (presumably).

But to give their reasons to delay it as "Players wanted A, B, and C." is rubbish.

[Jeffythequick]

All LOTRO Game Worlds and Websites are now available. Thank you for your patience while we worked to resolve issues that resulted from a power outage at our data center. In order to provide the best possible experience for players, we will be delaying the release of Helm’s Deep until Wednesday November 20th.

As a result, we will be extending our Helm’s Deep pre-purchase offers as well as the 100% XP boost and Bounder’s Bounty events until the 20th as well.

My rumor is that it was Sapience multiboxing his 25 hour game-a-thon that caused this whole snafu.

Thanks Rick!

[EruReborn]

The selfishness and childish nonsense seen here and elsewhere on these forums have to be seen to be believed.

Recovery from a hard crash (assuming that was the case, which the reports that have been made suggest it was) is not as simple as "plug it in, switch it on". All sorts of hardware / software / database integrity checks have to be made to sensure that the game Worlds are both playable and stable. It is not surprising that some are still suffering glitches, and I am sure these will get sorted out in the coming hours, though I hope the sysadmins are now enjoying some well earned pie and ale for their sterling efforts in fixing a problem that was not of their making today (and quite different from the extended downtime they had planned).

If some of the less mature players can't cope with a few hours of downtime in such extreme circumstances then they really need to take a long, hard look in the mirror before griping about the situation (or starting the 37th thread on the matter without even bothering to check what is already in the forums).

I was really looking forward to playing HD tonight, having participated in the entire beta, but I can't. No great shakes. Tomorrow (hopefully after some well-earned rest) the techies will regroup and make sure that everything is ready for Wednesday's update - there is zero point in patching an unstable system with a major update like HD and the delay is the only responsible thing to do.

It is probably safe to say that the silent majority, though understandably disappointed, is behind the decision that Sap and team have taken, but as usual, the loudmouths and crybabies dominate.

Thanks Devs/Admins and everyone else for your above and beyond efforts today and to make sure everything goes as planned on Weds.

[rosey21]

All I'm gonna say is some of you people are truly unbelievable.

[Brdrker]

As I've actually run environments in a data center, the initial event was not Turbine's fault. That is, I believe the story line because I have experience that backs this up.

I've seen data centers, on multiple occasions, lose an A or B side circuit because, despite recommended vendor maintenance to their batteries (which hold the data center up until the generators come on to cover for the lost power), the batteries have, in essence, lied about their state of being and that entire side experiences a cascade failure, thus loosing all power to that circuit. Once I've seen the loss of one side circuit take down the other circuit so regardless of running dual power supplies on your computer hw, one powered by circuit A and one powered by circuit B, everything crashes and burns. The root cause could be something as simple as the data center vendor doing circuit maintenance and it goes horribly wrong. Murphy always gets a paycheck.

Depending on the hardware Turbine uses and how it is connected into power, there may be steps Turbine can take to survive a single circuit loss. Dual power supplies (and subsequently providing power to them to your cage/rack/etc.) costs money and power isn't cheap as data centers usually charge you something not entirely insignificant for that second circuit. But, this is a cost/benefit analysis. As this game is F2P at the moment, I'm not expecting a huge hw investment from Turbine. That's a cost/benefit reality.

Maybe Turbine does pull out the stops here. I have no way of knowing. I will say this. They are the ONLY F2P that within filing an in-game ticket that I get a response quickly, usually within 10 minutes. Blizzard never gets back to me the day of (and they aren't F2P). The other F2P games I play, in-game technical support is non-existent (i.e., black hole) for all practical purposes. I'm stunned by this so kudos to Turbine. The advice isn't always what I'm looking for but, hey, someone gets back to me quickly and tries to help.

Whatever the reason, computers do not take kindly to having their power ripped away from them (machine components die this way). The tendency of a database to take extreme exception to power loss (lose power in the middle of a write) is a serious problem. This means Turbine, while waiting for power to come back, weren't sleeping with visions of sugar plums dancing in their head. Some dB people probably lost a few years off their life sweating out what they were going to find.

As for the loss of an entire day. I played all morning and into the early afternoon, PST. Nobody lost the ENTIRE DAY. I was in a pile of orcs wrecking havoc and my biggest concern was waking up in the midst of a mob I could not survive, re-spawning who knows where and then having to find my way back. As it was, Turbine put me back in a safe spot. They did a minor rollback. Took me 5-10 minutes this morning to recover what I was doing (and found something new to do along the way).

My attitude, for what is worth, having fought what Turbine is going through is, yes, I'm somewhat disappointed but I'm going to cut them some slack here. I don't pay for this game. I don't pay for Turbine Points (because I'm one cheap SOB). They have more than proven that they are worth staying with so I'm going to give them the benefit of doubt here and wait a few days.

Maybe I can pick up 150 tokens to get that dang horse in Delving before I get my tokens taken. The game will still be there either way. Hang in there.

[Grimdoin_EU]

In Turbine's defence - there was a WHOLE lot of whining about bounders tokens on Facebook, they've taken the right decision. You simply can't please everyone.

If the delay is only done to compensate people for the 12 hours of +100%XP and opportunity to hand in the Bounder Tokens then the only right decision would have been to lauch HD, give another day of +100% XP and put the Bounder NPC back in for a while. THAT would have been considerate, not delaying the expansion...

[Glod]

Thanks a lot for delaying the game to provide the best possible experience for me, Turbine.

Maybe you should consider delaying Helm's Deep (including the class changes) for another month.

I'm sure that would enable you to provide an even better experience for us players.

Thanks again for your effords.

[Glumposneak]

Isn't it a bit convenient that there is a power outage the exact same day HD is supposed to launch... And this is US not some third world country like Germany or France... Shame on you Turbine... Cutting the power to a large area just for your personal benefit...

And i will bet anything that it was Sapience idea..

Actually I used to work for an oil exploration company with a big computer installation in Houston as well as ours in England and others in Egypt and Norway.

I was shocked on a visit to Houston as to how unstable their power actually is and talking to our engineers it seems that they have seen this time and time again over their careers (both were over 50). They said that certain cities like New York were OK but most of the country had overhead power lines and power cuts in storms was very very common, something that is virtually unheard of inmost of Europe.

Sure we might have the odd occasion when power os cut off from a village or part of a town, but it's very rare. If there's a full scale lightning storm overhead, the thought that I could loose my power wouldn't even enter my head, yet many times my US kinnies have said just that, there's a storm so don't be surprised if they get cut off from loosing their power.

US power does not seem to be anywhere near as good as the power in most of the EU.

In 49 years, ignoring the power companies strikes in the 70's, I don't recall the power ever going off anywhere I used to live and it's only gone off twice in 13 years where I live now, once when something exploded under the pavement outside my house that caused the whole street to be covered in yellow smole and the emergency services to turn up, that took about 12 hrs for them to dig up and fix, the other time for about 2 hrs due to a fire in a local distribution point (don't know what it's called)

[FoxMcCloud]

Anybody else having trouble to log in the game? Mine's stucking in Loading screen every time.

[Darlgon]

Just a FYI for the tech guys. On Meneldor, about five min ago, I was able to log into my Mini, went to swap to my Warden, was stuck on the loading screen with the gold bar at zero until the game timed out. I was unable to do anything but reboot the machine.

[catofnine]

So the vocal minority told you they waited till the last second to do those things, and you decided that instead of releasing the xpac that people have been waiting months for instead you will delay it another day so they can catch up??????

Whaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaa?????? ?

Easy there Lobelia...did you honestly expect them to release HD immediately given how hinky the servers are right now? Wed is a reasonable compromise. And no, the delay wasn't because of the laggards with the bounder's tokens.

[k3nn3th]

The selfishness and childish nonsense seen here and elsewhere on these forums have to be seen to be believed.

Recovery from a hard crash (assuming that was the case, which the reports that have been made suggest it was) is not as simple as "plug it in, switch it on". All sorts of hardware / software / database integrity checks have to be made to sensure that the game Worlds are both playable and stable. It is not surprising that some are still suffering glitches, and I am sure these will get sorted out in the coming hours, though I hope the sysadmins are now enjoying some well earned pie and ale for their sterling efforts in fixing a problem that was not of their making today (and quite different from the extended downtime they had planned).

If some of the less mature players can't cope with a few hours of downtime in such extreme circumstances then they really need to take a long, hard look in the mirror before griping about the situation (or starting the 37th thread on the matter without even bothering to check what is already in the forums).

I was really looking forward to playing HD tonight, having participated in the entire beta, but I can't. No great shakes. Tomorrow (hopefully after some well-earned rest) the techies will regroup and make sure that everything is ready for Wednesday's update - there is zero point in patching an unstable system with a major update like HD and the delay is the only responsible thing to do.

It is probably safe to say that the silent majority, though understandably disappointed, is behind the decision that Sap and team have taken, but as usual, the loudmouths and crybabies dominate.

Thanks Devs/Admins and everyone else for your above and beyond efforts today and to make sure everything goes as planned on Weds.

If their reasoning was because stuff wasn't ready, absolutely. And obviously.

But that's not what Sapience shared, he told us what "we wanted", and told us now we get what we want and "still get HD this week".
I'll take HD later since that's where we are left - but I do not take responsibility for it and shame on Turbine for releasing their reasoning for delay as "No you."

It implies that the expansion is ready to launch and only isn't launching/hasn't launched because of the tears of the vocal minority.

[Tyveil]

Thank you. I hate Monday releases since I only get to game on the weekends! Now I can only hope for another delay and the actual release will be sometime Friday morning. That would work perfectly for me.

[Disaac]

In Turbine's defence - there was a WHOLE lot of whining about bounders tokens on Facebook, they've taken the right decision. You simply can't please everyone.

It would've been alot better had the XPAC been released on schedule.

[jensil]

Thanks Sapience and team for working hard at getting back all the fun and enjoyable turbine worlds up ! Looking forward to HD on Wednesday and crossing fingers for you guys that no other hiccup occurs :P