Game Services and Helm's Deep Update

[Amberjack]

I am glad for your consumers and for your employees (and employers) that your stuff is all back up and running.

But can we please not point fingers towards the players?

...

Turbine/WB/Sapience and the Devs are the ones who wanted/need "more time", not the playerbase.
Obviously, we wanted no issues, to get our #### done last night (or be able to finish it up after launch) and to have HD launch hours ago.
THAT is what we wanted.

No, some of "us" never wanted the HD changes at all.

If you're determined to represent "the players", en masse, then "the players" - ALL OF US - would like you to get it right.

[furtim]

I very much doubt that the new launch date was influenced in any way by a perceived necessity to keep the Bounder event open.

There are three things about this situation that would realisitically influence the launch date:

1. Scheduling arrangements with the hosting provider.
2. Letting the servers run for 24 hours to ensure that there are not any undetected issues remaining from the unexpected shutdown.
3. Allowing the folks on the operations team to go home and have a full night of rest after the all-hands-on-deck emergency. They've probably been on-site in the office or at the hosting provider since Outage Time + Driving Time O'Clock last night, with no sleep since then. And they probably won't be able to leave for a couple more hours yet, even though things seem pretty stable.

US power does not seem to be anywhere near as good as the power in most of the EU.

Oh, boy. If you think our power grid is bad, you don't even want to hear about our internet infrastructure. And you really, really don't want to hear about the state of our road bridges...

[Kerin_Eldar]

All I'm gonna say is some of you people are truly unbelievable.

And people round here sneer at the WOW community as being whining brats.

[cutlasssdfwewe]

In a vacuum a 2 day delay of an expansion is not a big deal, but this isn't a vacuum, some people still remember how poorly the RoR launch went, and how there has been unscheduled down time associated with just about every expansion since then. At some point the excuses don't matter anymore, no matter how valid. This needed to go well, because so many other things have not gone well over the last year.

[Bel-Astarte]

If their reasoning was because stuff wasn't ready, absolutely. And obviously.

But that's not what Sapience shared, he told us what "we wanted", and told us now we get what we want and "still get HD this week".
I'll take HD later since that's where we are left - but I do not take responsibility for it and shame on Turbine for releasing their reasoning for delay as "No you."

And a whole bunch of people DID want that, even if you or I did not need it.

Here's my speculation-they decided they needed the extra day or so to make sure the system has a stable period before deploying HD. Keeping in mind the many commenters on twitter and facebook, as well as here on the forums, who were disturbed that they had missed the deadlines for the xp boost and token turn in, they decided to turn those back on in the meantime.

It makes as much sense at least as your speculation that there's a conspiracy to "blame" the players for a delay.

I'm really kind of appalled at some of the reaction here to this news. It's like watching extremly entitled 3 year olds pitch a fit because someone else is getting something.

[Tobias_R]

Umm I only have read one of those things, and that was only because the servers went offline mid day yesterday. had that not happened none of this would be necessary.

Now people can cite all the stuff they want, 'act of God' 'mother nature', whatever. facts are this happens a lot, as in every time. How many server issues have there been? 10-12? The excuses ran out a long time ago on this sort of thing. But the defenders here show there are still enough people here who believe it.

When you have an actual DEADLINE people take heed, to come in and use these three excuses as a reason is really poor. Players would have been more than ready had the game run as expected.

In the end it doesnt really matter, it just delays the inevitable, which is this expansion that was the one everyone has looked to to 'save' Lotro is going to be the worst one yet. (I dont count Mirkwood because we all know what that was).

This whole fiasco is somehow poetic in that it sort of foreshadows everything as well as puts it into a nice neat package of reference.

Now with all this being said how about YOU also utilize these two days. If it is released with a ton of bugs and broken content then that puts it squarely back onto you guys. But really who are we kidding at this point.

Believe what? Do you seriously believe Turbine switched power off for all their game servers to prevent anyone complain about HD being a day late? I don't.

I was a bit surprised a couple of hours ago when it turned out loads of players had problems with not turning in their tokens. But seriously, if many of my fellow Lotro-players feel their gaming-experience gets a boost from this then so be it. HD will be my gaming-world for 10-12 months, another day or 5 without it is pretty unimportant. I don't care if the reason for the delay is the powerloss or being nice to people who had a lot of bounders tokens. Both reasons works and are valid to me.

Regards
/T

[Conancrush]

Easy there Lobelia...did you honestly expect them to release HD immediately given how hinky the servers are right now? Wed is a reasonable compromise. And no, the delay wasn't because of the laggards with the bounder's tokens.

Tuesday would be fine right ? Read what sapience said, they talked about it and listened to all the people that wanted more time. Learn to read

[JBomber]

Umm I only have read one of those things, and that was only because the servers went offline mid day yesterday. had that not happened none of this would be necessary.

Now people can cite all the stuff they want, 'act of God' 'mother nature', whatever. facts are this happens a lot, as in every time. How many server issues have there been? 10-12? The excuses ran out a long time ago on this sort of thing. But the defenders here show there are still enough people here who believe it.

When you have an actual DEADLINE people take heed, to come in and use these three excuses as a reason is really poor. Players would have been more than ready had the game run as expected.

In the end it doesnt really matter, it just delays the inevitable, which is this expansion that was the one everyone has looked to to 'save' Lotro is going to be the worst one yet. (I dont count Mirkwood because we all know what that was).

This whole fiasco is somehow poetic in that it sort of foreshadows everything as well as puts it into a nice neat package of reference.

Now with all this being said how about YOU also utilize these two days. If it is released with a ton of bugs and broken content then that puts it squarely back onto you guys. But really who are we kidding at this point.

To be fair on these points it would appear this is more than just an excuse, if you go to their twitter feed they re-tweeted a tweet by an entirely unrelated group that runs Minecraft servers who lost their access at the same time and according to this companies tweet they share the same data center as Turbine.(Links Turbines Twitter, Aklizhosting's Twitter both can be viewed without having a twitter account since I have none and am viewing them)

As to your other points, I would completly agree that the majority probably was ready for the release plenty in advance and would have cared less losing the one day.(though since neither of us are privy to everyone's actual opinions on the mater this statement what other players want is very washy from both you and me) I have seen a handful of people who were posting on the twitter yesterday during the downtime asking if the bounders token stuff would be extended so people could turn them in and I have seen a similar number of people posting here that they could care less and want the expansion now. Honesly it is probably for the best that they delayed it until later anyways since there servers are still a bit glitchy, I have heard several people complaining about login and timeout/crash issues (at least on my server riddermark) and I myself have now 4 characters who cannot log in without a timeout since I attempted to either enter the boar fountain area or swift travel on those toons, so i have decided to stay off for a few hours and try again tonight. Honestly, would I have preferred the update now, yes(despite the fact that I am one of those people who forgot to turn in my bounders tokens and still cannot due to issues on my server); do I think it would have been a good idea, probably not given the number of problems on my server alone atm. They need to make the servers stable again before pushing a rather large update to the game on us, imagine the uproar if when the expansion went live half the people could not log in (after having made it past the update servers) and many lost connection when they passed through certain areas or tried doing any form of port/swift travel stuff. *shudder*

[andracy]

Well the players have an point:
01:50 GMT - power has been restored and they are starting up their servers
~ 04:0x GMT - the launcher is searching for patch server
~ 05:4x GMT - servers are closed
~ 06:3x GMT - many servers will be available shortly, but actualy you cannot log into them
~ 07:1x GMT - servers will be available shortly, but once entered you will be in *searching for logon server* and after 20 attempts you are out.

after 7 hours ... - We continue to investigate login issues for the following servers ... bla bla bla

30 min later - All LOTRO Game Servers are now available. Thank you for your patience. Funny, Snowbourn is NOT working hehe
30 min later, Snowy is working.

all these took ... hmm, 21 h? And i am thinking ... hell, i worked today 7 h. Was long day since I usualy work 6. Mondays... I hate mondays. And these guys are working for 21 h. Well, i feel sorry for them. And i would love to go there, bring them an cofee and raise their morale. I realise is only bad luck at the end.
The decision to delay HD seem correct. Now dont shoot me. But just check lotro twitter and forums for example to see how many ppl are complaining about servers issues and disconnects. For sure all these little problems MUST be solved before HD. Atm is just NO stability. This day is already passed so only remains one day for fixing these. I am sure they will not use this time to improve HD, no one have time to touch that hehe.

Mates, you have an horrible bad luck. Turbine must break this habit to have bad luck on an launch day. You really need to improve your karma hehe. Now go home to your wifes and kids, you did good for today.

Best regards

[Edlenn]

A two day delay sounds just about right to me.

[Glumposneak]

As I've actually run environments in a data center, the initial event was not Turbine's fault. That is, I believe the story line because I have experience that backs this up.

I've seen data centers, on multiple occasions, lose an A or B side circuit because, despite recommended vendor maintenance to their batteries (which hold the data center up until the generators come on to cover for the lost power), the batteries have, in essence, lied about their state of being and that entire side experiences a cascade failure, thus loosing all power to that circuit. Once I've seen the loss of one side circuit take down the other circuit so regardless of running dual power supplies on your computer hw, one powered by circuit A and one powered by circuit B, everything crashes and burns. The root cause could be something as simple as the data center vendor doing circuit maintenance and it goes horribly wrong. Murphy always gets a paycheck.

Depending on the hardware Turbine uses and how it is connected into power, there may be steps Turbine can take to survive a single circuit loss. Dual power supplies (and subsequently providing power to them to your cage/rack/etc.) costs money and power isn't cheap as data centers usually charge you something not entirely insignificant for that second circuit. But, this is a cost/benefit analysis. As this game is F2P at the moment, I'm not expecting a huge hw investment from Turbine. That's a cost/benefit reality.

Whatever the reason, computers do not take kindly to having their power ripped away from them (machine components die this way). The tendency of a database to take extreme exception to power loss (lose power in the middle of a write) is a serious problem. This means Turbine, while waiting for power to come back, weren't sleeping with visions of sugar plums dancing in their head. Some dB people probably lost a few years off their life sweating out what they were going to find.

I don't disagree with anything above except for the F2P comment.

I know what it can take to repair databases etc I know the frustration the IT workers will be going through.

Turbine chose to go F2P and according th their last public figures have made a nice proffit from doing so, that is why I disagree with your F2P comment, I'm not saying you are not right, just that they offer a 24/7 game, it's not entirely free, many people spent 100s on TP or other accounts etc and they expect a company supplying a game such as this, to have proper backups in place. If Turbine wont pay for propper backups then they cant lay the blame at anyones feet but their own, Last time if memory serves me correctly it was totally nothing to do with them and was all the datacentres fault. That I do not agree with, either the datacentre isn't providing the service turbine is paying for, in which case Turbine should be looking elsewhere, or Turbine isn't paying for a good service in the first place. Either way, ultimately the buck stops wioth Turbine.

Had the tornadoes etc been in Boston, of course anyone would understand, but from what I've read, they were not.

I am not joking when I say that some of the companies I've worked for, while I agree with you 100 in your description of what could go wrong, if I gave that as a reason, it would be unacceptable, I would get a severe telling off etc and that would be if it happened just once. They would want assurances I have personally made sure the data centre has done regular tests etc so that if another power outage happens, all events have been covered.

The fact that you know exactly what can happen, only goes to show that any data centre worth its while would also be aware of this scenario hence should have procedures in place to make double sure it's never going to.

[Avasa]

Do most of you that are moaning not realize that the Turbine team need to sleep at some point?
They can't survive purely on the (small) amount of gratitude we give them and (hopefully) overtime pay.
Give the teccies a break, let them rest and regroup, and we'll still be able to play Wednesday... Yes, some people have booked time off work, and such, but it's not the end of the world, unfortunate as it is.

[Thorgrum]

Guys please, no name calling (in either direction!). People are upset, we understand that. There was an issue, we did everything we could to get all of our services restored. Not just LOTRO, but all of our services across all of our titles. Now that they're all up and available, we considered some of the things we heard frequently during the downtime.

• You wanted more time to get in your last minute per-orders
• You wanted more time to turn in your Bounder's Tokens.
• You wanted more time to get your characters to 85 with the 100% XP boost.

Now you have the opportunity to do all of that and you'll still get Helm's Deep this week.

Excellent recovery on turbine's part. This way the people with bounder tokens to turn in don't loose out, another couple of days isn't going to kill anyone. Good luck on wed sapience, Ill try to ease up on the tro... interaction to make your life a little easier.

[MightyKOko]

It would be extremely unlikely if there was no nature disaster on Turbines expansion launch day.

PS: I love the fanboys!

[Glod]

> You wanted more time to get in your last minute per-orders


Maybe the massive amout of last minute pre-orders caused the power blackout in the whole Boston area?

I remember the time when this forum almost crashed, due to the massive amount of people, asking for more time to place their pre-order.

Glad you could resove this issue. Thanks again for your amazing effords.

[cutlasssdfwewe]

Why are people using sleep as an excuse? LOTRO is a 24/7 operation, are we supposed to believe the tech staff all works the same shift?

[Gagmuk]

It would've been alot better had the XPAC been released on schedule.

if their was no issues beyond human control. then yes i would agree, it would be nice to have the expansion released on time.

but it did not happen. equipment failed. which is why we are in the current situation.

i have HD pre-ordered, Got all the bounders tokens earned and spent. And most of my toons are 85.

I would rather turbine take its current coarse and delay the release of HD like they did, rather than to try to push the update onto servers that are obviously still having issues. I think players from 3 servers are still reporting issues at this point, Snowbourn, meneldor and darrow?

I would rather they fixed the issues from this power outage, tested the servers and repairs and made sure it was fixed and stayed fixed. That way when HD is put on em we wont have constant disconnects when it is released.

[catofnine]

Tuesday would be fine right ? Read what sapience said, they talked about it and listened to all the people that wanted more time. Learn to read

That is PR spin to ameliorate a missed deadline. You learn to think. This downtime is not specifically for catering to people who wanted more time. If the servers hadn't come crashing down last night, we would be downloading HD right now and reading the outrage from people who didn't turn in their tokens for whatever reason. Why do you have a problem with a company trying to make sure everything's ok after an unexpected event before launching a new expansion?

[Darlgon]

Just a FYI for the tech guys. On Meneldor, about five min ago, I was able to log into my Mini, went to swap to my Warden, was stuck on the loading screen with the gold bar at zero until the game timed out. I was unable to do anything but reboot the machine.

Aaand.. logged back in to warden in 21st bank. Zoned to main 21st, OK. Zoned to 21st crafting and.. well.. "was stuck on the loading screen with the gold bar at zero until the game timed out. " At least this time I did not have to reboot.

Guess its back to Skyrim for a while.

[Finborhur]

Turbine epic fail, as always they lie about release days nad never keep the word. Servers has crashed before the release, but that still doesn't mean they can't release the expansion. Why to delay? I don't care about that bounder tokens, I never attend in-game events, because they are horribly boring.

November 21st is the release day of Assassin's Creed IV: Black Flag. On November 20th I'm working from 16:00 to 8:00 morning. It will be hard to decide which one to play first.

[trcanberra]

Although this isn't turbines fault I'm going to be depressed till the 20th, fkk!!!!!!!

Really wanted to play mainly because i love the new class changes (to my class at least). FU mother nature.

My feelings too. I know stuff happens - still a sad panda. I am happy in a way that all the bounder token complainers can finish their stuff - and hopefully those who were sitting on 2500 when the servers went down will learn to do things a little bit sooner next time.

[Gagmuk]

Why are people using sleep as an excuse? LOTRO is a 24/7 operation, are we supposed to believe the tech staff all works the same shift?

nope but my guess is that 1st shift has the full crew all hands on deck. 2nd shift has a reduced crew to finish up items not finished on 1st shift. and 3rd shift probably has a skeleton crew to monitor everything, and be active gms during the night unless something major happens like yesterday then they call in the Calvary,.

since it was sapience who was posting updates yesterday and last night on Facebook and Twitter. he was one of the ones who got called in.

just my guess tho.

[pistolshrimps]

As you know. A lot of players are very upset and agitated about the delay on the expansion. Personally I am not too upset but I am disappointed. I was very excited to play HD today and now... I can't. I'll stick with you, turbine. I won't cancel my huge premium pre-order plus legendary pack. But I have put a lot of preparation into the expansion making sure I got it done by today and now I have 2 more days to do it? That's frustrating. Try and get the expansion up as soon as you possibly can. Guessing Tuesday is a day for all the Turbine workers to sleep after many hours of working out problems. XD Try and fix it. At least the initial rush of players will be lessened a bit when it comes out. Hope everything is okay at turbine. But please get HD up soon
Ignislamina, Firefoot.

[Avasa]

Why are people using sleep as an excuse? LOTRO is a 24/7 operation, are we supposed to believe the tech staff all works the same shift?

...24/7 operation, sure, but that doesn't mean they have all the usual teccies in over-night. Why would they?
...Or did you think a 24/7 operation meant that they kept, for example, Dev's in developing those nice class renovations rotating 24/7 too? No, only in-game support probably, and token force everywhere else.

[trcanberra]

> You wanted more time to get in your last minute per-orders


Maybe the massive amout of last minute pre-orders caused the power blackout in the whole Boston area?

I remember the time when this forum almost crashed, due to the massive amount of people, asking for more time to place their pre-order.

Glad you could resove this issue. Thanks again for your amazing effords.

Maybe they couldn't pay their power bills because not enough people pre-ordered?