Offical Discussion : Service issues Sept 1- Sept 2

[Amberjack]

YES! It is awesome! I prefer it over the other bacon.

Ever try frying Canadian bacon in American bacon grease?

Best of both worlds, I'm tellin' ya.

[BanginFannings]

He's got a point. We Americans are usually the most vocal during unplanned outtages in any capacity. Usually we back up our anger with threats to sue, go to the BBB or in the extreme case, threaten bodily harm against the developers.

Although sometimes annoying that passion and zeal is what made America the world's sole superpower.

[BlueJacket]

Many thanks for the status update. I checked facebook/twitter every so often for an update but it's good to see a longer explanation here on the forums. My best guess would be the folks at Internap were involved somehow ... but that's only a guess. I will say I go a lot done while not being able to play LOTRO. Once again thanks for the update and to everyone at Turbine who worked at resolving the issue.

[Sapience]

The most important things said, so far, in these responses is that those who gave up their holiday to discover and solve the problem should receive compensation from their company. Is that Turbine's policy? Do we need to write letters (with postage stamps) to executives?

I can't speak for others, but as for the Community Team our job is 24/7/365. Though we do sometimes get yelled at to go to bed.

Firstly, thank you for posting this information in the forums, and for doing a decent job of keeping us up to date in addition to responding kindly to many PO'd people. Like you said, this is a Holiday Weekend for us and that certainly adds to the frustration.

Secondly, a huge thank you for giving a few technical details. As you are obviously aware, there are a lot of technical IT people who play online games, and we definitely appreciate "the details".

All in all it has been a frustrating 24 hours for everyone involved. Thanks for your perseverance and professionalism, Sapience, and all the Turbine Staff involved in this. Our votes in you were well-placed.

Regards,

US.

I'm a former IT professoinal so i do try to impart a bit of 'de-teched' info when I can once I've had the team explain the full issue to me.

There was 78 unique threads about the service outage because no one from Turbine started a official thread or posted anything for that matter. As far as the community team not having access to the forums, I'm not buying it. If all of us could login and post, then so could you.

Anyone knowledgeable about routing of internet traffic and backbone providers isn't buying the third party story either. We could plainly see routing for all users stopped within the same Turbine data center and Turbine owned IP address space that was hosting the working forums. This should fall under Turbines domain and direct control and not Internap or other third parties. It is interesting that the internet backbone route path to the Turbine data center is the same as it was yesterday. The only thing different today is one different IP address in the Turbine data center before gls.lotro.com. It is also interesting that once the supposed ISP backbone issue had been resolved there were still issues with logging in and intermittent 500 internal server errors on the forums. That must of been caused by the third parties too right? Whatever the real reason for the outage, it speaks of ineptitude on Turbines part for managing the service.

What bothers me most about this outage is:

1. Lack of official status or updates regarding the service (on lotro.com)
2. General disrespect for paying subscribers by feeding them bits and pieces of mis-information on FB and Twitter. (Don't believe this garbage people).

I don't do FB or Twitter. I don't pay them to play Lotro. I pay Turbine to play Lotro. When I have a problem with the service I go to www.lotro.com to find out what is going on.

All that said, thank you to the people who actually worked and resolved the outage.

The great thing about facts is they require no belief and belief requires no facts. I've given the facts and you're certainly entitled to your beliefs.

If you can't access the forums normally, okay then. Regardless, I think forums must be more important than Facebook or Twitter and I hope you don't go the Facebook or Twitter only route in the future (not saying you have).

Thanks for addressing.

As a general rule they are, we use Facebook to expand our reach (as any good Community Manager and Team does), and we have made no secret of the fact that those our are primary channels in times when we cannot reach the launcher and forums.

[Startrekman1of9]

I can't speak for others, but as for the Community Team our job is 24/7/365. Though we do sometimes get yelled at to go to bed.



I'm a former IT professoinal so i do try to impart a bit of 'de-teched' info when I can once I've had the team explain the full issue to me.




The great thing about facts is they require no belief and belief requires no facts. I've given the facts and you're certainly entitled to your beliefs.


As a general rule they are, we use Facebook to expand our reach (as any good Community Manager and Team does), and we have made no secret of the fact that those our are primary channels in times when we cannot reach the launcher and forums.

Ok I will then give you the benefit of the doubt that even though all of us could post on the forums that you could not.
Then why couldn't you get on the forms if we were able to.
What kept the Turbine staff from accessing it just like we did?
A question, not a criticism.

[robbie1435]

I can't speak for others, but as for the Community Team our job is 24/7/365. Though we do sometimes get yelled at to go to bed.



I'm a former IT professoinal so i do try to impart a bit of 'de-teched' info when I can once I've had the team explain the full issue to me.




The great thing about facts is they require no belief and belief requires no facts. I've given the facts and you're certainly entitled to your beliefs.


As a general rule they are, we use Facebook to expand our reach (as any good Community Manager and Team does), and we have made no secret of the fact that those our are primary channels in times when we cannot reach the launcher and forums.

[Walterdagreat]

still having issues with loggng my 85 lvl toon takes for ever to tell that it has lost conection to server then have to start at the begging to have it happen again help please

[Sapience]

Ok I will then give you the benefit of the doubt that even though all of us could post on the forums that you could not.
Then why couldn't you get on the forms if we were able to.
What kept the Turbine staff from accessing it just like we did?
A question, not a criticism.

I can't discuss it at any length, but employee accounts are different.

[Grimmar2]

I logged in at 1525 est.

[Jsper]

I can't discuss it at any length, but employee accounts are different.

This is not due to a technical limitation but a policy choice.

[Startrekman1of9]

I can't discuss it at any length, but employee accounts are different.

OK that I can accept. I used to work for Microsoft in the late 1990's and we had similar restrictions.

It was just hard to have another patch go sour on us last Monday as the last few have for some of us and then this issue yesterday.
I know the 2 were not related but it felt like salt in the wound so to speak.

I do commend the people that took there Holiday and got us going again.

Can you speak also about the contract renewal that allows Turbine to keep the game going for another few years.
I heard something about a Contract that has to be renewed this year.
I understand if you cant say anything about it but it would be nice to know if we will get to go to Gondor and Mordor in the next few years

[Redamber]

First, as we didn't need 78 unique threads on the issue, most saying some version of "fix it", they've been removed and this thread will be used going forward.

Hopefully this tones down all the crazy conspiracy theories we've seen recently (pando, hacking, shut downs, contract issues. It's a pretty impressive list!). We appreciate those who tried to share back the information we were posting elsewhere.

lol gamers and conspiracy theories... the gaming community is at least as prone to that as student journalists are and just as imaginative when it comes to causes Glad it's fixed and thanks for the info - I even re-subscribed this evening for another year once I could get back on

Red

[LethalLethality]

Even though the servers are back up, I just can't pull myself away from The Sims 2.

Been meaning to invest myself into it but LotRO has been holding me back. This break was a good thing as it's letting me play something different.

[Tarkit]

Thanks for the update Sapience

[BanginFannings]

The store isn't up. How do I know TP is being added to my account if I can't see my TP?

[Im_Mereniel]

I'm not sure about other people, but I click on Dev Tracker about 20x as often as I click on Announcements. Also, I believe anything posted in Announcements would show on Dev Tracker anyways. Additionally, most of the discussion of this incident (at least the portion discussed on the forums) was in the Gen.Disc forum, and where people would expect to see more. Lastly, this is the forum people have been going to before starting a new thread about the outage, so its exactly where it needs to be to catch the eye of those doing that.
It's just fine where it was posted.

I always go straight to General Discussions cause at the time I am asking my questions, there usually has not been an official response yet.

Thank you very much for all your hard work, team! Any chance you can tell us the name of the backbone provider? Just curious.

[Startrekman1of9]

The store isn't up. How do I know TP is being added to my account if I can't see my TP?

The store is up now but you will get an email when you buy points

[Duathrandir]

Thank you for the updates, Sapience, what a way for you to spend the holiday weekend, ugh!

I found this site that tracks outages, http://www.outageanalyzer.com/ and there's one listed that's almost 22 hours old, pd.ed10.net. I wonder if the problem is something like this.

What holiday weekend? It's an extra day off. I'm sure he's being very handsomely compensated for his time.

[Nishka]

Thank you so much for fixing the issues. I appreciate all of the effort the Turbine team put into putting us back in the game. Now everyone go sleep :P

[ColSanders]

This issue is one we've seen before with a slight twist. This was again a routing issue with a very large internet backbone provider and how it routes traffic in and out of our data center provider (so you're talking 2 layers removed from our direct ability to 'fix it').

Considering this is a repeated issue with a vendor (or in this case a vendor's vendor), I sure hope your contract team is working on a contract with more serious outage penalties as the current (if any) do not seem to be enough to deter poor change practices. You might want them see if they have any ITIL certified people working for them. It's not the best methodology, but it's a start.

[Gildassen]

Thank everyone involved getting the game back online for giving up their holiday weekend to get all the whinny people back ingame. My ISp also had issues overnight and yes I was up all night trying to play LOTRO or gasp anything else. When I saw LOTRO was back I immediately quit playing the other game and came back. I have lost many holidays to unplanned network outages at my work so I feel for you.

I like so few others monitored facebook and twitter w/o complaining about it knowing that if the game had issues the forum site would too. Silly Rabbits.

Thanks again!!!!!
Get some well earned rest.

[Sapience]

Can you speak also about the contract renewal that allows Turbine to keep the game going for another few years.
I heard something about a Contract that has to be renewed this year.
I understand if you cant say anything about it but it would be nice to know if we will get to go to Gondor and Mordor in the next few years

It was answered a couple months ago actually. We're not going anywhere.We've got a lot of stuff we still want to do!

[whitealaska]

Thanks for working thru the night on this fix! I hope you took notes and recorded the chain of events that caused this problem and make a step-by-step list of how you fixed it. Great job guys!

[Duathrandir]

Thank you for your efforts in resolving the issue in a timely manner. Props to all those who are working on a holiday.

Huh?! Resolving in a timely manner? I still cannot play. How exactly has this issue been resolved, at all, much less in a timely manner?

[FittyBolger]

What do you expect? Canadians are all jealous because their only contribution to the world is taking a slice of ham, cutting it in a circle and calling it bacon

Tread lightly friend... for far north in the frozen wastes the legend stirs from it's long slumber... long forgotten by those in warmer climes.

Forged in an igloo...
Tempered in REAL Canadian Maple Syrup...
Hammerd on an anvil by beaver tails...
Commanding a horde of singing Mounties...
And emblazoned with a flaming red maple leaf...

Fear it!
Fear! Fire! Foes!

POUTINE