Was there a big outage or was it just me who couldn't connect?

[whheydt]

Here is an article on Twitter's problems today...

As for the whole pager/on call business...been there, done that.

One job I had was back before anyone but upper level executives or doctors even carried pagers (cell phones didn't exist) and I was the "on call" person for the company payroll systems. SystemS, plural? Yes. We had three. One for hourly employees that ran weekly, one for salaried employees that ran twice a month and one for retirees that ran monthly.

If the weekly payroll had a problem and I couldn't fix it over the phone, I had to drive in the the office (no remote access in those days) because the weekly payroll *had* to get out on time. There was one company plant on one occasion when the payroll checks arrived two *hours* late...and by then the wildcat strike had been going for 90 minutes. It was *that* critical to the company.

If the salaried pay couldn't be fixed over the phone, then it could be set aside until the next morning to solve the problem. If salaried paychecks came out a day or two late, no one was going to get worked up over it. Likewise...plenty of time to get the retiree checks out.

On the whole...if a game is down due to technical issues caused by weather, then we get to wait. Emergency services have first call on response agencies. Nobody is at risk of life or limb because they can't play LotRO.

--W. H. Heydt

Old Used Programmer

[Valamar]

Just for the record.... According to News Radio here in Philadelphia, Twitter had a major world-wide outage that ended around noon today. (I haven't looked up all the details (if I can even find any) yet.)

Given the changes in weather in the past 24 hours around here, it makes one wonder if all of these communication issues are part and parcel of this current Sunspot cycle. ... gives a different twist to the concept of a "perfect storm."

And yes, I'm another "Old Timer" System Administrator (now retired) quite used to the 24/7/365 on-call cycle.

[Fredelas]

I was totally behind this description! Until we got to the 'must be destroyed' part. Kinda lost some of its luster then.

I am convinced now that you must be a Trekkie. Primary and secondary system failures unfolding to create cascades is something that usually happens aboard starships.

So I have to ask, were you channeling Scotty, Geordie, O'Brien, B'Elanna, or Trip?

[Valamar]

Hmm... just found another listing -- apparently Google Talk was also down today! That makes 3 major outages in the past 24 hours.

Starting to sound like "somebody" getting ready to attack Olympic coverage tomorrow.

[fezzick]

We're still investigating the root cause for the outage, but we do know that this was a sort of 'pefect storm' that impacted both primary and secondary sytems.

There are a lot of assumptions being made in this thread and honestly most of them are wrong. The nature of last night's issues are entirely based in hardware and connectivity failures. The way they unfolded caused a cascade of issues that lead to communication issues both internal and external (with our International GM and CS teams).

For the record, the entire community team is on call 24/7. Community Managers like myself and Tolero (DDO) are on call 24/7/365 including vacations and Holidays with very rare exceptions. No one ever hesitates to call us at 2am if need be. It's the job and we're happy to do it.

That said, the nature of last nights issues was such, as noted above, that communications were impacted.

We'll continue to investigate the root cause. We'll address the issues and failures we discover, and will adjust policies where we discover they are needed as a result of those investigations.

We'll share what information we can when we can and if we can. It's probably worth noting that investigations like this usually are not done in a day, but over several days.

I was glad when LOTRO came back up. Since there was no indication of it being worked on (twitter and facebook) or when it might come up, I was sitting there wondering what I should do (fortunately I didn't get past that phase of withdrawal) and then to deal with it, watched some youtube vids of instance runs. Not quite the same.

Sapience may want to post his response and sticky it as a separate posting with a link to this thread. Someone else's thread was closed because he posted in this one but no link was provided and I spent 15 minutes looking for a threat started by Sapience. A thought.

[Crell_1]

I am convinced now that you must be a Trekkie. Primary and secondary system failures unfolding to create cascades is something that usually happens aboard starships.

So I have to ask, were you channeling Scotty, Geordie, O'Brien, B'Elanna, or Trip?

Now think about that for a minute, which one is a 'miracle worker'?

...Also, that twitter doesnt seem to understand the 'Did you tell him how long it would really take' concept..

*grin*

Things just go so smoothly around here most of the time, people forget just how responsive and interactive Turbine is when they can be.

[Mark_J]

There are some datacenters out there that can and do offer 100% up-time, it can be achieved but at a premium. Most companies calculate the cost of their revenue loss for downtime and take action accordingly. Many don't feel the difference between 99.5% (which is relatively easy and inexpensive to achieve) and 100% up-time (which is very expensive to achieve) is worth the increased cost. Turbine's revenue loss for an hour of downtime is insignificant when compared to the cost of providing 100% up-time.

Has anybody even considered the possibility that whatever caused the crash also crashed ALL links whereby the techs could talk to us??????? In other words, maybe none of them have power or maybe even the cell towers aren't working to post on Twitter???

I've only ever experienced a total blackout like that once and it lasted 3 days, it was right after Hurricane Katrina in Biloxi Mississippi.

[Iktomi]

I am convinced now that you must be a Trekkie. Primary and secondary system failures unfolding to create cascades is something that usually happens aboard starships.

So I have to ask, were you channeling Scotty, Geordie, O'Brien, B'Elanna, or Trip?

''

Scotty is too laid back and cool for that kind of techno-babble; it has to be one of the others. I nominate Geordie.

[Sapience]

I am convinced now that you must be a Trekkie. Primary and secondary system failures unfolding to create cascades is something that usually happens aboard starships.

So I have to ask, were you channeling Scotty, Geordie, O'Brien, B'Elanna, or Trip?

I proudly confess to being a Trekkie. While I'm always a Scotty fan (had the fantastic opportunity to meet him, Shatner, Nichelle, George(shared an elevator with him. he is an incredibly nice guy!), and Nimoy and most of the other cast members including all of the 'captains' at various times), I have to admit I'm a big Trip fan.

Though I think if you asked me what my prized Trek possessions are I'd have to say my signed copy of Leonard Nimoy's "I am Spock" (because a very good friend went through a lot of trouble to get it signed for me, in person) and a photo of Chase Masterson (Leeta from DS9), she gave me after a children's AIDS charity event we both worked on.

[Adder]

I was glad when LOTRO came back up. Since there was no indication of it being worked on (twitter and facebook) or when it might come up, I was sitting there wondering what I should do (fortunately I didn't get past that phase of withdrawal) and then to deal with it, watched some youtube vids of instance runs. Not quite the same.

Sapience may want to post his response and sticky it as a separate posting with a link to this thread. Someone else's thread was closed because he posted in this one but no link was provided and I spent 15 minutes looking for a threat started by Sapience. A thought.

LMAO! Ya gotta expect them from mighty, mobile, marauding megacorpbots like Sapience.

[danno8]

I was totally behind this description! Until we got to the 'must be destroyed' part. Kinda lost some of its luster then.

Well for the longest time I thought you were a girl. I think it stemmed from when there was another community manager around here by the name of Patience. Then I think either yours or hers (at least I think she was a she) signature had a line that read "Patience is not Sapience" or something that, although clear, somehow confused me further.

Needless to say I was shocked when I clicked on one of the community update youtube thingies and saw a big orange beard staring at me!

[Adder]

I proudly confess to being a Trekkie. While I'm always a Scotty fan (had the fantastic opportunity to meet him, Shatner, Nichelle, George(shared an elevator with him. he is an incredibly nice guy!), and Nimoy and most of the other cast members including all of the 'captains' at various times), I have to admit I'm a big Trip fan.

Though I think if you asked me what my prized Trek possessions are I'd have to say my signed copy of Leonard Nimoy's "I am Spock" (because a very good friend went through a lot of trouble to get it signed for me, in person) and a photo of Chase Masterson (Leeta from DS9), she gave me after a children's AIDS charity event we both worked on.

WOW. I don't get jealous very often but DAAHAM!

[Erebthoron]

There is no 100% security. The company I work for run one of Europe biggest data center and by odd circumstances they manage to shut it down by accident including the backup. It happen before, it will happen again. Keep in mind that every modern jet fly with software thats 99.9% secure and enjoy your next trip.
And Sapience: as a Trekkie the Force will never be with you

[Graycient]

Maybe Sapience was running around in the server room and tripped on some ice and broke his scrotum on the computer. Would explain the outage. lol I kid, I kid.

I actually missed the outage. I was working on Free People's Manual... Was it really as bad as you make it sound? xD

[djheydt]

[QUOTE=whheydt;6316396One job I had was back before anyone but upper level executives or doctors even carried pagers (cell phones didn't exist) and I was the "on call" person for the company payroll systems. SystemS, plural? Yes. We had three. One for hourly employees that ran weekly, one for salaried employees that ran twice a month and one for retirees that ran monthly.

If the weekly payroll had a problem and I couldn't fix it over the phone, I had to drive in the the office (no remote access in those days) because the weekly payroll *had* to get out on time. There was one company plant on one occasion when the payroll checks arrived two *hours* late...and by then the wildcat strike had been going for 90 minutes. It was *that* critical to the company.
[/QUOTE]

Yup. Earlier than that, when whh and I were first married, he had a job whose headquarters were in San Francisco (we lived across the bay, in Berkeley) and it was all batch processing. You know, the programs and the data were both on decks of punch cards? And he was again the guy who got called at weird o'clock in the morning to fix it when it wouldn't run.

Sometimes he would have to get up, get dressed, and drive into the City to fix things. Sometimes he would pick up the phone, mumble "Hello?" listen, and say "Well, go to line 783 and change the /*/ to a //*, and try it again,and if it doesn't work call me back," and hang up and I swear to Manwe, he'd be asleep again before his head hit the pillow. And it would work and they wouldn't call back.

Several jobs later, he was working for the phone company, and for reasons that were never made clear to me, he had to have a second phone line to call the system with if it needed fixing at strange hours. So a phone tech came in to install the second line. He looked around our house, which was ... sort of shabby and disorganized ... and finally asked me, "What does your husband do for the company?" "He's a programmer." "Oh. Oh really? Because I've never been sent to install a second phone line for anybody below the rank of vice-president."

The programmer's life.....

[Adder]

Thanks for the update Sapience.

---Deleted---

I give up. I get positive rep for flaming against some things Turbine does, but I stand behind them on an issue and get hit with negative rep.

Anyway, it'll be interesting to hear the results of the investigation.

Pretty sure where it came from too. 'Nuff said.

[Norwrei]

I guess I was one of the lucky ones that never got disconnected. There was a huge lag spike last night and then people started dropping like flies from the server.. then the forums were down. Strangely enough I managed to stay on the server.. was a ghostly experience lol since I seemed to be the only one on Eldar but I continued to level my baby warden.

[djheydt]

Well for the longest time I thought you were a girl. I think it stemmed from when there was another community manager around here by the name of Patience. Then I think either yours or hers (at least I think she was a she) signature had a line that read "Patience is not Sapience" or something that, although clear, somehow confused me further.

Needless to say I was shocked when I clicked on one of the community update youtube thingies and saw a big orange beard staring at me!

The problem is that names like Patience, Sapience, Charity, Fortitude, and essentially all other abstract nouns, including all names of qualities, are feminine nouns in both Latin and Greek. The qualities were worshipped as goddesses in Hellenistic times, and to this day we expect somebody with an abstract noun for an ekename to be feminine. With no defensible reason, since few of us know any Latin or Greek (I have a smattering, not enough to be really useful), and English lost its grammatical genders about a thousand years ago.

[I got several fantasy stories out of this fact a couple decades ago; my characters included Physis (Nature), Sophia (Wisdom), Elpis (Hope), and especially Arete (Virtue). You could tell they were goddesses because they never sweated, never got dusty, and the flowers they wore never withered. ]

[Cappie.evg]

So...basically...the same thing that happened to twitter recently happened to the Turbine servers.

[djheydt]

I guess I was one of the lucky ones that never got disconnected. There was a huge lag spike last night and then people started dropping like flies from the server.. then the forums were down. Strangely enough I managed to stay on the server.. was a ghostly experience lol since I seemed to be the only one on Eldar but I continued to level my baby warden.

Yes, from all accounts, most of the servers stayed up, but communication between them and the outside world was hosed. Whh and I had characters standing around in Combe, getting ready to log out actually, when all of a sudden we couldn't move ... and then we got "Connection to the chat server has been lost!" and we said "The heck with it" and logged out.

I'm going to take another look at the weather page ... apparently they're expecting ANOTHER big storm in this evening.

[Minulinnwen]

Yes, from all accounts, most of the servers stayed up, but communication between them and the outside world was hosed. Whh and I had characters standing around in Combe, getting ready to log out actually, when all of a sudden we couldn't move ... and then we got "Connection to the chat server has been lost!" and we said "The heck with it" and logged out.

I'm going to take another look at the weather page ... apparently they're expecting ANOTHER big storm in this evening.

I remember from last year when I was flying into Logan International (Boston) from Amsterdam, that we were put on a holding pattern because the airport had closed. We needed to divert to Bradley International in Connecticut because of a storm.

Later on, after I landed there was another massive thunderstorm with high winds and rain and I was sure the house would be destroyed. I never experienced weather this bad before. When I asked my friends they said it wasn't that bad, the one the day before (that caused my plane to be diverted) was a lot worse.

I guess us Europeans aren't used to anything, weather wise. I have to fly into Atlanta and than DFW in a few weeks, so I'm most interested how that will be like.

[Erebthoron]

No we Europeans are not used to bad weathers, we have here 25 degree all year only with some rain in between And this counts from Italy to Norway, from Spain to Ukraine...

[probitas]

I was finally able to connect to the forums just now, then the game crashed in the background.....figures.

[ElvenKing_of_Old]

not only this. the crickhollow server had a server wide lag yesterday, everything was out of place and no one had connection to the chat servers and when we thought it was over it happened again until at last after 10 or 20 minutes it got restored. not sure if it happened to any other server

[guguzza71]

Is there a website that says when the servers are up or down?

I think it woudl be a good idea to have something like this so readily available and up to date so that if there is an issue we can always check it out. It would make things simpler.