A Word on Hardware Improvements

[YLearn]

Well.. it seems I am outnumbered that Turbine could have done things better. Most of you dissers seem to think that "backup/restore" is the way to go here but that just tells me you are not up on current storage archetecture. An application as large as LOTRO must be using SAN storage which is perfect for replicating data (realtime) on multiple sites.... a write data on one site results in the same data getting written to the other). If they replicated data at both datacenters all they have to do is shutdown the online application at the old site and bring the new site online. This does not take twelve hours. For those of you still in "backup/restore" land there is still a use for backup but it's primary use these days is for protecting data. Using backup/restore to move data from one place to another is not applicable for critical online applications.

I am up on current storage architecture, but I can think of a number of reasons why backup/restore could still be preferrable. For instance, can you attest to the speed an reliability of the connection between the two datacenters? In any event, most SANs will not replicate changes in real time, they create a journal of changes and transfer them at scheduled times. Realtime copies of SANs have multiple potential problems.

Then you have the issues of IP/DNS propagation. For instance, did they move IPs from one DC to the other? Did they change DNS entries? Potentially some of both? They need time to make sure that the routing tables update properly, DNS changes propagate and verify everything is working properly.

Perhaps there are coding changes that needed to be made and or other maintenance they are doing at the same time? If you have a nice large downtime window, it may be a good time to reoptimize db tables, etcetera and further improve performance.

On top of all that, they will want to double and triple check that everything is running smoothly. What is worse than a long maintenance window? FUBARing it and having to move back to the old hardware, dealing with customer support calls, etc.

One final note, since the servers are up already, is that I much perfer they say we will be down from 4am-4pm and get the servers up early rather than saying 4am-10am and then have to update 2-3 times that they are extending the downtime.

[BerensBane]

The EULA which everyone signes in order to play is a way for Turbine to cover themselves from frivolous litigation.

Now.. under the covers, I am sure Turbine measures themselves on uptime and although some downtime is unavoidable, the less downtime they incurr the better they, as a business look.

Yeah.. no one's life is depending on LOTRO being online 24/7 but this IS a "critical application" if you look at it from Turbine's business objectives. If LOTRO stopped working, how long would their company/department remain in business? That makes it critical (to them). I know they could have planned this differently, switched over to the new servers in a minimal time (certainly less than 12 hours) and earned a big feather in their cap in doing so. Instead, once again, they show their true colors and provide a less than stellar service to the end users... us.

???? If its critical for them (ie their business/Jobs) but not critical for us (ie ... just a game) then why do you care? I'm sure they have the resources and capabilities to take care of their business ....

[Bewell]

???? If its critical for them (ie their business/Jobs) but not critical for us (ie ... just a game) then why do you care? I'm sure they have the resources and capabilities to take care of their business ....

...because I was dissed for using the term "critical application" in the first place..

I care because I do not like planning my next game experiance only to see the server status window when I try to log on.

Quote: I'm sure they have the resources and capabilities to take care of their business"

How do you know? Have you visited their datacenter? Have you reviewed their business plan?

The only thing we have to go on out here is whether the game is available when we expect it to be and how many and how long the downtimes are... and by those standards I do not rate Turbine very high. It's reasonable but certainly not world class.

[Squallalaha]

Pssst... we opened the servers 10 minutes ago.

Thanks,

lottery is up too.

[Bewell]

[QUOTE=YLearn;4259085]I am up on current storage architecture, but I can think of a number of reasons why backup/restore could still be preferrable. For instance, can you attest to the speed an reliability of the connection between the two datacenters? In any event, most SANs will not replicate changes in real time, they create a journal of changes and transfer them at scheduled times. Realtime copies of SANs have multiple potential problems.

Ask the New York Stock Exchange and other Stock exchanges around the world. Ask lotteries and many large hositals etc, how reliable the redundant connections between their "active/active" clustered datacenters are. I suspect you have not worked in a real critical environment with OS's other than windows. SAN's do replacate realtime data between multiple locations all the time as a normal business practice... we learned that from 9/11. Many of the businesses that didn't do that are not here today. There are many businesses that can remain online even if they loose an enture datacenter. This is the environment I consult in.. don't tell me it's not possible.

[Squallalaha]

I care because I do not like planning my next game experiance only to see the server status window when I try to log on.
...
The only thing we have to go on out here is whether the game is available when we expect it to be and how many and how long the downtimes are... and by those standards I do not rate Turbine very high. It's reasonable but certainly not world class.

It has been posted for DAYS in advance, that the servers would be down today.

Why dont you post here YOUR play schedule. I am sure that Turbine can adjust their future downtime around your play schedule so they can be seen in your eyes as "World-Class". Altough , correct me if I am wrong, but they are #1 in 'Middle-Earth', how many other Worlds do you want them to be #1 in? Azeroth?....

[Mithrandir3]

Armchair developers 0, Turbine +1.

Heh. As a developer myself, I read that comment, then read Turbine's response, and thought "man, this is why I'm not in PR..."


I totally woulda not been so nice.

[Bewell]

It has been posted for DAYS in advance, that the servers would be down today.

Why dont you post here YOUR play schedule. I am sure that Turbine can adjust their future downtime around your play schedule so they can be seen in your eyes as "World-Class". Altough , correct me if I am wrong, but they are #1 in 'Middle-Earth', how many other Worlds do you want them to be #1 in? Azeroth?....

Take your blinders off and look at the big picture... I refer you to the last downtime that got extended.. twice.

[BerensBane]

...because I was dissed for using the term "critical application" in the first place..

I care because I do not like planning my next game experiance only to see the server status window when I try to log on.

Quote: I'm sure they have the resources and capabilities to take care of their business"

How do you know? Have you visited their datacenter? Have you reviewed their business plan?

The only thing we have to go on out here is whether the game is available when we expect it to be and how many and how long the downtimes are... and by those standards I do not rate Turbine very high. It's reasonable but certainly not world class.

Hmmm... no I haven't visited their data center nor have I reviewed their business plan. [sarcasm] Perhaps you can enlighten us on your visit to the old and new turbine datacenters. I'm sure your just upset that they did not take your advice when they hired you as a consultant to evaluate their business plan and now your out to trash them on the forums [/sarcasm] .... Oh wait the servers are up 4 hours ahead of schedule ... guess you were wrong .... Don't worry be happy ... Go play the game ....

[Bewell]

Hmmm... no I haven't visited their data center nor have I reviewed their business plan. [sarcasm] Perhaps you can enlighten us on your visit to the old and new turbine datacenters. I'm sure your just upset that they did not take your advice when they hired you as a consultant to evaluate their business plan and now your out to trash them on the forums [/sarcasm] .... Oh wait the servers are up 4 hours ahead of schedule ... guess you were wrong .... Don't worry be happy ... Go play the game ....

No BB... it's nothing personal to me other than my time... however, if I was managing/implementing this migration, perhaps Turbine could have used this message instead: " Dear Users.. in order to execute a failover to our new UBER servers you may notice a momentary disruption in service. If you get disconnected, please log back in and continue your online experience. Please read "xxxx" for more information."

[Ignotus]

At least the Youtube video didn't link to a Rickroll.

[Bewell]

I guess everyone's back playing the game which I am going to do right after I get this in.
If anyone doubts that a business can remain functional with the loss of an entire datacenter.. here is a video for you to enjoy.

Happy Trails... it's been a fun discussion.

http://h71028.www7.hp.com/enterprise...solutions.html

[YLearn]

Ask the New York Stock Exchange and other Stock exchanges around the world. Ask lotteries and many large hositals etc, how reliable the redundant connections between their "active/active" clustered datacenters are. I suspect you have not worked in a real critical environment with OS's other than windows. SAN's do replacate realtime data between multiple locations all the time as a normal business practice... we learned that from 9/11. Many of the businesses that didn't do that are not here today. There are many businesses that can remain online even if they loose an enture datacenter. This is the environment I consult in.. don't tell me it's not possible.

Never said real time replication wasn't possible. Never said that it doesn't happen. What I did say is that it introduces a number of potential problems. Stop to remember that exchanges and trading floors are an entirely different beast than most entitites. For most companies, replication of the journal every 5-10 minutes (or even an hour or two) is more than sufficient, less costly to implement and avoids problems of real time replication.

As for active/active datacenters, who cares? In this case they are MOVING from one datacenter to another. They are not an active/active situation. Why would they incur the additional expenses of setting up potential multiple high speed links between the datacenters for a move? It may be they have sufficient bandwidth and reliability between the two datacenters, but frankly I don't know and I seriously doubt you do. That was my point.

The whole point of my post is that yes, they could potentially do it faster, but from my experience, there are many reasons to do things in many different ways. I do not know their exact situation or everythinng that is involved for them, nor was I involved in any of the planning of the project or know all the goals and taks to accomplish. So I am not going to claim I know the "best" way they should be doing things and that 12 hours is far too long. Additionally, the final point was that it is better to under promise and over deliver than the opposite.

Since you started the flames, let me just say that you must be new enough to IT that you believe that your hammer is the only tool in your box and the best tool for every job. Don't start on my experience as you have no idea and will lose on that front. Especially since your stongest point there seems to be the "Windows sucks you loser" card. I could list the platforms/companies/OSes/whatever that I am very familiar with, but I don't feel a need to throw around my credentials to feel better about myself, besides I think my first post well established that I know a bit more than the average bear.

So what if you consult? At least 80% of the consultants I have worked with in my lengthy IT career haven't been worth a tenth of their cost to the businesses that hire them. I will say that most consultants have a large plus associated with them from my viewpoint. They ensure I never have to worry about work, as I often come behind to fix what they poorly implemented in the first place. And since I have more experience and knowledge, I get paid more than they did while I do it.

[LordWeaver]

All I have to say is well done Turbine for a great transition period. Shaving off 3 hours off the time slot was great work. Thumbs up here! Lets just hope that there are no teething problems hehe.

[BerensBane]

No BB... it's nothing personal to me other than my time... however, if I was managing/implementing this migration, perhaps Turbine could have used this message instead: " Dear Users.. in order to execute a failover to our new UBER servers you may notice a momentary disruption in service. If you get disconnected, please log back in and continue your online experience. Please read "xxxx" for more information."

Except for this scenario ... shortly after you made your announcement, an unavoidable fiber break occurs between data centers. The local provider apologizes but it could take 4 to 6 hours to repair the break. In your scenario you have told your 1000's of users that they will have uninterrupted service. In the real world stuff happens and as stated before by others, Turbine is better off promising 12 and delivering 4 than promising a hiccup and then having egg on their faces. Always expect the unexpected is my motto ....

[Bewell]

Since you started the flames, let me just say that you must be new enough to IT that you believe that your hammer is the only tool in your box and the best tool for every job. Don't start on my experience as you have no idea and will lose on that front.

Ok Junior... since you had to drag me back into this with a personal attack.. here is my reply.

1967-1971 - Us Navy - Maintained missle systems radar and fire control computer maintenance. We used Oscilloscopes, DVM's and Data Analyzers back then
1972- 1973 - Honeywell Computer systems. Field Service Technician
1973-1974 - Data General - Nova disk repair
1974 - Current - Employed with a major computer manufacturer. Spent 5 years building and doing Final Acceptance and Test of mainframe systems. Went to the field as a service engineer. Three years later promoted to district support. Next went to regional support. Spent 5 more years doing "Availability Reviews" for the owners of some of the worlds major datacenters. The results being a ~200 page document covering all of the single points of failures in both the technical solution as well as the business plan and management of that plan. Currently doint 3rd lvl support of my companie's customers on a global scale. I consult as needed. Usually because one of our customers ask for me personally.

I feel I have sufficient background to discuss this topic on a technical level but am not interested in being dragged back in to adress a personal attack.

[Sapience]

We're done. Our Net Ops team deserves high praise for putting together a plan and executing it in such a fashion that the servers came up three and a half hours ahead of schedule. Enjoy the game everyone!