A Word on Hardware Improvements

[Celebrimborn]

Hi Bewell,

That's exactly what we did. Today is the cutover to the new hardware that's already been set up in the new datacenter. Thanks for your input!

-Meghan

Me thinks you just got delt. P

[Sonsy]

I am not sure that Turbine realizes that they have a critical application and we are paying for 24/7 access. You sure don't treat it like one.

Heh...actually, it's just a game. I wouldn't call it a "critical application" by any means. No nuclear missile silos or brain surgery going on here. It's actually much more important to them (their jobs depend on it) than it is to us. I might have to turn on the TV or take a walk or do something else to entertain myself. Oh well.

You are paying for the right to access it whenever it is available, nothing more.

[MrBloodworth]

Hi Bewell,

That's exactly what we did. Today is the cutover to the new hardware that's already been set up in the new datacenter. Thanks for your input!

-Meghan

Armchair developers 0, Turbine +1.

[Seedly]

Ah folks... I hate to be a stick in the mud but if you are upgrading your servers and at the same time populating a new data center it would have been better to put new servers in the new data center and get them running while we users are enjoying the game on the old servers... then switch over to the new servers during a much shortened downtime. I am not sure that Turbine realizes that they have a critical application and we are paying for 24/7 access. You sure don't treat it like one.

Everybody's an expert...

You know, Turbine seems to do a damn fine job in terms of uptime if you ask me. I didnt remember there ever being anything in the EULA about "COMPANY promises to maintain X% server availability". But you know what, they still keep the things up a vast majority of the time. I know they probably wont post actual numbers, but I bet planned uptime is over 99% and total uptime is over 98% (for perspective, thats 87 unplanned hours down and 174 planned hours down respectively, with this major upgrade only taking 12 planned hours). What do you want? Perfection?

Besides, its not like you can do a rolling backup of this kind of content without causing more issues than you solve. What happens when you log in during the backup towards the end after your character had been rolled to the other server but before those shiny new servers go live? Your gonna loose all the xp, loot, and quest completion during that time.

Besides, you and I dont know what database system they use or what the data actually looks like (as in table structure, schemas, etc). Its entirely possible that allowing people to play while rolling over could break characters entirely if part of the character's data was moved while that person was logged in.

So a sysadmin has to choose between a rolling backup that will most likely result in broken characters, spotty access, and 'lost' playtime (not to mention taking much longer), or they can shut it down entirely during low system load times (4am to 4pm, when the vast majority of the world is either asleep or at work), and make a fast and clean break to a new set of servers.

Which would you choose?

/Was planning to play this morning during some downtime, but lashing uselessly at trolls was almost as fun
//Hey Turbine, thanks for the upgrade. Yall do a good job!

[Kusa99]

I cant login whats going on

[Angadan]

Hi Bewell,

That's exactly what we did. Today is the cutover to the new hardware that's already been set up in the new datacenter. Thanks for your input!

-Meghan

Heehee, here's the Un-Patience (Impatient?) translation: Bewell, please don't think that everyone in the industry is dumber than you, because we're actually at least as smart, prolly a whole lot smarter (since you're sitting around armchair quarterbacking us). Love, Meghan.

Patience is so nice, she really earned her Forums nickname.

[gildhur]

I cant login whats going on

[Draugrim]

This is some good news -- perhaps the days of Raid v. Raid are not gone after all... I feared people would forgo PvP completely with the addition of skirmishes.

Either way, it'll be nice to see things running smoother across the board. Two thumbs up Turbine. (Did the Ettenmoors get their own server blade? :O )

[gradivus]

Tuesday December 15th will be a big day for The Lord of the Rings Online™, as we complete our most substantial hardware upgrade since the game launched in North America in April 2007.

Well, what are the new specs? They promised comments from you. What's the new hardware configuration you're using? What platform/servers, how many, and with what capacity?

_____________________

[Valendros]

Ah folks... I hate to be a stick in the mud but if you are upgrading your servers and at the same time populating a new data center it would have been better to put new servers in the new data center and get them running while we users are enjoying the game on the old servers... then switch over to the new servers during a much shortened downtime. I am not sure that Turbine realizes that they have a critical application and we are paying for 24/7 access. You sure don't treat it like one.

I'm confident that's what they DID do. But I'm sure they didn't upgrade every single tiny server. I've been part of many-a-server relocation & upgrade projects and you always get everything as set up as you possibly can before you hinder anybody's ability to work (or in this case, game). But there's always straggler servers or services in general that don't get upgraded, and those still need to be moved.

Also, if they upgraded the character database server (which would be a good bet, with the time it takes to open up the completed quest selection at level 60), they'll be taking most of the 12 hour downtime making 2 comprehensive backups (with double-verification each), replicating the data, then verifying the data on the new server. Since, when you think about it, the character data is their money, they're not going to be taking any chances. And the best way to not take chances, is to kick everybody out for 12 hours while it does a cold, hum-drum, bit-by-bit, no suprises, low risk replication.

[Valendros]

Anyway, I want to thank Turbine on giving us this new hardware. A lot of companies (you know who you are!) take money from content packs and paid features and pad their bottom line, especially in this economy. But you guys are doing the right thing and giving back to your customer, and I appreciate that.
Thank you.

p.s. if you guys can get it cleared… I wouldn’t mind seeing a couple pictures of the new datacenter… ::drool::

The old datacenter looked like this...

didn't it. don't lie now. All 'before' datacenters look like that.
(especially when the admins find out they're getting new hardware... god I miss that new hardware smell)

[Forestmist]

Exciting and fabulous that the servers are being upgraded!!! Thank you Turbine. I am looking forward to the improved performance and that the lag issues are being addressed. It is fantastic that Turbine is continually making changes and upgrades to the software and infrastructure. From a 2 lifetime account holder perspective I think it says a lot about Turbine's commitment to the user base!! Way to go!!!

[Comstrike]

They did release a video illustrating the change in the LOTRO YouTube channel.

http://www.youtube.com/watch?v=PUiO7Ouon9c

[Briannas]

I am so amazed how the best just keeps getting better.... Lotro Rocks.

Tuesday December 15th will be a big day for The Lord of the Rings Online™, as we complete our most substantial hardware upgrade since the game launched in North America in April 2007. This new server hardware gives us additional speed and capacity to support the growing world of Middle-earth, from the battles of Mirkwood to new Skirmishes across Eriador. It also provides the room to support the growing population of Middle-earth – there has been a huge response to the launch of Siege of Mirkwood™, and we are delighted to see both returning players and new explorers to the world. Thank you for your patience, and we look forward to opening the doors again once this transition is complete.

[Bewell]

Well.. it seems I am outnumbered that Turbine could have done things better. Most of you dissers seem to think that "backup/restore" is the way to go here but that just tells me you are not up on current storage archetecture. An application as large as LOTRO must be using SAN storage which is perfect for replicating data (realtime) on multiple sites.... a write data on one site results in the same data getting written to the other). If they replicated data at both datacenters all they have to do is shutdown the online application at the old site and bring the new site online. This does not take twelve hours. For those of you still in "backup/restore" land there is still a use for backup but it's primary use these days is for protecting data. Using backup/restore to move data from one place to another is not applicable for critical online applications.

[Rumfigger]

They did release a video illustrating the change in the LOTRO YouTube channel.

http://www.youtube.com/watch?v=PUiO7Ouon9c

This link took me to a video of hamsters running in wheels.

[Galadh]

I liked Boxer's comment on SETI (had to know current news events to get it)

[ju-ju]

So now what do we do about bring the freeps back to the moors...You took away the chief badges from the SOP..thet got them to come in there the most.. Well now there is hardly any freeps at all in the moors..So far i find the moors to be BORING. You need to do something fun and interesting..Like open up a town for us for a day or 2 or 3 ,,random..But if you let us in the town you make it possible that the Creeps are unable to attack 39 and below ..and they cant attack us..40 and above woud be open game..i think it would be fun ...Yes there would be some to complain but that is there nature..This is a game of fun and not real life.Yes there are some problems you all wont fix ...like the Creep mapping problem we had alot of us lost some of the experience of the maps ..As in A Creep would be working on there good map to TA for example and we have to start at like maybe 10 instead of 25 depending on how many Bad quests we turned in . i know that may not be important to turbine ..but that is what makes the game boring when you do not fix things like that..And you get a comment like oh we are looking into it ..How long does that take.. I know this game is for fun but when you get GM's that dont care thats a big proble...and yes i know that pvp is not the first thing on your list you better at least look at some of the Problems that you have or you get..Pvpers pay there money just like the pveier's. maybe you should look at why people have left this game..And not because they went to Aion or another PVP game but why they left Turbine. This is a great game BUT you might need to look at how your GM's take care of there customers.. I do have to say thank you to the GM that gave me my missing goat...Now that was the best Customer service i had ever in this game..thanks again to him.

[Squallalaha]

Thanks for the update:

We will also be running a special “Moving Day Lottery” on my.lotro.com during the move. Be sure to sign up for your chance to win some great in game loot!

When can we sign up for the lottery?

[Unique]

I am not sure that Turbine realizes that they have a critical application and we are paying for 24/7 access. You sure don't treat it like one.

Wrong, you acknowledged it not being available all the time when you clicked I agree on the Terms of Service Agreement.

14. TURBINE DOES NOT WARRANT THAT ACCESS TO OR USE OF THE GAME WILL BE UNINTERRUPTED OR ERROR-FREE, THAT YOU WILL BE ABLE TO ACCESS THE GAME AT ANY TIME OR IN ANY GEOGRAPHIC AREA, OR THAT THE GAME WILL MEET ANY PARTICULAR CRITERIA OF PERFORMANCE OR QUALITY.

[Kusa99]

My game wont load

[BerensBane]

So now what do we do about bring the freeps back to the moors...You took away the chief badges from the SOP..thet got them to come in there the most.. Well now there is hardly any freeps at all in the moors..So far i find the moors to be BORING. You need to do something fun and interesting..Like open up a town for us for a day or 2 or 3 ,,random..But if you let us in the town you make it possible that the Creeps are unable to attack 39 and below ..and they cant attack us..40 and above woud be open game..i think it would be fun ...Yes there would be some to complain but that is there nature..This is a game of fun and not real life.Yes there are some problems you all wont fix ...like the Creep mapping problem we had alot of us lost some of the experience of the maps ..As in A Creep would be working on there good map to TA for example and we have to start at like maybe 10 instead of 25 depending on how many Bad quests we turned in . i know that may not be important to turbine ..but that is what makes the game boring when you do not fix things like that..And you get a comment like oh we are looking into it ..How long does that take.. I know this game is for fun but when you get GM's that dont care thats a big proble...and yes i know that pvp is not the first thing on your list you better at least look at some of the Problems that you have or you get..Pvpers pay there money just like the pveier's. maybe you should look at why people have left this game..And not because they went to Aion or another PVP game but why they left Turbine. This is a great game BUT you might need to look at how your GM's take care of there customers.. I do have to say thank you to the GM that gave me my missing goat...Now that was the best Customer service i had ever in this game..thanks again to him.

Easy now ju-ju ... Easy big fella ... drink another gallon of java and re-type again please ...

[ButrBiscut]

Well.. it seems I am outnumbered that Turbine could have done things better. Most of you dissers seem to think that "backup/restore" is the way to go here but that just tells me you are not up on current storage archetecture. An application as large as LOTRO must be using SAN storage which is perfect for replicating data (realtime) on multiple sites.... a write data on one site results in the same data getting written to the other). If they replicated data at both datacenters all they have to do is shutdown the online application at the old site and bring the new site online. This does not take twelve hours. For those of you still in "backup/restore" land there is still a use for backup but it's primary use these days is for protecting data. Using backup/restore to move data from one place to another is not applicable for critical online applications.

I see a lot of text here, but nothing acknowledging that Turbine might need/want to do some testing as part of the move. I would posit that much testing has already been done in the new datacenter while preparing for today's events. Once the game data is ready to go, regardless of which method(s) Turbine may have used to actually move all of it, spending as much time as is deemed necessary to do some additional testing is the smart way to go.

When Turbine upgraded to SoM, they didn't give us an expected time of completion. They didn't want another MoM launch on their hands, and I can understand why. Everything has gone much smoother with SoM, and if that trend continues today, then it's entirely possible we'll be back up and running earlier than expected.

[Patience]

Everything has gone much smoother with SoM, and if that trend continues today, then it's entirely possible we'll be back up and running earlier than expected.

Pssst... we opened the servers 10 minutes ago.

[Bewell]

Wrong, you acknowledged it not being available all the time when you clicked I agree on the Terms of Service Agreement.

The EULA which everyone signes in order to play is a way for Turbine to cover themselves from frivolous litigation.

Now.. under the covers, I am sure Turbine measures themselves on uptime and although some downtime is unavoidable, the less downtime they incurr the better they, as a business look.

Yeah.. no one's life is depending on LOTRO being online 24/7 but this IS a "critical application" if you look at it from Turbine's business objectives. If LOTRO stopped working, how long would their company/department remain in business? That makes it critical (to them). I know they could have planned this differently, switched over to the new servers in a minimal time (certainly less than 12 hours) and earned a big feather in their cap in doing so. Instead, once again, they show their true colors and provide a less than stellar service to the end users... us.