Data Center Update

[Daguest]

So unexpected. Turbine promise one thing, don't do it, go silent for months, and then claim it's better that way.
I'm shocked, really, but it's all in the inside, you can't see it.


Obviously having to cross an ocean, and numerous node is better than if it was in Amsterdam next door. Pure logic right there.

[matalan]

Dear Players,
Now that we have been in the new datacenter in New Jersey on the new hardware for several weeks now, we have learned a lot about how our service works with domestic and international ISPs. Early efforts at tuning have been showing progress, leading to measurable improvement in latency for both US and EU players. Based on this early information we believe that players will be best served by us focusing on improvements to the new datacenter, rather than moving the EU servers to Amsterdam.

Going forward we’ll continue to work on with the technology team at our datacenter as well as key regional ISPs to improve the performance of LOTRO for all players worldwide. The next step of this process will have us on a new routing system in the next few weeks. This is expected to improve performance for players in much of the EU, and certain parts of North America as well.

For those players experiencing issues in PvMP we are adding in client side optimizations with Update 18 that we hope will lessen the strain on client and server alike when in battles with large groups. We are also looking into the cause of client crashes when traveling and changing instances. We will continue to optimize the configurations of our servers to improve service stability. These efforts will not be over night, but should improve the situation over time so that your play experience will be even better than it was prior to the move.

Once again we thank you for your patience and understanding while we work through this construction.
Thank you,
Athena “Vyvyanne” Peters”

Sorry to hear that. Thing is, and forgive me, I don't believe your reasoning for not moving EU to Amsterdam. Myself and others pretty much predicted this was what was going on; you were simply playing for time so you could use the 'issues' of the new data center as the reason for not wanting to move EU to Amsterdam, rather than having the courage to say you had simply scrapped the move to Amsterdam even before the move to NJ. It's effectively a cover-up, a thin and obvious one at that.

Scrap the move to Amsterdam, but we've already promised it - how can we get away with that?
Either A - "the NJ servers are rubbish, we tell them we have to focus on those issues and don't want to risk moving to Amsterdam and everything falling to pieces"
Or B - "the NJ servers are awesome, so there's no NEED to move to Amsterdam anyway!"

Brilliant, either situation you could have decided BEFORE moving into NJ that you'd have valid reasons to use instead of the truth.... COST and or lack of capability.

Some could say this is all conspiracy theory, I understand that entirely, thing is either way I can't trust anything you say anymore, not after this and that's even IF you are telling the truth. Truth, or lies, you have massively f-ed up, and I didn't catch a sorry in that post of yours.

Remember when you said (paraphrased) "Don't judge us on the old team, we aren't them...." Well you've just proven you're not the same, but the results certainly are. Bravo!


So is the new raid still on the cards? Or is that going to magically vanish as you "believe players will be best served by us focusing on gimmicks and pretty unicorn horms for warsteads in the Store" Sorry to come over snide, but at this point literally nothing could make me trust a word you speak.


Thanks for the info though, really enlightening stuff.

[Spilo]

Welp, these PvMP lag adjustments are too little too late. In addition, Turbine continues to fail on individual class balance in the PvMp zone. Too little, too late- I just canceled my subscription.

[peachykins]

..............

[Ulanor_EU]

What a surprise, I am sure we are all shocked the promised Datacentre move isn't happening...

Wonder when these improvements they are making will actually get my latency down to even the latency I had on the old US servers as I am still having higher latency value since the move than I did before.

[Snowlock]

Look at all the players who left over the HD fiasco, did Turbine wake up & do something, nope. I don't see them changing.

THis is the truth. They have the IP, as long as they do, they've got a certain level of revenue built in. If this were a game with an original IP, handled in this way, I can't imagine it lasting. It's a good thing for those players who love touring the landscape of middle earth, it will probably be providing that opportunity for years to come, regardless if there's any updates or not. But if the player really wants anything more than that, it sure seems like they're SOL.

Vy seems like a LOTRO fan, as did Frelorn (has he left LOTRO or Turbine? It almost seems like we're on the cusp of a new CM announcement). But they are mismanaging their failures. I think they're victims here of over promising and under delivering, while they continue to let the open wound of bad performance, especially in the ettenmoors, fester.

[Reximus]

If All the servers are permanently staying in New Jersey now, is there any chance of amalgamating the servers so us non-US players have a wider choice of EN servers available to transfer to?

No, this was addressed a long time ago. It had to do with how Codemasters and Turbine stored account data that would allow that data to overlap if they tried to allow players to cross from EU to US or vice versa. This could result in data loss and/or accounts being merged.

[Ulanor_EU]

No, this was addressed a long time ago. It had to do with how Codemasters and Turbine stored account data that would allow that data to overlap if they tried to allow players to cross from EU to US or vice versa. This could result in data loss and/or accounts being merged.

despite the fact both lots of shared data can copy to bullroarer of course

[Macdui]

Originally posted 21 May 2015
"New Servers and Datacenters: We spoke about this Turbine wide upgrade at the beginning of the year and had expected to have it done by now, but weather and other obstacles set us back. We finally have some progress and we can safely say that by the end of September of 2015 we should have the servers moved to our new datacenters with upgraded hardware and Operating Systems. The US servers will operate from our new datacenter in New Jersey while our EU servers will move back to Europe and reside in our new datacenter in Amsterdam."

From Nov 9th 2015 it's been "datacenter" singular.
In the interim Infinite Crisis was switched off.

ref:
https://lotro-wiki.com/index.php/Pro...-_January_2015
https://lotro-wiki.com/index.php/Pro...ter_-_May_2015

It took them 2 days to realise (be told) that the original premise to get us playing on geographical closer servers wouldn't work for many.
Players were advised to start afresh if they wanted a closer server.

Piaz stuck the knife in and Peters cut the artery taking it out.

[maartena]

It seems pretty clear to me that EU players are a pretty small portion of the overall population- just like raiders, and PvMPers, and crafters, and role-players, and housers, and festivalers.

I think you are wrong there, otherwise they would no longer support 5 EU servers, of which 2 in German, 1 in French. There are obviously plenty of players in the EU. Evernight numbers have consistently shown to be equal, if not more to Brandywine in daily/hourly logons. The players are there for sure. Perhaps some of those are players from other continents than Europe, but the high population on the German servers does suggest that there are more than plenty in the EU to warrant a server move to the EU.

I too am disappointed. Sure, the few toons I moved from Snowbourn to Evernight will experience less lag because they stay in the US, but I hardly ever play those, and they are low level. I am disappointed for the EU players who were promised a server move to the EU, only to see that promise broken.

I am also not surprised. I saw this coming. And I also believe that the reasons are not technical alone..... Cost may have something to do with it as well - although of course I do not have a good overview of what they have been quoted for Amsterdam.

As for Australian players: They are better served with servers in the US, as the NA continent is closer for them than Europe is.... so they should probably applaud this if they have toons on EU servers. Still, it is a shame that they came back on their original promises.

[Srale]

I think you are wrong there, otherwise they would no longer support 5 EU servers, of which 2 in German, 1 in French. There are obviously plenty of players in the EU. Evernight numbers have consistently shown to be equal, if not more to Brandywine in daily/hourly logons. The players are there for sure. Perhaps some of those are players from other continents than Europe, but the high population on the German servers does suggest that there are more than plenty in the EU to warrant a server move to the EU.

I too am disappointed. Sure, the few toons I moved from Snowbourn to Evernight will experience less lag because they stay in the US, but I hardly ever play those, and they are low level. I am disappointed for the EU players who were promised a server move to the EU, only to see that promise broken.

I am also not surprised. I saw this coming. And I also believe that the reasons are not technical alone..... Cost may have something to do with it as well - although of course I do not have a good overview of what they have been quoted for Amsterdam.

As for Australian players: They are better served with servers in the US, as the NA continent is closer for them than Europe is.... so they should probably applaud this if they have toons on EU servers. Still, it is a shame that they came back on their original promises.

maartena I think bastiat1 was sarcastic :-)

[Schinderhannes]

Stuff that keeps me away from this game, or make it really hard to enjoy playing it atm:

Broken Mounted Combat
I hoped the server transfer to amsterdam would have a positive effect on the sluggish and buggy movement mechanics.

Game Design
Really hard to describe in a few words, but I think the game needs a polished item progression and a motivating and in some ways more immersive and coherent itemization. (low lvl imbued weapons, Essence-Gear-Crafting, mixed sources for max/best gear etc)
I also miss new, modern gameplay with more sandbox elements and general agency. Instead, the game repeats its old back and forth quests design over and over and over...

Invisible weapons bug
Can't believe it is still a thing.

and some other stuff... (User Interface, difficulty/balancing, dungeon design, GRIND)

I know the game since July 2007
I will not rest until the Ring gets destroyed! (maybe )

I really want to make it with you to mordor!

[Finranduil]

Welp, this sucks. Really hoped for a move. But things change over time. And it's not like we had servers in Europe for the past few years... Been playing on servers that are US based ever since I started this game. I hope you manage to reduce lag/latency a bit for the EU players

Lorim

[maartena]

With regards to the New Jersey datacenter only decision:

They made that decision last year. You don't develop two environment that are geared to hosting 5 servers each, and consolidate them into one location geared to hosting 10 servers. They claimed they had hardware in Amsterdam, and they probably did have leased hardware.... but you don't just decide a week, or even 2 weeks before the anticipated move date on Jan 11 that they decided to "start" with New Jersey.

I believe the reason the move got pushed back, and pushed back again.... was indeed because that decision had been made MONTHS ago, and they now needed to increase the capacity to be able to host all the hardware for 10 game servers as opposed to just 5. (And, obviously, Bullroarer, Palentir, and other internal test servers). DOUBLING the amount of capacity needed for servers requires planning. You need double the amount of switch ports, double the amount of bandwidth, double the amount of IP addresses, double the amount of rack space, CPU's, storage space, double everything to be able to host all 10 game servers.

They had to have started planning for this in October or earlier, and probably still had hoped to finish before Christmas, which obviously did not happen. This was not a sudden change of heart, this was a well calculated decision that was made probably in Q3 of 2015, and had to be planned and thought out, designed, and implemented.

I'm not personally affected by the EU servers moving or not moving, but Turbine should not claim that this was a "sudden" change "because it works better" in New Jersey. You don't just move 10 servers worth in an environment that was designed for 5 US servers, even if they would want to to cut costs or something, they needed to have the capacity ready and built weeks in advance to do this.

[Macdui]

With regards to the New Jersey datacenter only decision:

They made that decision last year. You don't develop two environment that are geared to hosting 5 servers each, and consolidate them into one location geared to hosting 10 servers. They claimed they had hardware in Amsterdam, and they probably did have leased hardware.... but you don't just decide a week, or even 2 weeks before the anticipated move date on Jan 11 that they decided to "start" with New Jersey.

I believe the reason the move got pushed back, and pushed back again.... was indeed because that decision had been made MONTHS ago, and they now needed to increase the capacity to be able to host all the hardware for 10 game servers as opposed to just 5. (And, obviously, Bullroarer, Palentir, and other internal test servers). DOUBLING the amount of capacity needed for servers requires planning. You need double the amount of switch ports, double the amount of bandwidth, double the amount of IP addresses, double the amount of rack space, CPU's, storage space, double everything to be able to host all 10 game servers.

They had to have started planning for this in October or earlier, and probably still had hoped to finish before Christmas, which obviously did not happen. This was not a sudden change of heart, this was a well calculated decision that was made probably in Q3 of 2015, and had to be planned and thought out, designed, and implemented.

I'm not personally affected by the EU servers moving or not moving, but Turbine should not claim that this was a "sudden" change "because it works better" in New Jersey. You don't just move 10 servers worth in an environment that was designed for 5 US servers, even if they would want to to cut costs or something, they needed to have the capacity ready and built weeks in advance to do this.

You are somehow under the illusion that they did double up! Do we know this, if performance is anything to go by?

Mac

[Thoroval]

despite the fact both lots of shared data can copy to bullroarer of course

From my experience, copies to BR are just a tiny fraction of all created characters. I'd speculate well below 0.01% (Turbine would have numbers). Even those few copies to BR fail due to data conflicts. I've personally experienced a failed character transfer to BR within the first 24 hours of a database wipe. That's very likely because a character from the EU was already copied that had something in inventory with a conflicting ID. It only takes one single conflict to cause a copy/transfer failure.

Now imagine trying to allow for the merging of all data from all characters from all worlds. I'd hate to even begin to calculate the possible number of conflicts. It would certainly be way beyond the ability to correct via manual means. And manually is how Turbine is currently handling failed transfers due to data conflicts within regions and they can't even correct those failures without falling months (months!) behind on their support tickets.

No, Turbine would have to engineer an automated solution that would essentially allow the conversion and merging of all the data. I don't have any insight into their schema, but I do have some small amount of knowledge and experience in database design. It is likely technically possible, but also likely at tremendous cost. Far more cost than they'd ever hope to recover from their Premium Service fees and they obviously don't have the operating cash to just absorb that cost.

Transferring between regions is something that I truly wish Turbine had accounted and planned for, but unfortunately it appears to be way, way too late for that now. All that said, Turbine has surprised me with features that they swore up and down in the past were technically impossible.

[ShireDweller]

You are somehow under the illusion that they did double up! Do we know this, if performance is anything to go by?

Mac

Game performance since the move does feel like trying to stuff 10 pounds of ..erm.. "droppings" into a 5-pound sack, doesn't it?

[Valindor]

After reading a lawyer crafted reply from the community manager previously......Knew they were not going to do anything to try and hang on to the EU players.............By moving the servers

Ettenmoors is unplayable !!!!!!

If I was a bookie I would not be offering odds on the playability of a PVE raid due to lag etc..........

Disappointed that you could be straight with us the community. But this back up a point that the majority of my gaming mates (20-30) views of Warner Brothers controlled games, they avoid like the PLAGUE! WB have an appalling rep in the wider gaming community. End of the day word of mouth etc...........

[Dadislotroguides]

I am going to remain optimistic (cue trolls) that Turbine will get the data centers sorted out, and performance will be better than when they started on this venture.

I really do not think that people appreciate the monumental tasks that were involved with this move/upgrade; issues were inevitable. Am I happy that it is taking as long as it is to fix the lag, disconnects, etc. that players are experiencing; absolutely not !!! However, I have seen progress and it sounds from this announcement that U18 will bring more performance.

The revenue stream is no where close to WoW, so people need to stop comparing LOTRo to it; they are apples and oranges. Speaking of which, if anyone thinks WoW is without lag, crashes, etc. they need to do their homework. Their forums are riddled with complaints about performance just like every other MMO out there. There is no such thing as an MMO with awesome performance, great storyline, fantastic community, superb mechanics, low cost, and longevity.




As for the EU servers, it is no surprise that after the issues they faced in Jersey that they are reluctant to move servers again to the EU (and it looks like the decision may have been made prior to that). I am certain there are also business reasons to keep them stateside. Turbine is a subsidiary of Warner Brothers. Warner Brothers decides how much funding Turbine gets for LOTRo, and from there Turbine decides how to allocate it to their resources. If you want to hate on someone/something, direct it to WB. The Turbine team is immensely talented but under funded IMHO.

They definitely need to do a much better job with how they communicate, no arguments from me there.

[Arnenna]

Well turbine, you got me on this one. I was one of the players banking on your delivery on this, and hoping naysayers were wrong.

Very very disappointing indeed.

While I'm seeing improvement for most part since a week after the server move to NJ, it's still a million miles from the performance I had when we were in Boston.

If you can get it back to what it was then, - fine.
If not, then you really need to reconsider the opening statement on this thread because, nobody (and I mean nobody) is ever going to be willing to pay the same money for a lesser service.

You promised Amsterdam to the EU players - you should deliver exactly that.

[kagorsa]

You know you could just wait until the end of turbines financial year, and look at their financial records, which would show that they did or did not spend currency on a data centre in Amsterdam.

[Hurin]

Well, don't we all feel silly now. I mean, all this hub-bub about a new EU datacenter when all along it's been the case that the NJ datacenter is just as good!

One wonders why there were ever plans to move servers to Amsterdam in the first place!

[MessyR]

The only way to hurt turbine is to stop playing, most won't, so the lies will continue.

[Thorgrum]

Well, don't we all feel silly now. I mean, all this hub-bub about a new EU datacenter when all along it's been the case that the NJ datacenter is just as good!

One wonders why there were ever plans to move servers to Amsterdam in the first place!

I dont feel silly. I genuinely hoped the EU players got a data center, they had to deal with the codemasters consolidation that was negative enough. Also, this new team at lotro have, mostly, delivered on things I never thought would come to fruition (new pvp map, creep mail, LI's that grow with you, crafting at your house). Sure we can argue the viability of those things but they are now "in game".

So it wasnt a stretch for me anyway to think that when she said they were moving them -at the same time- that it was an honest statement.

Had Paitz or campbell made the statement and we all believed it, ya we would have all been silly fools.

I mean Im not shocked at all, but the hope wasnt misplaced based on current conditions.

[maartena]

Stuff that keeps me away from this game, or make it really hard to enjoy playing it atm:

Broken Mounted Combat
I hoped the server transfer to amsterdam would have a positive effect on the sluggish and buggy movement mechanics.

Game Design
Really hard to describe in a few words, but I think the game needs a polished item progression and a motivating and in some ways more immersive itemization.
I also miss new, modern gameplay with more sandbox elements and general agency. Instead, the game repeats its old back and forth quests design over and over and over...

Invisible weapons bug
Can't believe it is still a thing.

and some other stuff... (User Interface, difficulty/balancing, dungeon design, GRIND)

I know the game since July 2007
I will not rest until the Ring gets destroyed! (maybe )

I really want to make it with you to mordor!

They leave stuff unfinished.

Someone builds a new Bree in their spare time.... and he got canned, no one picked it up.
The person that developed the hobby system got canned, and no one picked it up.
The level 50 mobs in the Northern Low Lands were supposed to be a zone with quests, no one picked it up.

There are all sorts of pet projects that were started, but never finished.... some of which were exciting things that players generally really loved, such as the new Bree.

Message to Turbine: FINISH things before you do new things.