Data Center Update

[maartena]

I've found that the best way to eliminate political debates in /world is to bring up another topic (preferably game-related) that distracts people from it.

That helps sometimes. Or people try to diffuse the situation by shouting back: Bilbo Baggins 2016 - Free Pies for everyone! - or something somewhat game related. It helps, but there is always trolls that insist on continuing to issue their political opinions, and the heated elections are not helping.

It's not just politics either.... during the superbowl we had an individual who was calling virtually every play of the game. If we wanted to know about what was happening in a sports match that is.... you know, broadcast for free everywhere, don't you think we would have gone and watched it ourselves? He was told that several times by several people, but continued nonetheless. Of course he wasn't breaking any terms of service, and since he never alted (or so it seemed) an ignore did the trick there....

Of course you can't expect harsh moderation and NO offtopic chat..... That's why I would want to suggest to split it in two channels: /world for everything game related, and /offtopic for everything not game related. Make both channels auto-join, and set a rule that anything not game related should go into /offtopic. Now make a "wrong channel for content" report option, and maybe it can be dealt with that way. People can leave the /offtopic channel, and those wanting to discuss offtopic stuff should not do it in /world.

If that can be done, together with an account-wide ignore, I think you solve 90% of problems.

[GerkinPickle]

We are looking into all of it. If there is something that will tweak performance in a positive direction without destroying the integrity of the game, we are looking at it whether it is new since the move or from before. We have found that the move has exacerbated issues we had before but we infrequent or not as noticeable until we performed our upgrades. So we are looking into these things as well. I dont see us doing a complete overhaul on our game engine code as one player mentioned. The cost in time and resources as well as stability risk there would be very high, but there may be adjustments that can be made without that needing to be necessary.

Thanks for the updates and honesty. In my most humble opinion, Turbine's best friend right now is honesty. It disarms haters and people will want to help rather than complain.

[GerkinPickle]

Regarding client side issues: I think something has changed lately in the client as well and who knows, it could be related to the server (or what we think are server) issues.


Basically, my CPU spins up for no reason whatsoever lately when the game is running. Even during the character selection screen, my CPU runs at 15-20%. The weirdest thing of all, much like the character selection screen, if my character in game is not moving, the CPU again spins up and as soon as I move or start panning the camera, it calms back down.

I'm pretty sure I would have noticed this in the past because the fan kicks up a few notches now to the point where it's distracting.

[Snowlock]

That's why I would want to suggest to split it in two channels: /world for everything game related, and /offtopic for everything not game related. Make both channels auto-join, and set a rule that anything not game related should go into /offtopic. Now make a "wrong channel for content" report option, and maybe it can be dealt with that way. People can leave the /offtopic channel, and those wanting to discuss offtopic stuff should not do it in /world.

If that can be done, together with an account-wide ignore, I think you solve 90% of problems.

I often get you mixed up with the other Mar because you both are well meaning, but both your suggestions always carry with it a perfect world requirement and so they never ever would work.

If it's not moderated, which I agree it shouldn't be (for the average discussion) then it won't matter how many channels you create, one of them will just become the dumping ground for all of the topics, like /world is now for trade.

If you want to moderate but not moderate a global channel (keep it on topic but not get all dystopian with the participation), then you make it subscriber only. If you cut out the F2P'er, you're going to cut out the kids, and it's the kids who are the ones spamming /world with nonsense because they lack the maturity to see how stupid their topics are. They can read it, so they can respond to the lff stuff, but that's it.

[Freawaru]

For those players experiencing issues in PvMP we are adding in client side optimizations with Update 18 that we hope will lessen the strain on client and server alike when in battles with large groups.

Is that so, eh? Some thoughts for you guys, if you haven't already considered them:

1. PLEASE, FOR THE LOVE OF ERU, GIVE US A UI OPTION TO DISABLE THE DISPLAY OF BUFFS/ACTIVE SKILLS/STANCES/ETC. ON TARGETED CHARACTERS. Target forwarding helps reduce the lag somewhat, but not for the poor player through whom we are target forwarding.

2. Speaking of buffs, please consider removing keep/OP/Delving buff icons from individual character portraits. They're completely unnecessary. Relocating the buff icons to something that's similar to the PVMP+ bar would work better and reduce buff icon-related UI load: http://www.lotrointerface.com/downlo...o728-PvMP.html

3. DISABLE MULTIBOXING IN THE MOORS. One player should not be able to add a half a raid's worth of lag to the area! I'm so tired of crashing because I've been targeted by six minstrels or six RKs at the same freaking time, knowing that they're controlled BY ONE PLAYER.

[Father_of_Tillogrim]

traceroute to gls.lotro.com (198.252.160.30), 64 hops max, 72 byte packets
1 192.168.1.1 (192.168.1.1) 4.118 ms 1.024 ms 0.998 ms
2 79-69-128-1.dynamic.dsl.as9105.com (79.69.128.1) 5.743 ms 6.082 ms 6.163 ms
3 host-78-151-224-3.as13285.net (78.151.224.3) 9.063 ms 10.058 ms 8.252 ms
4 host-78-151-225-144.static.as13285.net (78.151.225.144) 8.755 ms 8.026 ms 8.020 ms
5 host-78-144-12-111.as13285.net (78.144.12.111) 15.323 ms * 18.074 ms
6 host-78-144-11-22.as13285.net (78.144.11.22) 15.840 ms 15.154 ms 14.922 ms
7 ix-8-0.thar1.lrt-london.as6453.net (195.219.100.77) 15.652 ms 15.221 ms 15.771 ms
8 if-3-2.tcore2.ldn-london.as6453.net (80.231.62.73) 86.369 ms 86.047 ms 85.961 ms
9 if-26-2.tcore1.ldn-london.as6453.net (80.231.62.58) 86.554 ms 87.814 ms 86.534 ms
10 if-8-2.thar1.njy-newark.as6453.net (66.198.70.173) 87.917 ms 88.121 ms 86.972 ms
11 if-5-2.tcore1.n0v-new-york.as6453.net (66.198.70.82) 85.729 ms 84.351 ms 84.964 ms
12 63-235-41-97.dia.static.qwest.net (63.235.41.97) 87.246 ms 88.156 ms 86.008 ms
13 ewr-cntr-11.inet.qwest.net (205.171.17.2) 86.666 ms 88.360 ms 86.706 ms
14 206.103.215.50 (206.103.215.50) 86.971 ms 110.707 ms 103.619 ms
15 63.236.3.130 (63.236.3.130) 85.830 ms 86.203 ms 85.029 ms
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *

[Munzel]

Hey, the Brand New EU Datacenter with Extreme good Routing with 5 EU Servers at Amsterdam, don`t forget it, i have told You...

[Milogson]

Dear Players,
Now that we have been in the new datacenter in New Jersey on the new hardware for several weeks now, we have learned a lot about how our service works with domestic and international ISPs. Early efforts at tuning have been showing progress, leading to measurable improvement in latency for both US and EU players. Based on this early information we believe that players will be best served by us focusing on improvements to the new datacenter, rather than moving the EU servers to Amsterdam.

/snip

Once again we thank you for your patience and understanding while we work through this construction.
Thank you,
Athena “Vyvyanne” Peters”

Dear Vyvyanne,

a few questions (yes, I may be naive to still have hopes for a move to Amsterdam at some time in the future).
And I am asking for clear, concise answers:

1) Does Turbine still intend to move the EU-servers to Amsterdam AT ALL?

2) If #1 is a "yes" is your quoted information to be interpreted as "Yes, Turbine still intends to move the EU-servers to a European based data-center AT SOME TIME IN THIS YEAR, yet currently we are simply looking to patch up the NJ data center disaster first"?

3) Is the move of EU-servers to Europe still part of this or the next quarter year's production plan?

4) If there is NO EU-data-center planned anymore, why was this not communicated more clearly and earlier?
If you are required to evade the truth due to company policy, please simply reply with "Aaargh" to this question.


Consider this an instructional manual for honest, truthful, non-corp-speak communication.

Have a 'thank you' back, completely unused and recycled, for hopefully truthful and clear concise answers.
A PO'ed soon-to-be ex-customer.

[maartena]

Regarding an EU datacenter, I don't think it is going to happen...... Of course it is speculation, but you don't just put 10 servers (which probably means anywhere from 10-15 pieces of hardware and/or virtualized hardware per server) in an environment that was initially designed for just 5 servers.

They would have to plan for this, both from a physical space point of view, as well as bandwidth availability, available cage power, available cage racks, available switches and ports on those switches, cabling, storage, amount of memory in virtual hosts, etc.... you pretty much need double that. The decision to not move to the EU was NOT made in early January, RIGHT before the January 11th move.... it was probably made in Q3, or at the latest in early Q4 2015, so they had 2-3 months to prepare, to build out the environment to host 10 servers instead of 5, etc.

I guarantee you that decision was made MONTHS before it was announced. As to why Turbine decided to not communicate that at that time, I don't know..... My personal opinion is that it would have been much better to communicate this WAY before the move was actually done..... saying: Look, we looked into Amsterdam but the tests on Bullroarer showed it will be better to keep everything in one datacenter. If they had done that THEN, right after the Bullroarer tests, I think it would have been received a lot better.

I know this was a COMPLICATED move, with resources being migrated from physical to virtual, AND from 32 to 64 bits, AND on new hardware, AND on new provider links, so there is a lot of things that can go wrong... but I also feel Turbine could have handled it a lot better. They really should have hired a group of temp-engineers to do some of the trouble shooting and configuration.... At my work we just hired two Cisco TEMP engineers to setup a few networks in China that have a deadline, because our internal staff isn't able to meet the date. That is just stuff you DO to make sure projects get delivered on time.... I know Turbine probably has a lot more budget-constraints then the company I work for, and Turbine (not WB) is a lot smaller than the company I work for, but I think a lot of things could have been handled better.

I am disappointed for the EU players, because that is the biggest thing you guys were looking forward to, and it got canned.

[Thundertrain]

4) If there is NO EU-data-center planned anymore, why was this not communicated more clearly and earlier?
If you are required to evade the truth due to company policy, please simply reply with "Aaargh" to this question.

Thank you for the laugh, Milogson. It's even more amusing because I think there's at least a 15% chance of Vyv ACTUALLY responding with "Aaargh".

[Vilan]

Since the primary rule of corporate communication is to take good news and immediately trumpet it from every rooftop, and take bad news and ignore it as long as humanly possible until having it shoved relentlessly in your face and then mumble something about "We're looking into the situation" and hope everybody forgets about it, I think it's safe to say there is absolutely no chance of an EU data center.

[Munzel]

I have start playing Lotro after a Month of Release at May 2007 on an European Server by Codemasters and played on it almost 4 Years until June 2011, because on 26 April 2011 came the Announcement, that the 10 EU Servers ( i think 5 German servers, 3 English, 2 French) from Codemasters Europe goes Global to Turbine Servers at Boston. So in 2011 the Transfer for 10 EU Servers to Boston tooks about 6 Weeks, and now 5 Years Later in 2016 it tooks over 6 Months to transfer 10 Servers From Boston to New Jersey...I have no Knowledle about Server Technics, but i think, the Lotro Server Technics 2011 was 6 Times Faster then in 2016...and i did not understand this...
The Announcement in August 2015 was, World Transfer and 5 US Servers and 5 EU Servers, but now there are Standing 10 US Servers at New Jersey...Players in Europe from England, French and Germany and maybe Denmark or the Netherlands now plays on the US Servers Evernight, Sirannon or Belegaer...They have only the Title EU Server, but they stands right behind the US Servers in one Datacenter at New Jersey..
Lotro runs over 8 Years on 29 Servers, and now they havn`t the Money for only 10 Servers, 5 in the EU, 5 in the US? Okay, and one Bullroarer...

[Raffaella]

I was hoping to relocate to the European servers would solve my problems:
if I participate in a raid are immobilized, I can not even log out;
in instances of Pelargir may happen that you can not move, and only ranged Allowed to hit the boss;
it happens that while I play I get thrown out of the game and I have to relog.


I had the same problems in bullroader, when we were playing too much at the same time.
So you want to play well that half of us leave the game?

[Srale]

It was funny today on Frelorn Twitch stream - few people were asking some info about Amsterdam/EU datacentar and he did not give any answer or atleast acknowledge question about it...

[Marita]

It was funny today on Frelorn Twitch stream - few people were asking some info about Amsterdam/EU datacentar and he did not give any answer or atleast acknowledge question about it...

maybe he thinks if he doesn't reply the issue will go away and us in the EU who have issues will shut up and go away....

[gripply]

So the EU move was just a carrot on a stick to get EU players to return, in short: a money grab.

Everyone cut your losses and stop supporting this unethical, corrupt company

You're confused the unethical corrupt company that owns an MMO game... its name begins with C not T. :P I've already spoken with my wallet about that, to the benefit of Turbine.

Vyv, thank you for the update and for being specific about the issues you are working on. Especially on Mac, we get a lot of crashes just after completing an instance when we leave. The PC just has wierd freeze-go effects, which are interesting since I've been playing an RK a lot and those hots/dots are frozen, then apply all at once. For the most part though, thankfully for us the game is playable. I'm sorry for those who have a worse experience, but I"m glad to hear it's being worked on.

As an aside, I worked with people from Amersterdam back in my IT days and they impressed me with their professionalism and helpfulness. The people I worked with were very direct but also were completely lacking in malice or cynicism. It was refreshing after working in an American IT environment. I think if my experience is any indication that you will have a much smoother ride with the EU servers.

[jfdiogddsfsd]

Just a quick post for Vyvs attention.
.
Evernight Ettenmoors is showing no improvement in gameplay , reduced awareness/rubberbanding/skill lock everytime more than 12 people ( or one mini spamming 0 second BC) are in the same area.

Sending bug reports is very inconvenient in a pvp raid situation, do you still want constant bug reports? It can be organised again but most people are now bored of the stop start ( usually complete wipe involved as we all stop to fill in the report which can take up to 3 mins to initiate , never mind fill in ) action.

Just don't want you to think the problem has gone away because we arn't complaining as much.

[Marancil]

Yesterday evening (GMT 18-20 approximately) I was in the Shire. My level 86 CAP had a very simple mission; complete a few Slayer Deeds. Started off at Brockenborings killing Goblins in the area to the north, Spiders in Bindbole Wood. The we took care of the Slugs at Rushock Bog. After that we went East to get them Brigands and Harvest-flies. The Brigands were taken care of in the usual place, Narrowcleeve, but when approaching the Yale-height to take out them flies, all hell broke loose.

Jagged movements, rubber-banding, time-outs, freezes. On a scale I have not EVER encountered to date. Took several seconds between attacking a fly to actually see my poor Captain starting to chop at it. Took several seconds to start picking at a copper-ore, took several seconds for my faithful Ally Archer to react on my commands.

I am not the one to complain easily about the game, but this time it was worse than I have ever seen it. I tried to /bug, but it did not react, after a while the game timed out and I lost connection. There was no warning via the "Reduced awareness" function. There were not many players around and I was not in an area that is known for poor performance, like Rohan and Gondor are.

Turbine, get your act together!

Arkenstone, via Europe, Sweden.

[Macdui]

The next step of this process will have us on a new routing system in the next few weeks.

Having calmed down from the EU server situation I wanted to pick up on this point.

I couldn't understand what new system you could have in mind so I assumed this was some "laymen speak". Seems we have CenturyLink supplying the datacentre and their network map for New Jersey datacentres doesn't include a 100 Gigabit Ethernet (100GbE) setup, are you implying that our datacentre is going to be upgraded by CenturyLink and join their 100GbE network?

If it is I think this is great news so might have been delivered with more details, including it's potential for zero losses, to off-set the disastrous handling of the EU datacentre move. If this is not the case I'm intrigued as to what this new system might be?

Frelorn mentioned much work being done with ISPs to encourage more direct routing: I myself had problems routing via San Jose (from the UK) and that has been fixed but it has had little or no effect of the game performance other than perhaps I'm 60ms quicker to notice the lag. However with peering into NY then a hop NJ it's unlikely my latency will improve significantly now, it's a few ms slower than what I got to old servers. Understandably you might have many still playing with less than optimal routes and they should be helped, but I can't help but think that these "aren't the issues you are looking for" (to paraphrase) to deal with performance/stability.


Mac

[Thurallor]

Yesterday evening (GMT 18-20 approximately) I was in the Shire. My level 86 CAP had a very simple mission; complete a few Slayer Deeds. Started off at Brockenborings killing Goblins in the area to the north, Spiders in Bindbole Wood. The we took care of the Slugs at Rushock Bog. After that we went East to get them Brigands and Harvest-flies. The Brigands were taken care of in the usual place, Narrowcleeve, but when approaching the Yale-height to take out them flies, all hell broke loose.

Jagged movements, rubber-banding, time-outs, freezes. On a scale I have not EVER encountered to date. Took several seconds between attacking a fly to actually see my poor Captain starting to chop at it. Took several seconds to start picking at a copper-ore, took several seconds for my faithful Ally Archer to react on my commands.

I am not the one to complain easily about the game, but this time it was worse than I have ever seen it. I tried to /bug, but it did not react, after a while the game timed out and I lost connection. There was no warning via the "Reduced awareness" function. There were not many players around and I was not in an area that is known for poor performance, like Rohan and Gondor are.

Turbine, get your act together!

Arkenstone, via Europe, Sweden.

What was your latency and packet loss during this time? I'm not saying this isn't an instance of server performance problems, but it doesn't sound like it. Did you ask if anyone else was experiencing problems? I hate to say it, but occasional network problems like this may be an unavoidable result of playing on servers located halfway around the world from you.

[botoo]

On the bright side of this unhappy 'no EU servers' situation: players close to the current datacenter who are playing with low ping / no loss still experience a likewise amount of lag/skill-lock/freezes as about everybody else. So if (and thats a very big IF) Turbine is able to sort the current lag, even EU players should experience the benefit of it. From my personal point of view: a latency of ~100 without loss is a very good playable situation. However, I am quite sure the lag is not by routing/datacenter/location alone, imho the worst part of it is caused by bad coding and by adding layer on top of layer of new effects (skills/procs) in a bad way.

[Marancil]

What was your latency and packet loss during this time? I'm not saying this isn't an instance of server performance problems, but it doesn't sound like it. Did you ask if anyone else was experiencing problems? I hate to say it, but occasional network problems like this may be an unavoidable result of playing on servers located halfway around the world from you.

I did not have that active at the time, but I can say so much, despite what anyone else might argue, that my network has always performed very well and I had no recent changes on it. The only thing that might affect my network is a temporary grave loss of bandwidth but I have measurement and alerts in place for that and it was not the case as far as I can tell. I very rarely have noticeable lag. South Bree mostly, which I tend to avoid.

[maartena]

Having calmed down from the EU server situation I wanted to pick up on this point.

I couldn't understand what new system you could have in mind so I assumed this was some "laymen speak". Seems we have CenturyLink supplying the datacentre and their network map for New Jersey datacentres doesn't include a 100 Gigabit Ethernet (100GbE) setup, are you implying that our datacentre is going to be upgraded by CenturyLink and join their 100GbE network?

More bandwidth doesn't mean better latency. Typically, datacenters are supplied by a good many providers, and they probably have redundant links. A common OC (Optical Carrier) speed for datacenters like that are OC-192 and OC-768 which are respectively around 10 GBit/s and around 40 GBit/s links, but they could have bundled a lot of them. That is a HUGE amount of bandwidth.

Turbine probably doesn't need more than about 2 Gbit/s for ALL the servers, considering they contracted out to akamai for game and patch downloads. Remember, the amount of bandwidth per player is fairly low, someone on a 64 kbit/s ISDN line could play Lotro without too much hassle, provided the quality of the line is good. You can probably host a 1000 players on a 100 Mbit/s line, or 10,000 players on a 1 Gbit/s circuit. If they have a 2 Gbit/s circuit hand-off on to a 10 Gbit/s GBIC on one of their firewalls they are probably in good shape. I wouldn't be surprised their circuit hand-off in their DC cage is 1 Gbit/s, and it isn't hitting that ceiling.

It doesn't matter that I have a nice fat 300 Mbps pipe to my home, Lotro is never coming close to even using 0.2 Mbps when I play. Information transferred is nothing more than coordinates and skills, all the animations to represent those are done locally, even if there are 50 players around me moving about.

[maartena]

On the bright side of this unhappy 'no EU servers' situation: players close to the current datacenter who are playing with low ping / no loss still experience a likewise amount of lag/skill-lock/freezes as about everybody else. So if (and thats a very big IF) Turbine is able to sort the current lag, even EU players should experience the benefit of it. From my personal point of view: a latency of ~100 without loss is a very good playable situation. However, I am quite sure the lag is not by routing/datacenter/location alone, imho the worst part of it is caused by bad coding and by adding layer on top of layer of new effects (skills/procs) in a bad way.

Before the data center move, ISP move, and OS upgrades for the Lotro servers..... even players who were physically the FURTHEST away, in Australia and New Zealand, were able to raid and PVmP properly, even though their regular ping would never be better than 250-350 or so. I used to be in groups with an Australian (on voice chat) when I played in early morning US times due to my work schedule, and we had no problems doing complicated raids and PVP fights.

That was, of course, last year, before the moves, and on a now closed server.

Ideally, it will get back to THAT performance.... where an Australian can yet again do everything in the game. I had heavy lag inside Isengard yesterday, playing solo. I was once again, captured by Saruman's minions and forced to clean up their slop. My guess is that because this zone is probably not used a lot, it is placed on a server that hosts these "less important" zones in order to free up as much resources for "more important" zones such as popular instances and cities like Bree.

[Macdui]

More bandwidth doesn't mean better latency. Typically, datacenters are supplied by a good many providers, and they probably have redundant links. A common OC (Optical Carrier) speed for datacenters like that are OC-192 and OC-768 which are respectively around 10 GBit/s and around 40 GBit/s links, but they could have bundled a lot of them. That is a HUGE amount of bandwidth.

Turbine probably doesn't need more than about 2 Gbit/s for ALL the servers, considering they contracted out to akamai for game and patch downloads. Remember, the amount of bandwidth per player is fairly low, someone on a 64 kbit/s ISDN line could play Lotro without too much hassle, provided the quality of the line is good. You can probably host a 1000 players on a 100 Mbit/s line, or 10,000 players on a 1 Gbit/s circuit. If they have a 2 Gbit/s circuit hand-off on to a 10 Gbit/s GBIC on one of their firewalls they are probably in good shape. I wouldn't be surprised their circuit hand-off in their DC cage is 1 Gbit/s, and it isn't hitting that ceiling.

It doesn't matter that I have a nice fat 300 Mbps pipe to my home, Lotro is never coming close to even using 0.2 Mbps when I play. Information transferred is nothing more than coordinates and skills, all the animations to represent those are done locally, even if there are 50 players around me moving about.

Thanks for this. My thinking was these latest switches have the potential for zero losses, I read, and LoTRo has always ground to a halt with fairly minimal packet loss figures. But my real question was what was this "new routing system" we might get in a few weeks? Have you got any ideas? Is it something else they are hanging their hopes on, that will again prove fruitless? Or is it more vapourware? I'm so suspicious of Turbine right now esp the Queen of Communication Flow and I'd like to know how far to spin when the finger is raised once more!

But above all, it's the constant references to the small team and lack of resources as to the reasons that something can't be done or fails to be fixed but it's the poor decisions and huge amounts of wasted development money at Turbine that effects this game. This last year it was a PvMP map that no one plays, the cash grab imbuement system, a laggy MT with quests that really take the biscuit (what the hell is my heroic char doing washing clothes in a laundry for?) and server transfers/migration. How well has all this turned out?

Mac