Data Center Update

[Thurallor]

When are you going to enforce another set of new rules upon us? You can easily make it so that all forum-users get banned or unable to login for certain type of players, that fit your description?
I just want to know why aren't you closing this or making sure the negative comments won't get displayed and warn with bans, if this negativity persists?

You can do whatever and don't have to justify anything or communicate about anything, like let's say for instance, in-game tickets?

What made your company suddenly change their minds?
I wish I could speak just as well English as my native tongue, because short on words for this, but I think you get the point.

You're complaining that you're not being censored?

[Pipo]

What do you mean? We (the players / users / customers) actually have something to say?!
We really getting listened to, by Turbine? LOTRO has never been anything else but censorship!
This whole thread is about it, really..
That's what I meant, because not getting listened to (being ignored, like Turbine has really been doing) is kind of the same as getting censored, as both won't make it to the drawing boards, but it was meant sarcastically.

Thank you for all the great bits and bytes that makes LOTRO, but.... now you've really messed up for me. Last straw... @ Turbine

[Viriel]

The end of this game started some time ago so this post doesn't change anything in my opinion but what is sad is the way in which its happening. You would think that Turbine should learn on their own mistakes made before but they keep acting like a kid who is not able to admit that he made one.Shame to see the game i was playing since almost beginning ending this way. Good thing for me i gave up on this kid some time ago. Sorry for all those who are sill playing.

Btw i found this interesting. It's got great potential and its still in development if you would be looking for anything new. At this point you would also have a chance to say what you would like to see in end product.

http://www.navalaction.com/#ageofsail

Anyway, farewell LotRO. Have fun people.

[Palanthas]

I came back to this game at the end of last year after hearing the promises of a new improved server platform and servers in the EU once more. I also figured there would be a general rush back to the game from other players hearing the news.

The server move was an absolute debacle from start to finish. Instead of improving on the old, tired server platform, as was promised, the performance has actually gotten WORSE. As such, instead of players returning, more have simply turned their backs on the game.

And now, the nail in the coffin for EU players. No EU servers, just laggy, overloaded US servers.

This game now has nothing to offer me. I've maintained a VIP subscription for the last 6 years, I've just cancelled it this morning.

This game is now dead, in my opinion. It doesn't have much time left, I will waste no more of my own precious time on it.

It was fun for a time, back in Moria. It will never be so again.

Goodbye LOTRO.

[Macdui]

This is actually something we are in the process of doing.

If your doing some house-keeping the Turbine Inc site currently has your games listed in this particular order.
Infinite Crisis
LoTRo
DDO
Asheron's Call

Deleted the cynical comments

Mac

[ShireDweller]

If your doing some house-keeping the Turbine Inc site currently has your games listed in this particular order.
Infinite Crisis
LoTRo
DDO
Asheron's Call

Deleted the cynical comments

Mac

Somewhat related, my local Gamestop still has Infinite Crisis points cards for sale.

If Turbine can't afford the shipping to get those items returned to HQ to destroy, at least let the stores know to take them off the shelves.

That's the decent thing to do.

[Snowlock]

I came back to this game at the end of last year after hearing the promises of a new improved server platform and servers in the EU once more. I also figured there would be a general rush back to the game from other players hearing the news.

The server move was an absolute debacle from start to finish. Instead of improving on the old, tired server platform, as was promised, the performance has actually gotten WORSE. As such, instead of players returning, more have simply turned their backs on the game.

And now, the nail in the coffin for EU players. No EU servers, just laggy, overloaded US servers.

This game now has nothing to offer me. I've maintained a VIP subscription for the last 6 years, I've just cancelled it this morning.

This game is now dead, in my opinion. It doesn't have much time left, I will waste no more of my own precious time on it.

It was fun for a time, back in Moria. It will never be so again.

Goodbye LOTRO.

Translation: I heard the PERFORMANCE WAS GOING TO IMPROVE SO I CAME BACK.

I told you, Vy, there are alot of players like this. We don't care how the bad performance is FIXED; whether it's data center moves, or removing some of the layers of glop on the engine, or fixing routes, or some other thing. You have players that are waiting at the gates to come back to LOTRO. Theyr'e just waiting for the all clear. This guy is not alone; they just rarely post on the forums.

[Fratonia]

Also, it may not have been Internap's problem nor Telia's problem, but an issue with a backbone provider in between, such as Level3 or whatnot. Since the problems you described seemed to affect ONLY the clients of the ONE provider you mentioned, I am actually inclined to say the issue was with THAT provider, not with anything on the US side of things.

Actually Telia Sonera IS the backbone provider. There is nothing "in between". All hops were responding with 0 percent packet loss all of the time up to the interconnect between Telia Sonera and Internap. There is no doubt where the problem was located. Trust me, we have run extensive traces.
As normal user I couldn't talk about such things with Telia Sonera directly because I'm not their customer although I have to say that the support guy was helpful. I also couldn't talk with Internap about it but Turbine could. Unfortunately they didn't seem interested in solving the problems and/or communicating with us players so we had to live with the problems for a year.
*sigh* The old problem with Turbine and communication... I'm still wondering why Turbine's bug reporting system allows to select email progress notifications for bug reports. There were never any notifications for bug reports except the initial "thank you for submitting blabla"...

[Vyvyanne]

Somewhat related, my local Gamestop still has Infinite Crisis points cards for sale.

If Turbine can't afford the shipping to get those items returned to HQ to destroy, at least let the stores know to take them off the shelves.

That's the decent thing to do.

New websites are in the works with the marketing team.

Retailers have been made aware of the IC closure, but they control their own inventory. All we can do is inform them those items are no longer valid, which we have done.

To those concerned that we are working on multiple things at once, the LOTRO team is a group of people all with specializations in different areas.

We have a team that works on the networking, server maintenance and datacenter. They are actively working with our vendors to improve the ISP routing and looking into any problems created by the set-up or capabilities of the hardware itself (which is fully configured for all 10 worlds as well as set-ups to keep each of the closing worlds running until April and character data from them available even after that.)

We have game side engineers who are looking into the issue with the login server that occasionally starts giving the "Server Full" message and preventing logins when the server is not actually full. Additionally they have stopped any work they may have been doing on Update 18 to troubleshoot the problems we are seeing with transitions between landblocks since switching over to the new OS and hardware in the new datacenter.

We have game engineers working with QA, designers and artists to work on reducing over all client memory footprints in our newer areas as well as ways to reduce memory usage in PvMP.

QA has already completed the testing required to come to new recommended game specs. This is a process that takes months and an outside hardware compatibility lab to run, we are just in the process of compiling those results and running them through the various teams here for approval and they will be posted once they have completed that process and translations.

And we still have designers and artists whose talents do not lean on the side of performance improvements that continue to work away at U18 content.

Getting the new datacenter to be even better than what we had before is still our goal and we still have folks working heavily on it, but everyone has to work within their own strengths and due to the number of things that changed with this move it will take multiple approaches to get us back to where we want to be.

[GozmitJosh]

This may or may not be the right thread to ask, but what is turbines statement on the client side issues that come up for players.

Mainly how the client doesnt fully utilize multi-core CPU's, Lotro only uses maybe 20% at most of my CPU and 10% of my GPU, which to me seems like i should be getting much higher FPS with my hardware, and less of the FPS jumping all over the place.

Is the client really not utilizing the extra cores of our CPU's due to it's age?
Is it a known issue that really cant be fixed without a huge investment to recode, that Lotro does not have right now?

Or is it in the works for later down the road?...etc...etc

That seems to be my number 1 issue in Lotro when it comes to performance.

[Nenuiale]

We have game side engineers who are looking into the issue with the login server that occasionally starts giving the "Server Full" message and preventing logins when the server is not actually full. Additionally they have stopped any work they may have been doing on Update 18 to troubleshoot the problems we are seeing with transitions between landblocks since switching over to the new OS and hardware in the new datacenter.

We have game engineers working with QA, designers and artists to work on reducing over all client memory footprints in our newer areas as well as ways to reduce memory usage in PvMP.

Very good to know the team tries to take us to a place where the (server-side) performance is better than the old datacentres and all this fuss about new hardware does not have the old server performance as endgame. But...

In this very welcome adress of the client-side performance issues which are related with the game engine, I do not see any mention of the longstanding problem of buff/debuff icon performance drop. Is that something the team is working on, given we're expecting a new raid? I think I've never seen any turbine official commenting on this issue.

And an afterthought: does the above quote mean that U18 will be pushed further away than what you originally planned? So, should we expect new stuff in Q3 of 2016 as opposed to Q2 (around the anniversary, when usually the first major annual update was implemented)?

[siipperi]

--

Very informative, thank you.

[Mirillin]

And an afterthought: does the above quote mean that U18 will be pushed further away than what you originally planned? So, should we expect new stuff in Q3 of 2016 as opposed to Q2 (around the anniversary, when usually the first major annual update was implemented)?

Do you really want another update when the last one is still unplayable by so many due to crashing and lag?

I personally feel that should sort out old bugs before introducing new ones. However that has never been Turbine's style so I imagine we will get U18 sooner rather then later.

[Wibz]

We have game engineers working with QA, designers and artists to work on reducing over all client memory footprints in our newer areas as well as ways to reduce memory usage in PvMP.

QA has already completed the testing required to come to new recommended game specs. This is a process that takes months and an outside hardware compatibility lab to run, we are just in the process of compiling those results and running them through the various teams here for approval and they will be posted once they have completed that process and translations.

And we still have designers and artists whose talents do not lean on the side of performance improvements that continue to work away at U18 content.

Do these improvements include fixing the horrendous buff/debuff lag/client performance hit that happens in fellowships and raids? This performance hit can send the client from running 60fps (or 100+, etc) down to less than 10 fps on very powerful PCs. Also when this happens all the buffs/debuffs of the selected mobs start to flash, jump around and sometimes totally disappear.

The issue has been raised multiple times over the last 4 years and has been totally ignored by Turbine.

[Vyvyanne]

Do these improvements include fixing the horrendous buff/debuff lag/client performance hit that happens in fellowships and raids? This performance hit can send the client from running 60fps (or 100+, etc) down to less than 10 fps on very powerful PCs. Also when this happens all the buffs/debuffs of the selected mobs start to flash, jump around and sometimes totally disappear.

The issue has been raised multiple times over the last 4 years and has been totally ignored by Turbine.

We are looking into all of it. If there is something that will tweak performance in a positive direction without destroying the integrity of the game, we are looking at it whether it is new since the move or from before. We have found that the move has exacerbated issues we had before but we infrequent or not as noticeable until we performed our upgrades. So we are looking into these things as well. I dont see us doing a complete overhaul on our game engine code as one player mentioned. The cost in time and resources as well as stability risk there would be very high, but there may be adjustments that can be made without that needing to be necessary.

[Macdui]

Good to see the response and change in appreciation from two weeks ago.

Since we were asked for /bug reports at that time and it seems to have had the desired effect should we continue repeating the same old reports from the same old /locs or would you prefer only reports from new /locs we come across? Ofc if you have enough to work with and understand how wide spread it is now and the /locs are now unnecessary we can hold back on them, until it's all "fixed" again.

On a positive note should we post any improvements we see in the server/hardware thread?

Mac

[dstini]

New websites are in the works with the marketing team.

Retailers have been made aware of the IC closure, but they control their own inventory. All we can do is inform them those items are no longer valid, which we have done.

To those concerned that we are working on multiple things at once, the LOTRO team is a group of people all with specializations in different areas.

We have a team that works on the networking, server maintenance and datacenter. They are actively working with our vendors to improve the ISP routing and looking into any problems created by the set-up or capabilities of the hardware itself (which is fully configured for all 10 worlds as well as set-ups to keep each of the closing worlds running until April and character data from them available even after that.)

We have game side engineers who are looking into the issue with the login server that occasionally starts giving the "Server Full" message and preventing logins when the server is not actually full. Additionally they have stopped any work they may have been doing on Update 18 to troubleshoot the problems we are seeing with transitions between landblocks since switching over to the new OS and hardware in the new datacenter.

We have game engineers working with QA, designers and artists to work on reducing over all client memory footprints in our newer areas as well as ways to reduce memory usage in PvMP.

QA has already completed the testing required to come to new recommended game specs. This is a process that takes months and an outside hardware compatibility lab to run, we are just in the process of compiling those results and running them through the various teams here for approval and they will be posted once they have completed that process and translations.

And we still have designers and artists whose talents do not lean on the side of performance improvements that continue to work away at U18 content.

Getting the new datacenter to be even better than what we had before is still our goal and we still have folks working heavily on it, but everyone has to work within their own strengths and due to the number of things that changed with this move it will take multiple approaches to get us back to where we want to be.

WOW, wonderful post! Love to see the increased communication recently Turbine!

[Snowlock]

We are looking into all of it. If there is something that will tweak performance in a positive direction without destroying the integrity of the game, we are looking at it whether it is new since the move or from before. We have found that the move has exacerbated issues we had before but we infrequent or not as noticeable until we performed our upgrades. So we are looking into these things as well. I dont see us doing a complete overhaul on our game engine code as one player mentioned. The cost in time and resources as well as stability risk there would be very high, but there may be adjustments that can be made without that needing to be necessary.

Gosh you guys are hard to nail down on this issue. Lets be as specific as we can be:

ISSUE: Skill lag. Have you seen this video? You don't have to watch long:



You can see the players moving but no skills firing. This has been going on since the Great River Update, March of 2012. Before, there was "lag" in the game. When the Great River update hit, there was this skill lock stuff. This skill locking has been happening ever since. When you say it's been exacerbated by the move, that's incorrect. It's been here since the aforementioned update. It's what caused the massive end-gamer exodus in 2013-14. Its the perf issues that players are referring to when they say "They were hoping to come back".

It has been neither "infrequent" or "not as noticeable". What has happened is by and large it's not been reported as much because this generation of players have largely come to accept it. So "infrequent" to you would be "not bothering reporting it for the 4th year in a row" to us. OR, the players that truly were irritated enough to report it, have QUIT THE GAME.

I'd refer you to the numerous threads about this skill locking lag from back when it started in 2012, but the Previous CM was actively deleting the threads, telling us it was our ISP's and in many cases perma-banning the players reporting the issue.

The reason you're hearing about it now so much after the data center move was because we were hoping the data center move would finally resolve this issue for us. But now that it hasn't, we're bringing it up again.

There may be posters following up on my post saying they've never experienced this. They are either lying, mistaken, or never been to the Ettenmoors under the conditions it arises.

When does it arise? When there are groups fighting over a total number of around 20 or higher, often it is preceded by a reduced awareness notification, but not always. Often it is accompanied but stuttering, hitching, rubber banding, CTD's, and getting stuck on load screens. All of those things suck, but in PVP, when you can fire your skill when you want to, your experience is ruined.

Please let me know if you are specifically addressing this issue. Please.

[maartena]

We are looking into all of it. If there is something that will tweak performance in a positive direction without destroying the integrity of the game, we are looking at it whether it is new since the move or from before. We have found that the move has exacerbated issues we had before but we infrequent or not as noticeable until we performed our upgrades. So we are looking into these things as well. I dont see us doing a complete overhaul on our game engine code as one player mentioned. The cost in time and resources as well as stability risk there would be very high, but there may be adjustments that can be made without that needing to be necessary.

One thing I have noticed is bad performance in instances that are (most likely!) hardly used....

Example: Hall of Lights in Mirkwood (not sure if instance, but its a separate zone), which is only used in the Vol II. Epilogue quests. When I did it last week, I was probably the ONLY one using that particular zone on the server, which makes me think that "underutilized" zones may have gotten a lower priority. Since this is a PVE quest zone, it's definitely not an important zone and and performance issues are not a big issue. BUT.... it is noteworthy, because that zone, underutilized as it is, should run at a high speed if there is no one there anyways

Another Example is a Lothlorien Epic quest where you have to visit Lady Galadriel and use the mirror. Moving inside the zone towards Galadriel I was hopping and a skipping all over the place. Again, not that important, and only used for short times as part of the epic quests.... but noteworthy as a underused zone like that should not have performance issues at all.

Just wanted to note that, I have done several /bug reports over the last few weeks.

[Snowlock]

I actually found one of the old threads. It was created from a previous thread, and that previous thread is one of the deleted ones.

https://www.lotro.com/forums/showthr...-the-lag/page7

The players are reporting all kinds of lag issues, but it's this skill lock stuff at the heart of it. You can also see Sapience saying it's not Turbine's fault as the players get madder and madder until the final, most telling post on the thread.

[Vyvyanne]

Good to see the response and change in appreciation from two weeks ago.

Since we were asked for /bug reports at that time and it seems to have had the desired effect should we continue repeating the same old reports from the same old /locs or would you prefer only reports from new /locs we come across? Ofc if you have enough to work with and understand how wide spread it is now and the /locs are now unnecessary we can hold back on them, until it's all "fixed" again.

On a positive note should we post any improvements we see in the server/hardware thread?

Mac

/bug reports are always helpful as they increase the amount of data we have to work with. Please do be aware that a /bug will never get a response, it is just a way to send data for us for our QA team to parse and use to verify/ reproduce issues but it gives them valuable data from which to do this.

We also could still use traceroutes from players experiencing latency issues. We present these to our ISP partners to help them diagnose problems on that end. You can send those to tech support or post them here.

And yes feedback if you see improvements is helpful as well. It lets us know if we are on the right track. For example it would be good to know if you have seen any improvement since Wed. Last week vs this week since we restarted the servers. We are investigating if we have a degradation over time issue. We know the login server issue presents after it reaches a thresh hold of time so we are resetting those before the weekend to prevent weekend interruptions while we work on a longer term fix. We would like to see if the game servers have a similar issue. If they do then there should be an improvement in gameplay this weekend vs last weekend, but if you are also in the group of folks having ISP route issues, you might not notice.

More specific problems like the client-side problems in PvMP will take a patch. Those will go into beta on Bullroarer once we have enough changes to test out effectiveness. We will announce testing periods for those on those forums once they are ready.

[Amorey]

New websites are in the works with the marketing team.

That is really lovely, thanks for letting us know


I hope this includes an update to the LOTRO site and maybe these forums too, we could do with a bit of 'spring cleaning'

[rannion]

Traceroute for Online (dutch ISP)

(In-game pings varying between 100-220ms)

C:\WINDOWS\system32>tracert gls.lotro.com

Tracing route to gls.lotro.com [198.252.160.30]
over a maximum of 30 hops:

1 1 ms 1 ms 1 ms TL-WR1043ND.lan [192.168.0.1]
2 2 ms 5 ms 1 ms 192.168.1.1
3 25 ms 28 ms 18 ms bng1-asd7.nl.euro.net [194.134.5.64]
4 17 ms 17 ms 17 ms ae6.cr1-asd8.nl.euro.net [194.134.161.50]
5 17 ms 18 ms 18 ms ae1-br1-asd6.nl.euro.net [194.134.162.221]
6 1699 ms 1666 ms 1471 ms lag-111.ear3.Amsterdam1.Level3.net [213.19.197.77]
7 94 ms 98 ms 97 ms ae-1-51.edge3.NewYork2.Level3.net [4.69.138.196]
8 94 ms 97 ms 94 ms ae-1-51.edge3.NewYork2.Level3.net [4.69.138.196]
9 93 ms 95 ms 93 ms jfk-brdr-04.inet.qwest.net [63.146.26.5]
10 94 ms 93 ms 108 ms ewr-cntr-11.inet.qwest.net [205.171.17.2]
11 95 ms 95 ms 104 ms 206.103.215.50
12 102 ms 94 ms 95 ms 63.236.3.130
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

Trace complete.

C:\WINDOWS\system32>tracert gls.lotro.com

Tracing route to gls.lotro.com [198.252.160.30]
over a maximum of 30 hops:

1 1 ms 1 ms 1 ms OPENWRT [192.168.0.1]
2 2 ms 1 ms 2 ms 192.168.1.1
3 22 ms 26 ms 35 ms bng1-asd7.nl.euro.net [194.134.5.64]
4 20 ms 18 ms 19 ms ae6.cr1-asd8.nl.euro.net [194.134.161.50]
5 32 ms 17 ms 19 ms ae1-br1-asd6.nl.euro.net [194.134.162.221]
6 30 ms 18 ms 18 ms lag-111.ear3.Amsterdam1.Level3.net [213.19.197.77]
7 93 ms 93 ms 93 ms ae-1-51.edge3.NewYork2.Level3.net [4.69.138.196]
8 93 ms 102 ms 99 ms ae-1-51.edge3.NewYork2.Level3.net [4.69.138.196]
9 95 ms 116 ms 102 ms jfk-brdr-04.inet.qwest.net [63.146.26.5]
10 116 ms 109 ms 114 ms ewr-cntr-11.inet.qwest.net [205.171.17.2]
11 104 ms 107 ms 150 ms 206.103.215.50
12 94 ms 100 ms 150 ms 63.236.3.130
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

Trace complete.

[ShireDweller]

...
Retailers have been made aware of the IC closure, but they control their own inventory. All we can do is inform them those items are no longer valid, which we have done.

So you're saying it's GameStop's fault for continuing to sell worthless items? Good to know, I'm sure they'll be happy to hear that.

Do you have a contact name that I can send a PM to regarding this? There must be someone at Turbine who would be interested in reminding the store that by continuing to sell these points cards, they'd be taking good money for a game that no longer exists.

[Pontin_Finnberry]

New websites are in the works with the marketing team.

Retailers have been made aware of the IC closure, but they control their own inventory. All we can do is inform them those items are no longer valid, which we have done.

To those concerned that we are working on multiple things at once, the LOTRO team is a group of people all with specializations in different areas.

We have a team that works on the networking, server maintenance and datacenter. They are actively working with our vendors to improve the ISP routing and looking into any problems created by the set-up or capabilities of the hardware itself (which is fully configured for all 10 worlds as well as set-ups to keep each of the closing worlds running until April and character data from them available even after that.)

We have game side engineers who are looking into the issue with the login server that occasionally starts giving the "Server Full" message and preventing logins when the server is not actually full. Additionally they have stopped any work they may have been doing on Update 18 to troubleshoot the problems we are seeing with transitions between landblocks since switching over to the new OS and hardware in the new datacenter.

We have game engineers working with QA, designers and artists to work on reducing over all client memory footprints in our newer areas as well as ways to reduce memory usage in PvMP.

QA has already completed the testing required to come to new recommended game specs. This is a process that takes months and an outside hardware compatibility lab to run, we are just in the process of compiling those results and running them through the various teams here for approval and they will be posted once they have completed that process and translations.

And we still have designers and artists whose talents do not lean on the side of performance improvements that continue to work away at U18 content.

Getting the new datacenter to be even better than what we had before is still our goal and we still have folks working heavily on it, but everyone has to work within their own strengths and due to the number of things that changed with this move it will take multiple approaches to get us back to where we want to be.

Thanks for the info always good to see info on work in progress