SSG - Time to invest in your servers....

[Zyot]

Could you guys show the least bit of respect for your community and invest in your servers?
Going into a new expansion with this server performance, is plain stupidity...

[Harvain]

There is ongoing work for the LOTRO Servers, some issues did arise such as having VRAM issues for Belegaer (pretty sure it was Belegaer, still working on the Severlin Q&A from October so can't quote such directly about what was stated about LOTRO Servers so I am summarizing the rest of this post), doing work to try and improve performance (Arkenstone and Landroval had this first), Update 30.3.2 had "game performance improvement(s)" however this seems like it didn't have a positive effect based on LOTRO Community Feedback so far (plus not knowing what was adjusted to compare), Server Hardware to be upgraded, and more.

It won't be an overnight thing, flip a switch, so on kind of change(s) to improve things. Just the constant battle of trying to counter LOTRO's aging and ever increasing technology outside of LOTRO.

[Zyot]

Evernight had issues for at least 2 years.
No one asked for changes overnight.

[LagunaD2]

We just need more server hardware - which they could deploy today - not better servers.

I think they are simply cutting corners to save money, and blaming nebulous performance issues for their failure to provision enough computing and network resources for the game to run smoothly.

In effect, they are cheating us out of the better, lag-free game experience we deserve (and some of us pay for).

[OghranNasty]

Upgrades to the servers, both in hardware and software, are coming, if plans pan out this year even.

You wil have to account for the fact that it is a complicated operation with many moving parts, unexpected stuff will arise, so maybe a little later before we see real improvements. Sadly, that is how these things work: there is no magic button. Maybe we'll see something noticeable by springtime, earlier if we are lucky.

[LagunaD2]

Sadly, that is how these things work: there is no magic button.

Yes there is - pay for more computing power and network bandwidth. Now.

[DKenny]

Yes there is - pay for more computing power and network bandwidth. Now.

That is ultimately the solution. But why should they though? If the expansion sells well, the dollars are voting that the current situation is acceptable.

[Dredre999]

Could you guys show the least bit of respect for your community and invest in your servers?
Going into a new expansion with this server performance, is plain stupidity...

Pretty sure that is exactly what they are planning with the move to 64 bit servers as stated by devs.

[Harvain]

The biggest thing is that technology is always improving and LOTRO will always be degrading in performance.

LOTRO's, Turbine's and SSG's track records is that they are never ahead of such things and always playing catch up.

They are only using 1 Server Host Location, this will ALWAYS have it's own problems especially as all European Data has to travel through 1 Internet Location which SSG has no control over and can cause problems for all "Across the Atlantic" Clients while North American & South American Clients might not encounter such.

Hardware should have more overhead than what is needed so that if things arise where more of the Server that LOTRO Servers are run on has more horsepower in reserve.

LOTRO's Engine, Coding, etc is old and has it's own limitations. Pretty much akin to spreading butter over bread, eventually you can only do so much with what is available.

So on and so forth.



But this will always be various issues with LOTRO, even when SSG catches up and "improves things" for a time. Eventually, LOTRO will degrade and the process begins anew.

It's been going on through LOTRO's life cycle it's just been easier to see especially since before the LOTRO Closed Worlds and Transfers began.

[Raninia]

This article that IGN did a while back is pretty good in lots of ways, but here's the relevant part I want to call out:

"The internet screams, 'Add more servers,' and the number of servers is quite literally never the problem.

"People think of it like: a server can handle X players, Y people want to play, so Y / X = number of servers = problem solved, right? Simply: no. There are at least four more layers of complexity to a live game, and each one amplifies any of the previous layers' problems."

As other folks have mentioned in this thread, we're working on this from several different directions. We're seeing some improvement, but it's not broad or sustained enough yet, so we're going to keep poking at it from other angles. I'm aware of at least three performance-related initiatives that we're working on right now, and all the ones I can remember are targeted for this year still. So it'll continue going, but I fully expect that we'll have to keep working on it beyond this year, and we've already planned additional work beyond 2021 and into 2022.

[Snowlock]

This article that IGN did a while back is pretty good in lots of ways, but here's the relevant part I want to call out:



As other folks have mentioned in this thread, we're working on this from several different directions. We're seeing some improvement, but it's not broad or sustained enough yet, so we're going to keep poking at it from other angles. I'm aware of at least three performance-related initiatives that we're working on right now, and all the ones I can remember are targeted for this year still. So it'll continue going, but I fully expect that we'll have to keep working on it beyond this year, and we've already planned additional work beyond 2021 and into 2022.

How much of our lag issues arise from all of the account-wide things we have like our shared wallet and storage, Audacity, and account bound items? I was told in the past that these things essentially multiply the "footprint" the server sees us as because it's doing checks against all of our toons when accounting for these account-wide things.

[Braer]

This article that IGN did a while back is pretty good in lots of ways, but here's the relevant part I want to call out:



As other folks have mentioned in this thread, we're working on this from several different directions. We're seeing some improvement, but it's not broad or sustained enough yet, so we're going to keep poking at it from other angles. I'm aware of at least three performance-related initiatives that we're working on right now, and all the ones I can remember are targeted for this year still. So it'll continue going, but I fully expect that we'll have to keep working on it beyond this year, and we've already planned additional work beyond 2021 and into 2022.

"Pollock broke down the various layers for me, which include issues like the amount of information each server can hold for the entire playerbase, communication issues between servers, sharding, the actual ability to create more servers, and, aggravatingly, problems created by players themselves"

You keep adding plenty of items, a couple of thousand with new LIs alone, yet offer us inappropriately named storage options from the store that hardly scratch the surface in fulfilling players requirements. Can't even be bothered to increase their size to match item amounts let alone address functionality feedback. The Player derived issues could be as easy as a multi barter option for stackable items only and yet you choose to annoy your players by keeping the single item at a time barter only and enforce a delay if too "aggressive". You can't have landscape tracery drops because you failed to address raid farming with some brighter thinking rather than the usual inept whack-a-mole approach. The woeful loading in late game multi map areas and missions is down to your tech to cut corners by harvesting areas and changing them on the fly and you ignoring feedback and likely blundering into Gundabad with the same tech.

It's all down to you guys in effect.

[OghranNasty]

Yes there is - pay for more computing power and network bandwidth. Now.

And at least for servers, that is what they did. But if you order a bunch of servers now, you won't have them delivered tomorrow. Let alone have them installed, hooked up, configured or tested.
At my workplace we used to guesstimate 2 months for that. This year, with shortages and supply line issues, our last batch of servers took 5 months.

[LagunaD2]

And at least for servers, that is what they did. But if you order a bunch of servers now, you won't have them delivered tomorrow. Let alone have them installed, hooked up, configured or tested.
At my workplace we used to guesstimate 2 months for that. This year, with shortages and supply line issues, our last batch of servers took 5 months.

Data centers don't work like that.

One of the reasons they exist is to allow customers to quickly scale their resources to their needs (which is more difficult for in-house deployments, as you describe).

[LagunaD2]

This article that IGN did a while back is pretty good in lots of ways, but here's the relevant part I want to call out

You omitted these words before (literally the previous sentence...) those you quoted, which make the context irrelevant to what we're discussing:

"Every live game falls over day one," he said.

We're not talking about any surge of logins on "day one". We're talking about a steady-state situation where the resources you are paying for are entirely insufficient to handle normal, predictable player populations.

Day in, day out.

[Arantha]

This article that IGN did a while back is pretty good in lots of ways, but here's the relevant part I want to call out:



As other folks have mentioned in this thread, we're working on this from several different directions. We're seeing some improvement, but it's not broad or sustained enough yet, so we're going to keep poking at it from other angles. I'm aware of at least three performance-related initiatives that we're working on right now, and all the ones I can remember are targeted for this year still. So it'll continue going, but I fully expect that we'll have to keep working on it beyond this year, and we've already planned additional work beyond 2021 and into 2022.

Thanks very much Raninia, really appreciate it. Hope the troubles are soon in an accetable manner.

[oggbnvwnyc]

This article that IGN did a while back is pretty good in lots of ways, but here's the relevant part I want to call out:



As other folks have mentioned in this thread, we're working on this from several different directions. We're seeing some improvement, but it's not broad or sustained enough yet, so we're going to keep poking at it from other angles. I'm aware of at least three performance-related initiatives that we're working on right now, and all the ones I can remember are targeted for this year still. So it'll continue going, but I fully expect that we'll have to keep working on it beyond this year, and we've already planned additional work beyond 2021 and into 2022.

How desperate are you that you have to link a single article that talks in very broad strokes about servers in like a few sentences?

The SSG "We're working on it" PR slogan is truly utterly worthless.
Just for perspective: e.g. the Warmane WoW private servers, assuming the data is correct, manage to have 27k+ players online right now, when your own servers already go into meltdown with <1k non-anon people online on Evernight.
https://www.warmane.com/devlog

[Sodiuss]

Does anyone else remember when the new servers were touted and how they could run all the worlds on one server if they wanted to?

What utter BS that was

[OghranNasty]

Data centers don't work like that.

One of the reasons they exist is to allow customers to quickly scale their resources to their needs (which is more difficult for in-house deployments, as you describe).

It is a question of rent or buy. If you rent your servers your dc will prob be able to provision new gear quickly, although you still need to do installations, config and testing. But if you just rent rack space and buy your own gear, as SSG does, you do all the legwork yourself.

[LagunaD2]

There's every reason to expect that if they achieve improvements in server performance, they will use them to further reduce their costs by cutting back further on the hardware and keeping the player experience at as poor a level as they think they can get away with.

Since that's what they're already doing now.

[Harvain]

Does anyone else remember when the new servers were touted and how they could run all the worlds on one server if they wanted to?

What utter BS that was

At the time they could.

But technology is always gets better and older Tech and Coding always degrades. It's a constant battle, not something you can just "Buy this Server, throw a Game on and only Upgrade X amount of years down the road" especially in LOTRO's case.



The main thing is that SSG is playing catch-up versus trying to stay ahead of things. Which at the start and end of the day, it's more LOTRO itself than what Hardware they are using. Yes, having better Hardware will give more overhead for when things are more busy and/or for any reason(s) using more Resources on a Server Box.

But this has it's limitation and can only provide X amount of Performance before the Game, Coding, and the Game Server itself will hit their own limitations and hold things back. Every bit of Coding, every bit of Graphics (Trees, Bushes, Clouds, Buildings, Boxes, NPCs, Smoke, Fire, every little detail adds to the overall amount of things going on), Account + Character(s) logged in and their Data on said Server, Items, Systems, etc all affect how each individual Server will perform.

How any LOTRO Server handles this is how every Server will handle it. You have the extreme end with Evernight, which here lately at least when I've checked it usually averages 900+ non-anon Characters logged in at various times of the day and has more "lag" compared to any other Server. Then I guess you could use Shadowfax for an example where it has a much smaller amount of Characters logged in at any given time and how "lag" is here is different but hiccups akin to Evernight can occur. Both Servers operate the exact same way but Evernight is affected by how things are by X amount of Magnitudes more than Shadowfax. Obviously these numbers are purely a "LOTRO Community Member's Observations" of how Character Counts at any given time.



Now what is the solution? There is no one solution.

The Server Box or Boxes need better Hardware, better Internet Speeds, etc.

• This has limitations as SSG is at the mercy of their Data Center and how quickly they can get to them. 100% guarantee SSG is not even considered a "High Value Client" for those who use said Data Center. Time to setup a Server Box and all it entails will vary for every Client.

LOTRO's Coding must be improved to try and better deal with how things are.

• Huge limitations for various reasons especially with how LOTRO was built usually with a "make it work right now and not worry about next year" mentality that in all honesty has dominated a large majority of LOTRO's History.
  
  
• It's like going to an older house that 10 different families lived in and decide to "modernize" the wiring. Unless you plan to gut the whole building, you'll only be able to do so much within the confines of what you are working with.
  
  
• Really at the end of the day, the only way SSG will ever have a Game where they can more easily optimize their Coding, etc would be to make a brand new LOTRO with a "Make it work for today and next year" mentality but even this has it's limitations, there is no such thing as 100% future proof.

Finally SSG has to really think about what is worth having and supporting and how can they slim anything down to run at their base necessities and how they make and present the various Regions in LOTRO.

• This is every System: Housing, PvMP, Big Battles, Legendary Item Systems, Cosmetics, Skirmishes, Vaults, Wallets, Crafting, anything that has Data that has to be saved and dealt with. Finding ways to decrease their affects on a LOTRO Server is a key area.
  
  
• Regions and how they are represented are also a factor. Regions like the War of the Peaks and it's constant "War" does affect things granted nobody in the LOTRO Community could provide any formal Data or educated cases to how much this affects every LOTRO Server which has said Content on it and running beyond that it "does" add to what is happening.
  
  
  • Now what this means is how SSG will make future areas and their Content. More is not better. Minas Tirith has and still is one of the most extreme cases of how having so many Buildings, NPCs and Objects especially in a confined area will affect things in a more negative way.
    
    
  • Finding a way to present X Region with a minimal amount of "everything" will help short & long term Server Performance.
    
    
    • For example: How to represent a village. How many Buildings, Bushes, Trees, NPCs and other Objects would be the bare minimum to present a believable scenario for what is going on that both keeps said Village's impact on Server Performance to as low as it possibly can without fully restraining the ability to World Build any given Region.

And a whole lot more that could be stated, debated, argued and so on. The ball is firmly in SSG's court, they can and need to find ways to improve anything and everything but it's a matter of if it's worth doing such and are they willing to do such.

Which at least as of heading into Update 31: The Fate of Gundabad Expansion, SSG doesn't seem willing to go to what are extremes measures but are truly necessary for long term viability of LOTRO.

[LagunaD2]

But this has it's limitation and can only provide X amount of Performance before the Game, Coding, and the Game Server itself will hit their own limitations and hold things back. Every bit of Coding, every bit of Graphics (Trees, Bushes, Clouds, Buildings, Boxes, NPCs, Smoke, Fire, every little detail adds to the overall amount of things going on), Account + Character(s) logged in and their Data on said Server, Items, Systems, etc all affect how each individual Server will perform.

This is a really uninformed and silly argument. The server has nothing to with the graphical rendering of trees, bushes, clouds, etc. Do you really think the server is drawing the blades of grass?

Lag can occur due to network latency or server latency. The lag in LotRO is too general and widespread to be primarily due to network latency (no doubt it accounts for some lag experienced by some people some of the time). The main problem is server latency - the server has too many things to do, and it falls behind. Additional computing resources would undoubtedly improve the situation. They are just too cheap to pay for them, I believe.

I base my belief on 48 years of experience in programming, including cutting-edge real-time systems I have written, debugged and operated (data acquisition systems for elementary particle physics experiments).

Brute force isn't always the answer. But it's usually the answer.

[Harvain]

This is a really uninformed and silly argument. The server has nothing to with the graphical rendering of trees, bushes, clouds, etc. Do you really think the server is drawing the blades of grass?

Lag can occur due to network latency or server latency. The lag in LotRO is too general and widespread to be primarily due to network latency (no doubt it accounts for some lag experienced by some people some of the time). The main problem is server latency - the server has too many things to do, and it falls behind. Additional computing resources would undoubtedly improve the situation. They are just too cheap to pay for them, I believe.

I base my belief on 48 years of experience in programming, including cutting-edge real-time systems I have written, debugged and operated (data acquisition systems for elementary particle physics experiments).

Brute force isn't always the answer. But it's usually the answer.

Everything that occurs on a Game Server affects how a Server Box will run. Edit: Not all the "lag" we are encountering is purely from a "Internet Connection & Server Box Hardware" perspective, they are factors but aren't the key factors.

How much is happening per Game Server will affect how many Resources each Server will need and how each Server will be handling from a User's Perspective while playing within the Resources any Server Box can provide.



For example: I helped run several Servers for a game. We had a "really good" Server Box for our use case for an older Game which technically didn't need such. We could run 5 Servers with 200 Players each along with 1 of the 5 Servers having 200+ Bot NPCs running around and fighting all at the same time and had 4 Player PvP Event Servers plus an always going PvE Server.

Now we upgraded from a weaker Hardware Server which did allow for more Servers to be run from said Server Box but we also saw early on that the PvE Server was chugging due to the amount of things going on with that Server plus the other 4 Servers when all 5 Servers were running at the same time. We decided to have the PvE Server closed for just over an hour to ensure the PvP Event Servers could run well and then restart the PvE Server once Event time was over.

Later we made adjustments to the PvE Server with what we had on it: improved our maps for less clutter, change up our map styles to a more "Server Performance" minded way of making them where we used the minimum to present a scenario for what we were going for with Trees being a key World Object that we kept to a minimum, trying to find ways to improve Performance on our end in various ways like a finite amount of weapons on the terrain players could use, so on and so forth. This allowed us to run all 5 Servers again at the same time (plus varying amounts of Players as time went on so not having 1,000 Players + 200+ NPCs every time Event Servers were running).



Do not underestimate how things are occurring on any given Server and how it affects the Server Box as a whole.

[BigLotroFan]

"we're working on it"

We get that response for a lot of issues. Often over a long period of time. Rarely do we see any results.

[LagunaD2]

Rarely do we see any results.

The 64-bit client (which I admit I never believed they'd pull off) is the only example I can think of where some long-promised solution actually materialized and worked.

But even then it took years and years and years....