Load skirmish data ONLY when it is actually used by the player

[Jadowyn]

To the best of my knowledge, there are only 3 times when skirmish data is actually used by a character:
- when they open the skirmish panel to look at the data
- when they visit the trainer/captain to modify the data
- when they load their skirmish soldier in order to use it in landscape mode or in a skirmish

I know that over 90% of the time I play for weeks at a time and never even look at that data or use a skirmish soldier on any of my toons.

So why bother to load data that will never be used during most play session by most players?


If and when I first need the data during a given play session is the time to actually load it. This cuts down on the login time because less data is loaded and it prevents the waste of bandwidth for those players that have to worry about limits and may not skirmish much.

[banjolier]

To the best of my knowledge, there are only 3 times when skirmish data is actually used by a character:
- when they open the skirmish panel to look at the data
- when they visit the trainer/captain to modify the data
- when they load their skirmish soldier in order to use it in landscape mode or in a skirmish

I'm assuming you're talking about the recent patch notes mention of skirmish statistics calculations done during login. The suggestion to delay this is a good one, but I need to point out that you're barking up the wrong tree. The patch is not changing anything with regard to data about your equipped skirmish traits or skills, so your 3 points here are entirely irrelevant to the issue.


Here's the relevant section of the patch notes:

We've discovered that a very significant chunk of time is spent during logon to read and calculate your skirmish stats. Instead of doing these calculations during logon, we will now do them after. Your Skirmish stats will then update in the background, as you play. This should allow some characters who have been unable to login due to a timeout to get into the game. Please note, your Skirmish stats will not display correctly until all of the processing is completed after logging in.

-- from U11.5 Release Notes

The only place the player sees these statistics (to my knowledge) is the Skirmish Panel, Stats tab. That is the only display that will be affected by the change. Stats are collected on all instances, but presumably there's no need to do the calculations before running more instances that will add to the data. (I could be wrong about that presumption, however.)

As I've mentioned elsewhere, I don't see any reason these calculations could not be delayed until the user opens the Skirmish Panel directly, or an option given to toggle off the statistics calculations entirely.

[Raven-EU]

As I've mentioned elsewhere, I don't see any reason these calculations could not be delayed until the user opens the Skirmish Panel directly, or an option given to toggle off the statistics calculations entirely.

I think doing them in the background after loading up your character is just about the right time to do it (at least if done correctly). If they manage to properly thread the calculation process then game performance should not suffer in any noticeable way. If, on the other hand, the calculation is postponed to when the dialog is actually opened you might need to wait for a very long time before you can actually use the dialog. Let's assume you have a five minute load time now and 1 minute after the patch, then I would assume you'd need about 4 minutes to do the calculations... That's four minutes looking at an unresponsive UI and if there's one thing that freaks me out it's having to look at an unresponsive UI. (Yes, I hate the game-freeze you get when toggling between completed/active quests).

Doing it during log-in is obviously too early.
Doing it when opening the dialog is too late.
Doing it somewhere in between in a manner that do not reduce game performance is when it should be done.

[trancejeremy]

Well, this method apparently causes crashes if you log out of the character before it calculates the data.

Which makes more sense? Use a method that might crash your client if you log out of a character quickly (something that probably happens a lot if you have alts), or a slight pause/freeze when pulling up a panel to look at the thing being calculated?

Especially when you have that already in the game. Hit O for the social panel and it takes a few seconds to load, because it's presumably figuring out who is online at that instant.

[Khafar]

Well, this method apparently causes crashes if you log out of the character before it calculates the data

If so, that should be relatively simple to reproduce and fix. As long as the background thread used to calculate the stats doesn't cause any noticeable performance issues (and it shouldn't), that seems a perfectly fine way to deal with these stats.

Lazy evaluation is a good choice for many of these types of issues, but if it's going to cause the UI to "hang" for a minute or two while it calculates stats, that's too long. Even 30 seconds is borderline too long. Running a worker thread to handle the calculations would be a good option in that case - it can be run at low priority.

Khafar

[Yula_the_Mighty]

Turbine does load data when you tried to access it. Crafting used to be that way. The issue as a customer was the 10+ second client lock up when you opened the crafting panel. I would stare at my PC wondering if the game software had crashed.

The one aspect about the game that I remember being lock up is wanting to see your list of completed quests. That used to take forever with your client completely frozen. It only took 5-10 seconds today.

[Raven-EU]

Well, this method apparently causes crashes if you log out of the character before it calculates the data.

Given that it's a known issue even in the patch notes mean they're well aware of the issue and are probably already working on a fix. This patch will allow players to log characters that were previously unable to load and as such I would consider it a fairly important hot-fix even with the known issues.

Which makes more sense? Use a method that might crash your client if you log out of a character quickly (something that probably happens a lot if you have alts), or a slight pause/freeze when pulling up a panel to look at the thing being calculated?

Load times were up to several minutes for some players so we're not talking about a "slight pause/freeze".

Yes, those used to being able to do a quick log-in-check-mail-log-out rotation will now have to wait for a bit before moving on to the next character. That *is* an inconvenience which is why it was flagged as a known issue in the patch notes.

[Raven-EU]

Turbine does load data when you tried to access it. Crafting used to be that way. The issue as a customer was the 10+ second client lock up when you opened the crafting panel. I would stare at my PC wondering if the game software had crashed.

The one aspect about the game that I remember being lock up is wanting to see your list of completed quests. That used to take forever with your client completely frozen. It only took 5-10 seconds today.

Might be something similar that is causing client pause/freeze when crafting. I've found that the progress bar makes a significant pause when crafting big batches if you leave the crafting panel open but runs smooth if you close it. Go figure

[BangoTwinkletoes]

There's an easy and simple fix that's from the old school book of IT... run an overnight batch job when the servers are quiet and upload the stats once per day.

[Sapience]

Well, this method apparently causes crashes if you log out of the character before it calculates the data.

Which makes more sense? Use a method that might crash your client if you log out of a character quickly (something that probably happens a lot if you have alts), or a slight pause/freeze when pulling up a panel to look at the thing being calculated?

Especially when you have that already in the game. Hit O for the social panel and it takes a few seconds to load, because it's presumably figuring out who is online at that instant.

There is a wild and crazy possibility, too. That the crash issue is a bug and we called it out so players would be aware until we fixed it.

And no, the pause would not be 'slight' when you pull it up if you only load that data when you pull up the panel. For some players in some cases(those who do a lot of skirmishes) it would be minutes.

[Tarlywyn]

There is a wild and crazy possibility, too. That the crash issue is a bug and we called it out so players would be aware until we fixed it.

An interim fix to the crash would be to disable logging out until the the skirms stats update - not that difficult to program (in smaller programs at least). I'm sure the team at Turbine would be more than capable of applying it.

[Darlgon]

Interesting side note to this:

Now, I spend more time looking at the orange loading bar when I am headed into game or swap alts than I did before. Maybe the game thinks I am in and starts loading that data, but my video has not caught up yet.

[mad_ox1]

And no, the pause would not be 'slight' when you pull it up if you only load that data when you pull up the panel. For some players in some cases(those who do a lot of skirmishes) it would be minutes.

We already have to wait minutes to bring up our completed quest log, why not add one more to the equation?

[tanlis]

Thanks for clarifying this, Sapience.

In the future, I would suggest that in the release notes it is called out that something like this is a bug and will be fixed in a future release. Cause even as an software developer, I thought that this behavior was going to be permanent based on how the lack of wording.

There is a wild and crazy possibility, too. That the crash issue is a bug and we called it out so players would be aware until we fixed it.

And no, the pause would not be 'slight' when you pull it up if you only load that data when you pull up the panel. For some players in some cases(those who do a lot of skirmishes) it would be minutes.

[Zepherzaper]

To the best of my knowledge, there are only 3 times when skirmish data is actually used by a character:
- when they open the skirmish panel to look at the data
- when they visit the trainer/captain to modify the data
- when they load their skirmish soldier in order to use it in landscape mode or in a skirmish

I know that over 90% of the time I play for weeks at a time and never even look at that data or use a skirmish soldier on any of my toons.

So why bother to load data that will never be used during most play session by most players?


If and when I first need the data during a given play session is the time to actually load it. This cuts down on the login time because less data is loaded and it prevents the waste of bandwidth for those players that have to worry about limits and may not skirmish much.

good idea they probably will do this sense a lot of players are having log in issues.

[Raven-EU]

Thanks for clarifying this, Sapience.

In the future, I would suggest that in the release notes it is called out that something like this is a bug and will be fixed in a future release. Cause even as an software developer, I thought that this behavior was going to be permanent based on how the lack of wording.

To me it was very obvious that the crash issue was something they're fixing right now. I guess it has to do with past experiences with software distributors. Mine have been fairly good.

My reaction as a long time software developer was that they choose to release this patch because more and more players were starting to have log-in issues. With the increased number of instances being run this problem was probably accelerating in severity every day. They did include in the release notes, not in a separate "known issues document" but in the actual release note that they have a known issue. To me that smacks very high of "we seriously need to get this patch out asap to fix the log-in issues. The crash issue can be fixed for the next patch as there's a simple work-around for it (spend five minutes on a character before logging out)".

[Blister_Burster]

There is a wild and crazy possibility, too. That the crash issue is a bug and we called it out so players would be aware until we fixed it.

And no, the pause would not be 'slight' when you pull it up if you only load that data when you pull up the panel. For some players in some cases(those who do a lot of skirmishes) it would be minutes.

Why is it calculating the skirmish data ingame rather then calculated stats being stored on the server? I would imagine just pulling calculated stats from a server would have no issues....

[Khafar]

Why is it calculating the skirmish data ingame rather then calculated stats being stored on the server?

Probably because they have to be extremely careful about server loading, whereas client loading is close to irrelevant (the vast majority of the time, the game doesn't use all that much CPU).

That said, I'm not quite sure how it can possibly take "minutes" to calculate skirmish stats. My PC could probably calculate pi out to a million decimal places in "minutes". How on earth could skirmish stats need that much horsepower? It seems likely they could make major improvements to their algorithms...

Khafar

[Grimdi]

That said, I'm not quite sure how it can possibly take "minutes" to calculate skirmish stats. My PC could probably calculate pi out to a million decimal places in "minutes". How on earth could skirmish stats need that much horsepower? It seems likely they could make major improvements to their algorithms...

It seems excessive for sure (esp. compared to the limited usage), but then we don't know what actual data is transferred, what is calculated and what is finally displayed.

An interim fix to the crash would be to disable logging out until the the skirms stats update - not that difficult to program (in smaller programs at least).

Actually I'd guess this would be no easier than fixing the bug properly, and adds the possibility for people not being able to log out at all if the check doesn't trigger properly. Background threads can be a real pain in such situations as you never know when (or if) they trigger.

[banjolier]

(the vast majority of the time, the game doesn't use all that much CPU).

How on earth could skirmish stats need that much horsepower?

LotRO isn't CPU-bound on the system I usually use either (3.4ghz quadcore CPU). But it runs good-enough on a 32-bit 1.6ghz machine -- that's the system that will take a while to churn through the skirmish stats, particularly on a character that has done lots of group content.

The currently published minimum specs for the game still calls for a 1.8ghz Pentium 4. That's the class of client machine that will take time to crunch large data sets.

[Widoch]

There is a wild and crazy possibility, too. That the crash issue is a bug and we called it out so players would be aware until we fixed it.

And no, the pause would not be 'slight' when you pull it up if you only load that data when you pull up the panel. For some players in some cases(those who do a lot of skirmishes) it would be minutes.

This exactly.



I timed logins on my main who has run a lot of skirms over the years upwards to 6 - 8 min. It got so bad I would hit login and then go over to either youtube to watch something or turn on the xbox and play a bit. If there was a delay, even for only half that time when I tried opening the window, I'd either say "screw it" and go play another game or I would simply stop running instances.

[FromgalTheMinstrel]

LotRO isn't CPU-bound on the system I usually use either (3.4ghz quadcore CPU). But it runs good-enough on a 32-bit 1.6ghz machine -- that's the system that will take a while to churn through the skirmish stats, particularly on a character that has done lots of group content.

The currently published minimum specs for the game still calls for a 1.8ghz Pentium 4. That's the class of client machine that will take time to crunch large data sets.

I'm sorry, but there is just no way your CPU has anything to do with how slow the skirmish stats are calculated. Every single frame displayed for the game takes thousands and thousands times more data crunching for both CPU and GPU than the trivial sum of numbers that the skirmish stats consist of.
For each little triangle that forms up the body of an orc, a sword, a tree leaf ingame, matrices of decimal numbers have to be multiplied, inverted, deformed based on animation, etc. And every single frame could have, average, close to 20,000, 50,000, 100,000 of those triangles. And your computer does that 60 times per second. That without going into lighting, texturing and other parts that concern the GPU more than the CPU.

No. As they said above, it totally feels the same as with the completed quest log delay. It sounds database related, or that for some strange reason they don't calculate the stats at their servers, but they send all the raw log data to the clients and let the clients calculate it themselves. And what is stalling the clients isn't that it takes too long to calculate those stats, but that the raw data is taking forever to be all sent to the clients.

Or I could be mistaken and in order to add together a couple of hundred of numbers, instead of using the ADD operation, they are renting our CPUs to the NSA to decipher our mails or something like that. I somehow wouldn't be surprised... ¬_¬

[cdq1958]

I'm sorry, but there is just no way your CPU has anything to do with how slow the skirmish stats are calculated. Every single frame displayed for the game takes thousands and thousands times more data crunching for both CPU and GPU than the trivial sum of numbers that the skirmish stats consist of.
For each little triangle that forms up the body of an orc, a sword, a tree leaf ingame, matrices of decimal numbers have to be multiplied, inverted, deformed based on animation, etc. And every single frame could have, average, close to 20,000, 50,000, 100,000 of those triangles. And your computer does that 60 times per second. That without going into lighting, texturing and other parts that concern the GPU more than the CPU.

No. As they said above, it totally feels the same as with the completed quest log delay. It sounds database related, or that for some strange reason they don't calculate the stats at their servers, but they send all the raw log data to the clients and let the clients calculate it themselves. And what is stalling the clients isn't that it takes too long to calculate those stats, but that the raw data is taking forever to be all sent to the clients.

Or I could be mistaken and in order to add together a couple of hundred of numbers, instead of using the ADD operation, they are renting our CPUs to the NSA to decipher our mails or something like that. I somehow wouldn't be surprised... ¬_¬

I second this, for I have instrumented my system (except for the last part :P). There is a lot of data transfer then disk transfer then data transfer, rinse and repeat.

Game rendering typically has a lot on load then small updates, so the issues are different in scope, I think.

Now why they do this client side versus database side; well I can think of a very good reason. Database concurrency and integrity. Best to keep the personal stat calcuations off the DB server, I'd say.