Once it's all said and done, please share with us what broke the poor runekeepers

[Foxmaiden14]

For the people asking why they don't just do a rollback and restore the database. Consider all the people who had logged on -- everything they did or collected would be lost, so that's a no-go as well.

That's not really an issue... rollbacks happen in every mmo. It's unfortunate, but if devs have to rollback a database, people are going to loose progress.

[maartena]

That's not really an issue... rollbacks happen in every mmo. It's unfortunate, but if devs have to rollback a database, people are going to loose progress.

I think I would rather wait a day and NOT lose my progress. But that is just me, I happen to have a computer that has Steam installed and on it about 50 other games I purchased over the years, so I can just play something else for a day. Personally I think waiting a day so they can fix it properly is a much, much better solution than rolling back to a previous backup which would not include Update 33 either.

[Foxmaiden14]

Well, Cord has said that it's not a possibility anyway, so they are just wasting their time asking for a rollback.

[Rogdush]

IMHO devs need to rollback db to point before update. I am certain they do backup everytime before any update or even maintanance.
So.. I would guess db need to be rollbacked before update 33 (just after world shutdown so no loss of progress). LOTRO db is not any db, it contains 15 years of data of every player who played the game. Rolling it back takes loads of time, then need to reapply the update 33 with fix to satchels hopefully already done. Probably need to do db reindex or some other cleanup thingy, then launch everything up and its the process for every server separate. Some things cant be speed up even with best cpus.
No wonder it takes time. Just play something else for a day or two. This needs to be done perfect now and im sure devs gonna do it good.
Mistakes happens everywhere, tbf lotro dont have much of it. Last such accident was like year ago afair.
Nothing unusual, just need to be patient have fun guys.

[Tralfazz]

For the people asking why they don't just do a rollback and restore the database. Consider all the people who had logged on -- everything they did or collected would be lost, so that's a no-go as well.

A rollback is one of the last things an MMO manager wants to do. However, assuming they followed a decent professional procedure and performed a full backup right after the servers came down yesterday morning and before they started applying the u33 and other updates; people aren't going to lose much progress.

It was announced that the game worlds reopened at 11:52am EDT. It was announced that the game worlds were shut down at 12:50pm EDT. That's 58 minutes. Assuming the announcement of reopening came a little late, that's still barely more than an hour. With extremely rare exceptions, people hardly had time to open all their Anniversary gifts and ooh-aah over the pretties before the servers came down. There is somewhere between very, very, very few and zero people who might have completed something both challenging and important in that 58-90 minutes.

So hopefully the bug can be fixed, the fix tested thoroughly, a simple rollback where only a tiny minority of players lose no more than an hour and a half of "progress" and the servers come up and we're back to having fun together.

If a backup was not made between the servers coming down and the patches being applied yesterday morning, well, shame on somebody I guess. That would be very poor procedure.

[Elfrich]

I am suffering from LotRO withdrawal, but it doesn't appear to be life threatening

Admit it. This is the most productive day y'all have had in years!

[Aurehir]

For the people asking why they don't just do a rollback and restore the database. Consider all the people who had logged on -- everything they did or collected would be lost, so that's a no-go as well.

The server was only up for only two hours yesterday. Is this really such a big deal, a couple of hours playtime, whereas the rest of the community is forced to wait over a day?

Or are you suggesting that they don't have a server backup from Wednesday morning just before the update? I don't imagine this to be the case, but such a level of incompetence would naturally infuriate quite a lot of people.

[Bjornfree]

The server was only up for only two hours yesterday. Is this really such a big deal, a couple of hours playtime, whereas the rest of the community is forced to wait over a day?

Or are you suggesting that they don't have a server backup from Wednesday morning just before the update? I don't imagine this to be the case, but such a level of incompetence would naturally infuriate quite a lot of people.

They want to give people the anniversary gifts. They need to fix this to ensure everyone gets them.

[Aurehir]

They want to give people the anniversary gifts. They need to fix this to ensure everyone gets them.

Why not roll back the server and reset the giveaway to where it was on Wednesday morning, release Update 33 without the RK patch, then in the meantime work on a fix for the RK bug and release it with the following update?

All that's lost is two hours of a minority of the community's playtime. Hardly significant compared to the entire playerbase, not to mention new players anticipating this update, waiting more than a day to log on.

[Aurehir]

Looks like the game worlds are back up. Well done, SSG!

[Iktomi]

For the people asking why they don't just do a rollback and restore the database. Consider all the people who had logged on -- everything they did or collected would be lost, so that's a no-go as well.

I was early in and was there when they came back down. I estimate (wasn't watching the clocks too closely) maybe 30 to 40 minutes *at most* had passed. So there's not a lot of things did or collected that would be lost.

On the other hand, there are ties and hooks that extended out past the game server databases and into general account stuff.

[04/20 10:57:57 AM] Thank you for being a VIP! Your account has been flagged so that all current and future characters will receive Tome of the Chestnut Corgi
[04/20 10:57:57 AM] You have received the following account reward: Tome of the Chestnut Corgi
[04/20 10:57:57 AM] You have received the following account reward: Light Weapon Aura
[04/20 10:57:57 AM] Account item Mordor Collector's Edition has been purchased!
etc

depending on when the account level databases were last backup up (might not have been before a big game patch as it wasn't an 'affected system' in the overall scope of the upgrade. So rolling back the game servers to remove all the goodies awarded (primarily the once per account items) might make it extra problematic to re-award them. Log in after roll back, game server checks and the account DB says "yep, that person got the once only goodies - skip", and you've just created another potentially bigger problem.

We don't know how things are hooked around and a 'simple' rollback of the game servers might not be that simple in this particular case.



============================== =====
edit:

After getting in it looks like they did some form of rollback after all.
I didn't get the all the chat spam about account flagged this, account flagged that but all the boxes I had opened, the frame I had applied the pets I'd added to my skills (and renamed) all reverted back to a "starting" state/position.

So there is that.

[DoRonRon]

One or two times during the night I was imagining Orion screaming "who the ^$&& hacked my beautiful coding?".

But hey they revert to the "halo" effect trope, no one has to take responsibility. Cord will keep everyone safe from attack.

How bad are the hacks if you delete an item in changing how it's displayed?

Now just a choice to come clean on the last backup date or throw it at the beleaguered CS? Annoy everyone equally and reveal too much or push a few to their breaking point? A budget requiring technical replacement solution based on Stone resources used too much to even consider? Who maybe around to get it up and running in an hour or two?

EDIT: So glad to be proved wrong and there was a backup, but I hope it was yesterdays rather than this mornings.

[Tralfazz]

EDIT: So glad to be proved wrong and there was a backup, but I hope it was yesterdays rather than this mornings.

Pretty sure that Cordovan got his "todays" and "yesterdays" a little mixed up in the announcement here:
https://forums.lotro.com/forums/show...rom-April-20th

Probably because he got no sleep last night!

[Calding]

Pretty sure that Cordovan got his "todays" and "yesterdays" a little mixed up in the announcement here:
https://forums.lotro.com/forums/show...rom-April-20th

Probably because he got no sleep last night!

I am assuming he had it written last night when the decision was made and it was just sitting to be published. Unfortunately today is not the same day as it was when he wrote it, although it is likely hard to distinguish the two for those involved. Hope they get some rest soon.

[Cordovan]

Pretty sure that Cordovan got his "todays" and "yesterdays" a little mixed up in the announcement here:
https://forums.lotro.com/forums/show...rom-April-20th

Probably because he got no sleep last night!

Correct, the rollback was to 7:00 AM Eastern on Wednesday.

[Cordovan]

I am assuming he had it written last night when the decision was made and it was just sitting to be published. Unfortunately today is not the same day as it was when he wrote it, although it is likely hard to distinguish the two for those involved. Hope they get some rest soon.

Yes

[Palentian]

Correct, the rollback was to 7:00 AM Eastern on Wednesday.

So, is the underlying issue fixed, or is this just a temporary thing?

[SniperCT]

The issue is related to a halo effect of a relatively minor bug fix to Runekeeper to get their satchels to properly display on the avatar.

Thank you!