Hytbold and 64-bit client crash and lockout

[Tancanspanc]

Is anyone else experiencing a crash-to-desktop when entering Hytbold using the 64 bit client?

This happens every time (tried 3 different alts, Hytbold at different stages of rebuild) for me on both my desktop and laptop PCs (both Win 10, with decent modern gaming h/w specs).

It then becomes impossible for me to log in to that character, unless I revert to the 32-bit client, and once in game, move the character away from Hytbold.

[DoRonRon]

Sev mentioned that reverting to 32-bit to circumvent these types of issues is not acceptable so we can hope this has filtered down to a blue on the case. However Orion's lead hasn't motivated others to post more so you might be frustrated in getting any acknowledgement. How can Sev enforce behaviour he doesn't embrace himself?

I'm loosing interest in testing and confirming such matters for the good of the game when SSG aren't motivated to do it themselves or even say if they are.

[Oghran]

In my case, looking at Window's Event viewer (Custom Views/Administrative Tasks) gave me a clue. Try that.

[Tancanspanc]

In my case, looking at Window's Event viewer (Custom Views/Administrative Tasks) gave me a clue. Try that.

Good suggestion:

1. The following is from the event log.


Faulting application name: lotroclient64.exe, version: 3301.65.4364.4059, time stamp: 0x62ddd058
Faulting module name: ntdll.dll, version: 10.0.19041.1949, time stamp: 0xfe96c48e
Exception code: 0xc0000005
Fault offset: 0x0000000000063466
Faulting process id: 0x5fc4
Faulting application start time: 0x01d8cbfa1061e707
Faulting application path: D:\temp\The Lord of the Rings Online\x64\lotroclient64.exe
Faulting module path: C:\WINDOWS\SYSTEM32\ntdll.dll
Report Id: bea75ad6-3b74-4842-8fe7-daba5ad0f578
Faulting package full name:
Faulting package-relative application ID:




2. The following is from lotroclient64.log (in 'users...AppData\Local\The Lord of the Rings Online)


481829527.678: ----CRASH REPORT START----
481829527.678: Program fault: ACCESS_VIOLATION (0xc0000005) trying to write to 0x0000000280000028
481829527.678: Detailed report:

Current local time: Mon Sep 19 08:40:56 2022


Version Report generated by CoreVersion : 2.0.143.1 (0x0200008F):
Language: English (United States)
CompanyName : Standing Stone Games, LLC.
FileDescription :
FileVersion : 3301.0065.4364.4059
InternalName : lotroclient64
LegalCopyright : Copyright © 2016-2022 Standing Stone Games, LLC.
OriginalFilename : lotroclient64.exe
ProductName : lotroclient64
ProductVersion : 3301.0065.4364.4059 lotro_live
Comments : compiled Sun Jul 24 18:59:47 2022 : RELEASE
TurbineBuildVersion : 3301.0065.4364.4059.RELEASE
TurbineType : Admin External

Version information for error module C:\WINDOWS\SYSTEM32\ntdll.dll:
Language: English (United States)
CompanyName : Microsoft Corporation
FileDescription : NT Layer DLL
FileVersion : 10.0.19041.1741 (WinBuild.160101.0800)
InternalName : ntdll.dll
LegalCopyright : © Microsoft Corporation. All rights reserved.
OriginalFilename : ntdll.dll.mui
ProductName : Microsoft® Windows® Operating System
ProductVersion : 10.0.19041.1741



481829527.678: ----CRASH REPORT END----

3. The following are the relevant 'non successful' events taken from Process Monitor of lotroclient64.exe the second before the crash.

08:40:56.1209991 lotroclient64.exe 24516 QueryOpen D:\temp\The Lord of the Rings Online\x64\winnlsres.dll NAME NOT FOUND
08:40:56.1213781 lotroclient64.exe 24516 CreateFileMapping C:\Windows\System32\winnlsres. dll FILE LOCKED WITH ONLY READERS SyncType: SyncTypeCreateSection, PageProtection: PAGE_EXECUTE_READWRITE
08:40:56.1215406 lotroclient64.exe 24516 RegOpenKey HKLM\Software\Microsoft\Langua geOverlay\OverlayPackages\en-US NAME NOT FOUND Desired Access: Read
08:40:56.1216983 lotroclient64.exe 24516 CreateFileMapping C:\Windows\System32\en-US\winnlsres.dll.mui FILE LOCKED WITH ONLY READERS SyncType: SyncTypeCreateSection, PageProtection: PAGE_EXECUTE_READWRITE
08:40:56.1223690 lotroclient64.exe 24516 CreateFileMapping C:\Windows\System32\en-US\ntdll.dll.mui FILE LOCKED WITH ONLY READERS SyncType: SyncTypeCreateSection, PageProtection: PAGE_EXECUTE_READWRITE
08:40:56.1224902 lotroclient64.exe 24516 RegQueryValue HKLM\System\CurrentControlSet\ Control\WMI\Security\540dc156-e9d6-42dc-a225-29794149a495 NAME NOT FOUND Length: 528
08:40:56.1229860 lotroclient64.exe 24516 QuerySecurityFile C:\Windows\System32\ntdll.dll BUFFER OVERFLOW Information: Owner, Group, DACL
08:40:56.1259633 lotroclient64.exe 24516 QueryOpen D:\temp\The Lord of the Rings Online\x64\faultrep.dll NAME NOT FOUND


Pick the bones out of that lot...

[Moonaz]

Check that your add ones are working

[alwaysbroke]

481829527.678: Program fault: ACCESS_VIOLATION (0xc0000005) trying to write to 0x0000000280000028

Based on this error you might try a couple things:

- open an admin command prompt and run "chkdsk c:/f", then run "sfc /scannow"
- update device drivers, or if you've already done this, try rolling back to a previous version that worked
- check for malfunctioning memory by searching for: Windows Memory Diagnostic (in start menu)
- check your page file: open task manager, performance tab, select your system (c) drive at the left, now on the right it should say

System disk: Yes
Page file: Yes

If not, run "sysdm.cpl", advanced tab, click performance, advanced tab, click change, choose "Automatically manage paging file size for all drives" (or set some sane non-zero defaults). Click OK. Switch to tab "Data Execution Prevention", and make sure the option "Turn on DEP for essential Windows programs and services only" is selected.

[Tancanspanc]

Based on this error you might try a couple things:

- open an admin command prompt and run "chkdsk c:/f", then run "sfc /scannow"
- update device drivers, or if you've already done this, try rolling back to a previous version that worked
- check for malfunctioning memory by searching for: Windows Memory Diagnostic (in start menu)
- check your page file: open task manager, performance tab, select your system (c) drive at the left, now on the right it should say

System disk: Yes
Page file: Yes

If not, run "sysdm.cpl", advanced tab, click performance, advanced tab, click change, choose "Automatically manage paging file size for all drives" (or set some sane non-zero defaults). Click OK. Switch to tab "Data Execution Prevention", and make sure the option "Turn on DEP for essential Windows programs and services only" is selected.

Thanks for taking the time to offer those suggestions. They are all perfectly reasonable approaches to troubleshooting the issue, but as I've now encountered the same, very specific and repeatable, crash on THREE different machines (and two different accounts) I think the likelihood of it being related to a hardware, driver or pagefile fault is incredibly small.

I've sent the bug report in to SSG through the formal channels and so will leave it for them to investigate further. Hopefully by posting here in general discussion a few more people may now be aware of it, and if they hit the same problem now know they can use the 32-bit client to retrieve their character. At least until SSG retire it in the near future.

[alwaysbroke]

Thanks for taking the time to offer those suggestions. They are all perfectly reasonable approaches to troubleshooting the issue, but as I've now encountered the same, very specific and repeatable, crash on THREE different machines (and two different accounts) I think the likelihood of it being related to a hardware, driver or pagefile fault is incredibly small.

I've sent the bug report in to SSG through the formal channels and so will leave it for them to investigate further. Hopefully by posting here in general discussion a few more people may now be aware of it, and if they hit the same problem now know they can use the 32-bit client to retrieve their character. At least until SSG retire it in the near future.

I have no progress in rebuild. Is there is minimum amount of rebuild?

[Tancanspanc]

I have no progress in rebuild. Is there is minimum amount of rebuild?

Nope.
One character had never been there before (valared from low level to 120).
One character was about 70% done
One character completed Thane of Hytbold

[Polymachos]

08:40:56.1209991 lotroclient64.exe 24516 QueryOpen D:\temp\The Lord of the Rings Online\x64\winnlsres.dll NAME NOT FOUND
08:40:56.1213781 lotroclient64.exe 24516 CreateFileMapping C:\Windows\System32\winnlsres. dll FILE LOCKED WITH ONLY READERS SyncType: SyncTypeCreateSection, PageProtection: PAGE_EXECUTE_READWRITE

Looks like a system file, why does your client look for that in the Lotro directory? And why is it a problem that it is read only? I wouldn't want a system file to be changed by anything except maybe an update implemented by Microsoft.

I probably missed the answer in this wall of text, but I will ask this anyway: did you already try the repair function of the client? Launcher, this tiny button on the upper edge, slightly to the right, with the upended triangle/arrohead downwards, Options, repair tab.


Greetings, Polymachos

[ArgoMonster]

Hi, I am currently working my way through the Hytbold series, and am using the 64 bit client - had a crash to desktop a couple days ago - VERY rarely if ever
happens - will keep an eye out and see if this is repeatable - Like the poster - Hytbold is part way thru its rebuild.

Jim

[Tancanspanc]

I probably missed the answer in this wall of text, but I will ask this anyway: did you already try the repair function of the client? Launcher, this tiny button on the upper edge, slightly to the right, with the upended triangle/arrohead downwards, Options, repair tab.


Greetings, Polymachos

Just to let folks know that this suggestion by Polymachos resolved the problem. Many thanks to him/her, and all who took time to respond. I wasn't aware (or had forgotten) about this option and never used it before. It seems that this Repair function, removes your installed dat files and re-downloads and installs a fresh set; so practically a clean reinstall (20+ GB for me).

Incidentally , SSG did get back to my bug report in good time ( a couple of days ) but passed it straight over to customer support. They too responded in good time but their suggestion (rename the existing LOTRO folder in users/documents and restart) didn't make any difference, and their subsequent suggestion to send them dxdiag and msinfo32 reports from my machine(s), whilst understandable for basic troubleshooting, would have been unlikely to narrow down the cause of the crashing.

[Arnenna]

Wish I'd read this thread earlier. Just took an alt to Hytbold to hand in the intro quest - just to make some room in the quest log. I didn't even get through the gate. Relogging now, so will see if I get the same issue.

Changed to 32bit and no joy.

Result = Another stuck character.

I give up. Time to log out and switch to another form of entertainment.

[Polymachos]

Just to let folks know that this suggestion by Polymachos resolved the problem.

Glad that I could help. I guess, there were some libraries involved that didn't cooperate as expected. In such a case, a complete replacement of all files by the new ones, them being all at the same version level, is probably a better way to fix it, than starting a lengthy wild goose chase after the faulty entry.


Greetings, Polymachos

[alwaysbroke]

What interesting about this solution is that the launcher is supposed to check requirements. Apparently the "check" only confirms a file exists.

SSG, you need to check the hash of each file. Don't just check if the required files exist and have an expected size. Had your launcher correctly checked the hash of the required files (a compute cost incurred by the client machine), this would have been fixed by the user reloading the launcher. It would have been fixed the first day without needing to submit a ticket.

Now we know better. A kludge would be to install cygwin on Windows, then hash the game folders ourselves.

Edit: Futhermore, if you check the hash of each file, you save yourself and the player bandwidth. The repair option treats the install as broken and downloads everything. However, it's well known that your launcher is capable of downloading individual files (if they're missing). So treat corrupt files as missing after checking the hash, then your client will download a couple gigabyte instead of a whole install.

[Arnenna]

Glad that I could help. I guess, there were some libraries involved that didn't cooperate as expected. In such a case, a complete replacement of all files by the new ones, them being all at the same version level, is probably a better way to fix it, than starting a lengthy wild goose chase after the faulty entry.


Greetings, Polymachos

Your fix worked for me, but yikes, what a palava.

@SSG Isn't your launcher capable of finding corrupt or missing files and then simply replacing them? Having to do a complete download and reinstall of the game is a bit manic. Especially when you've had three characters drop into Limbo within a week.

Edit: Ohh hold on . . . just ported out of hytbold and . . . disconnected from server. I swear if this character is stuck again, I won't be reinstalling the game, I'll be uninstalling it.

Edit: yup, stuck again. No way am I reinstalling the game again. It can stay stuck. If this doesn't get fixed (its not just Hytbold, it is also Rohan Homesteads, and probably other places that either nobody has brought to the forum, or people haven't discovered yet), then good luck keeping new or returning players, especially players that maybe left due to game instability.

Movie night tonight it is then.

[LagunaD2]

Checking the hash code of tens of GB of files takes a non-negligible amount of time; it would delay your start-up and is usually unnecessary.

I don't know of any MMO that does it every time you launch the game.

[alwaysbroke]

Checking the hash code of tens of GB of files takes a non-negligible amount of time; it would delay your start-up and is usually unnecessary.

I don't know of any MMO that does it every time you launch the game.

Maybe for mechanical HDD. It actually depends on storage performance and hashing algorithm. The speed of the hashing algorithm will be vastly faster than storage. Assuming a throughput of 100MB/s (mechanical HDD), I see your point. In that case add the hash check to the repair option instead. That would be the best of both worlds. As it stands the repair option treats the entire install as broken, requiring a 30GB download.

Spending 5min (with a mechanical HDD) in the repair option, to save a player and SSG 20+GB of download and time...priceless

I'll benchmark it to provide concrete numbers.

edit: Storage performance is the limiting factor. A rather slow SSD can do a complete hash of the LOTRO folder in 1m20s. A mechanical HDD may take 5m or more. NVMe can complete the hashing in <40s. This is using SHA1 and MD5 to confirm the problem is storage performance bound.

So, okay, don't do it every time. Add it to the launcher repair option. How many people come to these forums with "a problem" that we could fix by telling them to run the repair option to check their install. As it stands, the reason we don't eagerly recommend the repair option is because it doesn't check file hashes. I wouldn't want to haphazardly recommend the repair option because it redownloads/reinstalls everything. Many problems related to corrupt files get debugged using a slow and arduous process. Look at SSG's ticket response (to OP) They don't even want to recommend repair as a first option. Adding this to the launcher repair would be a substantial improvement. The first troubleshooting step would always be "check your install with repair".

[DoRonRon]

There was a change made to the launcher that sped up the update/download process a couple of years ago. This made a significant difference. I'm all for slicker coding but experience tells me that doesn't come about by not leaving out some key functionality.

When it's judged only on the result and not the compromises made to get there as apparent with mission creation over playability things are going to go wrong.

[alwaysbroke]

I think I may have a clue why the problem is happening. After the last patch (33.2.1) I went around and updated all my LOTRO boxes. I encountered a strange bug on my Windows 11-N install.

The patch didn't ask for admin privileges. It then proceeded to "update" the files anyway. In the process I got several errors:

- Could not patch file "Lotrolauncher.exe". error 5: Access is denied.
- Failed to restart the launcher, error 2: The system cannot find the file specified

I got variants of the above for x64/lotroclient64.exe too. Here's where things get weird. Noticing these errors, and lack of requesting admin, I relaunched forcing admin. Apparently the launcher thought everything was okay and took me to login.

But wait there's more. I patched a Windows 11 Pro install without any issue. I then hashed the contents of the LOTRO install folder from this machine. (The idea was to use the hash results to assist other players.) When I compared the hashes from this machine to the previous machine (with borkage) I found something interesting...

The patching had actually failed (on the one that didn't originally ask for admin), and the hashes where markedly different. So essentially, due to a bug in the patcher I have a Windows 11-N machine with different file hashes from another machine where the patching worked perfectly. In both cases the launcher currently loads but clearly one is broken.

[alwaysbroke]

Bad news for anyone experiencing this type of crash/lockout issue. tl;dr you should just run the repair.

OK so if this is the result of a patching failure like I experienced above read on. You basically have at least 31.2GB of unpatched, corrupted files, in 104 files, including most dat, datx, and patchcache.bin. Your install is haloed.

[Arnenna]

Character fixed and back in the world after server daily resets.

[Arnenna]

Character now in limbo again, after stabling from Car Bronach to Mattugard. /sigh /tantum /slapshead

This has got to be fixed. Putting up with some freezes, lag spikes, rubber banding or hitching in game is one thing, but this takes it into the realm of unplayability. Losing your character after less than 30 minutes play, is just - unplayable. Players paying a sub are losing their bought time.

[Tralfazz]

This is not happening to everyone. Besides playing my main in and out of Hytbold for the last couple of days anyway, I've brought other characters there to test by walking, by using a Milestone skill, and via Stable Master travel. Hytbold is in various states of rebuild for each character I tested. I've been experiencing no issues at all. And even though Hytbold isn't the busiest area I do see other players there.

I mention this because while your characters become "un-stuck" at 3:00am server time when the new game day starts, because the problem is not universal there must be something on your end which is causing it. The fact that a full game reinstall also resolves your issue seems to confirm this.

Now, when I say, "there must be something on your end," I do not mean that you have done something wrong or there is something wrong with your computer in any way. LOTRO is to blame. It can be extremely finicky about DX components/graphics drivers/hardware, file permissions, and other things which cause our favorite game to run poorly, not at all, or generate odd behaviors like the one which is the topic of this thread; even when every other game played on that computer works perfectly fine. There are other potential causes including the way Hytbold graphics and interactives are staged for individual characters but there is something specific to your computer (or its graphics card or drivers) or its Windows components which LOTRO doesn't like due to the specific nature of Hytbold. (I wonder if you would experience the same problem in another staged area.)

Did this begin to happen to you, on the same hardware, only after a particular game patch or after a specific graphics driver update?

I don't have a solution for you. You said you already submitted a ticket. I hope you'll submit a ticket, including a link to this thread, each time this happens to you. People reporting this frequently and consistently is the best way for this issue to get attention from the SSG engineers.

[Arnenna]

This is not happening to everyone. Besides playing my main in and out of Hytbold for the last couple of days anyway, I've brought other characters there to test by walking, by using a Milestone skill, and via Stable Master travel. Hytbold is in various states of rebuild for each character I tested. I've been experiencing no issues at all. And even though Hytbold isn't the busiest area I do see other players there.

I mention this because while your characters become "un-stuck" at 3:00am server time when the new game day starts, because the problem is not universal there must be something on your end which is causing it. The fact that a full game reinstall also resolves your issue seems to confirm this.

Now, when I say, "there must be something on your end," I do not mean that you have done something wrong or there is something wrong with your computer in any way. LOTRO is to blame. It can be extremely finicky about DX components/graphics drivers/hardware, file permissions, and other things which cause our favorite game to run poorly, not at all, or generate odd behaviors like the one which is the topic of this thread; even when every other game played on that computer works perfectly fine. There are other potential causes including the way Hytbold graphics and interactives are staged for individual characters but there is something specific to your computer (or its graphics card or drivers) or its Windows components which LOTRO doesn't like due to the specific nature of Hytbold. (I wonder if you would experience the same problem in another staged area.)

Did this begin to happen to you, on the same hardware, only after a particular game patch or after a specific graphics driver update?



I don't have a solution for you. You said you already submitted a ticket. I hope you'll submit a ticket, including a link to this thread, each time this happens to you. People reporting this frequently and consistently is the best way for this issue to get attention from the SSG engineers.

Re: the re-install. It worked, then while my character walked way from Edgar, after handing in, she disconnected from the server and went into Limbo again. I'm not really sure could have possibly happened my end in that 20 seconds.

I've only been experiencing this since last weeks patch. Never ever had it happen before. No other changes on PC, no Windows updates, driver updates or anything else.

I lost two characters late last week (after the patch), porting out of a Rohan homestead (daily reset brought them back), and this character that got stuck in Hytbold yesterday (reinstal brought her back, only for her to disconnect again seconds later), is now in limbo after stabling from one place in Gundabad, to another place in Gundabad.