PR_END_OF_FILE_ERROR accessing LOTRO forums on VPN

[alwaysbroke]

There are many connection errors that can occur accessing the forums. This type of error has some causes local to your computer, and one major SSG cause. In general this message means the HTTPS connection did not complete with a mutually agreed-upon cipher.

Local to your computer:
- Security software such as Avast, AVG, ESET. Try updating them or disabling them temporarily to see if the problem still occurs.
- Incorrect system clock. Secure connections require an accurate system clock. Try setting your system time to auto and then sync your network time.

The SSG cause:
- The IP address you're using is on a blocklist. You'll probably only see this when using a VPN. This isn't the same as an account-based ban. This means the IP address you're using was associated (on the internet) with abuse and/or spam, and is now on one of the prominent block lists. To check this I confirmed that a known "bad" IP was on the blocklists, and was receiving 403-Forbidden at several major websites. It was also receiving PR_END_OF_FILE_ERROR when accessing the forums. SSG uses these same blocklists on the forum and in-game. The easiest solution is to change to another endpoint. Even if it "works" in-game, you'll end up being disconnected from the world chat server, and experience "lost progress".

tl;dr -- A good rule of thumb is to check your VPN by loading the forums as a canary. You'll have fewer in-game issues if the forums are loading.

Technical Considerations (for SSG):
The actual cause of the PR_END_OF_FILE_ERROR is that the forum server refuses to finish the HTTPS handshake. It's not that the block prevents a connection. In wireshark the client initiates a connection, and goes through all the usual steps until it reaches the part where client/server agree on cipher. Then the server just sits there until the connection times out. I'd imagine you did this to prevent wasting CPU cycles negotiating a full HTTPS connection. When I check the forum server configuration on SSL Labs I see (rather hilariously) that you use TLS_DHE_RSA_WITH_AES_256_GCM_S HA384 as your preferred cipher. So most likely you did this to avoid the high-cost associated with Diffie-Hellman key exchange. However, this is even more hilarious (not just for preferring a cipher around since Android 4.4.2) because you still consume TCP buffers for the SYN-ACK-FIN. Which means your own method of preventing blocklist IP's from accessing the forums does nothing to prevent denial of service via resource exhaustion. You leave those resources in limbo for ~10seconds before the connection closes. A better approach would be to not negotiate a partial HTTPS connection (fail-fast), and not even ACK the initial SYN.

[alwaysbroke]

Firefox

Secure Connection Failed
An error occurred during a connection to forums.lotro.com. PR_END_OF_FILE_ERROR

---

Safari

Safari Can't Open the Page
Safari can't open the page "forums.lotro.com" because the server unexpectedly dropped the connection.

---

Safari Mobile

Safari could not open the page because the server stopped responding.

---

Opera, Chrome/Chromium, Brave, Vivaldi

This page isn't working
forums.lotro.com didn't send any data.
ERR_EMPTY_RESPONSE

---

Edge

This page isn't working right now
forums.lotro.com didn't send any data.
ERR_EMPTY_RESPONSE

[egarthur]

There are many connection errors that can occur accessing the forums. This type of error has some causes local to your computer, and one major SSG cause. In general this message means the HTTPS connection did not complete with a mutually agreed-upon cipher.

Local to your computer:
- Security software such as Avast, AVG, ESET. Try updating them or disabling them temporarily to see if the problem still occurs.
- Incorrect system clock. Secure connections require an accurate system clock. Try setting your system time to auto and then sync your network time.

The SSG cause:
- The IP address you're using is on a blocklist. You'll probably only see this when using a VPN. This isn't the same as an account-based ban. This means the IP address you're using was associated (on the internet) with abuse and/or spam, and is now on one of the prominent block lists. To check this I confirmed that a known "bad" IP was on the blocklists, and was receiving 403-Forbidden at several major websites. It was also receiving PR_END_OF_FILE_ERROR when accessing the forums. SSG uses these same blocklists on the forum and in-game. The easiest solution is to change to another endpoint. Even if it "works" in-game, you'll end up being disconnected from the world chat server, and experience "lost progress".

tl;dr -- A good rule of thumb is to check your VPN by loading the forums as a canary. You'll have fewer in-game issues if the forums are loading.

Technical Considerations (for SSG):
The actual cause of the PR_END_OF_FILE_ERROR is that the forum server refuses to finish the HTTPS handshake. It's not that the block prevents a connection. In wireshark the client initiates a connection, and goes through all the usual steps until it reaches the part where client/server agree on cipher. Then the server just sits there until the connection times out. I'd imagine you did this to prevent wasting CPU cycles negotiating a full HTTPS connection. When I check the forum server configuration on SSL Labs I see (rather hilariously) that you use TLS_DHE_RSA_WITH_AES_256_GCM_S HA384 as your preferred cipher. So most likely you did this to avoid the high-cost associated with Diffie-Hellman key exchange. However, this is even more hilarious (not just for preferring a cipher around since Android 4.4.2) because you still consume TCP buffers for the SYN-ACK-FIN. Which means your own method of preventing blocklist IP's from accessing the forums does nothing to prevent denial of service via resource exhaustion. You leave those resources in limbo for ~10seconds before the connection closes. A better approach would be to not negotiate a partial HTTPS connection (fail-fast), and not even ACK the initial SYN.

Thanks for looking into the forum "empty response" issue. I see it constantly, but it's been just a minor annoyance, and I haven't spent the time to fire up Wireshark and do some Root Cause Analysis.

Your diagnosis seems quite plausible, but I have one nagging question: why does this only happen some of the time? Reloading usually "fixes" it; sometimes it takes multiple reloads. Everything you explained should result in persistent failure.

Could it be that there is a load-balanced cluster, and only some of the nodes in the cluster are misconfigured in this way? Do your observations bear that out? Of course, if it's an L2 load-balancer, you can't trivially tell which node you're talking to in the member pool, but there might be something in the response headers (especially if they're running different webserver software, or different versions)...

With respect to your "canary" comments: it's decent advice, but I would separate it more clearly from the "technical considerations" SSL issue, maybe even a different thread, otherwise folks might be confused into thinking that the SSL issues somehow affects their game client, causes lag, etc. Those are really separate and distinct issues.

P.S. Not that it really matters, but I assume SYN-ACK-FIN is an abbreviation, since all of the flags for the complete lifecycle of a gracefully-torn-down TCP connection would of course be much longer than that, e.g. SYN-SYN/ACK-ACK-FIN-FIN/ACK or something like that.

[alwaysbroke]

Thanks for looking into the forum "empty response" issue. I see it constantly, but it's been just a minor annoyance, and I haven't spent the time to fire up Wireshark and do some Root Cause Analysis.

No worries. I have too much free time right now.

Your diagnosis seems quite plausible, but I have one nagging question: why does this only happen some of the time? Reloading usually "fixes" it; sometimes it takes multiple reloads. Everything you explained should result in persistent failure.

When I tested this problem I used a VPN IP that was already on several anti-spam/abuse blocklists. That may be the difference. SSG may also be doing some dynamic blocking. Most likely to prevent denial of forum service though SYN-ACK style resource exhaustion. What you're describing, about the refresh working, sounds like a dynamic block. Whereas, what I tested was a blocklisted IP.

Could it be that there is a load-balanced cluster, and only some of the nodes in the cluster are misconfigured in this way? Do your observations bear that out? Of course, if it's an L2 load-balancer, you can't trivially tell which node you're talking to in the member pool, but there might be something in the response headers (especially if they're running different webserver software, or different versions)...

Who can really say what SSG is doing here. Is something misconfigured? evidently yes but I can only measure the symptom. The connection stops before finishing the https handshake. Unfortunately the headers don't exist at this point (because the connection is basically dropped before completion). Chances are I really don't want to know how they're doing it. It's breaking TLS though. It's a completely hackish solution that depends on breaking the way things should work. Futhermore, SSG is resorting to forcing TLS 1.0 on every browser that connects to the forum, and using non-forward secure ciphers in the process (which I discussed in a separate thread).

With respect to your "canary" comments: it's decent advice, but I would separate it more clearly from the "technical considerations" SSL issue, maybe even a different thread, otherwise folks might be confused into thinking that the SSL issues somehow affects their game client, causes lag, etc. Those are really separate and distinct issues.

Quite true. Many people in-game have mentioned "lost progress", or disconnection from the world chat server. They typically blame the game. I'll create a separate thread for that part like you suggest. Thanks!

P.S. Not that it really matters, but I assume SYN-ACK-FIN is an abbreviation, since all of the flags for the complete lifecycle of a gracefully-torn-down TCP connection would of course be much longer than that, e.g. SYN-SYN/ACK-ACK-FIN-FIN/ACK or something like that.

Also quite true. I was trying to abbreviate (tl;dr style), but I couldn't think of anything simpler than just mentioning the primitive steps. The actual lifecycle of TCP can be much more complex as you've noted.

[maartena]

I use VPN from time to time to get around geographical blocks, such as being in the United States, and wanting to capture and save some shows from BBC iPlayer in the UK, or NPO/NLZiet in The Netherlands.

It has been my experience that with the VPN active, from time to time certain sites "ban" me. Not necessarily a ban, but the IP address I have acquired through the VPN has been blocked. When you use a commercially available VPN, you get a new IP address every time you connect, and as VPN's are also used to circumvent IP bans, absuse/troll forums of all sorts, utilize for gold selling or nefarious activities on games, etc, etc..... the chances of you connecting to a VPN node and getting an IP address that has in the past been used in a negative way, is actually quite high.

I know SSG blocks VPN IP addresses, but they do not block all ranges to all VPN's. If they find a certain IP address (or range) has been involved in nefarious activities, it will be added to their block list. The chances of you having such a IP address is very possible.

There is a solution with most VPN providers though: I use "PIA" aka "Private Internet Access" for my VPN service if I need to circumvent geographical blocks. The client installed offers a setting called "Split Tunnel" which allows you to decide whether traffic goes across the VPN tunnel, OR directly to the Internet using your own ISP IP address. What you can do is whitelist all known IP addresses owned by SSG (the ARIN DB entry will read Turbine) and force that traffic to not use the VPN.

These are (as far as I know) all IP addresses owned by SSG and used for both LOTRO and DDO: (This information is 100% publicly available and searchable by anyone)

#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/resources/registry/whois/tou/
#
# If you see inaccuracies in the results, please report at
# https://www.arin.net/resources/registry/whois/inaccuracy_reporting/
#
# Copyright 1997-2022, American Registry for Internet Numbers, Ltd.
#


NetRange:       198.252.160.0 - 198.252.160.255
CIDR:           198.252.160.0/24
NetName:        TURBINE-BSN3
NetHandle:      NET-198-252-160-0-1
Parent:         NET198 (NET-198-0-0-0-0)
NetType:        Direct Allocation
OriginAS:       AS22673
Organization:   Turbine Inc. (TURBI-1)
RegDate:        2012-10-16
Updated:        2021-12-14
Ref:            https://rdap.arin.net/registry/ip/198.252.160.0


OrgName:        Turbine Inc.
OrgId:          TURBI-1
Address:        117 Kendrick St.
Address:        Suite 350
City:           Needham
StateProv:      MA
PostalCode:     02492
Country:        US
RegDate:        2012-02-10
Updated:        2022-02-14
Ref:            https://rdap.arin.net/registry/entity/TURBI-1


OrgTechHandle: NETWO4941-ARIN
OrgTechName:   Network Engineering
OrgTechPhone:  +1-858-239-0500 
OrgTechEmail:  netadmin@standingstonegames.com
OrgTechRef:    https://rdap.arin.net/registry/entity/NETWO4941-ARIN

OrgAbuseHandle: NETWO4941-ARIN
OrgAbuseName:   Network Engineering
OrgAbusePhone:  +1-858-239-0500 
OrgAbuseEmail:  netadmin@standingstonegames.com
OrgAbuseRef:    https://rdap.arin.net/registry/entity/NETWO4941-ARIN


#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/resources/registry/whois/tou/
#
# If you see inaccuracies in the results, please report at
# https://www.arin.net/resources/registry/whois/inaccuracy_reporting/
#
# Copyright 1997-2022, American Registry for Internet Numbers, Ltd.
#

So in your Split Tunnel settings, you can exclude the entire range of 198.252.160.0 - 198.252.160.255 from the VPN, and force its traffic using your normal ISP. You should then be able to visit and use any service provided by SSG, whilst still using your VPN to secure your traffic for other sites.

[DoRonRon]

As a BBC License fee payer, I resent that people circumvent fees in this manner.

Programmes such as "Mock the Week" being axed by reason of expense has me particular annoyed just now.

[maartena]

As a BBC License fee payer, I resent that people circumvent fees in this manner.

Programmes such as "Mock the Week" being axed by reason of expense has me particular annoyed just now.

I don't think that has anything to do with people using VPN's. There is about a MILLION British expats and retirees in Spain and Portugal, and many of those no longer pay BBC license fees, but still use VPN to access BBC and ITV content.

If programs get cut because they are too expensive, you should probably look at who you vote for over there......

From Reuters, Jan 16 2022:

LONDON, Jan 16 (Reuters) - Britain's government will cut the BBC's funding by ordering a two-year freeze on the fee that people pay to watch the broadcaster, the Mail on Sunday reported.

The future of the licence-payer funded British Broadcasting Corporation is a perpetual topic of political debate, with Prime Minister Boris Johnson's government most recently suggesting its funding needs to be reformed.
Set against an inflation rate expected to reach a 30-year high of 6% or more in April, freezing the licence cost at its current 159 pounds ($217.40) would provide some relief to consumers battling sharply rising costs of living.

But it would also be a large blow to the BBC's finances as it tries to compete with privately funded news outlets and the likes of Netflix and other entertainment streaming services funded by consumer subscriptions.
In November, the government launched negotiations to agree how much the TV licence would cost, part of a five year funding settlement due to begin in April 2022.

Basically, Boris Johnson's (when he was still PM) government made a decision that saves the viewers some money on their yearly bills (no increase of the license fee), but caused the BBC a budget shortfall of GPB 285 Million. (US $388 Million)
Because of that, BBC was forced to cancel and/or axe a number of productions. They have also announced they need to slash 400 jobs, apparently....

If you want to blame someone, don't blame British expats watching from other countries, blame your government that is cutting BBC funding. People NOT paying the license fee isn't the issue here. I have no skin in the game when it comes to British politics, nor do I care..... but it seems to me that if this is an important issue to you, vote for the folks that are most likely to increase funding to the BBC in some way. It would probably mean you pay a higher license fee, but just with anything in life.... producing shows is getting more expensive, and if you don't increase the required income to produce those shows, some of them will end up on the chopping block. Doesn't work all that much different for commercial entities like Netflix or Amazon, only in this case it will be your house of commons that will have the power of decision around this.

Good luck though. I could always switch to torrents if you prefer..... It wouldn't cost the BBC any bandwidth from me, but on the flip side I would then help with re-distribution of said program, which I do not do now. By the way, I do funnel some money indirectly to the BBC by paying for BritBox.

[DoRonRon]

You could trump any of my political arguments I don't doubt.

Just because others do something does not absolve one of the wrongdoing.