Apache mangles HTTP requests, causing failure to load pages

Problems with the Windows version of XAMPP, questions, comments, and anything related.

Apache mangles HTTP requests, causing failure to load pages

Postby 4everJang » 23. September 2021 16:04

I am building an application which connects to my local web server using a Javascript socket. Note that this is not a browser, so I cannot use any web-based or even web-oriented libraries. I have been able to connect to a web server on the internet using my socket code. When I try to connect o my local Apache server (part of the XAMPP package for Windows, without any changes to the settings), every request I send causes a Bad Request response. When I look into the access log, I can see that the first newline following the HTTP/1.1 protocol identifier is not recognised and probably causes the Bad Request response.

The essential part of the code I am using:

Code: Select all
var sHost = 'localhost';
var oSocket = new Socket;
oSocket.open( sHost + ':80', 'UTF-8' );
oSocket.write( 'GET /index.html HTTP/1.1\r\nHost: ' + sHost + '\r\n\r\n' );
var sReply = oSocket.read( );
alert( sReply );


When I use this code with sHost = 'www.jang.nl' (one of my domains, hosted by One.com), I get the contents of the requested page (preceded by the usual headers). But running it with sHost set to 'localhost' gives the Bad Request response. Asking for localhost/index.html page in a local web browser gives the page contents without any problems.

Checking the access log, I see the following lines - first line for the request by the local web browser, second line when using the socket:

::1 - - [23/Sep/2021:09:19:32 +0100] "GET /index.html HTTP/1.1" 304 - "-" "Mozilla (etc)"
::1 - - [23/Sep/2021:09:19:52 +0100] "GET /index.html HTTP/1.1\n" 400 331 "-" "-"

My question: is there any setting for any of the Apache config files that may be causing this ? To me it seems obvious that the \n should not be there in the protocol identifier string, but why is that string not stripped as defined by the protocol ?
4everJang
 
Posts: 8
Joined: 21. June 2016 07:08
XAMPP version: 7.4.1
Operating System: Mac OS X Big Sur, Windows 10

Re: Apache mangles HTTP requests, causing failure to load pa

Postby Nobbie » 24. September 2021 12:49

I dont think that is an Apache problem. I think, its a problem of the connection / protocoll.

The HTTP specification RFC2616 states that each HTTP header line must end with a CR LF (carriage return / line feed), which is represented in Javascript as \r and \n. So that looks Ok in your code.

But i think, that "something" in your environment makes a translation from \r\n to only \n, as \r\n is the MSDOS-style end-of-line sequence, whereas only \n is the Linux/Unix-style end-of-line sequence. So for example, if you are running an FTP client and open a connection in so called "text mode" (vs. binary mode), the FTP protocoll translates each sequence of \r\n to \n (if transfering from MSDOS to Linux), or vice verse, if transfering from Linux to MSDOS.

As Xampp is running on Windows, i think the local Apache "reads" the Input from TCP/IP line by line (dont know which library function is used) and expects \r\n as end of line. But as only \n is transferred, it does not recognize the end of line and "thinks" that only \n is part of the token (in this case the information about HTTP version). That leads to bad header error.

The same string transferred to a Linux based Apache might work correctly, as Linux also handles text lines ending with \n only as a line. Therefore (I think) your test with http://www.jang.nl works correctly. Section 19 of RFC2616 says that only \n instead of \r\n may also work for compatibility reasons in some environments. I think thats what happens in that environment.

But the question remains, why and what translates \r\n into \n? I think that is already done on your client site, maybe the JavaScript environment (you did not specify it and further), maybe you can specify kind of mode in the open() event of the Socket. I dont know that class any further. Maybe you should not specify UTF-8 (what is a character mode), instead specify a binary mode or so?!
Nobbie
 
Posts: 13171
Joined: 09. March 2008 13:04

Re: Apache mangles HTTP requests, causing failure to load pa

Postby 4everJang » 05. November 2021 13:46

After trying all imaginable scenarios, I am still stuck with what I perceive to be a serious design flaw in Apache. It should simply accept both \r\n and \n as end of line characters. I have tried with a native Xampp on Windows with the same result. If the port listener code of Apache would just be able to handle either way of encoding a newline, there would not be any problem. As Apache is open source software, I could even try to fix this myself, but I have no idea where to start looking for this piece of code. If someone knows where the code is that interprets a connection request on a port, please let me know. It would save me a lot of search time.
4everJang
 
Posts: 8
Joined: 21. June 2016 07:08
XAMPP version: 7.4.1
Operating System: Mac OS X Big Sur, Windows 10

Re: Apache mangles HTTP requests, causing failure to load pa

Postby Azhrei » 05. November 2021 17:38

4everJang wrote:After trying all imaginable scenarios, I am still stuck with what I perceive to be a serious design flaw in Apache. It should simply accept both \r\n and \n as end of line characters.

Even after Nobbie pointed out the RFC that says header lines should end with \r\n, you think requiring \r\n is a flaw?

Wow, that's impressive.
I have tried with a native Xampp on Windows with the same result.

My first question would be, what is actually be sent over the wire? If you used a packet sniffer (Wireshark and tcpdump come to mind), you'd be able to see what was actually contained within the packet (assuming you're not use HTTPS; if you are, you'll need a proxy with its own certs, but since you're using port 80 I think this is unneeded).

Since you didn't provide any information about your runtime environment (which JS engine? which version? what locale and encoding is it running in? etc), I don't think you'll be able to get much help with this. Without any other information, all anyone here can do is guess. There's little point in that.

I hope you figure out the issue — if you do, please return and post your solution. Good luck!
Azhrei
 
Posts: 1
Joined: 05. November 2021 17:31
XAMPP version: 8.0.12
Operating System: macOS Big Sur

Re: Apache mangles HTTP requests, causing failure to load pa

Postby 4everJang » 08. November 2021 15:06

My first question would be, what is actually be sent over the wire? If you used a packet sniffer (Wireshark and tcpdump come to mind), you'd be able to see what was actually contained within the packet


I downloaded Wireshark and tried to figure out what is sent over the (internal) network but this is extremely technical and I do not have the hours I would need to figure out how to handle this app and look at what is sent to and received on localhost port 80 (which is what my local Apache is listening to). I am not a TCP/IP engineer and should not need to become one to solve this issue. If you have clear instructions how I can tell Wireshark to show me what is going back and forth on localhost:80 (and not show anything else) that would be useful.

Since you didn't provide any information about your runtime environment (which JS engine? which version? what locale and encoding is it running in? etc)


I am running ExtendScript (basically Javascript but with some limitations and a large number of objects for Adobe applications) from inside Adobe FrameMaker. This means there is no browser and none of the usual resources offered by a web environment are available. I did find out - simply by trying - that the Socket object works. Now that I am using a local Apache server and connecting to that via localhost port 80, everything works except that Apache claims it is receiving a bad request. This is clearly caused by the \r\n being mangled into \n. But where this mangling takes place is unknown to me. The fact that the same request works fine when connecting to an Apache server on a Unix machine may be explained by that Apache being less picky in receiving either \n or \r\n.

All in all, I think it is not that much to ask for Apache to have the same behaviour in Unix and Windows versions, i.e. accept both \r\n and \n as meaning the same thing.
4everJang
 
Posts: 8
Joined: 21. June 2016 07:08
XAMPP version: 7.4.1
Operating System: Mac OS X Big Sur, Windows 10

Re: Apache mangles HTTP requests, causing failure to load pa

Postby 4everJang » 08. November 2021 16:44

OK, sorry for being a little harsh in my comments. I am just getting so frustrated by not being able to solve a problem that apparently nobody else has - as everyone and their mother are using JS inside web browsers.

So far, I have figured out that the Socket object I am using is translating any combination of newline and carriage return into a single newline character. Not good if that is not accepted by Apache running on my Windows machine, but it is also misinterpreted on my Apache on Mac OS X, which is supposed to be the Unix variation. The same Apache server (or it it a different one?) is running on my hosted domain where the \n does not give any problems at all.

Question 1: Is there a configuration item somewhere in Apache that tells it whether to handle only \r\n or also allow \n ?
Question 2: What part of the Apache code handles the incoming HTTP requests and looks for the \r\n ?

Thanks in advance for any pointer that helps me finally get this problem solved. I could look for an IIS implementation instead but I really want to avoid going down that low road.
4everJang
 
Posts: 8
Joined: 21. June 2016 07:08
XAMPP version: 7.4.1
Operating System: Mac OS X Big Sur, Windows 10

Which module handles HTTP port traffic ?

Postby 4everJang » 09. November 2021 09:11

I need to overcome an issue with HTTP requests that get misinterpreted by Apache. The issue is that the \r\n sequence is expected but only \n is sent. Unfortunately, there is no way I can get my socket connection to send the expected \r\n, so I will have to work on the Apache port listener side to allow both \r\n and \n in the HTTP request. Amazingly, the 'incorrect' \n newlines are being accepted by Apache running on a hosted domain, so there might be a configuration setting that controls the encoding for a newline.

If anyone out there knows either the configuration file or the module in which I should check for a solution please let me know. I have been struggling with this issue for a long time and there seems to be no other solution than to make Apache accept both \r\n and \n as newlines. Or ditch Apache altogether (but I REALLY do not want to start using IIS for my local server).

[EDIT by Altrea: merged - because still the same issue]
4everJang
 
Posts: 8
Joined: 21. June 2016 07:08
XAMPP version: 7.4.1
Operating System: Mac OS X Big Sur, Windows 10

Re: Apache mangles HTTP requests, causing failure to load pa

Postby Altrea » 09. November 2021 09:24

I am not fully into the topic, but maybe this might help:
https://stackoverflow.com/questions/43574428/have-apache-accept-lf-vs-crlf-in-request-headers

and the part in the Apache manual
https://httpd.apache.org/docs/2.4/en/mod/core.html#httpprotocoloptions

Read more about the security issue forced Apache to change the default behavior with Apache 2.4.25 here:
https://nvd.nist.gov/vuln/detail/CVE-2016-8743

Be aware that changing the Option is a high security issue
We don't provide any support via personal channels like PM, email, Skype, TeamViewer!

It's like porn for programmers 8)
User avatar
Altrea
AF Moderator
 
Posts: 11926
Joined: 17. August 2009 13:05
XAMPP version: several
Operating System: Windows 11 Pro x64

Re: Apache mangles HTTP requests, causing failure to load pa

Postby Nobbie » 09. November 2021 09:51

First of all, glad to read that my assumption about the cause of the error seems to be correct.

Altreas Link also shows up, why that wrong end of line sequence works in other environments. I think, its simply an older Apache which hasnt been fixed already.

Last not least, even if this new runtime Option solves the issue, i would not be happy about it. You have a client which clearly has a bug or similar, as it translates CR LF into LF for an unknown reason, but instead of fixing the client, you are going to weaken your Apache Server. That is really strange in my mind.

Why dont you ask the Adobe Support instead of Apache Support, as this is clearly a problem of your runtime environment and not of Apache?!
Nobbie
 
Posts: 13171
Joined: 09. March 2008 13:04

Re: Apache mangles HTTP requests, causing failure to load pa

Postby 4everJang » 09. November 2021 16:05

@Altrea - Thanks for the pointers.

Reading about the Unsafe option I do understand that this would be a vulnerability. However, as I am only using Apache on my localhost and planning to use a private (undisclosed) port address for it, I feel the security risks can be kept in check. I can even add some code to prevent access to my localhost from another IP address.

@Nobbie - You probably know that getting an organisation like Adobe to make a change to software that nobody except myself is using would very probably lead to no result at all.

It still seems a little strange to me that there is no way to simply treat \n the same as \r\n. Now it looks like the only way to make this happen is to throw a whole lot of useful security checks overboard as well. As there are clearly two systems of encoding a newline on the planet and not a whole lot more, it still tells me that this is a design flaw. But at least I now have an option to try and make it work.
4everJang
 
Posts: 8
Joined: 21. June 2016 07:08
XAMPP version: 7.4.1
Operating System: Mac OS X Big Sur, Windows 10

Re: Apache mangles HTTP requests, causing failure to load pa

Postby Nobbie » 09. November 2021 16:33

4everJang wrote:@Nobbie - You probably know that getting an organisation like Adobe to make a change to software that nobody except myself is using would very probably lead to no result at all.


Maybe.

But anyway, i would give it a try, there is a good chance that already this behaviour can be changed or controlled either by an configuration option or a runtime switch or parameter. Its not necessarily a bug. And then you are fixing the "right" side instead of the wrong side.

4everJang wrote:it still tells me that this is a design flaw.


And it tells me that you have no idea at best.

It's definitely the easiest thing in the world to program a working client instead of putting in a bug like that. There is not the slightest reason to send only LF over the wire.

With absolute certainty not a single browser does such nonsense. It's a clearly defined protocol and a mistake if you don't follow it. Because coincidentally on Linux system text files also accept LF as end of line, has not the slightest meaning for the protocol HTTP.
Nobbie
 
Posts: 13171
Joined: 09. March 2008 13:04


Return to XAMPP for Windows

Who is online

Users browsing this forum: No registered users and 142 guests