We were experiencing problems where users would end up at our site with "empty GET requests". By this I mean that they originated on the external site as POST requests with several params, but the browser would only do a simple GET with no params to our server.
This happened more or less regardless of the browser etc and with numerous sites with various technologies.
For whatever reason renewing our certificate (that had worked for 18 months) solved the problem. The old cert was from Verisign and the new cert is from Thawte.
Below the horizontal rule is the original question.
We have a service where we expect various webstores to send us POST data over HTTPS and data payload as nvp POST parameters. Few percent of these are somewhere converted to GET requests and the payload is lost, so they come to our server as GET requests without any of the parameters.
When we get a GET requests there are no signs on our load balancer or firewall of a POST request that would be connected in any way to the GET we're receiving. The GET is usually the first incoming request that we can see from a given IP.
There does not seem to be any other common factor other than almost all the client computers are Windows of various versions. The browsers can be any of Firefox, IE or Chrome at least.
The other site submitting the data via the user browser (form with method POST) could be using any of but not limited to the following;
PHP, Java, Ruby on rails, .NET
Linux, Windows
Apache, Nginx, IIS, Tomcat, Cowboy
Drupal, Magento, Joomla, Prestashop, proprietary and in-house solutions
The problem affects clients from multiple different sites that don't seem to have anything in common. Also it does not seem to be a certain ISP that the users are coming from.
What seems to happen is that the POST request vanishes and never reaches any of our servers. Somehow the browser sends a GET request instead without any of the parameters. However the referer is intact which according to our testing means the user cannot have selected the browser address field and pressed enter as this loses the referer header. With a one or a few retries (browser back, reclick the submit on the referring site) the POST comes through as it should
Also according to what we've heard of the end users comments on this they get no browser error message.
Server to server calls don't seem to be affected of this problem. It looks like that the problem started on July 14th as after this date we have been getting these problem requests daily. It started with only a few a day and have been steadily rising since and has now settled to approx 5% per day of all requests we receive to our API.
So, this does not look like a problem on the referring sites, as it impacts sites with such a varying platform with no common factor. It does look like it's a windows problem as the problem user agents are practically all windows of various versions, but it affects all browsers, so what could it be?
Any suggestions are welcome at this point.
EDIT: We would see if it were a http -> https redirect on our end as we tested that and found the http request in load balancer logs. Non-www to www redirect would also happen at our load balancer AFAIK and we'd see that also. Also forgot to mention that we get mostly valid data from the affected sites, it's only a small portion of requests that behave erratically.
EDIT: * SOLVED (kind of) * As one of those "there's now way this will help, but let's try anyway as we have no clue what's wrong" we installed a new certificate. This problem has ceased to exist since the installation of the new cert.
We currently have no clue on how the certificate could have caused the behaviour we were experiencing. AFAIK if there are problems with the certificate the browser would prompt the user for action (from what we heard, nobody was receiving any prompts) and either not talk with the server at all or proceed normally. What I would not expect is for the browser to change POST to GET and dump all parameters. The old cert was installed correctly, or how else it could have been working flawlessly for a year and a half?
Either way, there has been no cases in about a week. The last entry in the log showing one of these problem cases is timed at roughly few minutes before installing the new cert...
