-1

I have just updated a (dynamic) website by sftp/ssh and all the new pages have been considered as 404 error not found. Actually, all the pages, including the old pages which was overwritten.

Also, the website is using cloudflare.

Let me show you a curl of a url:

curl http://example.com/something/

 HTTP/1.1 404 Not Found Date: Tue, 26 Jan 2016 01:25:08 GMT
 Content-Type: text/html; charset=UTF-8 Transfer-Encoding: chunked
 Connection: keep-alive Set-Cookie:
 __cfduid=de5a53a227295f9d9374fdf39bb45514f1453771507; expires=Wed,25-Jan-17 01:25:07 GMT; path=/; domain=.example.com; HttpOnly
 X-Powered-By: PHP/5.5.29 
 X-Pingback: http://example.com/folder-wp-blog/xmlrpc.php 
 X-UA-Compatible: IE=edge 
 Expires: Wed, 11 Jan 1984 05:00:00 GMT 
 Cache-Control: no-cache, must-revalidate, max-age=0 
 Pragma: no-cache Set-Cookie:
 PHPSESSID=c72b8d6fc18775ef85ac8ab1b8bf6e95; path=/ Server:
 cloudflare-nginx 
 CF-RAY: 26a86c0ee0f10f15-IAD

As you can see, the curl returns a 404 error. BUT the a URL is live and working like a charm.

The problem is on cloudflare I thought. So, I "purge everything" when I curled before but the 404 still exist. I added a custom rule to bypass caching on cloudflare on a spesific url but it didn't work at all.

Also, I thought that was some mistake on my .htaccess and I erase everything from there. I restart the apache but the 404 are still exist.

As a result, every search engine and every other bot see a 404 header. The problem is everywhere, except on the /index.html and on the wordpress blog which is on a /folder-wp-blog/ folder. This WP wasn't changed at all. So, every other html and php file which is in root is flaged as 404...

Any ideas? solutions?

Mark Stosberg
  • 3,771
  • 23
  • 27
Greg
  • 101
  • 5
  • Go to Settings > Permalinks, check that you have the right permalink structure set, and click Save Changes (even if you didn't change anything). Note whether any problems are reported. – Michael Hampton Jan 26 '16 at 01:40
  • An idea can you test your site at [Webpage test](http://www.webpagetest.org/) After doing that make sure the waterfall on the left has no red bars. If it does click in the left and you will see details of each URL on your site. Hope this helps. – smokingwheels Jan 26 '16 at 02:01

1 Answers1

0

These headers indicate that the request has gotten all the way through to your PHP/Wordpress install, and thus the problem is there, not at CloudFlare:

X-Powered-By: PHP/5.5.29 
X-Pingback: http://example.com/folder-wp-blog/xmlrpc.php

Take CloudFlare out of the picture by SSH'ing to the server and testing from inside the server:

curl -H 'Host: example.com' http://127.0.0.1/something/

That will have the same effect as requesting http://example.com/something, but with the need for the DNS resolution or going through CloudFlare.

A variation of the same testing technique would to be update /etc/hosts on the server and add a line that maps example.com to 127.0.0.1. Then could run curl http://example.com/something from inside the server with the request going first through CloudFlare. You may wish to remove the entry from /etc/hosts when you are done testing on the server, or it may be considered a useful optimization, perhaps also simplify WP-Cron calls that might call back to the server from within itself.

If you are getting different behavior in browsers vs curl the difference is likely in the HTTP request headers being sent. Using tools like Chrome Inspector, look carefully at the request headers that Chrome is sending. For example, if there is a cookie being sent in Chrome but not curl, that could make a different. A way to level the playing field is to use an "Incognito" or "Private Browsing" window which won't send prior browser cookies.

In the Chrome Network Inspector, you can see the IP address of the server that responded. Compare with this dig +short example.com run from the same machine the curl requests are coming from.

Mark Stosberg
  • 3,771
  • 23
  • 27
  • it returns 404 error page – Greg Jan 26 '16 at 02:14
  • Try the advice at http://codex.wordpress.org/Changing_The_Site_URL particularly the `wp-config.php` fix for changing site URLs, if need your site URL just changed. – Mark Stosberg Jan 26 '16 at 02:16
  • Also are you sure the URL is "working a charm" for all browser requests? Have you tried multiple browsers? Is this really just happening with `curl`? – Mark Stosberg Jan 26 '16 at 02:19
  • i tried firefox and safari + mobile versions and it works – Greg Jan 26 '16 at 02:22
  • I got in all pages now 404 and it is only visible with curl. – Greg Jan 26 '16 at 02:23
  • Did the site URL change? Did you try the tip above to address that? Also, I updated my answer re: curl vs. browsers. – Mark Stosberg Jan 26 '16 at 02:25
  • I added: define('WP_HOME','http://example.com/folder-wp-blog/'); define('WP_SITEURL','http://example.com/folder-wp-blog/'); but it didnt worked – Greg Jan 26 '16 at 02:40
  • I continue to have 404 (via Curl) the pages – Greg Jan 26 '16 at 02:40
  • Also, I noticed that one by one the blog url as going from 404 to 200. whaat??? and finally my index.html got 200 but the rest are still 404 – Greg Jan 26 '16 at 02:41
  • Your `wp-config.php` variables should have "http://" prefix in them. It sounds like DNS resolution is slowing changing over. See my updated question for comparing the DNS resolutions you are getting in curl vs browser. – Mark Stosberg Jan 26 '16 at 02:49
  • yes, sorry. I have added with http, but no luck – Greg Jan 26 '16 at 02:54
  • Also see the suggestions re: headers and DNS that were added to my answer. – Mark Stosberg Jan 26 '16 at 02:57
  • dig +short and from chrome network: boths IPs are cloudlfars IPs – Greg Jan 26 '16 at 03:11
  • Also, I saw that there is a small blank line at the top of the html source file (viewed from a browser). This empty row is not normal due to the fact the menu has to start for the top of the page.. It is like , something returns blank and after that the actuall page loads – Greg Jan 26 '16 at 03:19
  • in a file.php the first row is an include to the funct.php which has all the fuctions that I used to develope the website. and the first again row in the funct.php is include($_SERVER['DOCUMENT_ROOT'].'/the-folder-wp/wp-blog-header.php'); – Greg Jan 26 '16 at 03:21
  • I include that to call the WP functions to get the some categories.. – Greg Jan 26 '16 at 03:22
  • The problem is that even if I comment the //include(../../wp-functions.php) the 404 error still exists. If it was a problem due to the include and the WP blog's functions call, it should be ok when it is on comments. But it doesn't... – Greg Jan 26 '16 at 03:23
  • FIXED!! The problem was the include. I am so stupid that I included the wp-blog-header.php and not the wp-load.php . .. Oh my god! By the way thank you very very much for your time. http://wordpress.stackexchange.com/questions/22739/external-page-integrated-into-wordpress-returns-a-404-error and https://cooltrainer.org/fixing-false-404-headers-on-external-pages-including-wp-blog-header-php/ – Greg Jan 26 '16 at 03:32