Soft 404: when every wrong address on your site answers 200

What a soft 404 is

Ask a site for a page that does not exist and it should answer with status 404. A soft 404 is a site that answers 200, the status for success, and shows something else: often the home page, sometimes an empty template.

A person notices and moves on. A crawler takes the status at its word. It records that the address exists and holds a copy of your home page.

Test for it in one command

Ask for an address that cannot exist and print only the status:

curl -s -o /dev/null -w '%{http_code}' https://example.com/this-page-does-not-exist-41872

The answer should be 404. If it is 200, you have a soft 404. If it is 301 or 302, follow it: a redirect of every unknown address to the home page is the same problem with one more step.

Try a second address with a file extension, such as /nothing.txt. Some sites handle the two differently.

Where it comes from

Nearly always from the rule that makes clean addresses work. A web server set up for a PHP application is told: if no file matches the address, hand the request to index.php. In nginx that is one line:

try_files $uri $uri/ /index.php?$args;

The rule is correct. WordPress, Laravel and every framework with a router depend on it, and KLYRN writes the same line for a PHP site. What it assumes is that index.php is a router: a script that looks at the address and answers 404 itself when nothing matches.

When index.php is a plain home page, it never looks at the address. Every request that reaches it gets the home page and a 200.

What it costs

• Duplicate pages. Every mistyped link and every address a scanner guesses becomes a copy of your home page that a search engine may index.
• Wasted crawling. A crawler spends its visits on addresses that are not pages, and reaches your real pages later.
• Broken links you cannot see. A link checker reports nothing, because nothing answers 404.
• Noise in your logs. Scanners asking for /wp-login.php and /.env look like successful visits.

The fix, in the script and not in the web server

Leave the nginx rule alone. Make the home page check that it was asked for. At the top of index.php, before any output:

$path = parse_url($_SERVER['REQUEST_URI'] ?? '/', PHP_URL_PATH) ?: '/';
if ($path !== '/' && $path !== '/index.php') {
    http_response_code(404);
    require __DIR__ . '/404.php';
    exit;
}

Compare the path only, so that the home page with a query string, such as a campaign tag, still answers 200.

Give the 404 page the site's own header and two links: the home page and the thing most visitors came for. Mark it noindex. Then run the test again, and check that the home page and one real page still answer 200.

Static sites and custom error pages

A static site has no script to fall back to, so nginx answers 404 on its own. The question there is only what the page looks like. In KLYRN a site can serve its own page for a status from the Rules tab, and the status stays 404 while the page is yours.

One thing to avoid in any setup: an error page that redirects to the home page. It turns a correct 404 back into the problem above.