One Checkbox Can Hide Your Whole Site From Google's AI. Here's a Two-Minute Check.

The site works. That's the problem.

Your site loads. The phone number is there, the photos are there, the contact form sends. A customer who types in your address sees exactly what you meant them to see.

Somewhere in the head of that page, there may be one line of code that tells search engines to skip it entirely. Visitors never see it. Nothing looks broken. You just stop showing up, and nobody tells you why.

I can't tell you how often this happens. I haven't found a published number. Anybody who hands you a percentage is guessing. What I can tell you is how it gets there, what Google says it costs, and how to check your own site in about two minutes.

What noindex is

noindex is an instruction to search engines. Google's documentation defines it as "Do not show this page, media, or resource in search results." It usually lives in a meta tag in the page's head, and it can also be sent as an HTTP header called X-Robots-Tag. Google says any rule that works in a robots meta tag can be sent as that header too.

Here's why it matters for AI. Google's page on its AI features says a page has to be "indexed and eligible to be shown in Google Search with a snippet" to appear in AI Overviews and AI Mode. There are, in their words, "no additional technical requirements," and inclusion is never guaranteed. So if a page is noindexed, it doesn't meet the one requirement Google lists.

That is a precondition, not a prediction. Removing noindex doesn't earn you a citation. It just takes away the thing standing between you and the possibility.

Where it comes from

Nobody types this on purpose. It shows up four ways.

The first is WordPress. Under Settings, then Reading, there's a checkbox labeled "Discourage search engines from indexing this site." When it's ticked, WordPress outputs <meta name='robots' content='noindex,nofollow' /> in every page (that's been true since version 5.3). Older versions did it with a Disallow: / line in robots.txt instead. The box is unchecked by default, which means somebody checked it. A developer building the site on a staging copy is the classic reason, and it's a sensible thing to do while the site is under construction. Then launch day comes and the box stays ticked.

The second is an SEO plugin with a global setting that does the same job. The third is a staging setup that gets copied to the live site, settings and all. The fourth is a server or CDN setting that adds an X-Robots-Tag header, which never appears in the page itself.

If a freelancer or agency rebuilt your site at some point, that's the likeliest road to this. It's also the least likely thing anybody thinks to check afterward, because the site works.

WordPress's own documentation is blunt about the limits: neither of those methods (the checkbox or the older robots.txt Disallow) blocks access to your site, and it's up to search engines to honor the request. Google does.

What's documented for ChatGPT and Perplexity, and what isn't

OpenAI's page on its crawlers says OAI-SearchBot is used to surface websites in ChatGPT's search features, and that sites that opt out through robots.txt won't be shown in ChatGPT search answers. That page doesn't mention noindex or meta tags at all.

Perplexity's page recommends allowing PerplexityBot in robots.txt and says nothing I could find about whether it honors noindex.

Both companies document robots.txt, and neither documents noindex. Whether they honor it is unpublished, so anyone who tells you with confidence is guessing.

Three checks

You need a browser and your homepage address.

One: view the source. Put view-source: in front of your address, so view-source:https://yoursite.com, or right-click and choose View Page Source. Then use Ctrl+F or Cmd+F and search for noindex. Then search for none, because content="none" means the same thing as noindex, nofollow. Google reads the tag names and values without caring about capitalization, and it also honors crawler-specific tags like <meta name="googlebot" content="noindex">. So search for the word itself, not just the word robots.

If nothing turns up, good. If you find it on a page you want people to find, that's your answer.

Two: check the response headers. This one hides from view-source because the instruction travels in the HTTP response, not in the page. If you're comfortable at a command line, curl -sI https://yoursite.com prints the headers, and you're looking for a line starting with X-Robots-Tag. If you're not, your browser's developer tools show response headers under the Network tab, or ask whoever hosts your site. A hosting company or CDN can set this without touching a single file on your site.

Three: read your robots.txt, and watch for the trap. Go to yoursite.com/robots.txt. A Disallow: / under User-agent: * blocks everything. That's a different problem from noindex, but here's where the two collide. If robots.txt blocks a page, Google can't fetch it, which means it can never read the noindex rule on that page. Google says pages carrying those rules "cannot be disallowed from crawling." So blocking and noindexing the same page can leave you with a rule nobody reads. Fix them as a pair.

While you're at it, open Settings, then Reading in WordPress and look at that checkbox. It takes ten seconds. Check your SEO plugin's global settings too.

After you fix it

Don't expect the change overnight. For preview-control changes, Google's page warns that "crawling can take anywhere from several days to several months." That's about recrawl timing. It isn't a promise, and nobody gets to give you a date.

OpenAI and Perplexity both say robots.txt changes can take about 24 hours. That figure is about robots.txt. It tells you nothing about how long it takes after removing a noindex, so don't borrow it.

What the audit does and doesn't see

The AI Crawler Access module in the paid report has a line that checks for noindex. If your homepage carries a robots meta tag with noindex in it, that line shows FAIL, with the finding "The homepage is set to noindex," and it can drive one of the fixes in your report. WordPress's "Discourage search engines" output is exactly the case it catches.

That line carries no points. It will not lower your score or your grade, so a noindexed homepage doesn't show up as a worse number. It shows up as a FAIL you can read. A robots.txt that blocks the five AI bots we check (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Bingbot) is different. That does cost points, up to the full 15 in that module, since a blanket Disallow: / counts as blocking all five. A leftover staging robots.txt is the expensive version of this mistake.

The check reads the homepage's HTML meta tag only. It doesn't see X-Robots-Tag headers, which is why check two is a manual job. It also won't catch content="none" or a crawler-specific tag like name="googlebot". The Deep Scan adds a per-page noindex column in its action checklist, but Quick Scan and Full Audit don't.

The free Grade Check scans your homepage and nothing else, and it shows you a score and a letter grade, not the individual check lines. A noindexed homepage won't change that number, so the free check can't tell you whether you have this problem. Think of it as the fast first look at everything else. The view-source search is what answers the noindex question.

Do it before lunch

Run the view-source search today. Search for noindex, then none. Check the header. Look at the Reading box. If all of it comes back clean, you've ruled out one of the quietest ways a working site can disappear.

Then, if you'd like a number to go with it, the free Grade Check at aeocheck.net scores your homepage on the same engine as the paid report. It takes about a minute and costs nothing.