Author: Ben

The X-Robots-Tag is an HTTP response header that gives search engines the same indexing instructions as a robots meta tag — noindex, nofollow, nosnippet and more — but works on any file type, including PDFs, images, XML and plain text, where a meta tag can’t be placed.

What is the X-Robots-Tag?

Every time a browser or crawler requests a URL, the server replies with HTTP headers before the content itself: the status code, the content type, caching rules. The X-Robots-Tag is one of those headers. It tells search engines whether they may index the URL and how they may show it.

HTTP/2 200
content-type: application/pdf
x-robots-tag: noindex

That response says: this PDF may be fetched, but don’t show it in search results. It does the same job as <meta name=”robots” content=”noindex”> in an HTML page — which a PDF simply can’t contain. The header name, the user-agent names and the values are all case-insensitive, according to Google.

How is it different from a meta robots tag and robots.txt?

The last row is why X-Robots-Tag problems are hard to spot: you can view a page’s source and see nothing wrong, while the server quietly sends noindex. The difference between blocking and noindexing is covered in blocked by robots.txt and excluded by noindex tag.

Which values can you use?

Google supports these rules in both the meta tag and the header:

Bing is different on the last row. Bing supports the X-Robots-Tag too, and for Bing noarchive means “do not link in Chat and Copilot”, while nocache limits Copilot to URL, title and snippet. If you want to control how Microsoft’s AI uses your content, these are the values that do it — see AI crawlers and robots.txt.

You can also target one crawler by name and combine several headers:

X-Robots-Tag: googlebot: nofollow
X-Robots-Tag: otherbot: noindex, nofollow
X-Robots-Tag: unavailable_after: 31 Dec 2026 23:59:59 CET

When should you use the X-Robots-Tag?

• PDFs and documents you want downloadable but not ranking — price lists, internal manuals, duplicate PDF versions of HTML pages.
• Images you don’t want in Google Images.
• Machine files such as XML sitemaps, feeds, API responses and llms.txt, which are meant for bots and look odd as search results.
• Whole sections, such as a staging folder or a directory of print pages, where editing every template is impractical.
• Temporary content with unavailable_after, like event or offer pages.

For normal HTML pages, a meta robots tag set by your SEO plugin is usually simpler and easier for your team to see.

How do you set the X-Robots-Tag?

Apache (.htaccess)

This is Google’s own example for all PDFs:

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

NGINX

location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex, nofollow";
}

WordPress and PHP

Yoast SEO already sends noindex, follow on its XML sitemaps. For your own output, send the header before any content, for example in a small plugin:

header( 'X-Robots-Tag: noindex', true );

That’s how our site plugin marks /llms.txt as noindex. On a CDN such as Cloudflare you can add the header with a response-header transform rule instead of touching the server.

How do you check a URL’s X-Robots-Tag?

• Command line: curl -I https://example.com/file.pdf shows all response headers.
• Browser: DevTools → Network → click the request → Response Headers.
• Search Console: URL Inspection reports “Indexing allowed? No: ‘noindex’ detected in ‘X-Robots-Tag’ http header” when the header blocks indexing. The live test shows the current state; see how to check if a page is indexed.

Example: the headers on this site

On 25 September 2026 we requested eight URL types on indexurepoint.com and recorded the X-Robots-Tag:

The pattern is exactly what you want: machine files carry a header noindex, human pages are controlled by the meta tag, and nothing a visitor should find is blocked. Note that the sitemaps are noindexed but not blocked in robots.txt — Google still has to crawl them to read the URLs inside.

What goes wrong with the X-Robots-Tag?

• An invisible noindex. A server rule, CDN setting or security plugin adds noindex to every response, and nobody notices because the page source looks clean. A r/TechSEO thread titled “Hidden X-Robots-Tag: noindex reported by GSC – can’t find source” is a typical example. Check the live headers, not the HTML.
• Staging rules going live. A site-wide noindex header on the staging server gets copied to production during launch.
• Blocked and noindexed at once. If robots.txt disallows the URL, Google can’t read the header, so the URL may stay indexed. Remove the Disallow until the noindex has been processed.
• Conflicting rules. Header says noindex, meta tag says index: Google applies the most restrictive combination, so the page stays out.
• Relying on noarchive for Google. Google no longer uses it. It still matters for Bing and Copilot.
• Expecting instant removal. The header takes effect when Google recrawls the URL. For a fast temporary hide, see how to remove a URL from Google.

FAQ

llms.txt is a Markdown file at the root of a website that gives AI agents a short, curated map of its most important pages. It is a proposal, not a standard: Google Search says it doesn’t use it, Chrome’s Lighthouse now checks for it, and it helps agents far more than it helps rankings.

What is llms.txt?

llms.txt is a plain-text file written in Markdown and served at /llms.txt, the same place robots.txt lives. It gives a language model or AI agent a compact summary of what a site is about and a list of links to the pages that matter, so the agent doesn’t have to wade through navigation, scripts and ads to work that out.

The proposal was published by Jeremy Howard of Answer.AI on 3 September 2024. Its core idea is stated plainly: the file “stays small enough to fit in context”, and the detail “lives behind the links”. It is a community proposal — not a W3C, IETF or search-engine standard — and nobody is obliged to read it.

What does the llms.txt format look like?

The specification defines a fixed order, so the file can be read by a person, a model or a simple parser:

• H1 with the site or project name — the only required element.
• A blockquote with a short summary containing the key information needed to understand the rest.
• Optional paragraphs or lists (no headings) with more detail on how to interpret the site.
• H2 sections with “file lists”: each entry is a Markdown link [name](url), optionally followed by a colon and a note.
• An “Optional” H2 section, by convention, for secondary links an agent can skip when context is tight.

This is the start of the live file on this site:

# IndexUrePoint

> Independent, plain-English guides to getting web pages found
> and indexed on Google, Bing and AI search ... English and Dutch.

## Guides (English)

- [How to Remove a URL from Google](https://indexurepoint.com/remove-url-from-google/): Fast temporary hide versus permanent removal.
- [Orphan Pages: What They Are and How to Fix Them](https://indexurepoint.com/orphan-pages/): Pages nobody links to get crawled last.

The proposal also suggests offering clean Markdown copies of important pages at the same URL with .md appended. Some documentation platforms additionally publish an llms-full.txt with all content in one file; that is a common convention, not part of the core proposal.

Who actually uses llms.txt?

This is where most articles overpromise. Here is what can be verified.

• Google Search: no. Google’s guide to optimizing for generative AI features (updated July 2026) says you don’t need AI text files or Markdown to appear in Search or its AI features. Gary Illyes said at Search Central Live in July 2025 that Google doesn’t support llms.txt and isn’t planning to (Search Engine Land). When Google added llms.txt files to some of its own developer sites, John Mueller said plainly it was not an endorsement.
• Chrome Lighthouse: it checks. Lighthouse 13.3 moved its Agentic Browsing category into the default report, including an llms.txt audit (Search Engine Journal). The audit is about agents, not rankings: without the file, Chrome’s documentation says, “agents may spend more time crawling the site”.
• AI documentation sites: widely. Anthropic (code.claude.com/docs/llms.txt), OpenAI (developers.openai.com/llms.txt), Google (ai.google.dev/api/llms.txt) and Perplexity (docs.perplexity.ai/llms.txt) all returned HTTP 200 when we checked. Coding agents that read documentation are the clearest real-world audience.
• AI answer engines citing ordinary websites: unproven. We found no official statement from OpenAI, Anthropic, Perplexity or Microsoft that llms.txt influences which websites are cited in answers. Critics such as Mark Williams-Cook argue the evidence for it is no better than astrology.

How is llms.txt different from robots.txt and a sitemap?

The key point: llms.txt does not block or allow anything. If you want to control what AI crawlers can fetch, that is still a job for robots.txt rules per AI crawler. And if you want pages discovered by search engines, you still need a working sitemap and internal links.

Do you need an llms.txt file?

If your goal is visibility in Google’s AI Overviews or AI Mode, spend the time on indexing and content instead — the steps in how to get cited in AI search and AEO vs GEO. If your site is documentation that developers use with AI coding tools, an llms.txt (and Markdown versions of key pages) is a reasonable, low-cost addition.

How do you create an llms.txt file?

Option 1 — Yoast SEO (WordPress)

Yoast added llms.txt generation on 10 June 2025, free for all users. Turn it on under Yoast SEO → Settings → Site features; the plugin builds the file from your site data and refreshes it weekly. You can let it pick content automatically or choose pages yourself.

Option 2 — write it by hand

Create a UTF-8 text file named llms.txt, follow the order above, and upload it to your site’s root so it loads at yourdomain.com/llms.txt. Keep it short: your 10 to 30 most useful pages, each with a one-line note, beats a dump of every URL.

Option 3 — generate it dynamically

That’s what we do. Our site plugin builds /llms.txt on request from the published guides, grouped into English and Dutch sections, and uses each article’s excerpt as the note. New articles appear automatically — both guides we published earlier today were in the file the moment they went live.

Then: add a noindex header

John Mueller has said using noindex for llms.txt “could make sense”, because other sites might link to it and it could otherwise get indexed, “which would be weird for users”. Since it’s a text file you can’t add a meta tag, so send an HTTP header instead. Ours returns:

HTTP/2 200
content-type: text/plain; charset=utf-8
x-robots-tag: noindex

Check yours with the URL Inspection tool or a header checker; the header itself is explained in our X-Robots-Tag guide.

What goes wrong with llms.txt?

• Treating it as a ranking hack. Selling llms.txt as a way into AI Overviews contradicts Google’s own documentation.
• Server errors. A 500 on /llms.txt is the one thing Lighthouse fails you for. A 404 is fine; a broken file is not.
• Links to non-indexable pages. Listing redirected, noindexed or deleted URLs sends agents to dead ends. Keep it in sync with your sitemap.
• Dumping everything. A file with thousands of links defeats the purpose of fitting in context. Curate.
• Blocking the file in robots.txt. Then compliant AI agents can’t read it. Use noindex, not Disallow.
• Duplicate generators. Two plugins both creating the file can conflict. Pick one method.

Want to control AI Overviews instead? llms.txt won’t do it, but Search Console’s new opt-out control will: AI Overviews in Search Console.

FAQ

To remove a URL from Google permanently, change the page itself — 404/410, noindex, or a login — and use Search Console’s Removals tool only as the fast, temporary hide (about six months) while the permanent fix takes effect.

Pick the right method

• Page is gone for good: delete it and return 404 or 410. Remove it from your sitemap.
• Page should exist but not in Google: add a noindex meta tag or X-Robots-Tag header — and make sure it’s not blocked in robots.txt, or Google can’t see the noindex. See noindex.
• Private content: put it behind a login or password. Google can’t index what it can’t access.
• Page moved: 301-redirect it to the new URL instead of removing it.

Using the Removals tool

• In Search Console, open Indexing → Removals → New request.
• Enter the exact URL, or choose “Remove all URLs with this prefix” for a whole folder.
• Choose Temporarily remove URL (hides it and clears the cached snippet) or Clear cached URL (keeps it in results but refreshes the snippet).
• At the same time, apply a permanent fix from the list above.

When it’s not your site

You can’t remove other people’s pages, but you can ask Google to refresh them. If a page has already changed or been deleted but Google still shows the old version, use Google’s Refresh Outdated Content tool. For personal information or legal issues, Google has separate removal request forms.

A worked example

Say you published a page with a customer’s name in it by mistake at /case-study-acme/, and it’s already indexed. The right order: first edit or unpublish the page — in WordPress, setting it to draft makes the URL return 404. Then open Search Console → Removals → New request, paste the exact URL and choose Temporarily remove URL. Within about a day the result disappears. Because the URL now returns 404, Google drops it permanently on the next crawl, and the six-month clock on the temporary hide stops mattering. If the page had to stay online without the sensitive part, you would instead edit the content and use Clear cached URL so the old snippet stops showing while the fresh version takes over.

Removing files, not pages? PDFs and images can’t hold a meta tag. Use the X-Robots-Tag header to noindex them.

FAQ

An orphan page is a page no other page on your site links to: visitors cannot click their way there, search engines find it late through the sitemap at best, and it typically indexes last — the fix is simply linking to it.

Why orphan pages hurt

Internal links tell Google what matters on your site. A page nothing links to looks unimportant, even if it’s your best work.

How to find orphan pages

• Compare two lists: all URLs in your XML sitemap versus all URLs a crawler finds by following links from your homepage. URLs in the first list but not the second are orphans.
• Use a site crawler such as Screaming Frog, Sitebulb or Bing Webmaster Tools’ Site Scan, and connect it to your sitemap or Search Console data where supported.
• Check Search Console: pages with impressions or in the indexing report that your crawl never reached.
• Common culprits: old landing pages, pages removed from menus, tag pages, and posts only reachable through paginated archives.

How to fix them

• Worth keeping? Add contextual links from two or three related, indexed pages, and consider a hub or category page that links to the whole topic.
• Outdated or duplicate? Merge it into a stronger page and 301-redirect it — see page with redirect.
• Not needed at all? Delete it, return a 404 or 410, and remove it from the sitemap.

A worked example

A typical case from our own audits: a site moves to a new menu, and twelve older guides drop out of it. They stay in the sitemap, so Search Console still knows them — but months later most sit in “Discovered – currently not indexed” or have slid down the rankings. The repair took an afternoon: each old guide got two contextual links from the newer articles that superseded parts of it, plus a place on a topic hub page. Within weeks the majority were recrawled. The lesson is that orphaning rarely happens on purpose; it happens when navigation changes and nobody checks what fell out. Make “what stopped being linked?” a standard question after every redesign.

FAQ

AI companies run different crawlers for different jobs — training, search and user-initiated fetching — and robots.txt lets you allow or block each one separately: allow the search bots if you want citations, decide deliberately about the training bots.

The main AI crawlers (September 2026)

• OpenAI: GPTBot collects training data. OAI-SearchBot surfaces sites in ChatGPT search. ChatGPT-User fetches pages when a user asks — and since OpenAI’s December 2025 documentation change, robots.txt rules may not apply to it.
• Anthropic: ClaudeBot for training, Claude-SearchBot for search, Claude-User for user-initiated fetches.
• Perplexity: PerplexityBot for its search index, Perplexity-User for user requests.
• Google: Google-Extended (a control token, not a separate crawler) for Gemini training and grounding in Gemini Apps and Vertex AI.
• Apple: Applebot-Extended controls use of content for Apple’s AI training.

Google-Extended explained

Google-Extended controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google states it does not affect your inclusion in Google Search and isn’t a ranking signal. AI Overviews are part of Search and rely on Googlebot, so blocking Google-Extended doesn’t take you out of them — see what is Googlebot. Since 31 August 2026 there is a separate switch for that: the Search generative AI control in Search Console lets you leave AI Overviews and AI Mode while staying in normal results — see AI Overviews in Search Console.

Example robots.txt rules

Allow AI search, block AI training:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot are allowed by default when you don’t list them. Test any change in Search Console’s robots.txt report, and remember that robots.txt is a request, not a lock — see blocked by robots.txt.

What about llms.txt? It doesn’t allow or block anything — it only suggests which pages matter to AI agents. See what is llms.txt for who actually reads it.

Real data from this site. Our host’s AI crawler log for 24 hours on 25 September 2026 showed 21 requests from known AI crawlers — 17 from GoogleOther and 4 from OAI-SearchBot — and none blocked. Testing with faked crawler user agents gave a very different (and misleading) picture; we explain why in our 403 guide.

Check before you publish. Our free robots.txt tester shows in one click whether GPTBot, ClaudeBot, PerplexityBot and Googlebot are allowed or blocked by your rules.

FAQ

The Google Indexing API lets a site tell Google directly that a page was added or removed — but officially only for JobPosting and livestream pages, and Google has attached an explicit spam warning for everything else.

What the Indexing API does

It’s a programmatic way to tell Google “this URL was updated” or “this URL was deleted”, authenticated with a Google Cloud service account that’s an owner of your Search Console property. For job boards and livestream pages — content that changes or expires quickly — it gets Google to crawl sooner than a sitemap would.

Why people misuse it

Because it sometimes triggers a fast crawl. Tutorials and “instant indexing” plugins spread the trick, and some paid indexing services are built on it. For a site owner watching pages sit in Discovered – currently not indexed, it’s tempting.

The risk

• It’s against the documented use. Google states the API is for job posting and livestream pages only.
• Google added a spam warning. The documentation warns that abuse, including using multiple accounts or other means to exceed quotas, may lead to access being revoked.
• A crawl isn’t indexing. Even when the API triggers a crawl, Google still decides whether the page is worth keeping. A page Google rejected for quality stays rejected.

What to do instead

• Submit a clean sitemap and request indexing for key pages in Search Console — see how to submit your website to Google.
• Link new pages from strong, already-indexed pages.
• Use IndexNow for Bing — the legitimate, open “push” protocol. Google doesn’t support it, but Bing and others do: what is IndexNow.
• Make the page worth keeping. Unique, useful content is what turns a crawl into an index entry.

FAQ

AEO (answer engine optimization) and GEO (generative engine optimization) are two names for the same goal: getting your content used and cited in AI-generated answers — and the foundation under both is ordinary indexing.

AEO vs GEO vs SEO

SEO is the foundation under both. AI answers are built on pages that search engines have already crawled and indexed; if you’re not indexed, no amount of AEO or GEO helps. That’s why this site treats getting indexed as step zero.

What the work actually is

• Be indexed in Google and Bing. Google’s AI Overviews build on Google’s index; Microsoft Copilot builds on Bing. Set up Bing Webmaster Tools and IndexNow, not just Search Console.
• Answer first. Put the direct answer in the first two sentences under a question-style heading. Models lift clear, self-contained sentences.
• Be specific. Concrete steps, named tools, exact labels and sourced facts are more quotable than generalities.
• Show who’s behind it. A consistent brand, a clear about page and first-hand experience signal trust.
• Use clean structured data. Article and FAQ schema help machines parse the page.
• Let the right crawlers in. Blocking AI search crawlers opts you out of their answers — see AI crawlers and robots.txt.

How to measure it

• Bing Webmaster Tools → AI Performance (beta) shows visibility in Microsoft’s AI experiences.
• Google Search Console includes AI Overview appearances in the regular Performance data, without a separate filter.
• Referral traffic from chatgpt.com, perplexity.ai and copilot in your analytics shows real visits from AI answers.
• Manual spot checks: ask the assistants your target questions and note whether you’re cited.

Where to measure it. For Google, the Generative AI report in Search Console shows impressions in AI Overviews and AI Mode: how to read it.

FAQ

Googlebot is Google’s web crawler: the program that discovers URLs, fetches your pages, renders them with an up-to-date Chrome engine and hands the result to Google’s indexing systems — if it cannot fetch or render a page, that page cannot rank.

How does Googlebot work?

• Discover. Googlebot finds URLs through links, sitemaps and previously known pages.
• Crawl. It fetches the page, respecting robots.txt and adjusting its speed to how fast and healthy your server is.
• Render. It runs the page’s JavaScript in an evergreen Chromium, so content loaded by scripts can be seen — if the scripts work for Googlebot.
• Hand over for indexing. The rendered content goes to Google’s indexing systems, which decide whether to keep it.

What are Googlebot Smartphone and Googlebot Desktop?

Googlebot is the generic name for two crawlers: Googlebot Smartphone, which simulates a user on a mobile device, and Googlebot Desktop, which simulates a desktop user. Google states that the majority of crawl requests are made with the mobile crawler and a minority with the desktop one, because Google indexes the mobile version of your site.

Both crawlers obey the same robots.txt token, Googlebot. You cannot allow the smartphone crawler and block the desktop crawler, or the other way round. In URL Inspection, the “Crawled as” line tells you which one fetched the page; for most sites it says Googlebot smartphone.

What does Googlebot Image crawl?

Googlebot Image fetches image files for Google Images and for every Search feature that shows a picture. Its user-agent string is Googlebot-Image/1.0, and it listens to the robots.txt tokens Googlebot-Image or, failing a specific group, Googlebot.

Blocking it reaches further than most people expect. According to Google’s crawler list, it affects Google Images, Discover, Video mode and all features in Google Search where images, logos and favicons are shown. Block it on your whole site and your favicon and article thumbnails can disappear from results.

# Keep one folder of images out of Google Images, allow everything else
User-agent: Googlebot-Image
Disallow: /private-images/

To get credit for the images you do want shown, the metadata matters as much as access: see image metadata SEO.

What does Googlebot Video crawl?

Googlebot Video fetches video files and video-related content for Google Search. Its user-agent string is Googlebot-Video/1.0; it obeys the tokens Googlebot-Video or Googlebot. Google says blocking it affects video-related Search features and other products that depend on videos.

A page can be indexed while its video is not: Googlebot reads the page, Googlebot Video fetches the file. If your video never gets a video result, check that the video file itself is not blocked by robots.txt or a firewall rule that only lets the main Googlebot through.

# Keep raw video files out of video features
User-agent: Googlebot-Video
Disallow: /media/raw/

Is there a separate Googlebot News crawler?

Not as a separate visitor. Google writes that Googlebot-News has no HTTP user-agent string of its own: crawling is done with the normal Googlebot user agents. Googlebot-News exists only as a robots.txt token. Blocking it affects Google News (news.google.com and the Google News app), not regular Search.

Which crawlers belong to the Googlebot family?

Several Google crawlers and tokens look alike but do different jobs. The table lists what each one controls and what happens in Search if you block it.

W.X.Y.Z stands for the current Chrome version, which changes over time; Google advises using a wildcard when you filter logs. Blocking Google-Extended does not hurt your Search rankings. More on that trade-off in our guide to AI crawlers and robots.txt, and on testing your rules in the robots.txt tester.

How do you see what Googlebot sees?

• URL Inspection → Test live URL → View tested page. Shows the rendered HTML, a screenshot and any resources that failed to load. This test is run by Google-InspectionTool, so a firewall that only trusts the Googlebot user agent can make the live test fail while real crawling works.
• Crawl stats report (Settings in Search Console). Shows how often Googlebot visits, response times and errors — the best way to spot server problems.
• Server logs. The ground truth of which URLs Googlebot requests, and how often.

If the rendered page looks right but the URL is still missing, work through how to check if a page is indexed and then crawled vs. discovered — currently not indexed.

How do you find Googlebot in your server logs?

Filter your access log on the string Googlebot, then verify the IP addresses before you trust the result. A quick first pass on a standard Apache or Nginx log:

grep -i "googlebot" access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head

That lists the IPs claiming to be Googlebot, busiest first. Check the top ones with the DNS test below. Look at which URLs real Googlebot requests most: if it spends its visits on parameter URLs or redirects, that is where your crawl budget goes.

Real or fake Googlebot?

Anyone can put “Googlebot” in a user-agent string, and scrapers often do. Google describes two ways to verify:

• DNS check. Run a reverse DNS lookup on the IP. A real Google crawler resolves to a host in googlebot.com, google.com or googleusercontent.com (Googlebot itself looks like crawl-66-249-66-1.googlebot.com). Then run a forward lookup on that hostname and confirm it returns the same IP.
• IP-range check. Compare the IP with Google’s published JSON files: common-crawlers.json for Googlebot, plus special-crawlers.json and user-triggered-fetchers.json for other Google agents. The old googlebot.json address now redirects to common-crawlers.json.

host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

How often does Googlebot come back? There’s no fixed schedule; it depends on demand and your server. Details and real numbers in how often Google crawls a site.

FAQ