Category: Blog

If your Search Console page indexing report has a hole in it for June 2026, nothing is wrong with your site, and the data is not coming back. Google says it doesn’t back-fill indexing data. My take: stop treating that report as your only record of what is indexed, and start keeping your own.

What happened to the page indexing report?

The page indexing report is the one most of us use to see which URLs Google has indexed and why others were left out. In June 2026 it stopped moving. According to Search Engine Land’s coverage, as summarised by ALM Corp, the report stayed on data from 11 June for about three weeks. It updated on the morning of 3 July and showed figures up to 29 June. Google had not publicly explained the cause at that time.

For anyone fixing an indexing problem in that window, that was a miserable three weeks: you could change a noindex tag, repair a redirect or resubmit a sitemap and the report could not show whether it worked.

The second part of the story came on 11 September. Site owners, including Dan Lauer on Bluesky, noticed that a chunk of June data that had been visible hours earlier was gone. Barry Schwartz reported it on Search Engine Roundtable. John Mueller’s answer was that the gap most likely comes from the time in June when the data was delayed: the report was not updated for that period, so there is nothing to show. Google’s position, in his words, is “We don’t back-fill indexing data.”

Does a gap in the report mean my pages were deindexed?

No. A missing stretch in the chart tells you Google did not record report data for those days. It does not tell you pages were dropped from the index. The two things live in different systems: the index serves search results, while the report is a reporting layer on top of it.

What can actually hurt is the confusion. If you compare today’s indexed count with a June number that no longer exists, you may think you lost pages when you didn’t. Compare against dates that have data, and check individual URLs with the URL Inspection tool when the answer matters.

How do I keep my own indexing record?

The lesson from June is simple: a report you don’t control can be late, frozen or incomplete. A small log you keep yourself costs ten minutes a month.

• Export the report on a schedule. The page indexing report has an export option. Save the indexed and not-indexed lists with the date in the file name once a month, or after every big change.
• Keep a list of the URLs that matter. New articles, money pages and anything you fixed. Next to each, note the date you published it, the date you requested indexing and the status you saw.
• Inspect key URLs directly. URL Inspection shows the last crawl date and whether the page is on Google, independent of the aggregate chart.
• Read your server logs. Googlebot requests in your hosting logs tell you when a page was actually fetched, whatever Search Console shows. The crawl frequency guide explains what normal looks like.
• Add a second engine. Bing Webmaster Tools gives you a separate view of what is discovered and indexed, and supports IndexNow.

How much should I trust indexing reports right now?

Trust them for direction, not for a single day’s number. Search Console had reporting delays earlier this year and another reporting glitch in early September, so a one-day dip or a flat week is weak evidence on its own. Stronger evidence is a steady change over several weeks, or a change you can confirm with a direct URL inspection.

This also matters after the September 2026 spam update. If you want to know whether the update moved pages into “Crawled – currently not indexed”, you need a clean before-and-after. Your own export from before 24 September is worth more than a chart with gaps. If pages are stuck, start with why Google isn’t indexing your site and the difference between crawled and discovered. Realistic timeframes are in how long Google takes to index a page.

FAQ

Google reads who made an image, who to credit and where its license lives from two places: IPTC photo metadata inside the file and ImageObject structured data on the page. Add a license URL and your images become eligible for the Licensable badge in Google Images. We added this to every image on this site, and learned that your host can quietly strip half of it.

What image metadata does Google read?

Google uses image metadata to show who is behind an image and how it may be used, both in Google Images. It can read that information from the file itself (IPTC photo metadata, stored as IPTC or XMP) or from structured data on the page. These are the fields that count:

With structured data, the only required property is contentUrl, plus at least one of creator, creditText, copyrightNotice or license. Google uses contentUrl to match the metadata to the right image, so it has to be the actual image URL.

How do you get the Licensable badge?

The Licensable badge in Google Images links to your license and tells people how the image may be used. Google’s rule is short: with IPTC you must fill in the Web Statement of Rights, and with structured data you must include license. Both point to a page that explains the terms.

That page doesn’t have to be a paid licensing shop. Ours, Image use & credit, says our images are free to share with a credit link back to the source article. That’s a license too, and it turns every copied image into a potential backlink.

IPTC or structured data — which should you use?

Use both if you can. They do different jobs:

• Structured data is what Google trusts first. The documentation says: if IPTC and structured data conflict, “Google will use the structured data information.” It also lives on your page, so nobody can strip it.
• IPTC metadata travels with the file. When someone downloads your image and uploads it elsewhere, the creator and license URL can go with it — if nothing strips them on the way.

On WordPress with Yoast SEO, the article’s main image is already output as an ImageObject. We added creator, creditText, copyrightNotice, license and acquireLicensePage to it with a small filter on Yoast’s schema graph, so every existing and future image gets the same fields.

What happened when we added it to 42 images

On 26 September 2026 we rebuilt the hero image of every article on this site: a visible footer with the article’s own URL, the author and the year, plus metadata written with ExifTool into IPTC, XMP and EXIF. Then we downloaded the live files the way a browser gets them. Our host serves images through a CDN that re-encodes them, often as WebP, and that re-encoding decided what survived.

In other words, the optimizer kept exactly the rights information and threw away the rest. That’s good enough for crediting, but it means you can’t rely on the file alone for descriptions or keywords. Those belong in the alt text, the caption and the page itself.

Two smaller lessons. First, the © sign got garbled in the IPTC Copyright Notice after re-encoding, so we now write “Copyright 2026 IndexUrePoint” there and keep the © only in XMP. Second, IPTC fields have hard length limits: Credit Line and Source allow just 32 characters, so “IndexUrePoint.com” fits and a full URL doesn’t.

How do you check your own images?

Don’t trust the file on your computer; check the file your visitors and Googlebot receive. With ExifTool installed:

curl -s -o live.jpg https://example.com/wp-content/uploads/image.jpg
curl -s -o live.webp -H "Accept: image/webp" https://example.com/wp-content/uploads/image.jpg
exiftool -G1 -s -Creator -Rights -WebStatement -Credit -CopyrightNotice live.jpg live.webp

The second line asks for WebP, like Chrome does, which is often where metadata disappears. Then:

• Test the page in Google’s Rich Results Test to confirm the ImageObject has license and contentUrl.
• In Search Console, look under Enhancements for an Image metadata report. On our site it appeared within days of adding the metadata. You’ll need your pages crawled first, so check they’re indexed.

What about AI-generated images?

Google reads the IPTC Digital Source Type field and supports four values for AI involvement, including trainedAlgorithmicMedia (made by a generative model) and compositeWithTrainedAlgorithmicMedia (a composite that includes AI-generated material). The base artwork of our illustrations comes from an AI image model and every word and number on them is added by us, so we label them as composites.

Our host stripped that field, though. So we also say it in plain words on our image use page. If labelling AI images honestly matters to you, don’t rely on the file field alone.

Should you put your URL on the image itself?

Metadata is invisible and fragile; pixels aren’t. Screenshots, social platforms and AI tools routinely drop metadata, but a small footer with your domain and the article’s URL survives all of them. That’s why every image here now carries its own address at the bottom.

Image-led searches are growing enough that Google now reports them separately: Search Console’s new multimodal filter shows traffic from Lens and Circle to Search.

Keep it modest: a thin strip, not a watermark across the subject. Google’s image guidelines favor relevant, high-quality images, and a logo splashed over the content does the opposite. For social sharing, make a separate 1200×630 version so platforms don’t crop your footer off.

FAQ

Google’s September 2026 spam update started on 24 September and finished rolling out on 8 October, after 13 days and 16 hours — far longer than the three earlier spam updates this year. My take: if your plan for getting pages into Google is volume and shortcuts, this is the update that makes that plan expensive.

What happened?

Google posted the update on its Search Status Dashboard on Thursday: “Released the September 2026 spam update, which applies globally and to all languages. The rollout may take up to two weeks to complete.” That window ran to roughly 8 October, and Google’s dashboard now lists the rollout as complete as of 8 October 2026, after 13 days and 16 hours.

The length is the story. According to the same dashboard, the March 2026 spam update finished in 19.5 hours, June’s in about two days and August’s in under three. A two-week allowance usually means Google expects to process a lot — or is being cautious about how it lands. Either way, don’t read too much into ranking swings in the first few days; compare against the full window.

Why this matters for indexing

Most people think of spam updates as ranking events. For small and new sites, they’re often indexing events too. When Google decides a site’s pages are low value, the symptom in Search Console usually isn’t a penalty notice — it’s pages sliding into “Crawled – currently not indexed” and new pages never making it out of “Discovered”.

That’s why I’m wary of the “just force it into the index” mindset. Pushing hundreds of near-identical pages through indexing services, or abusing the Indexing API for blog posts, can get URLs crawled. It can’t make them worth keeping — and scaled, low-value pages are exactly what spam systems look for. Google has also said since May that the same rules apply to content aimed at AI Overviews and AI Mode, so “GEO content farms” are in scope too.

How to check if your site was affected

• Mark the dates. In Search Console’s Performance report, compare clicks and impressions from 24 September onwards with the weeks before. Wait until the rollout completes before drawing conclusions.
• Watch the Page indexing report. A sudden rise in “Crawled – currently not indexed” on pages that were indexed before is a stronger signal than a ranking wobble.
• Check Manual actions. Algorithmic updates don’t send notices, but a manual action would show up under Security & Manual actions.
• Review Google’s spam policies honestly: scaled content abuse, site reputation abuse, cloaking, doorway pages, link schemes.

What if you were hit?

Google’s guidance is blunt: sites that see a change “should review our spam policies”, and improvements come if Google’s systems “learn over a period of months that the site complies” (Google). For link spam it’s harsher: any ranking benefit from spammy links “cannot be regained”. That means removing or improving thin pages, consolidating duplicates with canonicals or redirects, and then waiting — not re-submitting the same URLs every day.

FAQ

Google Search Console now has two AI tools every site owner should know: the Generative AI performance report, which shows how often your pages appear in AI Overviews and AI Mode, and the Search generative AI control, which lets you opt your whole site out of those features. Both reached all sites worldwide on 31 August 2026.

What changed in Search Console?

Until this year, Search Console mixed AI Overview appearances into the normal Performance report without telling you which impressions came from AI. And keeping content out of AI Overviews meant blunt tools: noindex, which removes the page from Google, or nosnippet, which also strips your normal search snippets. Both of those limitations are now gone.

How do you read the Generative AI performance report?

Open Search Console, choose Performance and then the Generative AI report for Search (Discover has its own). Google defines an impression as a link to your site being shown to a user in a generative AI feature. The report covers AI Overviews and AI Mode together; you can’t separate the two, and experiments in Search Labs aren’t included.

• Pages — which URLs appeared, reported under their canonical URL. This is the most useful tab: it tells you which pages Google’s AI actually uses as sources.
• Countries and Devices — where and on what those impressions happened.
• Dates — the trend over time. The newest data is preliminary and shown as a dotted line.
• Search type — “Web: text-based” or the new “Web: multimodal”.

Two details trip people up. First, the chart counts per property: if two of your pages appear in the same AI Overview, the chart counts one impression, while the Pages table counts both — so totals can differ. Second, the usual Search Console limits apply, including a maximum of 1,000 rows in the table. Use Export for anything larger.

A practical workflow: export the Pages tab, then compare it with the same period in the normal Performance report. Pages with many AI impressions but few clicks are being used as sources without sending visitors; pages with no AI impressions at all may simply not be indexed — check them with how to check if a page is indexed. For raising your citations, see how to get cited in AI search.

How does the Search generative AI opt-out control work?

Go to Settings → Search generative AI. You get two choices (three for child properties):

According to Google, the choice applies to the whole property and to child properties, which inherit it unless their owner overrides it. So a domain property decision also applies to blog.example.com and https://example.com/shop/ unless those are configured separately.

Three things the control does not do, all stated by Google:

• It is not a ranking or inclusion signal for the rest of Search. Your normal results are unaffected.
• It doesn’t affect AI training. For that you still need Google-Extended in robots.txt — see AI crawlers and robots.txt.
• Your content may still help Google Search as a whole, for example in understanding the language of queries and pages.

Search generative AI control vs Google-Extended vs noindex

If you only want to protect specific snippets rather than whole pages, the nosnippet rule and data-nosnippet attribute still exist, but they also limit your normal search snippets.

Should you opt out of AI Overviews and AI Mode?

For most sites, no. Opting out removes your links from the answers that increasingly sit above the classic results, while your competitors’ content stays in. Google also notes that content from other sites “may appear similar to yours” once you’re excluded — so you may lose the visibility without removing the information.

The trade-off is real, though. A preprint field study by researchers from the University of Pennsylvania and Northeastern, reported by Search Engine Journal, found that people forced to use AI Mode clicked through to other websites 18.8 percentage points less often than people using regular Search. It’s a forced-adoption experiment, not normal behaviour, but it shows why some publishers want out.

Opting out can make sense if your content is the product — paywalled research, data, original reporting — and AI answers substitute for a visit. Before switching, look at your Generative AI report: if your AI impressions are high and your clicks from the same pages are steady, staying in is likely helping, not hurting.

What is the new multimodal filter?

Since 24 September 2026, both the Performance report and the Generative AI report have a search type option for web searches that started with an image. Google says it covers Lens, Circle to Search on Android, image uploads to Google Search and Chrome’s right-click “Search this image”. If you sell products or publish visual content, this is the first time you can see that traffic separately. Data only appears if your site actually gets such searches.

If people find your pages through images, make sure those images carry your name and license too: see image metadata for Google.

What we see on this site

Both features are available for indexurepoint.com, which launched this month. As of 10 October 2026, the Generative AI report shows 4 impressions in total for 22 September to 6 October, one each for four pages: the home page, our guides hub, the Indexing API article and the orphan pages article. Four impressions is too small a sample to conclude anything, but it shows the report is live for a site this new. The control is set to the default, include, and we’re leaving it there: an independent guide wants to be cited. We’ll update these numbers as the report fills.

For the Microsoft side of the same question, Bing Webmaster Tools has its own AI Performance report showing citations in Copilot — see Bing Webmaster Tools.

FAQ

This free robots.txt tester tells you whether Googlebot, Bingbot or an AI crawler may fetch a URL under your robots.txt, which rule decides it, and which lines Google will ignore. It follows Google’s published matching rules and runs entirely in your browser.

How do you use this robots.txt tester?

• Paste your robots.txt. Open yourdomain.com/robots.txt in a browser, copy everything and paste it into the box.
• Enter a URL or path, such as https://yourdomain.com/shop/?color=red or just /shop/.
• Pick a crawler and click Test URL. You get Allowed or Blocked, the exact rule and line number that decided it, and which user-agent group was used.
• Test all crawlers checks the same URL for Googlebot, Bingbot and the main AI crawlers at once — handy after adding rules for GPTBot, ClaudeBot or Google-Extended.
• Validate file flags lines Google will ignore: misspelled directives, rules outside a group, relative sitemap URLs, crawl-delay and noindex.

How does Google read a robots.txt file?

Source for all of the above: Google's robots.txt specification.

Example: the longest rule wins

User-agent: *
Allow: /page
Disallow: /*.html

For /page.html, both rules match. /*.html is longer than /page, so the URL is blocked — a case that surprises many site owners. Paste it into the tester to see the deciding rule.

Example: one group, not two

User-agent: *
Disallow: /carrots

User-agent: Googlebot
Disallow: /shrimp

Googlebot follows only its own group, so /carrots is allowed for Googlebot even though the * group blocks it. If you add a group for a specific bot, repeat every rule you want it to obey.

What replaced Google's robots.txt Tester?

Google announced a robots.txt report in Search Console in November 2023 and sunset the old Tester. The report (Settings → robots.txt) shows which robots.txt files Google found for your site, when it last fetched them, and any fetch errors or warnings. It doesn't let you test individual URLs, which is what this tool is for.

To check a single page from Google's side, use URL Inspection: it reports "Crawl allowed? No: blocked by robots.txt" when a rule stops Googlebot. See how to check if a page is indexed.

What are the most common robots.txt mistakes?

• Blocking to deindex. Disallow stops crawling, not indexing. A blocked URL can still appear in Google without a description. Use noindex instead — see blocked by robots.txt and the X-Robots-Tag for files.
• Blocking CSS and JavaScript. Google renders pages; blocking /wp-includes/ or theme assets can make pages look broken to it.
• A leftover Disallow: / from a staging site. One line blocks the whole site.
• Forgetting that groups don't combine, as in the example above.
• Case mismatches: Disallow: /Private/ doesn't block /private/.
• Relative sitemap URLs. Sitemap: must be a full URL.

FAQ

“Blocked due to access forbidden (403)” means your server refused Googlebot with an HTTP 403. Since Googlebot never logs in, Google says the server “is returning this error incorrectly” — usually because a firewall, CDN bot protection or security plugin mistook Googlebot for a bad bot.

What does “Blocked due to access forbidden (403)” mean?

It’s one of the “why pages aren’t indexed” reasons in Search Console’s Page indexing report. Google’s definition is short and blunt:

“HTTP 403 means that the user agent provided credentials, but was not granted access. However, Googlebot never provides credentials, so your server is returning this error incorrectly. The page will not be indexed.” — Google Search Console Help

In other words: a human in a browser can probably see the page, but when Googlebot asks for it, something between the internet and your content says no. The page stays out of the index until that changes. If it was already indexed, it can drop out after repeated 403s.

What causes a 403 for Googlebot?

A quick way to split the causes: open the URL in a private browser window. If you get a 403 too, it’s permissions or server rules. If the page loads for you but Google reports 403, it’s almost certainly security tooling reacting to crawlers.

Our test: why a fake Googlebot can fool you

A common tip is to “test as Googlebot” by sending a request with Googlebot’s user agent string. We tried that on this site — and extended it to AI crawlers — to see how much you can trust it. Each crawler name was sent five times to an uncached page:

Taken at face value, that says our site blocks GPTBot and sometimes OpenAI’s and Anthropic’s other crawlers. But our host’s AI crawler log for the same 24 hours showed 21 real AI-crawler requests — 17 from GoogleOther and 4 from OAI-SearchBot — and zero blocked. The difference: our requests came from an ordinary server, not from OpenAI’s or Anthropic’s IP addresses, so the CDN treated them as impostors. That’s exactly what good bot protection should do.

The lesson for 403 debugging: a spoofed user agent tells you what your firewall does with fake bots. To see what happens to the real Googlebot, use Google’s own tools and your security logs.

How do you fix “Blocked due to access forbidden (403)”?

1. Confirm it with a live test

In Search Console, inspect the URL and click Test live URL. If “Page fetch” fails with a 403, the problem is current. If the live test succeeds, the report may simply be showing an older crawl — see how to check if a page is indexed.

2. Find what is blocking

Check, in this order: your CDN or WAF security events, your hosting firewall, your WordPress security plugin’s log, and finally your .htaccess or NGINX deny rules. Look for blocked requests whose user agent contains “Googlebot” around the time of the failed crawl.

3. Allow verified Googlebot — not the user agent string

Never whitelist by user agent alone; anyone can type “Googlebot”. Google’s verification guide gives two methods:

• Reverse DNS: look up the visitor’s IP (for example host 66.249.66.1), check that the hostname ends in googlebot.com, google.com or googleusercontent.com, then do a forward lookup to confirm it points back to the same IP.
• IP ranges: match against the JSON lists Google publishes, such as common-crawlers.json for Googlebot.

Most CDNs and security plugins have a “verified bots” or “allow search engines” option that does this for you. Turn it on rather than writing your own rules.

4. Remove the block and re-test

Run the live test again. When it passes, click Request indexing for the most important URLs and use Validate fix in the Page indexing report for the rest. Revalidation can take days to weeks, in line with Google’s normal crawl frequency.

What about “Blocked due to unauthorized request (401)”?

A 401 means the page asked for authorization — typically a password prompt or login wall. Google’s advice: if you want the page indexed, “either remove authorization requirements for this page, or else allow Googlebot to access your pages by verifying its identity”. You can reproduce it by visiting the page in an incognito window.

Staging sites protected with a password are the classic case: they correctly return 401, and you should keep them out of sitemaps and internal links rather than open them up.

When can you ignore a 403 or 401?

• Admin and login pages such as /wp-admin/ — they’re supposed to be closed. On this site, /wp-admin/ redirects anonymous visitors to the login page and /.htaccess returns 403, as it should.
• Members-only content you never meant to rank.
• Old URLs Google remembers but you no longer link to.

Do make sure none of these appear in your XML sitemap — a sitemap full of blocked URLs sends mixed signals. See sitemap could not be read and, for the overall picture, why Google isn’t indexing your site.

FAQ

Google has no fixed crawl schedule: it recrawls each URL as often as it thinks the page changes and matters, within what your server can handle. Popular, frequently updated pages can be fetched several times a day; quiet pages on small sites may wait weeks. For sites under about 10,000 pages, crawl budget is almost never the problem.

How often does Google crawl a site?

There is no single number, because Google doesn’t crawl “a site” on a timer. Its crawler keeps a queue of known URLs and decides for each one when it’s worth fetching again. Your homepage and a page that changes daily may be fetched several times a day; a two-year-old article that never changes may be revisited only every few weeks or months.

Figures like “every three to four weeks” that circulate online are averages from specific sites, not Google policy. The honest answer is: as often as Google thinks a page has changed and is worth refreshing, limited by how much your server can take.

What decides how often Google crawls your pages?

Google’s crawl budget documentation names two forces.

In practice that translates into a handful of levers:

• Popularity: pages with more links pointing to them — from other sites and from your own pages — are crawled more.
• Change frequency: Google learns how often a URL actually changes. Updating a date without changing content doesn’t fool it for long.
• Server health: Google says that when a site responds with 5xx errors or 429 rate-limit signals, “the limit goes down and Google crawls less”.
• URL inventory: thousands of duplicate, filtered or low-value URLs dilute the attention your important pages get.

What is crawl budget — and do you need to care?

Crawl budget is the combination of those two forces: the set of URLs Google can and wants to crawl on your site in a given period. Google is direct about who should worry about it:

Google’s Crawl Stats help page says the same: “If you have a site with fewer than a thousand pages, you should not need to use this report or worry about this level of crawling detail.” If your small site has pages that aren’t indexed, the cause is almost always quality, duplication or missing links — see crawled vs. discovered – currently not indexed and why Google isn’t indexing your site.

Real data: how Google crawled a brand-new site

indexurepoint.com went live in September 2026. This is what Search Console’s Crawl Stats report showed on 25 September, covering the first two days it reported (23–24 September):

Three things stand out. First, only 45% of requests were for HTML pages: the rest were scripts, images, CSS and feeds that Google needs to render pages. Second, two-thirds of requests were refreshes of URLs Google already knew — even on a site a few days old. Third, the URL Inspection records show how quickly new pages were picked up: guides we published on 24 September were crawled between 03:57 and 09:35 the next morning, and by that afternoon eight of the nine were indexed. Each was linked from other pages on the site and listed in the sitemap; details in how long Google takes to index.

The 5% of 404s are worth a look on any site: check which URLs they are, then either fix the links pointing to them or confirm the pages are meant to be gone. On a small site it’s harmless; on a large one, those wasted requests add up.

How do you see when Google last crawled your site?

• One page: URL Inspection in Search Console shows “Last crawl” with date and time, plus whether it was crawled as smartphone or desktop. See how to check if a page is indexed.
• The whole site: Settings → Crawl stats shows requests per day, response times, status codes, file types and Googlebot types for the last 90 days.
• Every request: your server’s access logs list each Googlebot visit with a timestamp. Verify real Googlebot by reverse DNS or Google’s published IP ranges, since user agents can be faked.

How do you get Google to crawl your site more often?

• Link new pages from strong existing pages. Internal links are how Google finds and prioritises URLs; pages nobody links to are crawled last — see orphan pages.
• Keep your XML sitemap accurate, with only indexable URLs and a real lastmod date when content changes.
• Make the server fast and stable. Google’s documentation lists improving load speed and supporting 304 Not Modified caching among its best practices.
• Clean up your URL inventory: consolidate duplicates, return 404 or 410 for pages that are gone, fix soft 404s and avoid redirect chains.
• Update content for real. Pages that genuinely change get refreshed more often.
• Request indexing sparingly for key new or changed pages. There is a daily quota, and repeating a request doesn’t make it faster.
• For Bing, use IndexNow to push changes instantly. Google doesn’t support it.

What goes wrong with crawling?

• Blaming crawl budget on a small site. With a few hundred pages, “crawled but not indexed” is a quality or duplication signal, not a budget problem.
• Parameter and filter URLs. Faceted navigation can create endless URL combinations that soak up crawling; a r/TechSEO thread titled “Google prioritizing crawl budget on filtered URLs” describes this pattern. Block or consolidate them.
• Slow or failing servers. Timeouts and 5xx errors make Google back off.
• A failing robots.txt. If robots.txt returns neither a valid file nor a 404, Google says it will slow or stop crawling until it gets an acceptable response.
• Redirect chains. Every hop is another request; point links straight at the final URL — see page with redirect.
• Spamming “Request indexing”. It hits the quota and changes nothing.

Crawl requests failing with 403? Then Google can’t crawl those pages at all. See blocked due to access forbidden (403).

FAQ

The X-Robots-Tag is an HTTP response header that gives search engines the same indexing instructions as a robots meta tag — noindex, nofollow, nosnippet and more — but works on any file type, including PDFs, images, XML and plain text, where a meta tag can’t be placed.

What is the X-Robots-Tag?

Every time a browser or crawler requests a URL, the server replies with HTTP headers before the content itself: the status code, the content type, caching rules. The X-Robots-Tag is one of those headers. It tells search engines whether they may index the URL and how they may show it.

HTTP/2 200
content-type: application/pdf
x-robots-tag: noindex

That response says: this PDF may be fetched, but don’t show it in search results. It does the same job as <meta name=”robots” content=”noindex”> in an HTML page — which a PDF simply can’t contain. The header name, the user-agent names and the values are all case-insensitive, according to Google.

How is it different from a meta robots tag and robots.txt?

The last row is why X-Robots-Tag problems are hard to spot: you can view a page’s source and see nothing wrong, while the server quietly sends noindex. The difference between blocking and noindexing is covered in blocked by robots.txt and excluded by noindex tag.

Which values can you use?

Google supports these rules in both the meta tag and the header:

Bing is different on the last row. Bing supports the X-Robots-Tag too, and for Bing noarchive means “do not link in Chat and Copilot”, while nocache limits Copilot to URL, title and snippet. If you want to control how Microsoft’s AI uses your content, these are the values that do it — see AI crawlers and robots.txt.

You can also target one crawler by name and combine several headers:

X-Robots-Tag: googlebot: nofollow
X-Robots-Tag: otherbot: noindex, nofollow
X-Robots-Tag: unavailable_after: 31 Dec 2026 23:59:59 CET

When should you use the X-Robots-Tag?

• PDFs and documents you want downloadable but not ranking — price lists, internal manuals, duplicate PDF versions of HTML pages.
• Images you don’t want in Google Images.
• Machine files such as XML sitemaps, feeds, API responses and llms.txt, which are meant for bots and look odd as search results.
• Whole sections, such as a staging folder or a directory of print pages, where editing every template is impractical.
• Temporary content with unavailable_after, like event or offer pages.

For normal HTML pages, a meta robots tag set by your SEO plugin is usually simpler and easier for your team to see.

How do you set the X-Robots-Tag?

Apache (.htaccess)

This is Google’s own example for all PDFs:

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

NGINX

location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex, nofollow";
}

WordPress and PHP

Yoast SEO already sends noindex, follow on its XML sitemaps. For your own output, send the header before any content, for example in a small plugin:

header( 'X-Robots-Tag: noindex', true );

That’s how our site plugin marks /llms.txt as noindex. On a CDN such as Cloudflare you can add the header with a response-header transform rule instead of touching the server.

How do you check a URL’s X-Robots-Tag?

• Command line: curl -I https://example.com/file.pdf shows all response headers.
• Browser: DevTools → Network → click the request → Response Headers.
• Search Console: URL Inspection reports “Indexing allowed? No: ‘noindex’ detected in ‘X-Robots-Tag’ http header” when the header blocks indexing. The live test shows the current state; see how to check if a page is indexed.

Example: the headers on this site

On 25 September 2026 we requested eight URL types on indexurepoint.com and recorded the X-Robots-Tag:

The pattern is exactly what you want: machine files carry a header noindex, human pages are controlled by the meta tag, and nothing a visitor should find is blocked. Note that the sitemaps are noindexed but not blocked in robots.txt — Google still has to crawl them to read the URLs inside.

What goes wrong with the X-Robots-Tag?

• An invisible noindex. A server rule, CDN setting or security plugin adds noindex to every response, and nobody notices because the page source looks clean. A r/TechSEO thread titled “Hidden X-Robots-Tag: noindex reported by GSC – can’t find source” is a typical example. Check the live headers, not the HTML.
• Staging rules going live. A site-wide noindex header on the staging server gets copied to production during launch.
• Blocked and noindexed at once. If robots.txt disallows the URL, Google can’t read the header, so the URL may stay indexed. Remove the Disallow until the noindex has been processed.
• Conflicting rules. Header says noindex, meta tag says index: Google applies the most restrictive combination, so the page stays out.
• Relying on noarchive for Google. Google no longer uses it. It still matters for Bing and Copilot.
• Expecting instant removal. The header takes effect when Google recrawls the URL. For a fast temporary hide, see how to remove a URL from Google.

FAQ

llms.txt is a Markdown file at the root of a website that gives AI agents a short, curated map of its most important pages. It is a proposal, not a standard: Google Search says it doesn’t use it, Chrome’s Lighthouse now checks for it, and it helps agents far more than it helps rankings.

What is llms.txt?

llms.txt is a plain-text file written in Markdown and served at /llms.txt, the same place robots.txt lives. It gives a language model or AI agent a compact summary of what a site is about and a list of links to the pages that matter, so the agent doesn’t have to wade through navigation, scripts and ads to work that out.

The proposal was published by Jeremy Howard of Answer.AI on 3 September 2024. Its core idea is stated plainly: the file “stays small enough to fit in context”, and the detail “lives behind the links”. It is a community proposal — not a W3C, IETF or search-engine standard — and nobody is obliged to read it.

What does the llms.txt format look like?

The specification defines a fixed order, so the file can be read by a person, a model or a simple parser:

• H1 with the site or project name — the only required element.
• A blockquote with a short summary containing the key information needed to understand the rest.
• Optional paragraphs or lists (no headings) with more detail on how to interpret the site.
• H2 sections with “file lists”: each entry is a Markdown link [name](url), optionally followed by a colon and a note.
• An “Optional” H2 section, by convention, for secondary links an agent can skip when context is tight.

This is the start of the live file on this site:

# IndexUrePoint

> Independent, plain-English guides to getting web pages found
> and indexed on Google, Bing and AI search ... English and Dutch.

## Guides (English)

- [How to Remove a URL from Google](https://indexurepoint.com/remove-url-from-google/): Fast temporary hide versus permanent removal.
- [Orphan Pages: What They Are and How to Fix Them](https://indexurepoint.com/orphan-pages/): Pages nobody links to get crawled last.

The proposal also suggests offering clean Markdown copies of important pages at the same URL with .md appended. Some documentation platforms additionally publish an llms-full.txt with all content in one file; that is a common convention, not part of the core proposal.

Who actually uses llms.txt?

This is where most articles overpromise. Here is what can be verified.

• Google Search: no. Google’s guide to optimizing for generative AI features (updated July 2026) says you don’t need AI text files or Markdown to appear in Search or its AI features. Gary Illyes said at Search Central Live in July 2025 that Google doesn’t support llms.txt and isn’t planning to (Search Engine Land). When Google added llms.txt files to some of its own developer sites, John Mueller said plainly it was not an endorsement.
• Chrome Lighthouse: it checks. Lighthouse 13.3 moved its Agentic Browsing category into the default report, including an llms.txt audit (Search Engine Journal). The audit is about agents, not rankings: without the file, Chrome’s documentation says, “agents may spend more time crawling the site”.
• AI documentation sites: widely. Anthropic (code.claude.com/docs/llms.txt), OpenAI (developers.openai.com/llms.txt), Google (ai.google.dev/api/llms.txt) and Perplexity (docs.perplexity.ai/llms.txt) all returned HTTP 200 when we checked. Coding agents that read documentation are the clearest real-world audience.
• AI answer engines citing ordinary websites: unproven. We found no official statement from OpenAI, Anthropic, Perplexity or Microsoft that llms.txt influences which websites are cited in answers. Critics such as Mark Williams-Cook argue the evidence for it is no better than astrology.

How is llms.txt different from robots.txt and a sitemap?

The key point: llms.txt does not block or allow anything. If you want to control what AI crawlers can fetch, that is still a job for robots.txt rules per AI crawler. And if you want pages discovered by search engines, you still need a working sitemap and internal links.

Do you need an llms.txt file?

If your goal is visibility in Google’s AI Overviews or AI Mode, spend the time on indexing and content instead — the steps in how to get cited in AI search and AEO vs GEO. If your site is documentation that developers use with AI coding tools, an llms.txt (and Markdown versions of key pages) is a reasonable, low-cost addition.

How do you create an llms.txt file?

Option 1 — Yoast SEO (WordPress)

Yoast added llms.txt generation on 10 June 2025, free for all users. Turn it on under Yoast SEO → Settings → Site features; the plugin builds the file from your site data and refreshes it weekly. You can let it pick content automatically or choose pages yourself.

Option 2 — write it by hand

Create a UTF-8 text file named llms.txt, follow the order above, and upload it to your site’s root so it loads at yourdomain.com/llms.txt. Keep it short: your 10 to 30 most useful pages, each with a one-line note, beats a dump of every URL.

Option 3 — generate it dynamically

That’s what we do. Our site plugin builds /llms.txt on request from the published guides, grouped into English and Dutch sections, and uses each article’s excerpt as the note. New articles appear automatically — both guides we published earlier today were in the file the moment they went live.

Then: add a noindex header

John Mueller has said using noindex for llms.txt “could make sense”, because other sites might link to it and it could otherwise get indexed, “which would be weird for users”. Since it’s a text file you can’t add a meta tag, so send an HTTP header instead. Ours returns:

HTTP/2 200
content-type: text/plain; charset=utf-8
x-robots-tag: noindex

Check yours with the URL Inspection tool or a header checker; the header itself is explained in our X-Robots-Tag guide.

What goes wrong with llms.txt?

• Treating it as a ranking hack. Selling llms.txt as a way into AI Overviews contradicts Google’s own documentation.
• Server errors. A 500 on /llms.txt is the one thing Lighthouse fails you for. A 404 is fine; a broken file is not.
• Links to non-indexable pages. Listing redirected, noindexed or deleted URLs sends agents to dead ends. Keep it in sync with your sitemap.
• Dumping everything. A file with thousands of links defeats the purpose of fitting in context. Curate.
• Blocking the file in robots.txt. Then compliant AI agents can’t read it. Use noindex, not Disallow.
• Duplicate generators. Two plugins both creating the file can conflict. Pick one method.

Want to control AI Overviews instead? llms.txt won’t do it, but Search Console’s new opt-out control will: AI Overviews in Search Console.

FAQ