Category: Blog

To remove a URL from Google permanently, change the page itself — 404/410, noindex, or a login — and use Search Console’s Removals tool only as the fast, temporary hide (about six months) while the permanent fix takes effect.

Pick the right method

• Page is gone for good: delete it and return 404 or 410. Remove it from your sitemap.
• Page should exist but not in Google: add a noindex meta tag or X-Robots-Tag header — and make sure it’s not blocked in robots.txt, or Google can’t see the noindex. See noindex.
• Private content: put it behind a login or password. Google can’t index what it can’t access.
• Page moved: 301-redirect it to the new URL instead of removing it.

Using the Removals tool

• In Search Console, open Indexing → Removals → New request.
• Enter the exact URL, or choose “Remove all URLs with this prefix” for a whole folder.
• Choose Temporarily remove URL (hides it and clears the cached snippet) or Clear cached URL (keeps it in results but refreshes the snippet).
• At the same time, apply a permanent fix from the list above.

When it’s not your site

You can’t remove other people’s pages, but you can ask Google to refresh them. If a page has already changed or been deleted but Google still shows the old version, use Google’s Refresh Outdated Content tool. For personal information or legal issues, Google has separate removal request forms.

A worked example

Say you published a page with a customer’s name in it by mistake at /case-study-acme/, and it’s already indexed. The right order: first edit or unpublish the page — in WordPress, setting it to draft makes the URL return 404. Then open Search Console → Removals → New request, paste the exact URL and choose Temporarily remove URL. Within about a day the result disappears. Because the URL now returns 404, Google drops it permanently on the next crawl, and the six-month clock on the temporary hide stops mattering. If the page had to stay online without the sensitive part, you would instead edit the content and use Clear cached URL so the old snippet stops showing while the fresh version takes over.

Removing files, not pages? PDFs and images can’t hold a meta tag. Use the X-Robots-Tag header to noindex them.

FAQ

AI companies run different crawlers for different jobs — training, search and user-initiated fetching — and robots.txt lets you allow or block each one separately: allow the search bots if you want citations, decide deliberately about the training bots.

The main AI crawlers (September 2026)

• OpenAI: GPTBot collects training data. OAI-SearchBot surfaces sites in ChatGPT search. ChatGPT-User fetches pages when a user asks — and since OpenAI’s December 2025 documentation change, robots.txt rules may not apply to it.
• Anthropic: ClaudeBot for training, Claude-SearchBot for search, Claude-User for user-initiated fetches.
• Perplexity: PerplexityBot for its search index, Perplexity-User for user requests.
• Google: Google-Extended (a control token, not a separate crawler) for Gemini training and grounding in Gemini Apps and Vertex AI.
• Apple: Applebot-Extended controls use of content for Apple’s AI training.

Google-Extended explained

Google-Extended controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google states it does not affect your inclusion in Google Search and isn’t a ranking signal. AI Overviews are part of Search and rely on Googlebot, so blocking Google-Extended doesn’t take you out of them — see what is Googlebot. Since 31 August 2026 there is a separate switch for that: the Search generative AI control in Search Console lets you leave AI Overviews and AI Mode while staying in normal results — see AI Overviews in Search Console.

Example robots.txt rules

Allow AI search, block AI training:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot are allowed by default when you don’t list them. Test any change in Search Console’s robots.txt report, and remember that robots.txt is a request, not a lock — see blocked by robots.txt.

What about llms.txt? It doesn’t allow or block anything — it only suggests which pages matter to AI agents. See what is llms.txt for who actually reads it.

Real data from this site. Our host’s AI crawler log for 24 hours on 25 September 2026 showed 21 requests from known AI crawlers — 17 from GoogleOther and 4 from OAI-SearchBot — and none blocked. Testing with faked crawler user agents gave a very different (and misleading) picture; we explain why in our 403 guide.

Check before you publish. Our free robots.txt tester shows in one click whether GPTBot, ClaudeBot, PerplexityBot and Googlebot are allowed or blocked by your rules.

FAQ

An orphan page is a page no other page on your site links to: visitors cannot click their way there, search engines find it late through the sitemap at best, and it typically indexes last — the fix is simply linking to it.

Why orphan pages hurt

Internal links tell Google what matters on your site. A page nothing links to looks unimportant, even if it’s your best work.

How to find orphan pages

• Compare two lists: all URLs in your XML sitemap versus all URLs a crawler finds by following links from your homepage. URLs in the first list but not the second are orphans.
• Use a site crawler such as Screaming Frog, Sitebulb or Bing Webmaster Tools’ Site Scan, and connect it to your sitemap or Search Console data where supported.
• Check Search Console: pages with impressions or in the indexing report that your crawl never reached.
• Common culprits: old landing pages, pages removed from menus, tag pages, and posts only reachable through paginated archives.

How to fix them

• Worth keeping? Add contextual links from two or three related, indexed pages, and consider a hub or category page that links to the whole topic.
• Outdated or duplicate? Merge it into a stronger page and 301-redirect it — see page with redirect.
• Not needed at all? Delete it, return a 404 or 410, and remove it from the sitemap.

A worked example

A typical case from our own audits: a site moves to a new menu, and twelve older guides drop out of it. They stay in the sitemap, so Search Console still knows them — but months later most sit in “Discovered – currently not indexed” or have slid down the rankings. The repair took an afternoon: each old guide got two contextual links from the newer articles that superseded parts of it, plus a place on a topic hub page. Within weeks the majority were recrawled. The lesson is that orphaning rarely happens on purpose; it happens when navigation changes and nobody checks what fell out. Make “what stopped being linked?” a standard question after every redesign.

FAQ

AEO (answer engine optimization) and GEO (generative engine optimization) are two names for the same goal: getting your content used and cited in AI-generated answers — and the foundation under both is ordinary indexing.

AEO vs GEO vs SEO

SEO is the foundation under both. AI answers are built on pages that search engines have already crawled and indexed; if you’re not indexed, no amount of AEO or GEO helps. That’s why this site treats getting indexed as step zero.

What the work actually is

• Be indexed in Google and Bing. Google’s AI Overviews build on Google’s index; Microsoft Copilot builds on Bing. Set up Bing Webmaster Tools and IndexNow, not just Search Console.
• Answer first. Put the direct answer in the first two sentences under a question-style heading. Models lift clear, self-contained sentences.
• Be specific. Concrete steps, named tools, exact labels and sourced facts are more quotable than generalities.
• Show who’s behind it. A consistent brand, a clear about page and first-hand experience signal trust.
• Use clean structured data. Article and FAQ schema help machines parse the page.
• Let the right crawlers in. Blocking AI search crawlers opts you out of their answers — see AI crawlers and robots.txt.

How to measure it

• Bing Webmaster Tools → AI Performance (beta) shows visibility in Microsoft’s AI experiences.
• Google Search Console includes AI Overview appearances in the regular Performance data, without a separate filter.
• Referral traffic from chatgpt.com, perplexity.ai and copilot in your analytics shows real visits from AI answers.
• Manual spot checks: ask the assistants your target questions and note whether you’re cited.

Where to measure it. For Google, the Generative AI report in Search Console shows impressions in AI Overviews and AI Mode: how to read it.

FAQ

The Google Indexing API lets a site tell Google directly that a page was added or removed — but officially only for JobPosting and livestream pages, and Google has attached an explicit spam warning for everything else.

What the Indexing API does

It’s a programmatic way to tell Google “this URL was updated” or “this URL was deleted”, authenticated with a Google Cloud service account that’s an owner of your Search Console property. For job boards and livestream pages — content that changes or expires quickly — it gets Google to crawl sooner than a sitemap would.

Why people misuse it

Because it sometimes triggers a fast crawl. Tutorials and “instant indexing” plugins spread the trick, and some paid indexing services are built on it. For a site owner watching pages sit in Discovered – currently not indexed, it’s tempting.

The risk

• It’s against the documented use. Google states the API is for job posting and livestream pages only.
• Google added a spam warning. The documentation warns that abuse, including using multiple accounts or other means to exceed quotas, may lead to access being revoked.
• A crawl isn’t indexing. Even when the API triggers a crawl, Google still decides whether the page is worth keeping. A page Google rejected for quality stays rejected.

What to do instead

• Submit a clean sitemap and request indexing for key pages in Search Console — see how to submit your website to Google.
• Link new pages from strong, already-indexed pages.
• Use IndexNow for Bing — the legitimate, open “push” protocol. Google doesn’t support it, but Bing and others do: what is IndexNow.
• Make the page worth keeping. Unique, useful content is what turns a crawl into an index entry.

FAQ

Googlebot is Google’s web crawler: the program that discovers URLs, fetches your pages, renders them with an up-to-date Chrome engine and hands the result to Google’s indexing systems — if it cannot fetch or render a page, that page cannot rank.

How does Googlebot work?

• Discover. Googlebot finds URLs through links, sitemaps and previously known pages.
• Crawl. It fetches the page, respecting robots.txt and adjusting its speed to how fast and healthy your server is.
• Render. It runs the page’s JavaScript in an evergreen Chromium, so content loaded by scripts can be seen — if the scripts work for Googlebot.
• Hand over for indexing. The rendered content goes to Google’s indexing systems, which decide whether to keep it.

What are Googlebot Smartphone and Googlebot Desktop?

Googlebot is the generic name for two crawlers: Googlebot Smartphone, which simulates a user on a mobile device, and Googlebot Desktop, which simulates a desktop user. Google states that the majority of crawl requests are made with the mobile crawler and a minority with the desktop one, because Google indexes the mobile version of your site.

Both crawlers obey the same robots.txt token, Googlebot. You cannot allow the smartphone crawler and block the desktop crawler, or the other way round. In URL Inspection, the “Crawled as” line tells you which one fetched the page; for most sites it says Googlebot smartphone.

What does Googlebot Image crawl?

Googlebot Image fetches image files for Google Images and for every Search feature that shows a picture. Its user-agent string is Googlebot-Image/1.0, and it listens to the robots.txt tokens Googlebot-Image or, failing a specific group, Googlebot.

Blocking it reaches further than most people expect. According to Google’s crawler list, it affects Google Images, Discover, Video mode and all features in Google Search where images, logos and favicons are shown. Block it on your whole site and your favicon and article thumbnails can disappear from results.

# Keep one folder of images out of Google Images, allow everything else
User-agent: Googlebot-Image
Disallow: /private-images/

To get credit for the images you do want shown, the metadata matters as much as access: see image metadata SEO.

What does Googlebot Video crawl?

Googlebot Video fetches video files and video-related content for Google Search. Its user-agent string is Googlebot-Video/1.0; it obeys the tokens Googlebot-Video or Googlebot. Google says blocking it affects video-related Search features and other products that depend on videos.

A page can be indexed while its video is not: Googlebot reads the page, Googlebot Video fetches the file. If your video never gets a video result, check that the video file itself is not blocked by robots.txt or a firewall rule that only lets the main Googlebot through.

# Keep raw video files out of video features
User-agent: Googlebot-Video
Disallow: /media/raw/

Is there a separate Googlebot News crawler?

Not as a separate visitor. Google writes that Googlebot-News has no HTTP user-agent string of its own: crawling is done with the normal Googlebot user agents. Googlebot-News exists only as a robots.txt token. Blocking it affects Google News (news.google.com and the Google News app), not regular Search.

Which crawlers belong to the Googlebot family?

Several Google crawlers and tokens look alike but do different jobs. The table lists what each one controls and what happens in Search if you block it.

W.X.Y.Z stands for the current Chrome version, which changes over time; Google advises using a wildcard when you filter logs. Blocking Google-Extended does not hurt your Search rankings. More on that trade-off in our guide to AI crawlers and robots.txt, and on testing your rules in the robots.txt tester.

How do you see what Googlebot sees?

• URL Inspection → Test live URL → View tested page. Shows the rendered HTML, a screenshot and any resources that failed to load. This test is run by Google-InspectionTool, so a firewall that only trusts the Googlebot user agent can make the live test fail while real crawling works.
• Crawl stats report (Settings in Search Console). Shows how often Googlebot visits, response times and errors — the best way to spot server problems.
• Server logs. The ground truth of which URLs Googlebot requests, and how often.

If the rendered page looks right but the URL is still missing, work through how to check if a page is indexed and then crawled vs. discovered — currently not indexed.

How do you find Googlebot in your server logs?

Filter your access log on the string Googlebot, then verify the IP addresses before you trust the result. A quick first pass on a standard Apache or Nginx log:

grep -i "googlebot" access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head

That lists the IPs claiming to be Googlebot, busiest first. Check the top ones with the DNS test below. Look at which URLs real Googlebot requests most: if it spends its visits on parameter URLs or redirects, that is where your crawl budget goes.

Real or fake Googlebot?

Anyone can put “Googlebot” in a user-agent string, and scrapers often do. Google describes two ways to verify:

• DNS check. Run a reverse DNS lookup on the IP. A real Google crawler resolves to a host in googlebot.com, google.com or googleusercontent.com (Googlebot itself looks like crawl-66-249-66-1.googlebot.com). Then run a forward lookup on that hostname and confirm it returns the same IP.
• IP-range check. Compare the IP with Google’s published JSON files: common-crawlers.json for Googlebot, plus special-crawlers.json and user-triggered-fetchers.json for other Google agents. The old googlebot.json address now redirects to common-crawlers.json.

host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

How often does Googlebot come back? There’s no fixed schedule; it depends on demand and your server. Details and real numbers in how often Google crawls a site.

FAQ

The reliable way to check if a page is indexed is URL Inspection in Google Search Console: it answers “URL is on Google” or “URL is not on Google” and, crucially, tells you why — the site: operator is only a rough sample.

Four ways to check, ranked by reliability

• 1. URL Inspection (Search Console) — the source of truth. Paste the URL in the bar at the top. You’ll see whether it’s on Google, the last crawl date, the user-declared and Google-selected canonical, and the exact reason if it’s not indexed.
• 2. Page indexing report (Search Console) — how many of your pages are indexed and, grouped by reason, which ones aren’t. Best for spotting patterns across a site.
• 3. site: search — type site:yoursite.com/page-url into Google. If it shows up, it’s indexed. If it doesn’t, it might still be indexed — Google says the site: operator doesn’t show every indexed URL.
• 4. Third-party index checkers — convenient for bulk lists of URLs you don’t own (like backlinks), but they rely on search queries and can be wrong in both directions.

Reading the URL Inspection result

Common reasons and their guides: Crawled / Discovered – currently not indexed, Excluded by noindex, Duplicate without user-selected canonical, Soft 404 and Page with redirect. “URL is unknown to Google” simply means Google hasn’t found it yet.

Checking Bing too

Bing Webmaster Tools has its own URL Inspection with the same idea: indexed or not, and why. It’s worth checking both — a page can be indexed in one and missing in the other. See our Bing Webmaster Tools guide.

FAQ

You submit a website to Google through Google Search Console — add the property, verify ownership, submit your XML sitemap and request indexing for key pages. It is free, takes about 15 minutes, and there is no paid alternative worth buying.

Submit your website to Google in 4 steps

• 1. Add a property in Google Search Console (search.google.com/search-console).
• 2. Verify ownership. For a domain property, add the TXT record Google gives you in your domain’s DNS settings (at your registrar or host). Verification usually succeeds within minutes once the record is live.
• 3. Submit your sitemap under Indexing → Sitemaps. WordPress with Yoast or Rank Math: /sitemap_index.xml. WordPress without an SEO plugin: /wp-sitemap.xml.
• 4. Request indexing for key pages. Paste the URL into the inspection bar at the top, then click Request indexing. There’s a daily limit — use it for your homepage and most important pages; the sitemap covers the rest.

Domain or URL-prefix property?

What happens after you submit

Submitting doesn’t index anything by itself. It tells Google where your pages are. Google then crawls them on its own schedule and decides which ones are worth keeping. For a new site, days to several weeks is normal — see how long Google takes to index.

• Watch the Page indexing report for statuses like Crawled or Discovered – currently not indexed.
• Check individual pages — see how to check if a page is indexed.
• Add Bing too. Set up Bing Webmaster Tools and IndexNow the same day.

FAQ

Bing Webmaster Tools is Microsoft’s free counterpart to Google Search Console: it shows how Bing crawls, indexes and ranks your site — and, since February 2026, how often Microsoft Copilot cites your pages in AI answers.

What is Bing Webmaster Tools?

Bing Webmaster Tools is a free dashboard from Microsoft for site owners. It tells you which of your pages Bing has found and indexed, which queries you appear for, which sites link to you, and what technical problems Bing ran into while crawling.

Microsoft’s own Webmaster Guidelines now describe how Bing surfaces content “across Bing search experiences, Copilot, and grounding API results” — in other words, the same index feeds classic search, Microsoft’s AI assistant and third parties that ground their answers on Bing.

Why does Bing matter in 2026 if Google has 91% share?

Because the Bing index is used far beyond bing.com. Google still holds about 91% of worldwide search and Bing about 4.5% (StatCounter, August 2026), but AI assistants change the maths.

The Seer study compared ChatGPT’s citations with Google and Bing results for the same questions: Bing matched 87%, Google 56%. It isn’t the whole story — experiments reported by Semrush suggest paid ChatGPT also draws on Google — but the takeaway is simple: if Bing can’t index you, you’re missing from a large share of AI answers. That’s also why we treat Bing as a pillar of answer engine optimization, not an afterthought.

How do you set up Bing Webmaster Tools? (10 minutes)

Microsoft’s Getting Started Checklist lists the same core steps. Here’s how they played out on this site.

Step 1 — Sign in

Go to bing.com/webmasters and sign in with a Microsoft, Google or Facebook account. No separate registration is needed.

Step 2 — Add your site

You get two choices. Import from Google Search Console is fastest: your verified properties and their sitemaps come across without separate verification, but you grant Bing read access to your Search Console data. Add manually means entering your URL and verifying yourself. We chose manual, to keep the accounts separate. If you haven’t set up Search Console yet, start with how to submit your website to Google.

Step 3 — Verify ownership

Three methods: an XML file (BingSiteAuth.xml) in your site root, an msvalidate.01 meta tag on your homepage, or a CNAME record in DNS. On WordPress the meta tag is simplest — Yoast SEO has a field for it. We added the tag, clicked Verify, and the site appeared in the dashboard straight away.

Step 4 — Submit your sitemap

Open Sitemaps → Submit sitemap and enter your sitemap URL — with Yoast or Rank Math usually /sitemap_index.xml. Ours showed “Processing” within seconds. If Bing reports an error, the causes are the same as on Google: see sitemap could not be read.

Step 5 — Connect IndexNow

Install the official IndexNow plugin (or enable IndexNow in your SEO plugin) so every new or updated URL is pushed to Bing instantly. By the time of writing, this site had sent 100 IndexNow submissions with zero failures. Details in what is IndexNow.

How does Bing Webmaster Tools compare to Google Search Console?

They overlap on the basics and differ on the extras. Bing is more generous with free research data; Google has the bigger audience.

Practitioners notice the extras too: one r/bigseo thread titled “Anyone here actively using Bing Webmaster Tools?” includes a reply calling the backlink data “surprisingly good”.

What does the AI Performance report show?

Microsoft introduced AI Performance on 10 February 2026 as a public preview. It covers Microsoft Copilot, AI-generated summaries in Bing and select partner integrations, with five views: total citations, average cited pages, grounding queries, page-level citation activity and trends over time. On 13 October 2026 Microsoft added Topics Portfolio (beta), which groups those grounding queries into broader themes.

Its limits matter. As Search Engine Land noted, the metrics reflect citation frequency only — no clicks, no traffic, and no indication of how prominently you were used. Treat it as a visibility signal, then check real visits in your analytics. To raise your citation count, apply the playbook in how to get cited in AI search, and make sure you aren’t blocking Microsoft’s crawler — see AI crawlers and robots.txt.

How does IndexNow fit in?

Bing Webmaster Tools shows what Bing knows about your site; IndexNow is how you tell Bing about changes the moment they happen. Instead of waiting for Bingbot to recrawl, your site pings Bing with each new, updated or deleted URL, and the IndexNow section of the dashboard lists what was received.

Manual URL submission still exists — Bing raised the daily quota from 10 to up to 10,000 URLs, depending on how long a site has been verified (Search Engine Journal) — but IndexNow does the same job automatically. Google doesn’t participate in IndexNow; for Google you still rely on sitemaps and internal links.

Can you connect Bing Webmaster Tools to ChatGPT, Claude or BigQuery?

Not with an official one-click connector: Microsoft publishes none for ChatGPT, Claude, BigQuery, Google Sheets, Looker Studio or Power BI (checked 17 October 2026). What Microsoft does offer is the Bing Webmaster API, and every working connection runs through it.

You get access in two ways: OAuth 2.0, which Microsoft recommends for apps, or an API key. For the key, open Bing Webmaster Tools, click Settings (top right) → API Access → Generate API Key. Only one key can be generated per user. The key belongs to your user account, not to one site, so treat it like a password. If the key leaks, delete it in Bing Webmaster Tools and generate a new one; every app that still uses the old key stops working until you give it the new key.

The API covers the same data as the main reports. Useful methods in the official reference include GetQueryStats and GetPageStats (traffic for top queries and pages), GetPageQueryStats, GetRankAndTrafficStats, GetCrawlStats, GetCrawlIssues, GetUrlInfo, GetKeywordStats, SubmitUrl and SubmitUrlBatch. The AI Performance report is not in the API reference, so citation data stays in the dashboard for now.

For most sites the simplest useful setup is a weekly script that pulls query and page stats into a sheet or a BigQuery table next to your Search Console data. That gives you a side-by-side view of which pages Google and Bing actually index and which queries each engine sends.

What goes wrong — and how do you fix it?

• “Is it even worth it?” — a common doubt, e.g. this r/growmybusiness thread. Ten minutes of setup buys you Bing, Copilot and part of ChatGPT’s source pool. Yes, it’s worth it.
• Indexed in the dashboard, missing in search. One r/bing user reported their site was deindexed while the portal still showed it as indexed. Use URL Inspection’s live check, and confirm with a site: search on Bing — the same method as in how to check if a page is indexed.
• Bingbot blocked. A firewall or robots.txt rule that lets Googlebot in but not Bingbot silently keeps you out. Check both — see blocked by robots.txt.
• Verification disappears. Bing re-checks verification. Remove the meta tag during a redesign and you lose access.
• Pages found but not indexed. Bing, like Google, skips thin or duplicate pages. The fixes in crawled vs. discovered and orphan pages apply here too.
• Duplicate IndexNow pings. Several plugins submitting the same URLs wastes quota. Use one method.

FAQ

“Page with redirect” means the URL forwards visitors elsewhere, so Google indexes the destination instead — usually exactly what you want. Its broken sibling, “Redirect error”, always needs fixing.

What “Page with redirect” means

Google’s wording: this is a non-canonical URL that redirects to another page, so it will not be indexed. The redirect works; Google followed it and put the destination in the index instead.

Normal vs. worth fixing

• Normal: http → https, non-www → www (or the reverse), old slugs after you renamed a post, trailing-slash normalisation, deleted pages redirected to a true replacement.
• Worth fixing: redirecting URLs listed in your XML sitemap — a sitemap should only list final, indexable URLs.
• Worth fixing: internal links and menus pointing to URLs that redirect. Every hop wastes crawl effort; link to the final URL directly.
• Worth fixing: a page you want indexed that redirects unexpectedly — often a plugin rule, a language redirect or a login redirect.

“Redirect error” — the one you must fix

This separate status means Google couldn’t complete the redirect at all. Google lists the causes: a redirect chain that was too long, a redirect loop, a redirect URL that exceeded the maximum URL length, or a bad or empty URL in the chain. Test the URL with URL Inspection’s live test or any redirect-checker, and make each old URL go directly to its final destination in one 301 hop.

Clean-up checklist

• Sitemap: make sure it lists only final URLs (Yoast and Rank Math handle this automatically for normal posts).
• Internal links: update menus and in-content links to the final URL.
• Use 301 for permanent moves and keep redirects in place for at least a year after a migration.
• Flatten chains: A → B → C becomes A → C and B → C.
• Don’t redirect everything to the homepage — Google tends to treat that as a soft 404.

FAQ