Fix AI Crawlability: Why AI Ignores Your Site
AI ignores your site when it cannot retrieve readable HTML, is blocked by robots.txt or WAF rules, or cannot be discovered through links and sitemaps. This is a technical access problem, not a content quality problem. The basic audit takes 30 to 60 minutes. Simple WordPress corrections take 2 to 4 hours. Rendering or infrastructure changes can require development work.
By Maria Dykstra · AI Visibility Architect · Last updated: July 2026
AI systems have trouble using your website as current evidence when important pages are blocked, return incomplete HTML, fail at the network layer, or can't be discovered through links and sitemaps. Fixing those issues improves technical accessibility. It does not guarantee indexing, retrieval, recommendation, or citation. Accessible pages get a fair shot. Inaccessible pages never do.
The 5 reasons AI ignores your site
These are the structural failures that keep AI crawlers from reading your content. Most companies have at least two of them active at once.
- Blocked bots. Your robots.txt has a Disallow directive for PerplexityBot, OAI-SearchBot, or Claude-SearchBot. These are search and retrieval crawlers, not training crawlers. Blocking them takes you out of the real-time citation pool. OpenAI says access changes take about a day to propagate, and the effect on individual citations varies.
- JavaScript rendering. Your site uses client-side rendering and your content isn't also server-rendered. The major AI crawlers read your first server response and don't wait for JavaScript to render. If your content is injected by client-side JavaScript, it isn't in the HTML they read.
- WAF interference. Cloudflare's Bot Fight Mode or your host's anti-bot protection is challenging AI crawlers at the network layer, before they ever reach your robots.txt. Your permissions file says Allow. The security gate says no.
- Missing or incomplete sitemap. AI crawlers follow links and sitemaps to find content. Without a comprehensive sitemap, they default to your navigation structure. Most B2B navigation leads to pricing pages and contact forms, not expertise content.
- Slow response times. A slow Time to First Byte raises the chance that AI crawlers abandon the fetch before reading the page. They have shorter patience than human browsers, and there is no guaranteed revisit.
Key definitions
The extent to which a specific automated crawler can request a public URL and receive useful content in the initial server response, without requiring JavaScript execution, redirect chains, or login authentication. High accessibility serves readable HTML to AI crawlers instantly. Low accessibility delivers empty shells, blocked responses, or slow timeouts. Accessibility is one dimension. Discoverability, renderability, retrieval, and citation selection are separate.
An AI bot that fetches or indexes real-time content to ground AI answers. Examples: OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity AI). User-triggered fetchers like ChatGPT-User, Perplexity-User, and Claude-User also surface your pages in live answers, though robots.txt may not govern them the same way as automated crawlers. Blocking a search or retrieval crawler reduces your presence in real-time citations.
An AI bot that collects data that may contribute to model training. Examples: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google, a robots.txt control token rather than a distinct logged crawler). Blocking a training crawler affects your long-term presence in model knowledge. It does not affect real-time citations. Allowing or blocking training crawlers is a strategic decision. Blocking search and retrieval crawlers is almost always a mistake.
This guide addresses the prerequisite layer beneath the Algorithmic Authority Stack: the technical infrastructure that determines whether AI systems can physically reach and read your content. It sits below Layer 1. Nothing in the Stack functions until crawlability is clean. Market Identity Clarity (Layer 1), Semantic Density (Layer 3), and Training-Ready Content (Layer 4) all produce zero benefit if AI crawlers can't access your pages.
Fix crawlability first. Then move to Fix Identity Fragmentation to address how AI classifies you once it can read you.
If any of them appear next to a Disallow directive, you've been blocking AI from reading your site. If none of them appear at all, AI bots crawl you under the general rules, working through the same JavaScript-heavy, navigation-cluttered pages that slow down humans. Either way, you learn something.
Now open your server access logs. Search for the strings "GPTBot," "PerplexityBot," and "OAI-SearchBot." If you can't find them, you either can't access your logs or those crawlers haven't visited in meaningful volume. AI can't cite what it hasn't read. And it can't read what it can't reach.
Most B2B companies assume AI finds them the same way Google does. The fundamentals overlap, but the tolerance differs. Googlebot interprets JavaScript-rendered pages, follows complex navigation, and indexes content buried three clicks deep. The major AI crawlers read the first server response and are less forgiving of the shortcuts Google's rendering budget can absorb. They want clean, server-rendered HTML, clear content hierarchy, and explicit signals about which pages matter. The result on most B2B sites is a page that's technically online but hard to use as evidence.
The companies appearing in AI answers made their content technically accessible to machines that work differently than the ones they'd been optimizing for.
In the Algorithmic Authority Index Wave 1 study (20 B2B companies across 5 industries, Q4 2025), every company tested failed Layer 4 of the Algorithmic Authority Stack: Training-Ready Content. Crawlability failures upstream of Layer 4 were a contributing factor in several. Content that AI can't reach can't be evaluated for citation. It doesn't enter the pool of candidate sources.
PerplexityBot crawl volume growth year-over-year. The highest growth rate of any crawler tracked in 2025.
Cloudflare Bot Traffic Report, 2025The major AI crawlers in Vercel's analysis read the first server response and did not render client-side JavaScript. If your content needs JavaScript to appear, they don't see it.
Vercel AI Crawler Analysis, 2025B2B companies in Wave 1 that failed the content layer. In several, crawlability limited how much owned content AI could evaluate.
Algorithmic Authority Index, Wave 1, Q4 2025In one anonymized audit, a B2B infrastructure company had strong content and consistent identity signals, but 40% of their key pages were JavaScript-rendered. PerplexityBot was hitting the server, reading an empty HTML shell, and leaving. They had zero Perplexity citations. A direct competitor with half their content volume and full server-side rendering was cited repeatedly in the same quarter. The content hadn't changed between audits. The accessible competitor was the one getting cited.
Do you have an AI crawlability problem? Run this test.
You can diagnose the most critical accessibility failures in about 15 minutes. Run this before touching anything.
First: know which bots you're dealing with
Most B2B companies treat AI crawlers as a single category. They're not. The distinction determines your entire robots.txt strategy.
| Bot | Company | Purpose | Impact | Action |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | Search retrieval | Citations in ChatGPT search | Allow |
| Claude-SearchBot | Anthropic | Search indexing | Citations in Claude search | Allow |
| PerplexityBot | Perplexity AI | Index building | Citations in Perplexity | Allow |
| ChatGPT-User | OpenAI | User-triggered fetch | Live ChatGPT answers (robots.txt may not apply) | Allow |
| Perplexity-User | Perplexity AI | User-triggered fetch | Live Perplexity answers | Allow |
| Claude-User | Anthropic | User-triggered fetch | Live Claude answers | Allow |
| GPTBot | OpenAI | Model training | Long-term ChatGPT understanding | Strategic choice |
| ClaudeBot | Anthropic | Model training | Long-term Claude understanding | Strategic choice |
| Google-Extended | Gemini training (robots.txt token) | Gemini model knowledge | Strategic choice |
User-triggered agents (ChatGPT-User, Perplexity-User, Claude-User) are initiated by a person's request, so robots.txt may not govern them the way it governs automated crawlers. Blocking a search or retrieval crawler reduces your citations. Blocking a training crawler affects long-term model presence: slower burn, harder to recover. Two user agents you may still see referenced, Claude-Web and anthropic-ai, are deprecated and no longer used.
How to read your results
| Result | Score | What it means |
|---|---|---|
| Representative bots receive stable 200 responses with useful HTML, sitemap current, fast TTFB | Accessible | AI can reach your site. Your invisibility problem is upstream: identity, semantics, or content structure. |
| Some bots, paths, or content types fail: partial JS rendering, outdated sitemap, or missing bot directives | Partial | AI is reaching some of your content and missing significant portions. Fixable in one focused session. |
| Important pages consistently return blocks, errors, or empty HTML: Disallow on retrieval bots, JS-only rendering, no sitemap | Blocked | AI is functionally blocked from reading your site. Fix this before anything else. No other layer matters until crawlers can reach your content. |
Every page AI can't read is a page that can't build your algorithmic authority. Content that can't be reached can't generate citations regardless of how well it's written.
Why AI crawlability fails. And why your SEO setup can make it worse.
AI crawlers are not better versions of Googlebot. They share the fundamentals: accessible pages, readable HTML, clear structure, current sitemaps. But they're less forgiving of the shortcuts Google's rendering budget can absorb. Each comparison below shows where the tolerance differs. If you are still mapping how those two systems differ, how SEO, AEO, and GEO pull apart lays the three disciplines side by side.
Crawl philosophy: patience vs extraction speed
- Builds a comprehensive long-term index
- Revisits pages on a schedule over months
- Accumulates crawl history and trust signals
- Tolerates slow, complex sites because the investment pays off over time
- Patient. Has been crawling your domain for years.
- Often verifies a specific claim for a query happening now
- Revisits are less predictable than Googlebot's scheduled recrawls
- Less crawl history to draw on
- If content isn't readable in the fetch window, it may not be read before the answer is generated
- Less patient than Googlebot. The window may be the crawl.
The consequence: There may be no second chance in a given retrieval window. If your content isn't readable when the crawler fetches it, you can lose the citation to whoever is.
JavaScript rendering
- Executes JavaScript to render page content
- Has a second-pass rendering queue for JS-heavy pages
- Invests in rendering because indexing the entire web requires it
- Can index React, Vue, and Angular sites with proper configuration
- Reads the first server response
- Relies on the initial HTML, not client-side rendering
- JS adds latency a real-time retrieval system can't always absorb
- Client-rendered pages return an empty HTML shell
- That shell is processed as a page with no content
The consequence: Vercel's analysis found the major AI crawlers rely on the initial HTML response rather than full client-side rendering. Your React site can look perfect in a browser and show AI crawlers nothing but <div id="root"></div>.
Bot access: training vs retrieval
- GPTBot = "the AI bot," so block it to protect your content
- All AI crawlers are the same category
- Blocking one blocks them all
- robots.txt is enough to control AI access
- GPTBot = training crawler. Affects model weights. Not citations.
- OAI-SearchBot and Claude-SearchBot = search crawlers. Affect real-time citations.
- Blocking GPTBot does nothing to your citation visibility
- Blocking PerplexityBot removes you from Perplexity citations
- WAF rules run before robots.txt, so your permissions file may be irrelevant
The consequence: Most B2B companies who added AI bot blocks in 2023 and 2024 blocked training and retrieval crawlers indiscriminately. They protected their training data and eliminated their citation visibility at the same time.
Sitemap and content discovery
- Crawls by following links and sitemap signals
- Has tolerance for missing or incomplete sitemaps
- Builds its own understanding of site structure over time
- Will eventually find deep content through link following
- Follows navigation structure if no sitemap exists
- Navigation prioritizes homepage, pricing, contact, not expertise content
- Blog posts and guides may never get crawled
- A complete sitemap with accurate lastmod is how deep content gets found
The consequence: Without a comprehensive sitemap, AI crawlers default to your navigation. Most B2B navigation leads to conversion pages, not content. Your expertise content becomes invisible by structural default. Sitemap priority values don't help here: Google ignores them.
The silent killer: Cloudflare Bot Fight Mode
- robots.txt says Allow for PerplexityBot
- Content is server-rendered
- Sitemap is current
- Everything should be working
- Zero AI citations anyway
- Cloudflare Bot Fight Mode is active
- PerplexityBot crawls at high volume, looks like a scraper
- Cloudflare issues a challenge before the bot reaches your server
- Your robots.txt is never consulted
- The bot gives up. Cites a competitor instead.
The fix: Plain Bot Fight Mode can't be bypassed with a WAF custom rule; it runs in a separate pipeline. Either turn it off or move to Super Bot Fight Mode, which supports exceptions through WAF skip rules. When you do allowlist, validate by verified-bot signal or published IP ranges, not by user-agent string alone, since user agents are spoofable.
How to fix AI crawlability: the complete sequence
Total time: 2 to 4 hours for a typical WordPress site. Rendering or infrastructure changes can take longer. Ongoing maintenance: 20 minutes per quarter.
When complete: AI search and retrieval crawlers have explicit permission to access your site, your key content pages are server-rendered, your sitemap is current and comprehensive, and you can verify crawler activity in your logs.
User-agent: * policy permits access. Explicit per-bot directives are useful for clarity and governance, but they are not a ranking or crawl-frequency signal.For React, Next.js, or Vue sites: switch from Client-Side Rendering to Static Site Generation (SSG) or Server-Side Rendering (SSR) for all public content pages. Next.js
getStaticProps and getServerSideProps solve this. Application pages behind authentication can remain client-rendered.<priority>. Keep the sitemap to canonical, indexable content pages and keep lastmod accurate. That is what actually helps discovery and re-crawl.llms.txt at your site root. Write it in Markdown with a concise site description, then list your most important pages by section. See the code block below for the recommended structure. Upload via SiteGround's file manager to your site root directory. Verify at yoursite.com/llms.txt.Perplexity citations for a B2B infrastructure company after fixing robots.txt, switching to Next.js SSG, and expanding their sitemap. Roughly 6 weeks to improvement. Content unchanged. Anonymized audit observation, not a controlled test.
Algorithmic Authority Audit case study- Step 1: Audit robots.txt. Document every AI bot directive and categorize as training, search, or user-triggered.
- Step 2: Test Page Source on 5 key pages. Confirm readable HTML before JavaScript runs.
- Step 3: Add explicit Allow directives for all search crawlers. Make an intentional decision on training crawlers.
- Step 4: Move to Super Bot Fight Mode and skip verified AI crawlers, or disable Bot Fight Mode. A WAF rule can't bypass plain Bot Fight Mode.
- Step 5: Fix server-side rendering on product, service, and blog pages. Disable AJAX lazy-load on content blocks.
- Step 6: Submit a comprehensive sitemap of canonical pages with accurate lastmod. Priority values are ignored by Google.
- Step 7: Deploy llms.txt. Do this last. It's infrastructure, not a citation driver.
- Step 8: Monitor server logs quarterly. Check HTTP status codes and Page Source on pages bots hit.
What this looks like: before and after
Anonymized audit observations, not controlled experiments. Retrieval systems can change during the same window.
Example 1: The infrastructure company with zero Perplexity citations
B2B infrastructure monitoring company. Strong content. Active blog with 24 posts over 18 months. robots.txt had no AI-specific directives. JavaScript-rendered product pages using a React-based marketing site. Sitemap existed but only covered 12 of 47 content pages.
PerplexityBot was hitting the server regularly. HTTP 200 on every page. But product and service pages returned empty HTML: a React shell with no server-rendered content. Blog posts were server-rendered and readable. Perplexity could read the blog but couldn't classify the company's actual product. Zero Perplexity citations in the quarter.
Explicit PerplexityBot and OAI-SearchBot allow directives added to robots.txt. Key product and service pages migrated to Next.js SSG. Sitemap expanded to cover all 47 content pages. llms.txt deployed pointing to the top 8 content assets.
About six weeks later: Perplexity citations went from 0 to 7. ChatGPT began surfacing the company in category queries where it had previously been absent. The content hadn't changed between audits. The infrastructure had.
Example 2: The company that blocked itself out of AI answers
A competitor in the same compliance software category. No AI-specific robots.txt directives (default allow). Server-side rendered marketing site on a traditional CMS. Sitemap current and comprehensive. Slower content production (8 posts per quarter) but every page immediately readable by AI crawlers from the first response.
The larger company in the same category. More content. More brand recognition among humans. In 2023, they added a GPTBot Disallow to protect training data, but a formatting error blocked the entire /blog/ path for all AI user agents. JavaScript-rendered service pages. Sitemap not updated in 8 months.
Ask Perplexity AI "What are the best compliance software options for mid-market companies?" and Company A appeared with a cited blog post. Company B did not. Company B had far higher domain authority and content volume. Company A was readable. Company B wasn't. The block had been in place for 18 months before the compliance team found it during an AI visibility audit. By then, the competitor had built citation history that Company B had to recover from zero.
robots.txt: copy-paste configuration
Complete AI-aware robots.txt for a B2B content site. Update the Sitemap URL for your domain before deploying.
# =========================================================
# robots.txt: AI crawler configuration
# Last updated: July 2026
# Review quarterly: AI crawler user-agents change frequently
# =========================================================
# Traditional search engines
User-agent: Googlebot
Allow: /
Disallow: /wp-admin/
Disallow: /wp-login.php
User-agent: Bingbot
Allow: /
Disallow: /wp-admin/
# AI SEARCH + RETRIEVAL: Allow (these drive real-time citations)
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Claude-User
Allow: /
# AI TRAINING: your strategic choice
# Allow = long-term brand understanding in model weights
# Disallow = content stays out of training datasets
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
# Note: ChatGPT-User is user-triggered; OpenAI says robots.txt
# may not reliably govern it, so it is not listed as a lever.
# Claude-Web and anthropic-ai are deprecated and omitted.
# Sitemap location
Sitemap: https://yoursite.com/sitemap_index.xmlWordPress on SiteGround: access robots.txt via SiteGround File Manager at the root directory, or via Rank Math, then General Settings, then Edit robots.txt. If Rank Math is managing robots.txt, add AI bot directives through the Rank Math interface to prevent overwrites on plugin updates.
llms.txt: recommended structure
# [Your Company Name]: [One-line description]
> [2-3 sentence description of what you do, who you serve, and your core IP.]
## Core content
- [Your anchor article title](/url/): Brief description.
- [Framework or methodology page](/url/): Brief description.
- [Key guide or resource](/url/): Brief description.
## Services
- [Primary service](/url/): Description and price if applicable.
- [Secondary service](/url/): Description.
## About
- [About page](/about/): Background, credentials, methodology.Upload to your site root via SiteGround File Manager. Verify at yoursite.com/llms.txt in a browser. The file must be served as plain text (text/plain).
How to know the fix worked
Run the same diagnostic from the top of this guide. The fix registers when your server logs show AI search crawlers receiving HTTP 200 responses on your content pages, and when your Page Source for those pages returns readable HTML with content, not empty div structures.
What "working" looks like: PerplexityBot, OAI-SearchBot, and Claude-SearchBot appear in your logs with 200 status codes on blog posts and guide pages. Your key content pages return full readable HTML in View Source. Your sitemap is indexed in Google Search Console with all content pages listed. Any change in AI citation may take longer than the crawl fix itself and depends on indexing, relevance, authority, and platform behavior.
If the fix hasn't registered after 30 days, check in this order. First, Cloudflare Security Events. If OAI-SearchBot or PerplexityBot traffic shows as "Challenged" or "Blocked," your WAF is the problem regardless of what robots.txt says. Second, cross-reference bot log 200 responses against actual Page Source content on those pages. A 200 from a bot hitting a JavaScript SPA can mean content was never delivered despite the success status. Third, check for JavaScript rendering that wasn't fully resolved on inner pages.
Tracking tools: Server access logs (primary source; GA4 is blind to bot traffic). Cloudflare Radar for bot traffic at the network layer. Google Search Console for crawl coverage and rendering issues. Dark Visitors for current AI crawler user-agent strings.
The re-test: If your robots.txt explicitly allows PerplexityBot, OAI-SearchBot, and Claude-SearchBot, your key content pages return full HTML in View Source, and your sitemap covers your content pages, the infrastructure is in place. If AI citations haven't improved after several weeks, the crawlability layer is likely clean and the problem is upstream. Move to Fix Identity Fragmentation next.
What this reveals about your other AI visibility failures
This guide covers the prerequisite layer beneath the Algorithmic Authority Stack: the technical infrastructure that lets AI crawlers reach and read your content. Fixing crawlability is the prerequisite. Without it, nothing else matters. But it's only the prerequisite.
Crawlability without identity clarity means AI can read your site and still can't classify you. It reaches your pages, extracts your content, and encounters three different descriptions of what you do depending on which page it lands on. The content enters the pool as noise, not signal.
The AI Visibility Snapshot screens all 7 layers across ChatGPT, Perplexity, and Gemini in 48 hours and names the first one failing. It tells you whether crawlability is actually your primary problem or whether identity and semantic issues are compounding on top of a crawlability baseline you've already cleared.
Frequently Asked Questions
Why isn't ChatGPT or Perplexity finding my latest content?
Three causes account for most cases. First, your robots.txt may be blocking search crawlers, or a recent publish triggered a re-crawl that failed due to WAF interference. Second, your content may be JavaScript-rendered. New pages that were never server-rendered have never been read by AI crawlers. Third, your sitemap may not include the new page, or the sitemap hasn't been resubmitted. Run the diagnostic at the top of this guide and check which applies. After you correct access, confirm successful requests in server logs. Search and citation changes may take longer and depend on indexing, relevance, and platform behavior.
Does blocking GPTBot hurt my ChatGPT visibility?
Long-term, possibly. Immediately, minimally. GPTBot is a training crawler. It builds model weights, not real-time answers. OAI-SearchBot is the search crawler that grounds ChatGPT's real-time citations. Allow OAI-SearchBot and ChatGPT search can cite you regardless of your GPTBot policy. The two bots have different effects on different timescales. Block one, not the other, and document the decision.
What is llms.txt and do I need one?
llms.txt is a plain text file at your site root that tells AI systems which of your pages contain the most relevant content, structured in Markdown. As of mid-2026, no major AI platform has confirmed native support in their primary products. An analysis of 94,614 cited URLs found only 1 was an llms.txt file. Implement it as infrastructure and future-proofing, but fix robots.txt, WAF rules, server-side rendering, and sitemap completeness first. Those produce results now. llms.txt is the insurance policy.
What is the difference between llms.txt and robots.txt?
They serve opposite purposes. robots.txt is a crawler-policy file that tells compliant crawlers what they can and cannot access. It's read before content is fetched. Authentication, WAF rules, and server controls are what actually enforce access. llms.txt tells AI systems which content is most relevant and how the site is structured. robots.txt determines whether crawlers can reach you. llms.txt is useful only after crawlers can reach you.
My WordPress site renders fine in the browser. Why would AI crawlers see empty pages?
Because "renders fine in the browser" means JavaScript ran. The major AI crawlers read the first server response and don't wait for JavaScript. If your WordPress theme or page builder uses JavaScript to render content blocks, AI crawlers can see empty containers. Test by right-clicking and selecting "View Page Source" (not Inspect). If your main content isn't in that raw HTML, don't assume AI crawlers can extract it. Kadence blocks with lazy-load or AJAX content loading are the most common culprit on WordPress sites built on the Kadence theme.
Do JavaScript-heavy sites always fail in AI discovery?
Not if the content pages use server-side rendering while JavaScript powers the interactive application layer. Many sites use a hybrid: marketing pages, blog posts, and guides are server-rendered for fast delivery and AI accessibility, while dashboards, user accounts, and interactive tools use client-side JavaScript. That's the correct architecture. The failure occurs when a team applies client-side rendering to all pages uniformly, usually because their framework defaults to CSR and nobody configured SSR for the public content pages.
How do sitemaps and lastmod signals affect AI freshness and re-crawl?
Sitemaps give AI crawlers a map of what exists. The lastmod attribute tells crawlers when each page was last updated. When a crawler visits your sitemap, accurate lastmod dates help it prioritize recently updated content. Google ignores sitemap priority and changefreq but may use accurate lastmod. Keep your sitemap current, set accurate lastmod values via Rank Math's sitemap settings, and resubmit to Google Search Console when you publish significant new content.
How do I expose PDFs and gated content to AI assistants safely?
PDFs are readable by AI crawlers if they're publicly accessible via a direct URL and linked from your sitemap or from pages AI crawlers visit. A PDF behind a login wall, a form gate, or a robots.txt Disallow directive is invisible to AI. For gated content you want AI to partially surface: create a public summary page that describes the content, includes the key findings, and links to the gated version. AI can read the summary, extract the key claims, and cite you. The full document stays protected. A public abstract with your proprietary data is more useful to AI than a gated PDF it can't reach.
Do private or intranet spaces ever leak into AI training or retrieval?
Private content should be protected through authentication and authorization, not robots.txt. A properly configured authentication wall prevents unauthenticated crawling. A robots.txt Disallow does not make a URL private and can even disclose the path publicly. Two indirect risks remain: employees sharing private content on public platforms (LinkedIn posts, conference slides, public repos), and private documentation accidentally published to a public URL without access controls. Audit your public-facing URL space quarterly to confirm no private content has been inadvertently exposed.
How often do I need to update my robots.txt for AI bots?
Quarterly at minimum. The AI crawler landscape changes faster than any other web infrastructure segment. New bots launch, existing bots change user-agent strings, and some get deprecated. Set a 90-day reminder to check Dark Visitors for new user-agents and update your robots.txt accordingly.
How do I manage crawl budget priorities for AI bots?
Crawl budget for AI crawlers is managed through sitemap completeness, accurate lastmod dates, and internal linking, not sitemap priority values, which Google ignores. Keep a comprehensive sitemap of canonical content pages, exclude low-value pages (tag archives, author pages, thank-you pages), and link frequently to your most important pages from within body content. AI crawlers follow links from pages they've already read, so consistent internal linking to your key content means they reach it faster and more reliably than through navigation alone.
My competitor appears in AI answers despite having a slow site. How?
Crawlability is closer to a gate than a ranking factor. If your competitor's site is slow but server-rendered, returns readable HTML, and allows the crawlers, they can appear despite technical debt elsewhere. Once you both clear access, the difference comes from identity clarity, semantic consistency, and content structure. Fix crawlability first, then diagnose which other layers your competitor is winning on.