Updated
Crawler roles, Google AI Search eligibility, measurement guidance and the llms.txt status were checked against current first-party documentation on October 1, 2026.
Tested with
OpenAI · Anthropic · Perplexity · Google Search · RFC 9309 · Schema.org · KDD 2024 GEO paper
There is no universal switch that makes an AI search product cite a page. Citation is the outcome of a pipeline: the system must be permitted to access the content, discover or retrieve it for a particular query, select it over competing sources, extract evidence from it and attach that evidence to a generated answer. A site owner can improve eligibility and evidence quality, but no public provider documentation promises inclusion for a compliant page.
This distinction matters because “AI optimization” advice often mixes three different mechanisms: model-training crawls, search indexing and user-triggered retrieval. They use different agents and controls. Treating them as one system can produce the opposite of the intended policy—for example, blocking a search crawler while believing that only model training has been disabled.
Key takeaway
Optimize for verifiable eligibility and useful evidence, not for a mythical AI ranking tag. Crawl access is necessary for some products, but it is never sufficient for citation.
This article uses three levels of evidence:
- Provider-documented behavior. Crawler identities, access controls and product eligibility come from current first-party documentation.
- Peer-reviewed experimental evidence. Claims about generative-engine visibility come from the KDD 2024 GEO study and are reported with its experimental limits.
- Engineering recommendations. Advice about page structure and evaluation is presented as a testable practice, not as a hidden ranking factor.
The term AI search refers here to a product that retrieves current web sources and generates an answer with links or citations. It does not refer to facts stored only in a model's training parameters. Citation means an explicit source attribution or link associated with a generated answer; a brand mention without a source link is a different outcome.
The term generative engine optimization (GEO) was formalized by Aggarwal and colleagues in research published at KDD 2024. GEO is a useful research framework for measuring source visibility in generated answers, but it is not a provider-standard protocol and it does not replace conventional search eligibility.
A simplified generative-search system can be described as five stages:
- The product receives a user question and may reformulate it into related searches.
- A search or retrieval system returns candidate documents.
- A ranking stage selects a smaller evidence set.
- The product extracts passages and generates an answer.
- The interface associates claims with selected sources.
Google publicly describes a related query fan-out process for AI Overviews and AI Mode: the system may issue multiple related searches across subtopics and data sources. Commercial products differ, so the diagram below is a conceptual model rather than a claim about one provider's private architecture.

| Stage | What a publisher can influence | What it does not guarantee |
|---|---|---|
| Access | robots.txt, CDN and WAF rules, authentication, HTTP status | That the crawler will visit the page |
| Discovery | Internal links, sitemaps, stable canonical URLs | That the page will be indexed |
| Retrieval | Clear topical relevance and indexable text | That the page will enter the final evidence set |
| Evidence selection | Specific, supported and current claims | That the system will prefer the page over another source |
| Citation | Traceable evidence and accurate provenance | A fixed citation position, frequency or referral volume |
This is why a successful crawl in server logs is evidence of access, not evidence of ranking or citation.
As of October 1, 2026, the providers covered here document the following roles:
| Provider | Automated search or discovery | User-triggered retrieval | Training-related control |
|---|---|---|---|
| OpenAI | OAI-SearchBot surfaces sites in ChatGPT search results | ChatGPT-User may fetch a page for certain user actions; OpenAI says robots.txt may not apply | GPTBot collects content that may be used for foundation-model training |
| Anthropic | Claude-SearchBot supports search indexing and result quality | Claude-User retrieves content at a user's direction | ClaudeBot collects content that may contribute to model training |
| Perplexity | PerplexityBot surfaces and links sites in Perplexity search | Perplexity-User performs user-requested fetches and generally ignores robots.txt | Perplexity states that these two agents are not used to train foundation models |
| Google Search | Googlebot controls crawl access for Search, including AI Overviews and AI Mode | Product implementation is not exposed as a separate public user agent in the cited guidance | Google-Extended controls some other Google AI training and grounding uses, not Google Search inclusion |

The policy below allows the documented automated search crawlers while opting out of the two documented training crawlers. It is an example, not a universal recommendation; the correct policy depends on the publisher's legal, licensing and product goals.
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Googlebot
Allow: /
# Separate policy choice: opt out of documented training collection
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlTwo operational caveats are important.
- The Robots Exclusion Protocol is a crawler-coordination standard, not an authorization system. Sensitive content still requires authentication and access control.
- A CDN, bot-management product or WAF can reject a legitimate crawler even when
robots.txtallows it. OpenAI and Perplexity publish IP ranges; verify both the user-agent and the provider's current IP data before creating bypass rules.
Do not infer policy from a user-agent string alone
User-agent strings can be spoofed. Use the provider's documented IP verification method, inspect the requested URL and status code, and keep WAF exceptions narrower than a general allow rule.
For Google's AI Overviews and AI Mode, the requirement is explicit: a page must be indexed and eligible to appear in Google Search with a snippet. Google states that there are no additional technical requirements, no special AI schema and no required AI-specific machine-readable file.
That makes the conventional technical baseline the highest-confidence work:
- Return a stable
200response for the canonical page. - Keep important content in crawlable textual form and expose it in the initial HTML where practical.
- Use descriptive titles, headings and internal links.
- Maintain a sitemap and stable canonical URLs.
- Avoid accidental
noindex,nosnippet, restrictivemax-snippetor CDN rules that conflict with the intended visibility policy. - Keep publication dates, authorship and revision dates accurate.
Server rendering is a robust implementation choice because it exposes the article without requiring a crawler to execute application JavaScript. It should be treated as a reliability measure, not as a documented citation boost.
Valid Article structured data can make authorship, dates, publisher identity and images machine-readable. It must agree with the visible page. Google explicitly says that no special Schema.org markup is required for its AI features, so structured data should clarify the document rather than claim facts that are absent from it.
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "How to Get Your Site Cited by AI Search Agents",
"datePublished": "2026-09-29",
"dateModified": "2026-10-01",
"author": {
"@type": "Person",
"name": "Your Name",
"url": "https://example.com/about"
},
"publisher": {
"@type": "Organization",
"name": "Your Publication",
"url": "https://example.com"
},
"image": "https://example.com/images/ai-search-citations.jpg"
}Validate the markup, but evaluate success at the product-output level. Passing a structured-data validator demonstrates syntactic eligibility for supported uses; it does not demonstrate that an AI answer will cite the page.
No provider in the source set publishes a rule saying that “answer-first” paragraphs, question-shaped headings or a specific word count cause citation. They remain reasonable information-design practices because retrieved passages often lose surrounding context. Treat them as hypotheses to test.
For technical and academic material, the strongest practical pattern is:
- State one bounded claim at a time.
- Name the entity, version, date and population to which a claim applies.
- Place the primary source close to the claim it supports.
- Separate measured results from interpretation.
- Publish methods, sample sizes and failure cases for original experiments.
- Use tables for explicit comparisons and headings for stable section boundaries.
- Record meaningful corrections and update dates without silently rewriting history.
The KDD 2024 GEO paper provides relevant—but limited—experimental evidence. Its GEO-bench contains 10,000 queries. In the main setup, the researchers retrieved the top five Google results, generated five responses with gpt-3.5-turbo at temperature 0.7 and measured source visibility. Adding citations, relevant quotations and statistics improved their visibility metrics; the largest reported improvements reached roughly 40%. The paper also tested selected strategies on the then-current Perplexity product.
Those results should not be converted into a universal recipe. The authors explicitly note that generative engines and query distributions change, that their tests covered only two engine configurations, and that they did not evaluate effects on conventional search ranking. The experiment supports testing sourced, specific evidence; it does not prove that adding arbitrary statistics or quotations improves visibility in today's production systems.
Do not manufacture authority
A citation, quotation or statistic helps only when it is relevant and accurate. Invented numbers, decorative references and unsupported expert language reduce reliability and may violate search spam policies.
llms.txt is an open proposal for publishing a concise Markdown map of important resources, often alongside Markdown versions of documentation pages. Its current specification is useful for agents and tools that deliberately look for it, especially in software documentation.
It is not part of the Robots Exclusion Protocol, not a general indexing directive and not required by Google for AI Overviews or AI Mode. None of the cited OpenAI, Anthropic or Perplexity crawler references promises a citation advantage for publishing it. Add llms.txt when it improves agent navigation to canonical material; do not use it as a substitute for HTML, internal links, sitemaps or crawler access.
Generated answers are non-deterministic and product behavior changes over time. A single successful citation is therefore an observation, not a stable rank. Use a versioned evaluation protocol.

- Freeze a query set. Include informational, comparison and problem-solving questions that the page can genuinely answer.
- Record the environment. Store product name, model or mode when visible, account state, region, language, date and whether web search was enabled.
- Repeat each query. Three to five runs per condition is a practical minimum for observing variation, not a statistical guarantee.
- Capture the evidence. Save the answer, cited URL, claim associated with the citation and retrieval timestamp.
- Separate access from output. Store crawler logs, index status, citation observations and referral analytics as different datasets.
- Change one variable. Revise one section, source or technical control, then rerun the same query set after the product has had time to recrawl.
Useful metrics include:
| Metric | Definition | Main limitation |
|---|---|---|
| Citation rate | Runs that cite your domain divided by valid runs | Sensitive to query wording, time and product state |
| Cited-URL distribution | Share of citations assigned to each canonical URL | Does not measure whether the cited passage supports the answer |
| Claim-support rate | Cited claims judged to be supported by the referenced page | Requires a documented human-review rubric |
| Referral sessions | Visits carrying an identifiable assistant referrer | Misses unclicked citations and stripped referrers |
| Verified crawler requests | Requests matching a documented agent and provider IP range | Demonstrates access, not citation or user impact |
Google Search Console now provides dedicated generative-AI performance reports for AI features in Search and Discover, including impressions, pages, countries, devices and dates. For other products, first-party reporting remains uneven, so combine available dashboards with server logs, referral analytics and the repeated-output protocol above.
Eligibility, evidence, evaluation
Separate crawler purposes; keep the intended search paths accessible through both robots.txt and network controls; publish indexable canonical HTML; attach accurate provenance and structured data; write bounded claims supported by primary sources; then measure repeated citation outcomes without treating one answer as a ranking.
This approach is less dramatic than a secret GEO formula, but it is testable. It also survives product changes better: access policies remain explicit, evidence remains useful to readers and each claim about visibility can be traced to an observation rather than a marketing promise.
- OpenAI crawler documentation — current roles of
OAI-SearchBot,GPTBotandChatGPT-User, plus published IP ranges. - Anthropic crawler controls — roles of
Claude-SearchBot,Claude-UserandClaudeBot. - Perplexity crawler documentation — roles of
PerplexityBotandPerplexity-User, WAF guidance and IP sources. - Google: AI features and your website — eligibility, query fan-out, structured-data guidance and visibility controls.
- Google Search Essentials — technical requirements and people-first content guidance.
- Google Article structured data — supported Article properties and implementation guidance.
- Search Generative AI performance reports — first-party Search Console reporting available as of 2026.
- Aggarwal et al., “GEO: Generative Engine Optimization” — KDD 2024 methods, results and limitations.
- RFC 9309: Robots Exclusion Protocol — the standardized semantics and security limits of
robots.txt. - The llms.txt proposal — current proposal, format and intended use.
- Schema.org Article — vocabulary for article metadata.
