Skip to content

Search articles, repositories, topics and pages.

AI Search

How to Get Your Site Cited by AI Search Agents

An evidence-based guide to how crawler access, indexability, source quality and repeatable measurement affect whether AI search systems can cite a webpage.

TIAN NEWS AI

Published Updated 9 min read

A website entering crawl and search infrastructure, being retrieved among multiple sources, and supporting an AI-generated answer with traceable citations

Updated

Crawler roles, Google AI Search eligibility, measurement guidance and the llms.txt status were checked against current first-party documentation on October 1, 2026.

Tested with

OpenAI · Anthropic · Perplexity · Google Search · RFC 9309 · Schema.org · KDD 2024 GEO paper

There is no universal switch that makes an AI search product cite a page. Citation is the outcome of a pipeline: the system must be permitted to access the content, discover or retrieve it for a particular query, select it over competing sources, extract evidence from it and attach that evidence to a generated answer. A site owner can improve eligibility and evidence quality, but no public provider documentation promises inclusion for a compliant page.

This distinction matters because “AI optimization” advice often mixes three different mechanisms: model-training crawls, search indexing and user-triggered retrieval. They use different agents and controls. Treating them as one system can produce the opposite of the intended policy—for example, blocking a search crawler while believing that only model training has been disabled.

Key takeaway

Optimize for verifiable eligibility and useful evidence, not for a mythical AI ranking tag. Crawl access is necessary for some products, but it is never sufficient for citation.

Scope and standard of evidence

This article uses three levels of evidence:

  1. Provider-documented behavior. Crawler identities, access controls and product eligibility come from current first-party documentation.
  2. Peer-reviewed experimental evidence. Claims about generative-engine visibility come from the KDD 2024 GEO study and are reported with its experimental limits.
  3. Engineering recommendations. Advice about page structure and evaluation is presented as a testable practice, not as a hidden ranking factor.

The term AI search refers here to a product that retrieves current web sources and generates an answer with links or citations. It does not refer to facts stored only in a model's training parameters. Citation means an explicit source attribution or link associated with a generated answer; a brand mention without a source link is a different outcome.

The term generative engine optimization (GEO) was formalized by Aggarwal and colleagues in research published at KDD 2024. GEO is a useful research framework for measuring source visibility in generated answers, but it is not a provider-standard protocol and it does not replace conventional search eligibility.

Citation is a retrieval outcome

A simplified generative-search system can be described as five stages:

  1. The product receives a user question and may reformulate it into related searches.
  2. A search or retrieval system returns candidate documents.
  3. A ranking stage selects a smaller evidence set.
  4. The product extracts passages and generates an answer.
  5. The interface associates claims with selected sources.

Google publicly describes a related query fan-out process for AI Overviews and AI Mode: the system may issue multiple related searches across subtopics and data sources. Commercial products differ, so the diagram below is a conceptual model rather than a claim about one provider's private architecture.

A query branches into related searches, retrieves many candidate documents, narrows them to selected evidence and connects answer claims back to cited sources
A simplified generative-search pipeline. Discovery creates candidates; selection and evidence use determine which sources are cited. Image generated by AI.
StageWhat a publisher can influenceWhat it does not guarantee
Accessrobots.txt, CDN and WAF rules, authentication, HTTP statusThat the crawler will visit the page
DiscoveryInternal links, sitemaps, stable canonical URLsThat the page will be indexed
RetrievalClear topical relevance and indexable textThat the page will enter the final evidence set
Evidence selectionSpecific, supported and current claimsThat the system will prefer the page over another source
CitationTraceable evidence and accurate provenanceA fixed citation position, frequency or referral volume

This is why a successful crawl in server logs is evidence of access, not evidence of ranking or citation.

Separate crawler purposes before editing robots.txt

As of October 1, 2026, the providers covered here document the following roles:

ProviderAutomated search or discoveryUser-triggered retrievalTraining-related control
OpenAIOAI-SearchBot surfaces sites in ChatGPT search resultsChatGPT-User may fetch a page for certain user actions; OpenAI says robots.txt may not applyGPTBot collects content that may be used for foundation-model training
AnthropicClaude-SearchBot supports search indexing and result qualityClaude-User retrieves content at a user's directionClaudeBot collects content that may contribute to model training
PerplexityPerplexityBot surfaces and links sites in Perplexity searchPerplexity-User performs user-requested fetches and generally ignores robots.txtPerplexity states that these two agents are not used to train foundation models
Google SearchGooglebot controls crawl access for Search, including AI Overviews and AI ModeProduct implementation is not exposed as a separate public user agent in the cited guidanceGoogle-Extended controls some other Google AI training and grounding uses, not Google Search inclusion
A website exposes three independent paths for training collection, search indexing and a user-triggered fetch, each passing through separate policy and network controls
Training collection, search indexing and user-triggered retrieval are separate access paths. Provider behavior and applicable controls differ. Image generated by AI.

The policy below allows the documented automated search crawlers while opting out of the two documented training crawlers. It is an example, not a universal recommendation; the correct policy depends on the publisher's legal, licensing and product goals.

robots.txt, txt
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

# Separate policy choice: opt out of documented training collection
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Two operational caveats are important.

  • The Robots Exclusion Protocol is a crawler-coordination standard, not an authorization system. Sensitive content still requires authentication and access control.
  • A CDN, bot-management product or WAF can reject a legitimate crawler even when robots.txt allows it. OpenAI and Perplexity publish IP ranges; verify both the user-agent and the provider's current IP data before creating bypass rules.

Do not infer policy from a user-agent string alone

User-agent strings can be spoofed. Use the provider's documented IP verification method, inspect the requested URL and status code, and keep WAF exceptions narrower than a general allow rule.

Establish search eligibility first

For Google's AI Overviews and AI Mode, the requirement is explicit: a page must be indexed and eligible to appear in Google Search with a snippet. Google states that there are no additional technical requirements, no special AI schema and no required AI-specific machine-readable file.

That makes the conventional technical baseline the highest-confidence work:

  • Return a stable 200 response for the canonical page.
  • Keep important content in crawlable textual form and expose it in the initial HTML where practical.
  • Use descriptive titles, headings and internal links.
  • Maintain a sitemap and stable canonical URLs.
  • Avoid accidental noindex, nosnippet, restrictive max-snippet or CDN rules that conflict with the intended visibility policy.
  • Keep publication dates, authorship and revision dates accurate.

Server rendering is a robust implementation choice because it exposes the article without requiring a crawler to execute application JavaScript. It should be treated as a reliability measure, not as a documented citation boost.

Use structured data as provenance, not as a GEO switch

Valid Article structured data can make authorship, dates, publisher identity and images machine-readable. It must agree with the visible page. Google explicitly says that no special Schema.org markup is required for its AI features, so structured data should clarify the document rather than claim facts that are absent from it.

article-schema.json, json
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "How to Get Your Site Cited by AI Search Agents",
  "datePublished": "2026-09-29",
  "dateModified": "2026-10-01",
  "author": {
    "@type": "Person",
    "name": "Your Name",
    "url": "https://example.com/about"
  },
  "publisher": {
    "@type": "Organization",
    "name": "Your Publication",
    "url": "https://example.com"
  },
  "image": "https://example.com/images/ai-search-citations.jpg"
}

Validate the markup, but evaluate success at the product-output level. Passing a structured-data validator demonstrates syntactic eligibility for supported uses; it does not demonstrate that an AI answer will cite the page.

Write evidence that can survive retrieval

No provider in the source set publishes a rule saying that “answer-first” paragraphs, question-shaped headings or a specific word count cause citation. They remain reasonable information-design practices because retrieved passages often lose surrounding context. Treat them as hypotheses to test.

For technical and academic material, the strongest practical pattern is:

  • State one bounded claim at a time.
  • Name the entity, version, date and population to which a claim applies.
  • Place the primary source close to the claim it supports.
  • Separate measured results from interpretation.
  • Publish methods, sample sizes and failure cases for original experiments.
  • Use tables for explicit comparisons and headings for stable section boundaries.
  • Record meaningful corrections and update dates without silently rewriting history.

The KDD 2024 GEO paper provides relevant—but limited—experimental evidence. Its GEO-bench contains 10,000 queries. In the main setup, the researchers retrieved the top five Google results, generated five responses with gpt-3.5-turbo at temperature 0.7 and measured source visibility. Adding citations, relevant quotations and statistics improved their visibility metrics; the largest reported improvements reached roughly 40%. The paper also tested selected strategies on the then-current Perplexity product.

Those results should not be converted into a universal recipe. The authors explicitly note that generative engines and query distributions change, that their tests covered only two engine configurations, and that they did not evaluate effects on conventional search ranking. The experiment supports testing sourced, specific evidence; it does not prove that adding arbitrary statistics or quotations improves visibility in today's production systems.

Do not manufacture authority

A citation, quotation or statistic helps only when it is relevant and accurate. Invented numbers, decorative references and unsupported expert language reduce reliability and may violate search spam policies.

Treat llms.txt as an optional navigation layer

llms.txt is an open proposal for publishing a concise Markdown map of important resources, often alongside Markdown versions of documentation pages. Its current specification is useful for agents and tools that deliberately look for it, especially in software documentation.

It is not part of the Robots Exclusion Protocol, not a general indexing directive and not required by Google for AI Overviews or AI Mode. None of the cited OpenAI, Anthropic or Perplexity crawler references promises a citation advantage for publishing it. Add llms.txt when it improves agent navigation to canonical material; do not use it as a substitute for HTML, internal links, sitemaps or crawler access.

Measure citations as a repeated observation

Generated answers are non-deterministic and product behavior changes over time. A single successful citation is therefore an observation, not a stable rank. Use a versioned evaluation protocol.

A fixed query set is run repeatedly across several assistants, recorded in a citation ledger, compared, followed by one controlled page revision and another evaluation round
A repeatable citation evaluation loop: freeze the questions, sample repeatedly, record evidence, change one variable and run the same protocol again. Image generated by AI.
  1. Freeze a query set. Include informational, comparison and problem-solving questions that the page can genuinely answer.
  2. Record the environment. Store product name, model or mode when visible, account state, region, language, date and whether web search was enabled.
  3. Repeat each query. Three to five runs per condition is a practical minimum for observing variation, not a statistical guarantee.
  4. Capture the evidence. Save the answer, cited URL, claim associated with the citation and retrieval timestamp.
  5. Separate access from output. Store crawler logs, index status, citation observations and referral analytics as different datasets.
  6. Change one variable. Revise one section, source or technical control, then rerun the same query set after the product has had time to recrawl.

Useful metrics include:

MetricDefinitionMain limitation
Citation rateRuns that cite your domain divided by valid runsSensitive to query wording, time and product state
Cited-URL distributionShare of citations assigned to each canonical URLDoes not measure whether the cited passage supports the answer
Claim-support rateCited claims judged to be supported by the referenced pageRequires a documented human-review rubric
Referral sessionsVisits carrying an identifiable assistant referrerMisses unclicked citations and stripped referrers
Verified crawler requestsRequests matching a documented agent and provider IP rangeDemonstrates access, not citation or user impact

Google Search Console now provides dedicated generative-AI performance reports for AI features in Search and Discover, including impressions, pages, countries, devices and dates. For other products, first-party reporting remains uneven, so combine available dashboards with server logs, referral analytics and the repeated-output protocol above.

A defensible implementation checklist

Eligibility, evidence, evaluation

Separate crawler purposes; keep the intended search paths accessible through both robots.txt and network controls; publish indexable canonical HTML; attach accurate provenance and structured data; write bounded claims supported by primary sources; then measure repeated citation outcomes without treating one answer as a ranking.

This approach is less dramatic than a secret GEO formula, but it is testable. It also survives product changes better: access policies remain explicit, evidence remains useful to readers and each claim about visibility can be traced to an observation rather than a marketing promise.

References

  1. OpenAI crawler documentation — current roles of OAI-SearchBot, GPTBot and ChatGPT-User, plus published IP ranges.
  2. Anthropic crawler controls — roles of Claude-SearchBot, Claude-User and ClaudeBot.
  3. Perplexity crawler documentation — roles of PerplexityBot and Perplexity-User, WAF guidance and IP sources.
  4. Google: AI features and your website — eligibility, query fan-out, structured-data guidance and visibility controls.
  5. Google Search Essentials — technical requirements and people-first content guidance.
  6. Google Article structured data — supported Article properties and implementation guidance.
  7. Search Generative AI performance reports — first-party Search Console reporting available as of 2026.
  8. Aggarwal et al., “GEO: Generative Engine Optimization” — KDD 2024 methods, results and limitations.
  9. RFC 9309: Robots Exclusion Protocol — the standardized semantics and security limits of robots.txt.
  10. The llms.txt proposal — current proposal, format and intended use.
  11. Schema.org Article — vocabulary for article metadata.

Updated

Topics

Related reading

  • LLM NPC Dialogue: State, Latency, Safety and Cost

    An architecture for adding generated dialogue to a game without giving up deterministic state, responsive interaction or narrative control.

    4 min read

  • Understanding Context Engineering in AI Agents

    Why the context you assemble matters more than the prompt you write, and practical techniques to keep it relevant and within budget.

    3 min read

  • Building a Reliable MCP Server

    A production-minded guide to building an MCP server that stays predictable under bad inputs, upstream failures, cancellation and real clients.

    8 min read

  • Survival Rules for Working with Claude Code

    An evidence-graded field guide to using Claude Code safely and reproducibly: constrain the task, protect the context window, verify every change and automate only inside explicit boundaries.

    10 min read