The Discovery Gate: How LLM Crawlers Find Your Entity in 2026

The Binary Gate

An AI search engine can only cite content it can find. Discovery is the first and most binary gate in the DSCRI ARGDW framework: either the crawler reaches your page and resolves your entity, or you are not in the consideration set. Everything downstream, content quality, authority, citation worthiness, depends on first passing this gate.

Most YMYL sites optimize for Google’s discovery surface and assume Perplexity, ChatGPT, and Claude work the same way. They do not. The discovery surface for AI search is materially different from classical search: it weighs entity-level identity signals heavier than page-level signals, it depends on infrastructure conventions that did not exist three years ago, and it treats your absence from a small set of canonical sources as evidence that the entity does not exist.

What Discovery Actually Means

Discovery has two requirements. First, an AI crawler must be able to reach the page that contains your entity’s information. Second, the crawler must be able to resolve which entity the page is about.

The first requirement is plumbing: robots.txt that permits AI bots, a current sitemap.xml, addressable URLs, server-side rendering that delivers content to non-JavaScript-executing crawlers. The second requirement is identity: a stable identifier the crawler can carry into its index and use to disambiguate the entity later.

A site can fail Discovery by satisfying one without the other. A page that is reachable but has no entity identifier produces a “the crawler read this and could not tell what it was about” failure. A page with strong identity signals on a URL that no crawler can reach is invisible. Both failure modes are common in B2B finance sites we have audited.

Bingbot Pass vs LLM Context Window Pull

The crawler shape changed between 2024 and 2026. Classical search crawlers like Googlebot and Bingbot run continuous crawl, index, and rank pipelines, where the index is the durable artifact and serving from the index is the query path. LLM-based search systems split this differently.

Some engines, like Perplexity and ChatGPT search, run real-time fetches into a context window per query. Others, like AI Overviews and Claude with web access, blend an offline index with on-demand retrieval. The practical result: the same page can be discoverable to one engine and invisible to another within the same hour, depending on which crawler shape the engine uses for that query type.

Optimizing only for Googlebot’s discovery surface leaves systematic gaps. The five levers below cover both surfaces. The work is incremental rather than a rewrite.

The Five Discovery Levers

1. A Wikidata QID for the entity

Wikidata’s QID has become the de facto entity anchor for AI search systems that need to disambiguate similarly named entities. A firm called Capital Wealth Advisors that exists in Wikidata under Q123456789 is structurally different from one that does not appear in Wikidata at all. The first carries an identifier the engine can attach to claims; the second is one entry in a long list of similarly named firms.

Creating a Wikidata entry is a focused four-hour effort: claim the entity, populate the core properties (instance of, country, headquarters, founded, founders), add three to five cited external references, and submit for community review. Once accepted, reference the QID from your homepage schema via the Organization sameAs property. This single signal solves about half of the entity-resolution failures I see on audit.

2. Schema graph linking with @id

JSON LD’s @graph plus @id pattern produces a linked-entity structure that AI crawlers can traverse without ambiguity. Instead of three disconnected schema blocks on a page (Organization, WebPage, BlogPosting), one @graph array holds all three with explicit @id cross-references. The crawler reads the page once and walks the entity links rather than guessing which Organization the BlogPosting was authored by.

The Astro Foundation emits this pattern by default. Sites built without it should retrofit. The minimal viable graph for a B2B finance home page is four nodes: Organization with QID sameAs, WebSite, WebPage, and a single BreadcrumbList. Anything beyond this is optimization; without these four, Discovery has nothing to anchor to.

3. llms.txt placement and structure

The llms.txt convention emerged in 2024 and has been broadly adopted by mid-2026. At root level (/llms.txt), the file declares which content is canonical for AI extraction, which is supplementary, and which is excluded.

The minimum useful structure: the entity name as a top-level heading, a single-sentence description, an ## Important pages list of three to seven canonical URLs with one-line annotations, and an ## Optional list of supplementary pages. Sites with extensive content can supplement with /llms-full.txt, but the short file is the carry-everything anchor. Adoption is uneven across engines; Anthropic and Perplexity index it consistently as of mid-2026 and OpenAI’s coverage is improving. Cost to add: thirty minutes of writing.

4. AI bot user-agent enumeration in robots.txt

The 2026 AI bot list that matters for a B2B finance site: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Amazonbot, and CCBot. Each line in robots.txt should explicitly Allow or Disallow these user agents.

The mistake most sites make is leaving them implicit. A site that does not enumerate GPTBot does not technically block GPTBot, but it also does not signal that GPTBot is welcome, and crawlers that respect explicit allow patterns will sometimes treat absence as ambiguous and reduce crawl frequency. Five lines of robots.txt closes this gap. The same diagnostic surfaces in the Crawled, Currently Not Indexed pattern for sites where Discovery is partially working.

5. Sitemap.xml hints with current lastmod

Both classical and AI crawlers use sitemap.xml as a discovery accelerator. The signal that matters most for AI is lastmod accuracy. A sitemap that reports lastmod: 2024-09-01 for a page that changes weekly tells AI crawlers the content is static and reduces re-crawl frequency.

A sitemap with accurate lastmod values, including for the entity-anchor pages (home, about, services), keeps the AI index aligned with the live state. Astro emits this correctly by default if @astrojs/sitemap is configured. Verify by querying your live /sitemap-index.xml and confirming the leaf sitemaps show current dates on top-priority pages.

Worked Example

A mid-sized RIA we audited passed every classical SEO discovery check. Sitemap was current. robots.txt allowed Googlebot. The home page had Organization schema. Google indexed everything and ranked competitively on category queries.

Querying Perplexity for the firm’s name, however, returned a profile for a similarly named smaller firm in a different state. Asking Claude for the firm’s services returned a refusal to characterize the entity because it could not confirm which firm was being referenced. The diagnostic took one afternoon. The firm had no Wikidata QID. The Organization schema referenced no sameAs external identifiers. The home page emitted Organization plus WebPage schema without an @graph wrapper. llms.txt did not exist. Two AI bot user agents were missing from robots.txt.

The fix sequence: created the Wikidata entry (four hours including the community review wait), added sameAs references to the QID, LinkedIn URL, and Crunchbase URL on the Organization schema, restructured the schema into an @graph with @id links, added a thirty-line llms.txt at root, expanded robots.txt to enumerate the eleven AI bots. Fourteen days later, Perplexity correctly cited the firm on brand-name queries. Claude returned a substantive answer when asked. Discovery passed.

What Discovery Does Not Solve

Discovery is necessary, not sufficient. Passing Discovery moves your entity into the consideration set; it does not move it to the citation set. The remaining nine gates (Structure, Content, Reputation, Infrastructure, Authority, Recency, Granularity, Differentiation, and Weighted Citability) decide whether the engine selects your page once it can see it.

The argument that schema and discovery infrastructure cannot substitute for accumulated authority is made in detail in Schema Is Necessary, Not Sufficient. Discovery is the floor. Without it, the rest of the DSCRI ARGDW framework is unreachable. With it, the rest of the framework becomes your work to do.

Discovery Diagnostic

Five questions. If any answer is no, fix that lever before continuing to the next gate.

1. Does your entity have a Wikidata QID, and is it referenced via sameAs in your homepage Organization schema?
2. Is your homepage schema emitted as a JSON LD @graph with explicit @id cross-references, or as disconnected schema blocks?
3. Does /llms.txt exist at your site root, with the entity name, description, and three to seven canonical URLs?
4. Does robots.txt explicitly enumerate the eleven AI bot user agents that matter in 2026?
5. Does sitemap.xml report accurate lastmod values, including for the home, about, and services pages?

Frequently Asked Questions

How long does Discovery work take to show in AI search results?

Two weeks is a reasonable expectation for the first AI engine to reflect the changes. Perplexity tends to update fastest because it re-fetches per query. Claude and ChatGPT search update on slower cadences. Google AI Overviews can take four to six weeks. The Wikidata community review wait is the only step that is not under your control; budget two to seven days for that.

Can I pass Discovery without a Wikidata QID?

For entities whose names are unique within their category, yes. For any firm with a name that overlaps with another organization, the QID is the single signal that disambiguates the two. B2B finance firms in particular tend to have overlapping names (Capital, Wealth, Advisors, Partners, Group). The QID resolves the ambiguity in a way that other signals cannot.

Is llms.txt actually used by AI engines, or is it aspirational?

As of mid-2026, llms.txt is consistently indexed by Anthropic’s crawlers and Perplexity’s crawlers. OpenAI’s coverage is improving but inconsistent. Google’s AI Overviews currently treats llms.txt as supplementary rather than authoritative. Adoption is heading toward universal but is not there yet. Cost to add is small enough that the asymmetric upside justifies the work regardless of current coverage.

What happens if I disallow an AI bot in robots.txt?

The bot will skip your site if it respects robots.txt (most major AI crawlers do). This blocks training-data extraction for some engines and live-fetch citation for others. The right move for most YMYL sites is to allow citation-class bots (PerplexityBot, ChatGPT-User, Claude-Web) and selectively control training-class bots (GPTBot, CCBot, Google-Extended). Allow everything if you are unsure; the asymmetry favors visibility.

How do I verify my entity passed Discovery?

Three quick checks. First, query Perplexity directly for the firm name and confirm the result returns your entity, not a similarly named one. Second, ask Claude or ChatGPT to describe your firm’s services; a refusal to characterize suggests entity-resolution failure. Third, search for your firm in Wikidata and confirm the QID is referenced from your Organization schema. If all three pass, Discovery is clear.

Next Gate

Discovery is the cheapest gate to pass: four to six hours of focused work plus a Wikidata review wait. Time to effect: roughly two weeks for the first AI engine to reflect the changes. Benefit: every subsequent gate becomes addressable. The next gate in this series is Structure, which covers JSON LD versus HTML semantics and the practical difference between a parseable page and a citable one.