How AI search finds, ranks and cites information

For 25 years, finding information online meant typing a query and choosing from a list of links. That is changing. Search engines now place AI-generated summaries above the links, and many people ask AI assistants questions directly. Instead of ten blue links, they get a synthesised answer with a few citations, or with none.

For anyone who studies or publishes information, two questions matter. How do these systems decide what to read, what to trust and what to cite? And what does that mean for how information is published?

How classic search works

Classic web search has three stages:

1. Crawling. Programs called crawlers follow links and fetch pages.
2. Indexing. Pages are parsed and stored in an index of words, entities and other signals, much like the index at the back of a book.
3. Ranking. For each query, the engine retrieves candidate pages and orders them using hundreds of signals: relevance to the query, quality and reliability, freshness, usability and the links pointing to a page.

Links were Google’s founding insight. Sergey Brin and Larry Page’s 1998 paper described PageRank, which treats a link as a vote. Links from important pages count more. Links still matter, but modern ranking relies heavily on machine-learned models of meaning and quality.

How AI answers work: retrieval-augmented generation

A language model on its own answers from patterns learned during training. Its knowledge stops at a cutoff date, and it can produce fluent but false statements. Retrieval-augmented generation (RAG), described by Patrick Lewis and colleagues in 2020, fixes part of this problem. The system first retrieves relevant documents and then generates an answer based on them.

AI search products generally work like this:

1. Understand the question, often by rewriting it into several search queries (“query fan-out”).
2. Retrieve candidate passages from a web index or a specialised collection, using keyword matching, embeddings (numerical representations of meaning) or both.
3. Rerank the passages by relevance and reliability.
4. Generate an answer from the top passages, and cite some of them.

In this pipeline, the unit of competition is often the passage, not the page. A clear paragraph that directly answers a sub-question can be retrieved and cited even if the page as a whole ranks modestly.

What gets cited

Research and practitioner studies are still young, but some consistent patterns are emerging:

• Pages must be retrievable first. AI search systems lean heavily on conventional search indexes. Content that is not crawled and indexed, or that ranks poorly, is rarely cited.
• Clarity beats cleverness. Passages that state facts directly, define terms and answer specific questions are easier to extract and quote.
• Evidence helps. In a 2023 study that introduced the term generative engine optimisation (GEO), Pranjal Aggarwal and colleagues found that adding citations, quotations and statistics to content increased its visibility in generated answers. Keyword stuffing did not help.
• Authority and consistency matter. Sources that are widely referenced, and whose claims match those of other reliable sources, are more likely to be chosen.
• Structure helps machines. Descriptive headings, tables, lists and structured data (schema.org) make content easier to parse.

What this means for publishers

Google’s guidance for its AI features is short: there are no special requirements beyond being eligible for normal search. Publish helpful, reliable, people-first content, keep pages crawlable and use structured data that matches the visible content. Practical consequences:

• Answer real questions completely. Cover the sub-questions a reader would ask next.
• Put the answer first. Lead each section with a direct statement, then give the detail.
• Show your evidence. Cite primary sources, give dates and numbers, and keep pages updated.
• Be explicit about entities. Name people, organisations and concepts precisely and consistently, and describe them with structured data.
• Keep technical basics solid. Fast, crawlable pages, accurate sitemaps and no accidental blocking of crawlers. Proposals such as llms.txt, a plain-text guide to a site’s most useful content for AI systems, may help, although adoption is still uneven.
• Measure differently. Track visibility and citations in AI answers as well as rankings and clicks, because a citation may not produce a click.

The bigger picture

AI search changes the economics of the web. When an answer is synthesised, fewer users click through to sources. That threatens the business models of the publishers whose work the answers depend on. It also concentrates influence: a handful of systems decide which sources are presented as the answer.

That is why the long-standing concerns of information science matter more than ever: relevance, reliability, provenance, bias and the evaluation of sources. Users need to know where an answer came from, and publishers need to be credited for their work. Societies need these systems to reflect the best available evidence, not simply the most repeated claims.