How Google Search Works

Area: How big systems work
Related: Inverted Index, BM25, PageRank, PostgreSQL, Regular Expressions, IndexedDB

What is it?

Google Search is a pipeline with three stages that run on very different clocks:

1. Crawling. Programs (Googlebot) fetch pages from the web, all the time, and follow links to find more.
2. Indexing. Each fetched page is rendered, parsed, deduplicated and analysed, then added to a giant inverted index: a map from every word to the list of pages that contain it.
3. Serving. When you type a query, Google understands what you mean, pulls candidate pages from the index in a few milliseconds, ranks them with hundreds of signals and several layers of machine learning, and builds the results page.

Crawling and indexing happen before you ever search. At query time nothing touches the live web. Google only reads its own copy.

Search is a lookup in a pre-built book index, followed by a very expensive argument about which pages on that index entry deserve to be first.

The “algorithm” isn’t one algorithm. PageRank is the famous part, but today it’s one signal among many. Most of the work is in retrieval (finding a few thousand plausible pages out of hundreds of billions, fast) and ranking (ordering them), and ranking is now dominated by learned models.

A caveat for the whole note: Google publishes how the system works in broad strokes, and the original 1998 paper describes the early design in detail. The modern ranking internals are secret. What’s known comes from Google’s own docs and blog posts, research papers, and testimony from the 2023 US antitrust trial. I’ve marked which is which.

Why it exists

Before Google, search engines (AltaVista, Lycos, Excite) ranked mostly by what’s on the page: how often your words appear, whether they’re in the title. That’s easy to game. Repeat “cheap flights” 500 times in white text on a white background and you rank.

Larry Page and Sergey Brin’s idea at Stanford (paper published 1998) was to also use what other pages say about this page:

• Links as votes. A page linked from many pages is probably useful. A link from a page that is itself important counts more. That’s PageRank.
• Anchor text. The words in a link (<a>best sourdough guide</a>) describe the target page, often better than the page describes itself, and they’re harder for the target to fake.

Both signals come from people other than the page’s author, so they’re harder to manipulate. That gap in quality is what made Google win.

Stage 1: Crawling

A crawler is a loop: take a URL from a queue, fetch it, extract links, add new URLs to the queue.

const queue = ['https://example.com/'];
const seen = new Set(queue);
while (queue.length) {
  const url = queue.shift()!;
  if (!allowedByRobotsTxt(url)) continue;
  const html = await fetch(url).then(r => r.text());
  store(url, html);
  for (const link of extractLinks(html, url)) {
    if (!seen.has(link)) { seen.add(link); queue.push(link); }
  }
}

At web scale every line of that becomes a hard problem:

• Discovery. New URLs come from links on known pages, from sitemaps site owners submit, and from Search Console. A page nobody links to and no sitemap lists may never be found.
• Politeness. Hitting one server with thousands of requests a second would take it down. Crawlers keep per-host queues and rate limits, and back off when a site slows down or returns 5xx errors. Google calls the result a site’s crawl budget: how much it’s willing and able to fetch from you.
• Prioritisation. The queue isn’t FIFO. Pages that change often (news homepages) and important pages get re-crawled more. A static page from 2012 might be revisited rarely.
• robots.txt tells crawlers which paths not to fetch. It’s a request, not access control, and blocking a URL in robots.txt doesn’t remove it from the index if other pages link to it (it can show up with no snippet). To keep a page out of results, let it be crawled and serve noindex.
• Duplicates and traps. The same page lives at http/https, with and without www, with tracking parameters, with session ids. Calendars generate infinite “next month” links. The crawler normalises URLs and detects traps, or it would spend its budget on junk.
• Rendering. Many pages build their content with JavaScript. Google renders pages in a headless, up-to-date Chromium (its Web Rendering Service), but rendering is expensive, so it can happen later than the initial fetch. Content that only appears after a click or a scroll may never be seen.

Stage 2: Indexing

For each fetched page:

1. Render and parse. Get the final DOM, the visible text, title, headings, links, structured data (schema.org markup), images, language.
2. Canonicalise. Group duplicate and near-duplicate pages into a cluster and pick one canonical URL to show. Signals include rel="canonical", redirects, sitemaps, internal links and HTTPS. Only the canonical normally appears in results.
3. Analyse. Detect language, entities (people, places, things in the Knowledge Graph), spam signals, quality signals, page experience signals.
4. Add to the index.

Google says its index covers hundreds of billions of pages and is well over 100 million gigabytes. Not everything crawled is indexed. Low-quality or duplicate pages are often dropped.

The inverted index

A database index answers “what’s in document 42?”. Search needs the reverse: “which documents contain sourdough?”. So you invert it: for each term, a posting list of the documents that contain it.

I built a toy one over five pages. Real output:

bread     → [{doc:a, tf:1}, {doc:c, tf:7}, {doc:d, tf:2}]
sourdough → [{doc:a, tf:1}, {doc:b, tf:1}, {doc:d, tf:1}]

tf is term frequency, how often the term appears in that doc. Real systems also store positions (for phrase queries like "sourdough bread" and for proximity: are the words near each other?) and which field the term was in (title, body, anchor text, URL).

const tokenize = (s: string) => s.toLowerCase().match(/\p{L}+|\p{N}+/gu) ?? [];
const STOP = new Set(['how', 'to', 'at', 'your', 'and', 'for', 'the']);
const stem = (w: string) => w.replace(/(ing|s)$/, '');   // toy stemmer
const terms = (s: string) => tokenize(s).filter(w => !STOP.has(w)).map(stem);
 
terms('How to bake sourdough bread at home')   // ['bake', 'sourdough', 'bread', 'home']
 
const index = new Map<string, { doc: string; tf: number }[]>();
for (const [id, text] of Object.entries(pages)) {
  const tf = new Map<string, number>();
  for (const t of terms(text)) tf.set(t, (tf.get(t) ?? 0) + 1);
  for (const [t, n] of tf) {
    if (!index.has(t)) index.set(t, []);
    index.get(t)!.push({ doc: id, tf: n });
  }
}

Answering sourdough AND bread means intersecting two posting lists. Kept sorted by doc id, that’s a merge: two pointers walking forward together (see Two Pointers), with skip pointers to jump ahead in long lists.

Compression

Posting lists for common words are enormous, so they’re compressed. Doc ids are sorted, so you store the gaps instead of the ids, and gaps are small numbers. Then you encode small numbers in fewer bytes (variable-length integers: 7 bits of data per byte, high bit means “more bytes follow”).

doc ids  [3, 7, 21, 22, 150, 151, 152, 9000]
gaps     [3, 4, 14,  1, 128,   1,   1, 8848]
varint:  10 bytes   vs   32 bytes as 4-byte ints

Smaller lists mean more of the index fits in RAM, and RAM is what makes serving fast. Jeff Dean’s 2009 talk (in Sources) walks through how Google’s encodings evolved for exactly this reason.

Building the index at scale

The original Google rebuilt the index in big batches. MapReduce (published by Google in 2004) came out of that kind of work: map over every page to emit (term, doc) pairs, shuffle by term, reduce into posting lists.

Batch rebuilds meant new pages took days or weeks to appear. In 2010 Google launched Caffeine, which indexes continuously. The Percolator paper (2010) describes the incremental processing system behind it: small transactional updates as each page arrives, instead of reprocessing the whole web.

Stage 3: Serving a query

Google has said a large share of daily queries (around 15%) are ones it has never seen before, so it can’t just cache answers. A query goes through roughly this:

Query understanding

• Spelling. “sourdoh bred” → “sourdough bread”. Learned from query logs: what did people type next after a search with no clicks?
• Synonyms and stemming. bake, baking, baked. “How to change a tyre” also matches “replace”.
• Intent. Is jaguar the car or the animal? Is python list sort a coding question? Does weather need the user’s location? Does election results need fresh pages (Google calls this “query deserves freshness”)?
• Entities. obama age is recognised as a Knowledge Graph entity plus an attribute, and can be answered directly.
• Neural understanding. Since BERT (2019), language models read the whole query including small words. Google’s example: in “2019 brazil traveler to usa need a visa”, the word “to” matters. Older systems treated it as noise.

Retrieval: from billions to thousands

The index is sharded by document: each machine holds the full index for a slice of pages. The query is sent to every shard in parallel. Each shard walks its posting lists, scores documents with cheap signals, and returns its local top few. A root server merges them.

Tricks to make this fast:

• Tiered index. Higher-quality pages in a smaller, faster tier that’s searched first.
• Early termination. Posting lists ordered so likely-good docs come first, and algorithms like WAND that skip documents which can’t make the top k.
• Caching of popular queries and posting lists.
• Replication. Many copies of each shard serve different queries, and the slowest replica can be dropped (“tail tolerance”).

Since around 2018 Google has also used neural retrieval (“neural matching”), which finds pages whose meaning is close to the query even when they share few words, using vector embeddings of queries and pages.

Ranking: from thousands to ten

Ranking works in stages. Cheap scoring on many candidates, then more expensive models on fewer.

Classic relevance (the textbook part). How well does the text match? The standard formula is BM25: reward documents that contain the query terms, with three adjustments:

• IDF (inverse document frequency). Rare terms matter more. starter (in 1 of 5 docs) scores higher than bread (in 3 of 5).
• Saturation. The 7th occurrence of a word adds much less than the 1st. Stuffing doesn’t keep paying.
• Length normalisation. A long page mentions everything a few times. Divide that out.

Real output from my toy engine for the query sourdough bread, where page c is keyword-stuffed (“Bread bread bread bread bread bread best bread recipes”):

raw term counts: c=7, d=3, a=2, b=1     ← stuffed page wins
BM25:            a=1.261, d=1.227, c=0.964, b=0.512

const idf = (t: string) => Math.log(1 + (N - df(t) + 0.5) / (df(t) + 0.5));
function bm25(query: string, k1 = 1.2, b = 0.75) {
  const scores: Record<string, number> = {};
  for (const t of terms(query)) {
    for (const { doc, tf } of index.get(t) ?? []) {
      const norm = (tf * (k1 + 1)) / (tf + k1 * (1 - b + b * length[doc] / avgLength));
      scores[doc] = (scores[doc] ?? 0) + idf(t) * norm;
    }
  }
  return Object.entries(scores).sort((x, y) => y[1] - x[1]);
}

PageRank (the famous part). Imagine someone clicking random links forever. With probability 0.85 they follow a random link from the current page. Otherwise they jump to a random page anywhere. A page’s PageRank is the fraction of time they spend on it.

PR(p) = (1 - d) / N  +  d × Σ PR(q) / outlinks(q)    for every page q linking to p

You compute it by power iteration: start everyone equal, apply the formula repeatedly until it stops changing.

function pagerank(graph: Record<string, { links: string[] }>, d = 0.85, iters = 50) {
  const ids = Object.keys(graph), n = ids.length;
  let pr = Object.fromEntries(ids.map(i => [i, 1 / n]));
  for (let it = 0; it < iters; it++) {
    const next = Object.fromEntries(ids.map(i => [i, (1 - d) / n]));
    let dangling = 0;                                  // pages with no outlinks
    for (const i of ids) {
      const out = graph[i].links;
      if (out.length === 0) dangling += pr[i];
      else for (const j of out) next[j] += (d * pr[i]) / out.length;
    }
    for (const i of ids) next[i] += (d * dangling) / n; // spread their rank evenly
    pr = next;
  }
  return pr;
}

On my five pages (a links to b, c; b to a; c to nothing; d to a, b; e to c), real output:

a 0.320   c 0.279   b 0.246   d 0.077   e 0.077     (sums to 1.000)

d and e have no incoming links, so they only get the random-jump share. It converges fast: a was 0.3190 after 1 iteration and 0.3197 from 20 on.

Dangling pages (no outlinks, like PDFs) need special handling or rank “leaks” out of the system. The (1 - d) jump guarantees one stable answer and stops rank getting trapped in loops.

PageRank is gameable too. I added 20 empty spam pages that all link to e (the “cheap watches” page). Its score went from 0.077 to 0.280, the highest in the graph (a fell to 0.064, partly because there are now 25 pages sharing the total). That’s a link farm, and it’s why Google later added link-spam systems (Penguin in 2012, SpamBrain later) and rel="nofollow"/"sponsored"/"ugc" for links that shouldn’t pass credit. Google stopped showing public PageRank scores in 2016 but has said PageRank, in evolved forms, is still one of its signals.

Everything else (the part Google doesn’t publish). Google says it uses many signals, grouped roughly as:

• Meaning and relevance: does the page answer this query, not just contain its words?
• Quality: expertise, authoritativeness and trustworthiness of the page and site. The public Search Quality Rater Guidelines describe this as E-E-A-T (experience, expertise, authoritativeness, trustworthiness). Human raters don’t rank pages directly. Their ratings are used to evaluate changes to the algorithms.
• Usability / page experience: mobile-friendly, HTTPS, Core Web Vitals, no intrusive interstitials. Mostly a tie-breaker.
• Context: your location, language, and sometimes recent searches.
• Freshness for queries where it matters.

And the learned models stacked on top. Publicly announced milestones:

Year	System	What it did
2011	Panda	Demoted thin, low-quality content (content farms)
2012	Penguin	Link spam
2012	Knowledge Graph	Entities and facts, the boxes on the right
2013	Hummingbird	Rewrite of the core around meaning of whole queries
2015	RankBrain	First big ML ranking system, for unseen queries
2018	Neural matching	Embedding-based retrieval
2019	BERT	Language model for queries and passages. Google said it affected about 1 in 10 English queries in the US at launch
2021	MUM	Larger multilingual, multimodal model, used for specific features
2022	Helpful content system	Site-wide signal against content made for search engines first. Merged into core ranking in 2024
2024	AI Overviews	Generated summaries above results, built on Gemini

User interaction. Testimony in the 2023 US antitrust case (United States v. Google) described a system called Navboost that uses aggregated click data (which results people click, and whether they come back quickly) from roughly the past 13 months to adjust rankings. Google had long been vague about using clicks. A leak of internal API documentation in 2024 also mentioned many signal names, but Google said it lacked context and may be outdated, so treat details from it carefully.

Building the page

After ranking, Google still has to:

• Diversify. Don’t show ten pages from one site or ten copies of one answer.
• Generate snippets. Pick the passage that best matches the query (doc servers hold cached copies for this) and bold the matched words.
• Assemble features. Featured snippets, the Knowledge Panel, “People also ask”, maps, news, images, videos, shopping, AI Overviews. Each is its own system bidding for space.
• Filter. SafeSearch, legal removals, spam.

All of this usually finishes in well under a second.

How it’s evaluated

Google doesn’t change ranking by gut feel. Each change goes through:

• Offline evaluation with human raters following the Quality Rater Guidelines, comparing results before and after (side-by-side tests).
• Live experiments on a small slice of traffic, measuring behaviour.
• A launch review.

Google has said it runs thousands of experiments and ships thousands of changes a year. Large core updates a few times a year reshuffle rankings broadly, which is why sites see sudden traffic swings.

The metrics behind this are standard IR: precision@k (how many of the top k are relevant) and NDCG (normalised discounted cumulative gain: relevant results near the top count more than ones at position 9).

What this means for building search yourself

You won’t build Google, but every product with a search box uses the same ideas, scaled down:

Google concept	Your app’s version
Crawl	Index on write (a DB trigger, a queue job when a document changes)
Inverted index	Postgres tsvector + GIN index, SQLite FTS5, Elasticsearch/OpenSearch, Meilisearch, Typesense, or MiniSearch in the browser
BM25	The default ranking in Lucene-based engines and SQLite FTS5. Postgres ts_rank is a simpler formula
Query understanding	Typo tolerance, synonyms, stemming per language
PageRank	Popularity: views, likes, how many other docs link here
Neural matching	Embeddings + a vector index (pgvector, HNSW) for semantic search
Multi-stage ranking	Keyword + vector “hybrid” retrieval, then a reranker model on the top 50
Freshness	Boost recent documents
Personalisation	Boost docs from the user’s own workspace or team

A real-world example

A SaaS app with a few million documents across many workspaces adds search:

• Indexing runs from a job queue each time a document is saved, so search is a few seconds behind writes, like Caffeine’s continuous indexing, not a nightly batch.
• Retrieval is hybrid: a full-text index (BM25) catches exact names and codes, and an embedding index catches “that doc about onboarding” when the doc says “new starter checklist”. Results from both are merged.
• Permissions are a filter at retrieval time, not after ranking. Filtering after you’ve picked the top 10 can leave the user with 2 results, or leak that a hidden doc exists.
• Ranking signals are text relevance, recency, how often the user opened the doc, and whether it’s in their own workspace. The app’s version of links (other docs mentioning this one) acts like a small PageRank.
• Snippets highlight the matching passage, the way Google’s doc servers do.
• Evaluation uses a set of 100 real queries with known good answers, scored with precision@5 before every ranking change.

Gotchas

• “The algorithm” is not PageRank. It’s a pipeline with hundreds of signals and several learned models. PageRank is one input.
• robots.txt blocks crawling, not indexing. Use noindex (and let the page be crawled so Google sees it) to keep a page out.
• JavaScript-only content is indexed later, or not at all, if rendering fails or content needs interaction.
• Duplicate URLs split signals until canonicalisation merges them. Pick canonicals yourself.
• Raw term frequency is trivially gameable. Use BM25-style saturation and length normalisation.
• Link-based scores are gameable. Any popularity signal you add to your own search will be gamed once people know it exists.
• Most claims online about exact ranking factors are guesses. Prefer Google’s own docs, the original paper, and court records.
• In your own search, filter permissions before ranking, not after.

Practice

1. Build the toy engine: tokenize, inverted index with positions, AND queries by intersecting sorted posting lists.
2. Add BM25 and compare it to raw term counts on a keyword-stuffed document.
3. Implement PageRank by power iteration on a 1,000-node random graph. Measure how many iterations it takes to change by less than 1e-6. Then add a link farm and watch the target’s score.
4. Add phrase search ("sourdough bread") using positions.
5. Delta + varint encode your posting lists and measure the size difference.
6. Build Postgres full-text search on a real dataset (to_tsvector, GIN index, websearch_to_tsquery, ts_rank), then add pgvector embeddings and merge both result lists with reciprocal rank fusion.
7. Write 20 test queries with known good answers and compute precision@5 and NDCG@10 for each version you build.

Sources worth reading

• Brin and Page, “The Anatomy of a Large-Scale Hypertextual Web Search Engine” (1998). The original design, very readable.
• Google Search Central, “In-depth guide to how Google Search works”: https://developers.google.com/search/docs/fundamentals/how-search-works
• Google, “How Search works” (the public overview, including ranking systems): https://www.google.com/search/howsearchworks/
• Google’s “A guide to Google Search ranking systems” (lists the named systems)
• Google Search Quality Rater Guidelines (public PDF)
• Jeff Dean, “Challenges in Building Large-Scale Information Retrieval Systems” (WSDM 2009 keynote)
• Peng and Dabek, “Large-scale Incremental Processing Using Distributed Transactions and Notifications” (Percolator, OSDI 2010)
• Google AI blog, “Understanding searches better than ever before” (BERT, 2019)
• Manning, Raghavan and Schütze, Introduction to Information Retrieval (free online). Inverted indexes, compression, BM25, evaluation.