The search on this site, taken apart.

Tokenising on /\w+/ would split Next.js into two words and destroy C++ and C# entirely. On a developer's site those are not edge cases, they are the queries — so internal separators and trailing + and # stay part of the token.

Folding lowercases and strips diacritics, so résumé and resume are one term. It runs per token, never on the whole document: Unicode normalisation changes string length, and the character offsets that highlight search snippets would land on the wrong letters.

Stopwords are dropped — with one deliberate omission. Standard lists remove single letters and very short words, which on a site carrying a C programming series would make the single most important query on the corpus return nothing. c, go and r are kept.

Stemming is Porter's 1980 algorithm, so deployment, deployed and deploying collapse toward one index term. It is 60 lines of regular expressions that fail silently when edited wrongly, which is why the repository carries a test suite pinning its output word by word — including one case that pins a known wart in the algorithm rather than quietly patching a published one.

Positions are what let the ranker tell a phrase from a coincidence. Two query terms occurring three words apart and two occurring four hundred words apart are the same match to a bag-of-words model; the smallest window covering all of them is found by a k-pointer sweep over the position lists, which is linear in the number of occurrences rather than quadratic.

The vocabulary is kept sorted, so prefix expansion — what makes results appear while you are still typing — is a binary search for the range rather than a scan of every term on every keystroke.

The F in BM25F is fields. A term in a project's name is not worth the same as the same term in paragraph nine, so title, subtitle, tags and body are weighted separately and each is length-normalised against its own average — titles are short by nature, and normalising them like prose would punish a four-word name for being four words long.

Two multipliers sit on top of the sum. Coordination favours a document matching three of three query terms over one matching a single rare term very well, because BM25 sums and would otherwise prefer the second. Proximity rewards terms found close together. Both appear in the breakdown under every result.

A handful of terms appear everywhere and separate nothing; the long tail appears once or twice and identifies a document almost by itself. That is exactly what inverse document frequency encodes, and it is why a search for "stripe" is decisive while a search for "code" is not.