Autocomplete, typos, synonyms, and name relevance

Users type names into a search box, one letter at a time, with typos, in spellings that differ from the stored record. This chapter makes that work on user-profile-v1. You build three scratch indices over the same million names to compare Elasticsearch’s autocomplete options by size and behaviour, confirm the choice made in chapter 05, add typo tolerance, change the synonym list on a live index without reindexing, and read a relevance score line by line with the _explain API.

You need the lab with user-profile-v1 loaded (chapter 05) and the analysis concepts from chapter 06. The chapter takes about 45 minutes. The scratch indices are deleted at the end.

Three ways to autocomplete, built side by side

Elasticsearch offers three mechanisms for prefix search. They differ in what they index, what queries they support, and whether they can be combined with filters.

Why it matters at 300 million profiles. Autocomplete runs on every keystroke, so it is usually the highest-rate query the Search API serves, and its index structures are among the largest in the mapping. Choosing wrongly is expensive twice: once in disk and heap, and again in a reindex to change it.

Example. Create four scratch indices. They share a small analysis block, a plain name analyser n and a prefix analyser np, and differ only in how fullName is mapped. The baseline has no autocomplete structure at all, so it shows what each option adds.

1
PUT ch07-baseline
2
{
3
  "settings": {
4
    "number_of_shards": 1, "number_of_replicas": 0,
5
    "analysis": {
6
      "filter":   { "eg": { "type": "edge_ngram", "min_gram": 2, "max_gram": 15 } },
7
      "analyzer": {
8
        "n":  { "type": "custom", "tokenizer": "standard", "filter": ["lowercase", "asciifolding"] },
9
        "np": { "type": "custom", "tokenizer": "standard", "filter": ["lowercase", "asciifolding", "eg"] }
10
      }
11
    }
12
  },
13
  "mappings": { "properties": {
14
    "fullName": { "type": "text", "analyzer": "n" },
15
    "state":    { "type": "keyword" }
16
  } }
17
}

Create ch07-edge, ch07-sayt, and ch07-completion with the same settings block, changing only the fullName mapping:

1
{ "type": "text", "analyzer": "n",
2
  "fields": { "prefix": { "type": "text", "analyzer": "np", "search_analyzer": "n" } } }

1
{ "type": "search_as_you_type", "analyzer": "n" }

1
{ "type": "text", "analyzer": "n",
2
  "fields": { "suggest": { "type": "completion", "analyzer": "n" } } }

Copy the names and states into each index, then merge each to one segment so that sizes are comparable:

1
POST _reindex?refresh=true&filter_path=created,failures
2
{ "source": { "index": "user-profile-read", "_source": ["fullName", "state"] },
3
  "dest":   { "index": "ch07-baseline" } }
4

5
POST ch07-baseline/_forcemerge?max_num_segments=1

Repeat both requests for ch07-edge, ch07-sayt, and ch07-completion. Each reindex reports {"created":1000000,"failures":[]}. Then compare:

1
GET _cat/indices/ch07-*?v&h=index,docs.count,store.size&s=index

1
index           docs.count store.size
2
ch07-baseline      1000000     24.2mb
3
ch07-completion    1000000     84.5mb
4
ch07-edge          1000000     32.2mb
5
ch07-sayt          1000000       47mb

These are lab measurements on one million short synthetic names, each index merged to one segment, _source included. Over the baseline, edge_ngram added about 8 MB, search_as_you_type about 23 MB, and completion about 60 MB. Needs validation: absolute sizes depend on name length and diversity, so measure your own data as chapter 08 describes. The order is what these numbers show, not the ratios at scale.

How each option behaves

Run the same user intentions against each index. A user in Bihar has typed prashant ku.

edge_ngram, with the query analysed by the plain analyser and every word required:

1
GET ch07-edge/_count?filter_path=count
2
{ "query": { "bool": {
3
  "must":   { "match": { "fullName.prefix": { "query": "prashant ku", "operator": "and" } } },
4
  "filter": { "term": { "state": "Bihar" } }
5
} } }

1
{"count":345}

search_as_you_type, with a bool_prefix query across the generated sub-fields:

1
GET ch07-sayt/_count?filter_path=count
2
{ "query": { "bool": {
3
  "must": { "multi_match": {
4
    "query": "prashant ku", "type": "bool_prefix", "operator": "and",
5
    "fields": ["fullName", "fullName._2gram", "fullName._3gram"]
6
  } },
7
  "filter": { "term": { "state": "Bihar" } }
8
} } }

1
{"count":345}

Both agree with PostgreSQL, where first_name = 'Prashant' AND last_name LIKE 'Ku%' AND state = 'Bihar' counts 345. Without "operator": "and", the bool_prefix query returns 16,378 profiles, because any word may match. That is fine for a ranked list, and misleading for a count.

completion, through the suggest API:

1
GET ch07-completion/_search?filter_path=suggest.names.options.text
2
{ "suggest": { "names": {
3
  "prefix": "prashant k",
4
  "completion": { "field": "fullName.suggest", "size": 5, "skip_duplicates": true }
5
} } }

1
{"suggest":{"names":[{"options":[{"text":"Prashant Khan"},{"text":"Prashant Kumar"}]}]}}

The suggester returns distinct values, not documents. That makes it good for “which names exist?” and useless for “which profiles match?”. Now try the two things a profile search needs. First, a match at the start of the surname:

1
GET ch07-completion/_search?filter_path=suggest.names.options.text
2
{ "suggest": { "names": { "prefix": "kum", "completion": { "field": "fullName.suggest", "size": 5 } } } }

1
{}

No suggestions: the completion structure matches from the start of the whole value only. Both ch07-edge and ch07-sayt return 50,009 profiles for kum, every Kumar. Second, a filter. Send a query that matches nothing, alongside the suggester:

1
GET ch07-completion/_search?filter_path=hits.total,suggest.names.options.text
2
{
3
  "query": { "term": { "state": "Atlantis" } },
4
  "suggest": { "names": { "prefix": "pras", "completion": { "field": "fullName.suggest", "size": 3, "skip_duplicates": true } } }
5
}

1
{"hits":{"total":{"value":0,"relation":"eq"}},
2
 "suggest":{"names":[{"options":[{"text":"Prashant Chatterjee"},{"text":"Prashant Das"},{"text":"Prashant Fernandes"}]}]}}

Zero hits, and suggestions anyway. The suggester ignores the query entirely. The only built-in way to restrict it is context fields declared in the mapping, which must be chosen in advance and cannot express arbitrary combinations of state, city, status, and gender.

The recommendation for this index

Recommendation. Use an edge_ngram sub-field, which is what user-profile-v1 already has as fullName.prefix. The Search API filters every autocomplete request by at least accountStatus, and often by state or city, so the completion suggester is ruled out. Between the other two, edge_ngram was smaller in the lab, needs only a plain match query, and gives complete control over the analysis chain, including the accent folding from chapter 06.

Trade-off. search_as_you_type is a credible alternative. It requires no custom analyser, and its shingle sub-fields rank matches whose word order follows the input higher, which helps with longer, multi-word values. For two- and three-word personal names, that advantage is small. The completion suggester is the right tool for a different feature: a global, unfiltered “names in our system” suggestion list where memory-resident speed matters and filters do not.

Common mistake. Setting min_gram to 1. Every name then produces one-letter terms, and a single typed character matches a large share of the index. With min_gram: 2, autocomplete starts on the second keystroke, which users rarely notice.

Autocomplete on the real index

The same query works on user-profile-v1, with the status filter every Search API request will carry and highlighting to show what matched:

1
GET user-profile-read/_search?filter_path=hits.total,hits.hits._source,hits.hits.highlight
2
{
3
  "size": 2,
4
  "_source": ["userId", "fullName", "city"],
5
  "query": { "bool": {
6
    "must":   { "match": { "fullName.prefix": { "query": "prashant ku", "operator": "and" } } },
7
    "filter": [ { "term": { "state": "bihar" } }, { "term": { "accountStatus": "ACTIVE" } } ]
8
  } },
9
  "highlight": { "fields": { "fullName.prefix": {} } }
10
}

1
{"hits":{"total":{"value":237,"relation":"eq"},"hits":[
2
  {"_source":{"userId":"6000","fullName":"Prashant Kumar","city":"Gaya"},
3
   "highlight":{"fullName.prefix":["<em>Prashant</em> <em>Kumar</em>"]}},
4
  {"_source":{"userId":"11400","fullName":"Prashant Kumar","city":"Gaya"},
5
   "highlight":{"fullName.prefix":["<em>Prashant</em> <em>Kumar</em>"]}}]}}

237 active profiles, matching PostgreSQL. The state filter used bihar in lowercase, and chapter 05’s normaliser made it match Bihar. The whole word is highlighted, not just the typed ku, because the edge n-grams share the word’s start and end offsets. If you need to highlight only the typed characters, the client can compute that from the input; the highlighter cannot.

Clean up the scratch indices:

1
DELETE ch07-baseline,ch07-edge,ch07-sayt,ch07-completion

Typo tolerance with fuzziness

What it is. A match query with fuzziness also accepts terms within a small edit distance of each query term: one or two insertions, deletions, substitutions, or transpositions. "fuzziness": "AUTO" allows 0 edits for terms of 1–2 characters, 1 edit for 3–5, and 2 for longer terms, as the match query reference describes.

Why it matters at 300 million profiles. Support agents search while on the phone, and they mistype. Fuzziness recovers those searches, but each fuzzy term expands into every matching variant in the index, so its cost grows with the size of the name vocabulary, which is large at 300 million real profiles.

Example. prashnat swaps two letters:

The last row finds exactly the Prashant Kumars despite two typos. Short names are protected by AUTO: rao allows one edit and still counts 49,980 profiles, the same as the exact query, because no other surname in the data is one edit away.

Trade-off. Fuzzy matches score like exact ones unless you separate them. Chapter 11 puts an exact match and a fuzzy match in the same bool query, so exact spellings rank first and typo matches follow. For large vocabularies, prefix_length, the number of leading characters that must match exactly, and max_expansions cap the cost.

Common mistake. Adding fuzziness to the fullName.prefix field. Every edge n-gram becomes a fuzzy term, so pr with one edit matches almost every two-letter fragment in the index. Keep autocomplete exact and apply fuzziness to the whole-word field only.

Synonyms you can change on a live index

What it is. A synonyms set is stored in the cluster and referenced by a synonym_graph token filter. Because chapter 05 used it only in the search analyser and marked it updateable, changing a rule reloads the analyser immediately, with no reindex.

Why it matters at 300 million profiles. Name variants surface continuously from support tickets: Fatima and Fathima, Lakshmi and Laxmi. If adding one required rebuilding a 300-million-document index, the list would never be maintained.

Example. The set currently has no rule for Fathima:

1
GET user-profile-read/_count?filter_path=count
2
{ "query": { "match": { "fullName": "fathima" } } }

1
{"count":0}

Add one rule with the synonym rule API:

1
PUT _synonyms/profile-name-synonyms/fatima
2
{ "synonyms": "fatima, fathima" }

1
{"result":"created","reload_analyzers_details":{"_shards":{"total":2,"successful":2,"failed":0},
2
 "reload_details":[{"index":"user-profile-v1","reloaded_analyzers":["name_search"],"reloaded_node_ids":["..."]}]}}

The response confirms that name_search was reloaded on user-profile-v1. _analyze with name_search on Fathima now returns fatima and fathima.

The request-cache trap

Now repeat the exact same count request:

1
{"count":0}

Still zero. Change the request text in any way, for example FATHIMA instead of fathima, and the count is 33,333. The first request was answered from the shard request cache, which caches size: 0 results keyed by a hash of the request body. Elasticsearch invalidates that cache when a refresh picks up changed documents or when the mapping changes. A synonym reload is neither, so the cached answer from before the change survived.

Clear the request cache after every synonym change:

1
POST user-profile-v1/_cache/clear?request=true

After that, the original request returns {"count":33333}. Search requests that return hits (size greater than 0) are not cached by default, so they were correct immediately. Counts and aggregations are the ones at risk. This behaviour was observed in the Elasticsearch 9.5.4 lab; treat the cache clear as part of the synonym-change procedure.

Common mistake. Putting synonyms in the index analyser. Every rule change then needs a reindex, and the expanded terms inflate the index. Keep synonyms at search time unless you have measured a reason not to.

Why one profile outranks another: BM25 and _explain

What it is. Elasticsearch scores text matches with BM25 by default. For each query term that matches, the score grows with how rare the term is across the index (inverse document frequency, IDF) and with how often it appears in this document’s field, adjusted for field length. The explain API shows the full calculation for one document.

Why it matters at 300 million profiles. Relevance is corpus-wide. Chapter 01 quoted PostgreSQL’s documentation that its ranking functions do not use global information. Elasticsearch’s does, and that is visible in every explanation.

Example. Search for prashant kumar. The top profiles are named Prashant Kumar with a score of 6.3967; profiles named Prashant Jha score 3.4012. Ask why, for profile 600, a Prashant Kumar:

1
GET user-profile-read/_explain/600
2
{ "query": { "match": { "fullName": "prashant kumar" } } }

1
6.396736  sum of:
2
  3.4011931  weight(fullName:prashant), boost * idf * tf from:
3
    2.2         boost
4
    3.4011934   idf = log(1 + (N - n + 0.5) / (n + 0.5)), with n = 33333, N = 1000000
5
    0.45454544  tf  = freq / (freq + k1 * (1 - b + b * dl / avgdl)), with freq = 1, k1 = 1.2, b = 0.75, dl = 2, avgdl = 2
6
  2.995543  weight(fullName:kumar), boost * idf * tf from:
7
    2.2         boost
8
    2.9955432   idf, with n = 50009, N = 1000000
9
    0.45454544  tf, as above

The response is condensed here; it nests each line as a JSON object with value, description, and details. Three things are visible:

• The score is a sum over matched terms. A Prashant Jha matches only prashant, so it scores 3.4012, the first half.
• Rarer terms weigh more. prashant appears in 33,333 profiles and kumar in 50,009, so prashant contributes more. On real data, a rare surname in the query counts for much more than a common one.
• Field length matters. dl / avgdl compares this name’s length in terms with the average. A longer name that contains the same words scores a little lower. The 2.2 boost is BM25’s k1 + 1, folded in by Lucene.

Common mistake. Treating identical scores as a ranking. All 1,666 Prashant Kumars score exactly 6.396736, so their order is arbitrary unless the query adds a tiebreaker sort. Chapter 11 adds updatedAt and userId after the score, and chapter 13 depends on that for stable pagination.

What you built, and what comes next

You compared the three autocomplete mechanisms on the same million names, by size and by behaviour, and confirmed fullName.prefix with edge n-grams as the right choice for a filtered profile search. You added typo tolerance, added a synonym rule to a live index, and found that counts need a request-cache clear after synonym changes. You can read a BM25 score line by line.

The sizes in this chapter come from short synthetic names on one node. They show the relative cost of each option, not what your index will need.

Chapter 08 turns measurements like these into a capacity plan: shard counts, replicas, refresh and translog settings, and a sizing method for 300 million profiles, built from sample data and validated by load tests instead of by rules of thumb.