Lexical Search and BM25
Lexical search ranks based on term matches and their distribution. BM25 accounts for term frequency, inverse document frequency, and document length. It is strong for exact product names, error codes, identifiers, and rare terms, but may miss paraphrases. Analyzer choices, stemming, stop words, field boosts, and exact-match fields matter.
Keep lexical search as a baseline even when adding vectors. For code and IDs, punctuation and casing can carry meaning, so use appropriate analyzers and keyword fields.
Implementation example
Query "ERR_AUTH_401" -> lexical index -> matching runbook section💡 Exact identifiers are often a strength of lexical retrieval.
Key Takeaways
- Inspect the relevant pipeline stage instead of guessing.
- Preserve source provenance, versions, and access scope.
- Measure quality on representative queries.
Knowledge Check
Which query is especially suited to lexical search?