Apache Lucene | System Designer

What is Apache Lucene?

Apache Lucene is a Java library for building search into an application. It turns documents into index structures that can find words, exact values, phrases, numeric ranges, and nearest vectors without scanning every source record.

Lucene is not a search server. It provides indexing, analysis, query, scoring, and storage APIs; the application still owns its network API, document lifecycle, authorization, replication, sharding, backups, and user experience.

The core invariant is simple: a query can only use structures created at index time, and an IndexSearcher only sees the point-in-time reader it was opened on. Field design and reader refresh are therefore part of the product contract, not hidden tuning.

Write path

Index evidence

Choose fields and analyzers, then write searchable terms, typed points, doc values, and stored values into immutable segments.

Query path

Express intent

Build a Query whose semantics match the field representation: analyzed text, exact terms, phrases, ranges, prefixes, or vectors.

Reader lifecycle

Publish a view

Open or refresh a reader to expose a new point-in-time view while older readers continue using their existing snapshot.

Follow a document into the index

A Lucene Document is a collection of named fields. Each field type decides which physical access paths are written; one logical source value may need more than one field representation when the product asks different questions.

1
Product question
Define the field contract
Write down whether the value must support full-text relevance, exact filtering, phrase positions, numeric ranges, sorting, or result retrieval.
2
Term policy
Analyze or encode the value
An Analyzer builds token streams for human language. Exact identifiers and typed numbers use field representations that preserve their application meaning.
3
Index structures
Write access paths
Postings locate matching terms, points support typed spatial or numeric queries, doc values provide column-style per-document values, and stored fields return payloads.
4
Search boundary
Publish a reader view
Open or refresh a reader, construct an IndexSearcher, and execute queries against that stable point-in-time view.

Keep the structures distinct

Postings answer term-oriented search questions and may carry frequencies, positions, and offsets according to the field type.
Stored fields return selected values after another structure identifies a document; storage alone does not make a value searchable.
Doc values expose strongly typed, per-document values used by operations such as sorting and faceting.
Point fields encode typed values for exact, set, and range predicates.
Term vectors are optional per-document term statistics; they are separate from the ordinary postings used by search.

Lucene's current Field API lists the purpose-built field classes and their intended access paths.

Design fields from product questions

Do not start from a generic "index everything" preset. Start with the questions the product must answer, then create the smallest explicit set of access paths that answers them.

Rules that prevent common schema mistakes

Use TextField for tokenized full-text search. Its standard field type indexes positions, which enables phrase queries.
Use StringField when the entire string is one exact term, such as a country code or identifier.
Add doc values when the read path needs sorting or faceting; for example, SortedDocValuesField provides an ordered per-document byte value.
Use typed fields such as IntField for integer exact/range queries and per-document numeric behavior.
Store only values the result path must return from Lucene. Large source payloads can stay in the system of record when the application can hydrate by stable ID.
Treat analyzer and field-type changes as a new index generation. Existing segments retain the terms and structures written under the old contract.

Reject a field recipe that cannot answer its product contract

Construct queries at the correct boundary

Query construction begins with ownership:

Human-language text needs the analyzer associated with the indexed text field.
Application-generated identifiers, dates, numbers, and selected filters should be turned into query objects directly.
A phrase needs indexed positions, not merely the original stored string.
A required filter that should not affect relevance belongs in BooleanClause.Occur.FILTER rather than a scoring clause.
A multi-term query such as a prefix, wildcard, or regular expression needs explicit input bounds and measurement against the real term dictionary.

The official query API defines concrete types such as TermQuery, PhraseQuery, BooleanQuery, and PrefixQuery.

Lucene's QueryBuilder uses an analyzer to construct Boolean and phrase queries from plain text. The Classic QueryParser is intended for human-entered query syntax; program-generated values should normally use query APIs directly.

Keep exact, numeric, and human-text request boundaries separate

Separate search visibility from durable commit

Lucene writes changes through IndexWriter, but "accepted by the writer," "visible to a reader," and "durable after a crash" are different states.

Near-real-time index lifecycle

Reader refresh and durable commit solve different problems; neither should be inferred from the other.

Mutation

IndexWriter buffers

Add, update, or delete documents. An update identified by a term is implemented as a delete followed by adding the complete replacement document.

Files

Flush creates a segment

Move buffered state into index files. A flush is not an fsync commit and may also trigger background segment merges.

Visibility

Reader refreshes

Open a near-real-time reader from the writer or refresh a manager so searches can use a new point-in-time view without closing the writer.

Durability

Commit syncs a point

Commit pending changes and sync referenced files so that the commit point survives an operating-system or machine crash.

Lucene's IndexWriter documentation defines flush, commit, update, merge, locking, and sequence-number behavior. Use ReaderManager or another reference manager when the application needs safe acquire, release, and refresh behavior around shared readers.

Plan identity and deletes deliberately

Index a stable domain ID as an exact term.
Use that term for replacement or deletion rather than retaining Lucene's internal document number as business identity.
Carry a source version when events can arrive out of order, and reject stale writes before they replace newer documents.
Remember that deleted documents consume space until segment merges reclaim them.
Keep the source of truth replayable so a new analyzer, field contract, or Lucene major version can build a clean generation.

Internal Lucene document numbers are ephemeral and may change as documents are added, deleted, or merged. They are not stable external IDs.

Treat relevance as an evaluated product behavior

Lucene includes BM25Similarity and other Similarity implementations, but a score is not a universal probability or quality guarantee. It is the result of one query, similarity, field statistics, boosts, and indexed document collection.

Boolean semantics

Score only what should rank

Use FILTER for required eligibility clauses that should not contribute to relevance. Keep scoring clauses focused on evidence that should change ordering.

Diagnosis

Explain representative hits

Inspect query rewrites and explanations on bounded samples to understand which terms, statistics, boosts, and clauses produced a score.

Release gate

Evaluate with judgments

Maintain a query set with expected relevant and irrelevant results. Compare ranking metrics and failure slices before changing analyzers, boosts, or similarity settings.

The official similarities package documents available ranking models and the meaning of BM25 parameters. Tune them only against labeled product outcomes.

Operate measured segments, not guessed formulas

Document count alone cannot predict Lucene index size, memory, indexing throughput, query latency, or query throughput. Those outcomes also depend on field structures, term distributions, stored payloads, analyzers, query shapes, concurrency, filesystem, hardware, reader refresh, caches, deletions, and merge activity.

Build a representative benchmark

Sample real document sizes, languages, term distributions, update rates, and delete behavior.
Replay the actual query mix, result sizes, filters, sorts, phrase/prefix behavior, concurrency, and timeout policy.
Measure median and tail latency together with indexing rate, refresh time, segment count, merge work, disk bytes, cache behavior, errors, and rejected requests.
Include steady writes, burst writes, large merges, reader refresh, restart, and disk pressure rather than benchmarking an idle, fully warmed index only.
Compare one controlled change at a time and retain the corpus, query set, environment, and Lucene version with the result.

Production guardrails

Keep one IndexWriter per index directory; Lucene enforces a write lock.
Reuse thread-safe writers, readers, and searchers according to their documented lifecycle instead of opening them per request.
Set request deadlines and use a query timeout where cancellation semantics fit the product contract.
Bound user syntax, query length, Boolean clauses, prefix/wildcard expansion, result count, and deep pagination.
Monitor uncommitted changes, buffered documents, segment count, deleted documents, pending merges, merge I/O, refresh failures, commit failures, disk headroom, and tail latency.
Do not schedule routine forceMerge on an actively updated index. Let merge policy manage segments unless a measured, lifecycle-specific workflow justifies otherwise.
Keep independent backups or a replayable source. Lucene index files are a search projection, not automatically a replicated system of record.

Review the official contracts before shipping

This lesson is grounded in Apache Lucene 10.3.2 documentation:

API contracts can change between Lucene releases. Read the migration guide for the version you deploy, rebuild representative indexes, and rerun relevance plus operational tests before a major-version cutover.

No quiz questions available

Could not load questions file