langid.py
Use langid.py for candidate-scoped language identification, uncertainty-aware routing, evaluation, and safe multilingual fallbacks.
What is langid.py?
langid.py is a pre-trained language identification library: give it text and it returns the most likely language label plus a score. Language identification is the classification step that answers “what language is this document written in?” before a system chooses a tokenizer, search index, moderation model, translation path, or human-review queue.
The bundled model covers 97 ISO 639-1 language labels and was designed to work across multiple domains without application-specific training. Its original implementation uses byte n-gram features, a multinomial Naive Bayes classifier, and a compiled scanner to score text efficiently.
The core invariant is langid.py can choose only from its active candidate set. A high normalized score means one candidate outranked the others under that model; it does not prove that the true language was eligible, that the input contains one language, or that the prediction is safe to automate.
Trace one document through the classifier
The model turns byte patterns into one score per eligible language, then returns the highest-scoring label. Your application still owns the policy that decides whether to accept, defer, or reject that result.
From text to an operational decision
Candidate selection changes the competition before the application applies its own release gates.
Document boundary
Input text
Pass a meaningful document or message. Empty, tiny, mixed-language, template-heavy, and machine-generated strings need an explicit handling policy.
Features
Byte n-gram scanner
The original model counts selected byte sequences. Byte features let one model process many scripts without first choosing a language-specific tokenizer.
Candidate ranking
Naive Bayes scores
The classifier scores each active language. Restricting the candidate set can improve the decision boundary only when the true language remains eligible.
Accept or defer
Application gate
Use labeled evaluation, input checks, score thresholds, product scope, and a fallback path to decide whether the prediction may affect users or downstream data.
See how the candidate set changes the answer
The official project documentation classifies the same English sentence under three candidate-language configurations. Select each configuration and watch both the top label and normalized score change. One narrow set excludes English completely and therefore produces a confident but impossible-to-correct answer.
Loading decision scenarios
The lab validates its candidate sets, detection fixtures, and policy controls before rendering.
Loading language-identification scenarios...
Scope languages from product truth
- Restrict candidates when a product, tenant, corpus, or route has a trustworthy set of supported languages.
- Keep an
unknownor review path when evidence conflicts with that scope. Do not add an unsupported language merely to force every record into a known label. - Avoid changing the module-level candidate set per request in a concurrent service. Build a dedicated
LanguageIdentifierfor each stable policy boundary instead. - Re-evaluate candidate scope when a product enters a new market or starts accepting content from a broader source.
Interpret the score without pretending it is certainty
The classifier works in log-probability space. Normalization converts its output to a 0-to-1 value that is easier to compare within the active candidate set. That number is useful evidence, but an application threshold must be learned from representative, labeled data rather than copied from a generic tutorial.
What the score can say
Relative evidence
A normalized score summarizes competition among the active candidates for this input. Changing the candidates can change the score even when the text is identical.
What the score cannot repair
Missing truth
If the true language is unsupported or excluded, normalization still distributes belief across the remaining labels and can make a wrong winner look decisive.
What production must add
Decision policy
Combine the score with minimum evidence, product scope, mixed-language handling, labeled slice metrics, and a fallback that does not silently corrupt routing.
Do not publish a universal confidence threshold
Text length, language pair, script, domain, markup, and candidate set all affect the score distribution. Calibrate thresholds per decision and measure false accepts and false deferrals on data shaped like production traffic.
Design the acceptance policy, not just the classifier call
Use the illustrative labeled fixture below to compare three release policies. Change the score floor and inspect which records auto-route, which defer, and which wrong labels leak downstream. The fixture is for policy reasoning, not a langid.py benchmark.
Loading decision scenarios
The lab validates its candidate sets, detection fixtures, and policy controls before rendering.
Loading language-identification scenarios...
Choose the cost of each error
- A false accept sends a record to the wrong language-specific workflow. It may poison analytics, choose the wrong model, or hide content from the correct reviewers.
- A false deferral spends fallback capacity on a prediction that could have been used. That cost can be acceptable when the downstream action is difficult to reverse.
- An abstention is a valid product result. Route it to another detector, a broader model, a multilingual path, or human review instead of inventing certainty.
- Mixed-language content needs a declared unit of classification: whole document, paragraph, sentence, or segment. A document-level top label does not describe every span.
Keep classifier state inside a stable boundary
The official API exposes set_languages() on the module-level classifier, but that mutates shared state. A service with concurrent tenants or routes should create an identifier for each stable candidate policy instead of changing one global classifier around individual requests.
Make the API contract explicit
- Return the predicted ISO code, normalized score, model or package version, candidate policy version, and final routing decision.
- Bound input bytes and characters before classification. Reject empty input and avoid storing raw user text in logs when aggregate diagnostics are enough.
- Initialize classifier objects when a worker starts; do not rebuild the embedded model for every request.
- Treat package and model upgrades as behavior changes. Replay a labeled evaluation set before release and retain a rollback path.
Separate model output from the routing gate
This executable standard-library example applies a decision policy to fixed detection records. Keeping this layer independent makes threshold behavior testable without loading the model, and it lets a second detector or human review reuse the same downstream contract.
1 Representative evidence
Collect labeled slices
Include supported, unsupported, short, mixed, noisy, and high-impact inputs from each real source rather than relying only on clean example sentences.
2 Not only top-line accuracy
Measure the decision
Track confusion by language pair and source, false accepts, deferrals, coverage, input length, score distribution, and downstream correction rate.
3 Versioned policy
Release with a fallback
Version the candidate set and threshold policy, canary changes, preserve an abstention path, and compare labeled outcomes before widening traffic.
4 Traffic and product change
Watch for drift
Alert when language mix, unsupported-language rate, score distribution, or manual correction rate moves beyond the reviewed envelope.
Review the production contract
Before placing langid.py on a routing path, answer these questions:
- What exact text unit is classified: document, message, paragraph, or sentence?
- Which languages are eligible, and what happens when the truth is outside that set?
- How were score gates calibrated for each source, language pair, and error cost?
- Which inputs are deferred because they are empty, short, mixed, unsupported, or outside the evaluated envelope?
- Which labeled metrics detect wrong automatic routes rather than only low scores?
- How are model, package, candidate policy, and threshold versions recorded together?
- What fallback handles abstentions, and how is a bad release rolled back?
The official langid.py project documentation is the primary API reference for classification, candidate restriction, normalization, batch mode, and model training. The authors' ACL 2012 system paper describes the byte n-gram, feature-selection, scanner, Naive Bayes, training-data, and evaluation design.
The original langid package is mature but old. Verify runtime compatibility and maintenance requirements against the exact artifact you deploy.