LangChain Production
Deploy LangChain in production: orchestration patterns, monitoring, scaling, and production best practices.
What is production LangChain?
Production LangChain is the practice of operating a LangChain application as a reliable software system, not merely getting a model call to work. LangChain supplies model, tool, agent, and middleware abstractions; LangGraph supplies lower-level stateful orchestration; LangSmith can supply tracing, evaluation, and deployment services.
In plain language: the framework helps connect the steps, but your application still owns the contract. It must bound what a run may do, identify its state, survive dependency and worker failures, expose evidence, and reach a user-visible terminal outcome.
The core invariant is every accepted run needs a bounded authority, deadline, durable identity, and observable outcome. A retry, deployment restart, or resumed thread must not silently become a second business operation.
Separate the framework layers before choosing an architecture
The current LangChain stack offers related layers with different jobs. Treating them as one interchangeable product hides important ownership decisions.
Application framework
LangChain
Integrates models and tools and provides create_agent, middleware, structured output, streaming, and runnable composition. Use it when the supplied abstraction matches the control loop you need.
Orchestration runtime
LangGraph
Models explicit state and transitions for long-running or custom workflows. Checkpointers can persist thread state so a run can pause, resume, and recover from the last recorded step.
Operations platform
LangSmith
Records nested traces, supports offline and online evaluation, and offers managed or self-hosted deployment options. It is useful infrastructure, not a substitute for application policy.
These layers are optional and composable. A fixed pipeline may use LangChain without a custom graph, a graph may use model clients without LangChain, and either can emit telemetry to another observability system.
Choose the least autonomous architecture that completes the task
Start with the shape of the decision, not the popularity of an abstraction. A fixed transformation, a bounded tool loop, and a resumable business process have different control and state requirements.
Known path
Deterministic pipeline
Use typed stages when the order and allowed operations are known before execution. This is easiest to test, budget, cache, and explain.
Bounded choice
LangChain agent
Use an agent when the model must select among approved tools or decide how many loop iterations are needed. Keep tool authority, iteration count, and stop conditions explicit.
Durable state machine
Custom LangGraph
Use a graph when the application needs explicit branches, parallel work, checkpoints, approval pauses, or recovery that must resume at a known boundary.
Architecture selection lab
Choose a workload and commit to an execution model. The map shows what that decision makes deterministic, what state it introduces, and where the operational risk moves.
Loading architecture scenarios...
The strongest default is the smallest model that expresses the required control. Adding an agent to a fixed path creates probabilistic routing without a corresponding product need; forcing a resumable process into one request handler loses durable execution.
Put contracts around the orchestration layer
The orchestration graph should receive validated input and return a typed outcome. Authentication, authorization, quotas, and business invariants remain deterministic boundaries around model-controlled behavior.
1 Identity
Admit a run
Authenticate the caller, authorize the use case, assign a run and thread identity, and attach an idempotency key for any business operation.
2 Limits
Bind the budget
Set an absolute deadline, model and tool limits, maximum loop count, cancellation signal, and spend envelope before orchestration begins.
3 Control
Execute typed stages
Validate state transitions and tool arguments. Keep credentials and policy outside prompts, and checkpoint only state that can be serialized and safely resumed.
4 Evidence
Commit an outcome
Return a validated result, a truthful pending state, or a classified failure. Record release, trace, cost, and side-effect identifiers for diagnosis.
Model state by lifetime and owner
"Memory" is too broad to be an operational contract. Separate transient invocation context, resumable thread state, cross-thread application memory, and authoritative business records.
One invocation
Run context
Carries request-scoped identity, credentials, feature flags, deadlines, and cancellation. Do not persist secrets into prompts or checkpoints merely because tools need them.
One thread
Checkpointer
Stores graph snapshots and pending writes for conversation continuity, human review, time travel, and fault recovery. A stable thread_id selects the state to resume.
Across threads
Store
Holds application-defined long-term information such as preferences or facts. It needs tenant isolation, retention, deletion, and provenance rules.
Business truth
System of record
Owns payments, orders, tickets, or other effects. A checkpoint may record an operation ID, but it must not replace reconciliation with the authoritative service.
Persist the minimum state required to resume safely. Unbounded message histories increase storage and context pressure; missing effect identifiers make ambiguous completion impossible to reconcile.
Retry the smallest uncertain boundary
LangChain runnables and agent middleware can retry model or tool calls, but retry scope is a correctness decision. Retry only failures classified as transient, preserve the original deadline, add backoff and jitter, and avoid replaying already successful stages.
A retry-safe side-effect path
The workflow reconciles authoritative state before replaying an operation whose completion is unknown.
Intent
Run checkpoint
Records the thread, step, operation key, deadline, and the last confirmed transition.
Attempt
Tool boundary
Sends one scoped request with the stable operation key and records the dependency request ID.
Truth
System of record
May commit even if the response is lost. It exposes lookup by operation key or an equivalent reconciliation path.
Recover
Resume decision
Returns the existing result, retries only a proven non-commit, or escalates when the outcome cannot be established.
A timeout means the caller stopped waiting; it does not prove the dependency stopped working. Retrying the whole chain after an ambiguous write can duplicate the effect and repeat unrelated model spend.
Rehearse failure before choosing deployment controls
A production run crosses state storage, model and tool dependencies, telemetry, and compute. Inject a failure, then choose the controls that determine whether the run is resumable, diagnosable, and safe to continue.
Loading failure scenarios...
Observe both service health and answer quality
A successful HTTP response is not enough evidence for an LLM application. Capture a nested trace for the execution path and evaluate whether the output satisfied the product contract.
Trace every causal boundary
- Link the root request, model calls, retriever calls, tool calls, checkpoints, and external effects under one trace.
- Attach stable metadata such as environment, application release, prompt or graph version, tenant-safe cohort, and thread ID.
- Record latency, queue wait, token use, attributed cost, retries, cancellations, and classified errors at the stage that caused them.
- Redact or avoid sensitive inputs and outputs according to the product's retention policy.
Evaluate the behavior
- Run offline evaluation on curated and production-derived cases before a release.
- Sample production traces for online quality, safety, and format checks.
- Break results down by task, risk, customer cohort, model route, and failure mode rather than relying on one average score.
- Promote representative failures into regression datasets and compare the proposed release with the current one.
Match deployment shape to run lifetime
Deployment is more than packaging the Python process. It determines whether queued work survives restarts and whether request admission can scale independently from execution.
Short bounded runs
Application service
Run a fixed chain or short agent behind your existing API service when normal request timeouts, stateless replicas, and external persistence cover the workload.
Owned infrastructure
Standalone Agent Server
Package graphs with the Agent Server runtime and operate its PostgreSQL, Redis, workers, upgrades, and capacity. Avoid scale-to-zero for queued long-running work.
Durable operations
Managed or self-hosted platform
Use LangSmith Deployment when its threads, runs, persistence, task queue, streaming, and horizontal-scaling model fit your operational requirements and governance.
For higher concurrency, separate API servers from queue workers so admission and execution scale on different signals. A rolling deployment must drain or resume in-flight work, preserve compatible checkpoint schemas, and keep the previous revision available for rollback.
Ship with explicit production gates
Before release
- Pin and review LangChain, LangGraph, model-provider, and tool dependencies.
- Test deterministic stages, tool contracts, graph transitions, interrupts, retries, cancellation, and checkpoint migration.
- Run offline evaluations on happy paths, adversarial inputs, tool failures, and long-context cases.
- Load-test queue wait, model quotas, checkpoint storage, and downstream concurrency rather than only the API handler.
- Prove that a rollback can read or deliberately reject state written by the new revision.
During operation
- Alert on terminal failure rate, stuck or orphaned runs, queue age, retry amplification, checkpoint errors, and quality regressions.
- Maintain kill switches for models, tools, prompts, graph routes, and releases.
- Reconcile ambiguous mutations before replay and retain the operation evidence needed for support.
- Bound every run with deadlines, cancellation, tool authority, iteration limits, and a visible pending or terminal state.
Primary sources behind this lesson
- LangChain agents documents
create_agent, structured output, middleware, thread-scoped invocation, streaming, and fault-tolerance controls. - LangGraph overview distinguishes the low-level orchestration runtime and its durable execution, streaming, persistence, and human-review capabilities.
- LangGraph persistence defines checkpointers for thread state and stores for cross-thread application memory.
- LangChain runnable retry reference recommends keeping retries on the smallest failure-prone runnable and limiting them to transient errors.
- LangSmith observability concepts defines projects, traces, runs, threads, feedback, tags, and metadata.
- LangSmith Agent Server describes persistence, task queues, API and worker roles, and supported runtime configurations.
Framework APIs change. Pin the versions you deploy and verify code against the matching official reference rather than copying examples from a different release line.