Feature Engineering
Master feature engineering: feature creation, selection, preprocessing, and feature store architectures.
What is feature engineering?
Feature engineering is the work of turning raw observations into model inputs that express a useful, reproducible signal. A feature can be a direct value, such as account age, or a derived value, such as purchases in the previous seven days.
The goal is not to manufacture as many columns as possible. The goal is to represent what the model needs to know using information that was genuinely available when the prediction was made. A feature is only production-ready when its meaning, timing, transformation state, and serving behavior are defined together.
Read ML Fundamentals first if prediction units, labels, training splits, and generalization are unfamiliar. Review Data Pipelines for the systems that compute and transport these values.
A useful feature must satisfy four invariants
Information boundary
Available
The value exists when the prediction is requested. Future events, delayed labels, and late-arriving records cannot silently enter historical training rows.
Same definition
Reproducible
Given the same entity, timestamp, raw inputs, and feature version, the offline and online paths produce the same value.
Unseen evidence
Useful
The feature improves an appropriate holdout metric or important slice enough to justify its complexity and cost.
Owned contract
Operable
The feature has a type, owner, source, freshness target, null policy, lineage, monitoring plan, and safe fallback.
A feature can be highly predictive and still be invalid. If it contains information unavailable at prediction time, the model is learning from the future rather than from a deployable signal.
One feature definition, two materializations
The feature contract is shared; historical and online values are computed for different access patterns.
Preserve time
Raw observations
Store entity keys, event time, availability time, source lineage, and immutable raw values.
Define once
Versioned transform
Specify inputs, windows, learned state, types, null behavior, and output semantics.
Train
Offline materialization
Reconstruct each historical value at its prediction timestamp without future information.
Serve
Online materialization
Publish fresh entity values within the inference latency and availability budget.
Start from the feature's meaning, not a transformation menu
Ask three questions before choosing an encoding:
- What relationship should become easier to learn? A ratio may express affordability; a rolling count may express recent intensity; a sine/cosine pair may express a cycle.
- Which model will consume the value? Distance-based and linear models are sensitive to scale and geometry in ways many tree-based models are not.
- What must production reproduce? High-cardinality encoders, external enrichments, and long windows add state, latency, and failure modes.
Magnitude and shape
Numeric
- Scale when model geometry depends on distance or gradient size.
- Use a log transform for positive, heavily skewed values when relative differences matter.
- Add ratios or differences only when their denominator, units, and missing behavior are defensible.
Identity without false order
Categorical
- One-hot encode bounded nominal categories.
- Group rare values when sparse dimensions are unstable.
- Fit frequency or target-derived encodings inside each training fold, with unknown-category behavior defined.
History without the future
Temporal
- Express recency as time since a prior event.
- Compute windows using only eligible earlier events.
- Encode periodic variables with a representation that keeps the cycle boundary adjacent.
Fit learned transformation state after the split
Some transforms are stateless: extracting the hour from a timestamp requires no learned parameters. Others learn state from data: imputers learn replacement values, scalers learn distribution statistics, vocabularies learn categories, and selectors learn which inputs survive.
1 Split first
Create the evaluation boundary
Separate training and evaluation evidence by the correct unit and time before inspecting target-derived statistics.
2 Learn state
Fit on training evidence
Learn medians, means, vocabularies, bins, encodings, and selected columns from the training partition only.
3 Reuse state
Transform every partition
Apply the frozen training-fitted transform to validation, test, batch inference, and online inference inputs.
4 Release
Version the full pipeline
Bind preprocessing state, feature definitions, model artifact, and schema so they are promoted and rolled back together.
The same rule applies inside cross-validation: each fold must fit preprocessing and feature selection using only that fold's training portion. A pipeline makes this boundary explicit and prevents inconsistent transformations.
Point-in-time joins keep historical features honest
A historical training row asks: what could the system have known for this entity at this prediction timestamp? That question involves two clocks:
- Event time: when the source event happened in the real world.
- Availability time: when the event became usable by the feature pipeline.
A row that happened before the prediction but arrived afterward can still create training-serving skew. The offline job sees it during a later backfill; the online model could not have seen it at prediction time. A correct join also enforces entity keys, window boundaries, and any time-to-live policy.
Select features by evidence and total cost
Feature selection is a controlled model decision, not a one-time ranking exercise. Put selection inside the evaluation loop and compare the smallest useful set with larger candidates.
Fast screening
Filter methods
Rank inputs using statistics independent of the final model. They are inexpensive, but they can miss interactions and do not prove production value.
Subset experiments
Wrapper methods
Train repeated candidate subsets against the chosen metric. They capture model-specific interactions but can be computationally expensive and unstable on small data.
Selection during fit
Embedded methods
Use regularization, tree splits, or learned gates to reduce effective inputs while training. Validate stability across folds and time periods.
Evaluate each candidate across more than aggregate accuracy:
- holdout quality and important slice quality;
- stability across folds, seeds, and out-of-time windows;
- data acquisition and computation cost;
- online latency, freshness, and availability;
- privacy, policy, and explanation requirements;
- monitoring burden and ownership;
- ablation evidence showing what is lost when the feature is removed.
Feature importance is not causal evidence. A model can rely on a proxy, split credit across correlated inputs, or assign importance to a leaking feature.
Treat every production feature as a contract
For each released feature, record:
- Meaning: a plain-language definition, unit, entity key, and valid range.
- Timing: event-time rule, availability rule, window, freshness target, and time to live.
- Computation: source fields, transformation version, learned state, and dependencies.
- Serving: offline and online materializations, latency budget, null fallback, and default behavior.
- Governance: owner, lineage, access policy, sensitive-data classification, and retention.
- Monitoring: null rate, freshness, distribution shift, offline-online parity, and downstream model usage.
Monitor the mechanism, not only the distribution
Distribution drift can be legitimate, while a stable distribution can hide a broken pipeline. Pair statistical checks with operational invariants:
- compare offline and online values for sampled entity-timestamp pairs;
- alert on late data, stale values, schema changes, and default-rate spikes;
- track feature versions alongside model predictions and outcomes;
- rehearse fallback and rollback when a dependency or backfill fails.
Prefer the simplest feature set that meets the product target and can be reproduced under production constraints. Every additional feature adds a dependency that must remain correct for every prediction.
Primary references
- scikit-learn: common pitfalls and recommended practices explains inconsistent preprocessing, train-only fitting, and leakage-safe pipelines.
- scikit-learn: ColumnTransformer with mixed types demonstrates separate numeric and categorical preprocessing inside one model pipeline.
- Feast: point-in-time joins documents historical retrieval relative to each entity timestamp and feature time-to-live window.