AutoML
Master AutoML: automated machine learning, neural architecture search, and hyperparameter optimization.
What is AutoML?
Automated machine learning (AutoML) searches a bounded set of machine-learning pipelines under a declared budget. Depending on the tool, that set can include data transformations, model families, hyperparameters, ensembling, and stopping rules.
AutoML is useful because comparing those choices manually is repetitive and easy to make inconsistent. It does not turn an undefined business request into a valid ML problem. People still own the target, prediction time, allowed features, validation boundary, business costs, release policy, and production monitoring.
The core invariant is: automation may choose only among pipelines and evidence that the experiment owner intentionally made valid. A fast search over leaked data or the wrong metric only automates the wrong decision.
Human boundary
You define the experiment
State the prediction target, decision time, unit of observation, eligible features, split strategy, objective, constraints, and untouched final test set.
Search boundary
AutoML explores candidates
Build and evaluate allowed transformations, estimators, hyperparameters, and ensembles within explicit time, compute, memory, and inference limits.
Production boundary
Release remains governed
Review validity, protected slices, latency, resource use, reproducibility, ownership, rollback, and monitoring before any candidate receives traffic.
Follow the complete AutoML decision path
AutoML occupies the middle of an ML lifecycle. Calling the entire lifecycle "automated" hides the decisions most likely to invalidate the result.
1 Contract
Frame the decision
Name what is predicted, for whom, at what time, and what action consumes the prediction. Reject labels or features that would not exist at that moment.
2 Evidence
Freeze evaluation
Choose random, grouped, or chronological splitting from the data-generating process. Define the selection metric and lock a final test set.
3 Automate
Run bounded search
Evaluate only allowed pipeline configurations. Record data, code, search space, resource budget, folds, seeds, failures, and every candidate result.
4 Govern
Review and release
Recheck the selected pipeline on untouched evidence, verify operational constraints, approve a versioned artifact, canary it, and retain a rollback path.
Keep the boundaries explicit
- Problem boundary: AutoML cannot decide whether the target represents the product outcome, a convenient proxy, or a harmful feedback loop.
- Data boundary: it cannot infer every entity, time, geography, policy, or causal dependency that must stay on one side of a split.
- Cost boundary: a library metric does not know the business cost of false positives, false negatives, abstentions, latency, or manual review.
- Release boundary: the best validation score is not automatically deployable, fair, robust, reproducible, observable, or affordable.
Budget the search with measured inputs
A search budget is a capacity envelope, not a model-quality forecast. Begin with one representative pilot fit on the intended hardware. Use its observed duration to ask how many complete cross-validation candidates can fit into the budget.
Budget a bounded pipeline search
Loading the transparent capacity model.
The lab uses transparent arithmetic:
search minutes = wall-clock budget x (1 - reserve percentage)worker-minutes = search minutes x parallel workersfit slots = floor(worker-minutes / observed pilot-fit minutes)complete candidates = floor(fit slots / validation folds)search coverage = complete candidates / declared configurations
Those numbers assume the pilot is representative and workers remain usable. Startup, data loading, stragglers, failed trials, scheduler contention, model-specific fit times, and final ensembling can reduce realized capacity. Re-measure instead of turning the estimate into a service-level promise.
The lab deliberately never predicts accuracy, loss, memory, or the winning algorithm. Dataset size and feature count are not enough to infer any of those outcomes.
Current AutoGluon documentation exposes time_limit, eval_metric, presets, and inference constraints as separate controls. The right values come from the workload, not from a universal preset. See the official TabularPredictor.fit reference and tabular essentials.
Design the search space before the optimizer sees it
A pipeline configuration is more than a model name. It can contain transformations, feature selection, estimator parameters, calibration, thresholding, and an ensemble. Every extra choice expands the space and creates another way to violate a production constraint.
Preprocessing
Transform inside each fold
Imputation, scaling, encoding, feature selection, and target-aware transforms must learn from the training portion of each fold. Fit them before splitting and validation has already seen information it should not have.
Search
Constrain candidate families
Permit only estimators and parameter ranges compatible with data shape, sparsity, explainability, training budget, inference latency, memory, licensing, and runtime.
Selection
Measure more than one axis
Use one declared selection objective, then inspect uncertainty, important slices, calibration, inference cost, artifact size, and failure behavior before release.
Choose a search strategy for the budget
- Grid search evaluates every declared combination. It is understandable but grows multiplicatively and wastes work when many values are unimportant.
- Random search samples from declared distributions. It provides a hard trial count and often explores more distinct values of influential parameters.
- Model-based optimization uses prior trial observations to propose later candidates. It adds optimizer state and assumptions that must be recorded.
- Multi-fidelity search stops weak candidates early or allocates progressively more data, epochs, or resources. Its intermediate signal must be comparable and safe to prune.
- Ensembling combines selected fitted models. It can improve measured performance while increasing artifact size, inference work, and operational complexity.
An AutoML run should persist the exact search-space definition, budget, framework version, data identity, split indices or splitter configuration, metric implementation, candidate history, failures, selected pipeline, and environment lock.
Protect the evaluation boundary
Validation is part of the optimization loop. The final test set is not. Once a test result changes the pipeline, feature set, threshold, metric, or search space, it has become selection data and no longer provides an untouched estimate.
Protect evidence before searching
Loading synthetic failure scenarios.
The scenarios use synthetic counts so the consequences are visible. They do not claim that a particular split or metric is universally correct. The correct boundary comes from how observations are related and how predictions will be used.
Use these official scikit-learn references when implementing the boundary:
- Common pitfalls explains why preprocessing belongs inside a pipeline and must learn only from training data.
- The cross-validation guide documents grouped and time-aware splitters.
- The nested cross-validation example shows why evaluating a tuned search on the same data used for selection can be optimistic.
Turn the selected pipeline into a production candidate
The search winner is a candidate, not a release. Refit and promotion need their own recorded steps so retries cannot silently change evidence.
- Freeze the selected pipeline specification, framework and dependency versions, training-data identity, splitter, metric implementation, random seeds, and search history.
- Refit according to the declared protocol. Do not quietly include the final test set because more rows appear attractive.
- Evaluate once on untouched final evidence and report confidence or variability, important slices, calibration, and the business-relevant error matrix.
- Test input contracts, malformed values, missingness, prediction latency, throughput, memory, artifact size, startup, and dependency security on the target runtime.
- Require a release decision that records evidence, policy version, owner, approval, candidate identity, and rollback target.
- Canary or shadow the immutable artifact, monitor data and prediction behavior, and retrain only through another versioned experiment.
Common failure signals
- Search performance is far above final-test or production performance.
- The same customer, patient, device, account, or document appears across folds.
- Training rows occur after validation rows for a future-prediction workload.
- Feature computation uses data that arrives after the prediction timestamp.
- Overall accuracy improves while the costly class or protected slice deteriorates.
- Repeated test-set checks guide feature, threshold, or search-space changes.
- A candidate cannot be reconstructed from its stored data, code, configuration, and environment identities.
More trials increase the number of comparisons against validation data. They do not repair an invalid split, make the metric business-aligned, or grant permission to reuse the final test set.