Bootstrap aggregation, or bagging, is an ensemble method that trains several copies of a model on random samples of the training data, drawn with replacement, and aggregates their predictions to reduce variance. It works best on flexible models such as fully grown decision trees.
This page is for data scientists and technology leaders who want to know what bagging does, how it relates to random forests and boosting, and what to settle before using it in models that feed AI agents.
What is bootstrap aggregation?
The scikit-learn ensemble guide defines bagging methods as a class of algorithms that build several instances of a black-box estimator on random subsets of the original training set and then aggregate their individual predictions into a final prediction. When samples are drawn with replacement, the method is known as bagging.
The word bootstrap refers to that sampling with replacement: each model sees a slightly different version of the data, so its mistakes differ from those of the others.
Why does bagging reduce variance?
According to the guide, these methods reduce the variance of a base estimator, for example a decision tree, by introducing randomisation into its construction and then making an ensemble out of it. Because they reduce overfitting, bagging methods work best with strong and complex models such as fully developed decision trees.
A model with high variance changes a lot when the training data changes slightly. Combining many such models smooths out those swings, which is the stability gain that gives the method its value.
How does bagging work in practice?
The same guide describes the main controls: the size of the sample and feature subsets, and whether samples and features are drawn with or without replacement. When only part of the data is used for each model, the samples left out can estimate generalisation accuracy through out-of-bag scoring.
In practice you choose a base model, the number of copies, how much data each copy sees and how to evaluate the result. Related tuning topics are covered in our entries on hyperparameter tuning and automated machine learning.
How does bagging differ from pasting, random forests and boosting?
The scikit-learn guide lists several close relatives that differ in how they draw their subsets.
| Method | How it builds the ensemble | Note from the guide |
| Bagging | Random subsets of samples, drawn with replacement | Best with strong, complex models |
| Pasting | Random subsets of samples, drawn without replacement | A related variant |
| Random subspaces and patches | Random subsets of features, or of samples and features | Other ways to randomise the base models |
| Random forest | Each tree uses a bootstrap sample, with features also randomised at each split | Its stated purpose is to decrease variance |
| Boosting | A different ensemble family that the guide contrasts with bagging | Usually works best with weak models such as shallow trees |
How does bagging relate to AI agents?
Agents often depend on predictive models for scoring, routing or forecasting, and those models need to be stable when data shifts. Teams running agents on ElixirClaw, the AI agents computing platform from XenonStack can use bagging as one way to make such models steadier, while keeping the training data and settings recorded for review.
What should you do first with bagging?
- Pick a high-variance base model, such as a deep decision tree.
- Compare a single model against the bagged ensemble on held-out data.
- Use out-of-bag scoring as a quick check on generalisation.
- Try a random forest as a ready-made bagged option.
- Record the sampling settings and data version with the trained model.
To map one decision your agents make, and the data and policy it depends on, book a Context Workshop. See what happens in the workshop.
Frequently asked questions
What is bagging in machine learning?
Bagging builds several instances of a base estimator on random subsets of the training set, drawn with replacement, and aggregates their predictions into one. Its aim is to reduce the variance of the base estimator.
What does the bootstrap mean in bootstrap aggregation?
The bootstrap is sampling with replacement. Each model in the ensemble is trained on its own resampled version of the data, so the models make different errors that the aggregate can smooth out.
Is a random forest the same as bagging?
Not exactly. A random forest builds each tree from a bootstrap sample, as bagging does, but it also randomises the features considered at each split. Its stated purpose is to decrease variance.
When does bagging work best?
The scikit-learn guide says bagging works best with strong and complex models, such as fully developed decision trees, whereas boosting usually works best with weak models such as shallow trees.