Motivation

$\color{black}\rule{365px}{3px}$

Model is highly dependent on the dataset it is trained on!

</aside>

You can see the model is highly overfit to two different sampling (i.e. High model variance). Since model’s prediction varies a lot on different datasets.

As we’ve seen earlier, a fully-grown decision tree will almost always overfit the data. It has low model bias, but high model variance. In other words, small changes in the dataset will result in very a different decision tree. As an example, the two models below are trained on different subsets of the same data. We can see they come out very differently.

Bagging (Bootstrap AGGregatING):

$\color{black}\rule{365px}{3px}$

Technique Examples: Random Forests
Parallel Processing: In Bagging, multiple base models (usually decision trees) are trained independently and in parallel on different subsets of the data.
Reduces Variance: By averaging the predictions of multiple models, Bagging reduces the variance of the overall model, leading to more stable and generalizable predictions. Random Forests is a common example of Bagging, where many decision trees are trained on bootstrapped datasets, and their predictions are aggregated (e.g., majority voting for classification).