# Gradient Boosted Decision Trees

> Sequential tree ensembles optimized via gradient descent

- Type: Misc
- By: Jerome Friedman
- Year: 2001
- Topics: Machine learning, Ensemble, Boosting, Trees
- Page: https://aipedia.org/gbdt

## Resources

- [Greedy Function Approximation: A Gradient Boosting Machine](https://doi.org/10.1214/aos/1013203451): Paper, Friedman, 2001
- [XGBoost: A Scalable Tree Boosting System](https://arxiv.org/abs/1603.02754): Paper, Chen & Guestrin, 2016
- [dmlc/xgboost](https://github.com/dmlc/xgboost): Code, XGBoost library
- [LightGBM: A Highly Efficient Gradient Boosting Decision Tree](https://papers.nips.cc/paper/6907-lightgbm-a-highly-efficient-gradient-boosting-decision-tree): Paper, Ke et al., 2017
- [CatBoost: unbiased boosting with categorical features](https://arxiv.org/abs/1706.09516): Paper, Prokhorenkova et al., 2017

**Gradient Boosted Decision Trees (GBDT)** are an ensemble method that builds decision trees one at a time, where each new tree is trained to correct the mistakes the current ensemble is still making. The final model is the sum of all the trees.

> This is a classical machine learning method, not deep learning — no neural networks involved. But the word "gradient" here is the same gradient as in [Backpropagation](https://aipedia.org/backpropagation): both methods follow the derivative of a loss downhill.

## Why Students Should Care

- On tabular data (spreadsheet-style rows and columns), GBDT implementations like XGBoost and LightGBM are still the method to beat — they routinely win Kaggle competitions where deep learning does not.
- GBDT is the clearest example of **boosting**: turning many weak models into one strong one.
- The "gradient descent in function space" idea is a genuinely beautiful piece of ML theory that connects trees to the optimization you know from neural networks.

## A Concrete Picture First

Suppose you are predicting house prices. Your first tree is crude: it predicts 300,000 for a house that actually sells for 370,000 — an error of 70,000. Now train a *second* tree whose job is not to predict prices, but to predict those **errors**. Add its output to the first tree's prediction, and the combined model is closer to the truth. Repeat: every new tree cleans up whatever error is left. That is boosting.

## Core Idea: Gradient Descent in Function Space

Here is the elegant generalization. GBDT performs **gradient descent over functions** rather than parameters. Instead of nudging weights, we iteratively improve a model $F(x)$ by adding a new function (a tree) that points in the direction of steepest loss reduction.

At iteration $m$:

$$
F_m(x) = F_{m-1}(x) + \eta \, h_m(x)
$$

- $h_m(x)$ is a regression tree
- $\eta$ is the learning rate

You do not need to memorize the notation. The important idea is: **the model is a running sum of trees, and each new tree is a small step downhill on the loss.**

## Loss and Gradients

Why is "predict the errors" the same as "follow the gradient"? For squared error loss:

$$
L(y, F(x)) = \frac{1}{2}(y - F(x))^2
$$

The negative gradient of the loss with respect to the prediction is:

$$
-\frac{\partial L}{\partial F(x)} = y - F(x)
$$

That is exactly the residual — the leftover error. So each tree trained to predict residuals is literally fitting the negative gradient. For other losses (classification, ranking), the residuals are replaced by that loss's gradients, and the same machinery works.

## Interactive Visualization

The demo below shows:
- Data points and ensemble prediction curve
- Residuals as vertical lines
- Individual trees contributing to the final model
- Loss decreasing as trees are added

*Interactive demo: https://aipedia.org/gbdt*

## Key Hyperparameters

- **learning_rate (η)** – step size for each tree; smaller values improve generalization (but need more trees)
- **n_estimators** – number of trees in the ensemble
- **max_depth** – controls tree complexity and interaction order (how many features a single tree can combine)

The classic trade: lower the learning rate, raise the number of trees.

## GBDT vs Random Forest

Both are tree ensembles, but they combine trees in opposite ways: random forests train many independent trees in parallel and average them; GBDT trains trees sequentially, each depending on all previous ones.

| Aspect | GBDT | Random Forest |
|------|------|---------------|
| Training | Sequential | Parallel |
| Bias–Variance | Low bias | Low variance |
| Optimization | Gradient-based | Bagging |
| Overfitting | Controlled via learning rate | Controlled via averaging |

## When to Use GBDT

- Tabular data with mixed feature types
- Medium-sized datasets
- When strong performance and interpretability matter

For images, audio, and text, deep learning dominates — but for a table of customer records, reach for GBDT first.

## Common Confusion

- **Boosting vs. bagging**: boosting (GBDT) builds trees *sequentially*, each correcting the last; bagging (random forests) builds trees *independently* and averages them.
- **Gradient boosting vs. gradient descent**: same gradient idea, different space — gradient descent updates parameters; gradient boosting adds whole functions (trees).
- **GBDT is not deep learning.** No neural networks, no [backpropagation](https://aipedia.org/backpropagation) — though both follow gradients of a loss.
- **"Residuals" are a special case.** Trees fit plain residuals only under squared error; in general they fit the negative gradient of whatever loss you chose.

## Where To Go Next

- Read [Backpropagation](https://aipedia.org/backpropagation) to compare gradient descent in parameter space with the function-space version here.
- Read [Adam](https://aipedia.org/adam) for how gradient-based optimization is tuned in deep learning.
- Read [Scaling Laws](https://aipedia.org/scaling-laws) for how deep learning behaves in the regimes where it *does* beat trees.

## Libraries

GBDT is widely available in:
- `xgboost`
- `lightgbm`
- `catboost`
- `sklearn.ensemble.GradientBoosting*`