Gradient Boosting

Ahh, gradient boosting. In addition to having a totally kickass name, this family of machine learning algorithms is currently among the best known approaches for prediction problems on structured data. This series of posts strives to give a comprehensive understanding of gradient boosting by providing intuitive mathematical explanations, from-scratch implementations of key algorithms, and examples of how to apply modern gradient boosting libraries to solve practical data science problems.

I recommend reading through the series in order, since concepts tend to build on earlier ideas.

How to Build a Gradient Boosting Machine from Scratch

python

gradient boosting

from scratch

Understand the intuition behind the gradient boosting machine (GBM) and learn how to implement it from scratch.

Get Down with Gradient Descent

gradient boosting

Get down with the intuition for gradient descent via a fresh analogy, develop the mathematical formulation of the algorithm, and implement it from scratch to train a linear regression model.

How Gradient Boosting Does Gradient Descent

gradient boosting

Understand how gradient boosting does gradient descent in function space to minimize any differentiable loss function in the service of creating a good model.

How to Implement a Gradient Boosting Machine that Works with Any Loss Function

python

gradient boosting

from scratch

Summarize Friedman’s seminal GBM paper and implement the generic gradient boosting algorithm to train models with any differentiable loss function.

Consider the Decision Tree

gradient boosting

Understand the core strengths and weaknesses of the decision tree, and see how ensembling makes trees shine.

Decision Tree from Scratch

python

gradient boosting

from scratch

A detailed walkthrough of my from-scratch decision tree implementation in python.

XGBoost Explained

gradient boosting

xgboost

In-depth explanation and mathematical derivation of the XGBoost algorithm

XGBoost from Scratch

python

gradient boosting

xgboost

from scratch

A walkthrough of my from-scratch python implementation of XGBoost.

XGBoost for Regression in Python

python

tutorial

gradient boosting

xgboost

A step-bystep tutorial on regression with XGBoost in python using sklearn and the xgboost library

Gradient Boosting Multi-Class Classification from Scratch

python

gradient boosting

from scratch

How to implement multi-class classification for gradient boosting from scratch in python

XGBoost for Binary and Multi-Class Classification in Python

python

tutorial

gradient boosting

xgboost

A step-bystep tutorial on binary and multi-class classification with XGBoost in python using sklearn and the xgboost library