> For the complete documentation index, see [llms.txt](https://json007.gitbook.io/svm/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://json007.gitbook.io/svm/boosting/gradient_boosting.md).

# gradient boosting

优化目标：m个模型叠加起来最小化损失函数 $${{\beta\_m,\alpha\_m}}*{m=1}^M = \arg \min L(y, \sum*{m=1}^M \beta\_m h(x,\alpha\_m))$$ ,\
设$$F\_m(x) = F\_{m-1}(x) + \beta\_m h(x,\alpha\_m) = \sum\_{i=1}^m \beta\_m h(x,\alpha\_m)$$ ，\
则$$\beta\_m,\alpha\_m = \arg \min L(y, F\_{m-1}(x) + \beta\_m h(x,\alpha\_m))$$

每一步新建的模型使得损失函数下降为当前步的负梯度方向。\
$$g\_m(x) = - \[\frac {\partial L(y,F\_m(x))}{\partial F\_m(x)} ]\_{F\_m(x)}$$,

为了使当前模型能够与负梯度方向一致，故优化该式子\
$$\beta\_m ,\alpha\_m = \arg \min (g\_m(x) - \beta\_m h(x,\alpha\_m) )^2$$\
最后叠加到全部模型$$F\_m(x) = F\_{m-1}(x) + \beta\_m h(x,\alpha\_m)$$

![Gradient Boosting](https://2270971654-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M7DcNFhVrwIk3Tks_pB%2Fsync%2F5fe0f585ac19158815e2363e24766fc6fdeee4c3.png?generation=1589383933174021\&alt=media)

台湾大学的讲义：\
![](https://2270971654-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M7DcNFhVrwIk3Tks_pB%2Fsync%2Fb04cfb2c5815e87ac38047a247ea45afe7fa3d04.png?generation=1589383932448339\&alt=media)\
![](https://2270971654-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M7DcNFhVrwIk3Tks_pB%2Fsync%2F0557caeb49f04100df5f26456faf586f350827fe.png?generation=1589383933081820\&alt=media)

## GBDT(Gradient Boosting Decision Tree)

在gradient boosting框架下，每个基模型都是决策树。\
![GBDT](https://2270971654-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M7DcNFhVrwIk3Tks_pB%2Fsync%2F5e44bf6f954f63829bca44c087df495de4b5043a.jpg?generation=1589383932782969\&alt=media)\
![](https://2270971654-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M7DcNFhVrwIk3Tks_pB%2Fsync%2F907ff74895602e5e49f63d648a0950e58346935d.png?generation=1589383932765183\&alt=media)

**每添加一个模型，朝着误差的负梯度方向下降，不能朝着牛顿方向下降吗？**

> 泰勒展开
>
> $$f ( x ) = \frac { f \left( x \_ { 0 } \right) } { 0 ! } + \frac { f ^ { \prime } \left( x \_ { 0 } \right) } { 1 ! } \left( x - x \_ { 0 } \right) + \frac { f ^ { \prime \prime } \left( x \_ { 0 } \right) } { 2 ! } \left( x - x \_ { 0 } \right) ^ { 2 } + \ldots + \frac { f ^ { ( n ) } \left( x \_ { 0 } \right) } { n ! } \left( x - x \_ { 0 } \right) ^ { n } + R \_ { n } ( x )$$
>
> 常用的
>
> $$\begin{array} { l } { e ^ { x } = 1 + \frac { 1 } { 1 ! } x + \frac { 1 } { 2 ! } x ^ { 2 } + \frac { 1 } { 3 ! } x ^ { 3 } + o \left( x ^ { 3 } \right) } \ { \ln ( 1 + x ) = x - \frac { 1 } { 2 } x ^ { 2 } + \frac { 1 } { 3 ! } x ^ { 3 } + o \left( x ^ { 3 } \right) } \ { \sin x = x - \frac { 1 } { 3 ! } x ^ { 3 } + \frac { 1 } { 5 ! } x ^ { 5 } + \circ \left( x ^ { 5 } \right) } \ { \arcsin x = x + \frac { 1 } { 2 } \times \frac { x ^ { 3 } } { 3 } + \frac { 1 \times 3 } { 2 \times 4 } \times \frac { x ^ { 5 } } { 5 } + \frac { 1 \times 3 \times 5 } { 2 \times 4 \times 6 } \times \frac { x ^ { 7 } } { 7 } + \circ \left( x ^ { 7 } \right) } \ { \cos x = 1 - \frac { 1 } { 2 ! } x ^ { 2 } + \frac { 1 } { 4 ! } x ^ { 4 } + o \left( x ^ { 4 } \right) } \ { \frac { 1 } { 1 - x } = 1 + x + x ^ { 2 } + \frac { a ( a - 1 ) } { 2 ! } x ^ { 2 } + \frac { a ( a - 1 ) ( a - 2 ) } { 3 ! } x ^ { 3 } + o \left( x ^ { 3 } \right) } \end{array}$$

## LambdaMART

在排序中，大神直接定义了每步优化的梯度！！！然后套用了gradient boosting框架，就有了LambdaMART。

### 参考佳文

[Gradient Boosting](http://www.cnblogs.com/LeftNotEasy/archive/2011/01/02/machine-learning-boosting-and-gradient-boosting.html)\
[GBDT(Gradient Boosting Decision Tree) 没有实现只有原理](http://blog.csdn.net/dark_scope/article/details/24863289)\
台湾大学林轩田老师的讲课很清晰容易理解。

[GBM之GBRT总结](http://nanjunxiao.github.io/2015/08/05/GBM之GBRT总结/)

[GBDT 一个Python代码的解释](http://nbviewer.jupyter.org/github/liudragonfly/GBDT/blob/master/GBDT.ipynb)

[梯度提升算法](http://nbviewer.jupyter.org/github/liudragonfly/GBDT/blob/master/GBDT.ipynb)