> For the complete documentation index, see [llms.txt](https://json007.gitbook.io/svm/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://json007.gitbook.io/svm/lr/softmax.md).

# softmax

Logistic Regession宗旨是以线性分割面分割各类别 $$\log \frac {Pr(Y=i|x)}{ Pr(Y=j|x)}$$ .同时保持各类别的概率和为1.

LR依从 **bayesian** 法则，即它对样本的分类是看哪个类别的概率最大:

$$
G(x) = \arg \max\_k Pr(Y=k|x)
$$

因此类别i和j的分界面由决定

$$
Pr(Y=i|x) =  Pr(Y=j|x)
$$

**各类别概率**

$$
\log \frac{P(Y=1|x)}{P(Y=k|x)} = w\_1 \cdot x  \\
\log \frac{P(Y=2|x)}{P(Y=k|x)} = w\_2 \cdot x  \\
..... \\
\log \frac{P(Y=k-1|x)}{P(Y=k|x)} = w\_{k-1} \cdot x  \\
\\
P(Y=1|x) = P(Y=k|x) \exp(w\_1 \cdot x) \\
P(Y=2|x) = P(Y=k|x) \exp(w\_2 \cdot x) \\
..... \\
P(Y=k-1|x) = P(Y=k|x) \exp(w\_{k-1} \cdot x) \\
\\
P(Y=k|x) = \frac {1}{1+\sum\_{k=1}^{K-1} \exp(w\_k \cdot x)} \\
P(Y=1|x) = \frac {exp(w\_1 \cdot x)}{1+\sum\_{k=1}^{K-1} \exp(w\_k \cdot x)} \\
P(Y=2|x) = \frac {exp(w\_2 \cdot x)}{1+\sum\_{k=1}^{K-1} \exp(w\_k \cdot x)} \\
$$

这个过程都是$$P(Y=k|x)$$倍数，若

$$
P(Y=k|x) = \frac {\exp(w\_k \cdot x)}{\sum\_{k=1}^{K} \exp(w\_k \cdot x)} \\
\text{则} \\
P(Y=1|x) = \frac {\exp(w\_1 \cdot x)}{\sum\_{k=1}^{K} \exp(w\_k \cdot x)} \\
P(Y=2|x) = \frac {\exp(w\_2 \cdot x)}{\sum\_{k=1}^{K} \exp(w\_k \cdot x)} \\
$$

则，这就是softmax。

> 怎么感觉像无向图模型中一般用的势能函数一样。给每个类别赋一个势能，然后归一化。然后求势能最大的那个类别。

所以

$$
p\_j = \frac {\exp(w\_jx)} {\sum\_i^K \exp(w\_i x)}
$$

## softmax loss

一般分类的估计就是使用（负对数）极大似然估计（negative log-likelihood ）， 二分类的最大似然就是交叉熵。

$$
L(f(x\_i,\theta),y\_i) = -\frac {1}{m} \log \prod\_{i=1}^m \prod\_{k=1}^K p\_{i,k}^{1(y\_i = k)} + \lambda \sum\_{k=1} |\theta\_k| \\
\= -\frac {1}{m} \sum\_{i=1}^m \sum\_{k=1}^K {1(y\_i = k)} \log \frac {\exp(f(x\_i,\theta))} {\sum\_j \exp(f(x\_i,\theta))} + \lambda \sum\_{k=1} |\theta\_k| \\
$$

来个简化版的：

$$
L(x\_i,\theta\_k,y\_i) = -\frac {1}{m} \log \prod\_{i=1}^m \prod\_{k=1}^K p\_{i,k}^{1(y\_i = k)}  \\
\= -\frac {1}{m} \sum\_{i=1}^m  {1(y\_i = k)} \log \frac {\exp(x\_i \theta\_k)} {\sum\_j \exp(x\_i \theta\_k)}  \\
$$

求导得：

$$
\frac {\partial L}{\partial \theta\_k} = -\frac {1}{m} \sum\_{i=1}^m x\_i \[ {1(y\_i = k)} - p(y\_i = k | x\_i; \theta\_k) ]
$$

### 向量化表示

$$
h(\eta) = \frac {1}{\sum\_k e^{\eta\_k}} \begin{bmatrix}
e^{\eta\_1}      \\
e^{\eta\_2}      \\
\vdots \\
e^{\eta\_K}
\end{bmatrix} \\
J = -\sum\_{k=1}^K 1{y=k} \log(h\_k) \\
\frac {\partial J}{\partial \eta} = h-e\_y \quad e\_y \text{是第y个元素为1，其余为0的列向量}
$$

然后对于全部样本，优化函数为：

$$
L = -\sum\_{n=1}^N e\_y \log(h(\eta)) \\
\frac {\partial L}{\partial \eta} = \sum\_{n=1}^N (h-e\_y) \\
$$

### 参考佳文

[探究最陌生的老朋友Softmax](https://mp.weixin.qq.com/s?__biz=MzA3Mjk0OTgyMg==\&mid=2651123524\&idx=1\&sn=0546ceca3d88e2ff1e66fbecc99bd6a7)

[Softmax vs. Softmax-Loss: Numerical Stability](http://freemind.pluskid.org/machine-learning/softmax-vs-softmax-loss-numerical-stability/)\
[softmax回归](http://blog.csdn.net/acdreamers/article/details/44663305)\
[Softmax回归](http://deeplearning.stanford.edu/wiki/index.php/Softmax回归)\
[Caffe Softmax层的实现原理](https://www.zhihu.com/question/28927103)

[Softmax与交叉熵的数学意义](https://zhuanlan.zhihu.com/p/78711135)

[ArcFace，CosFace，SphereFace，三种人脸识别算法的损失函数的设计](https://zhuanlan.zhihu.com/p/285598652)