> For the complete documentation index, see [llms.txt](https://json007.gitbook.io/svm/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://json007.gitbook.io/svm/me/iis.md).

# IIS

一般介绍最大熵的文章都会IIS。凸优化中迭代方法有很多，目前收敛速度比较快的是拟牛顿法，常用L-BFGS，没必要用IIS，但是IIS的思想可以学习学习。

$$
\begin{align}
L\_{\widetilde p}(P\_w) &= \log \prod\_{x,y} P(y|x)^{\widetilde P(x,y)} \\
&= \sum\_{x,y} \widetilde P(x,y) \log P(y|x) \\
&= \sum\_{x,y} \widetilde P(x,y) \log (\exp(\sum\_{i=1}^n w\_i f\_i(x,y)) / Z\_w(x)) \\
&= \sum\_{x,y} \widetilde P(x,y) \sum\_{i=1}^n w\_i f\_i(x,y) -  \sum\_x \widetilde P(x) \log Z\_w(x) \\
\end{align}
$$

$$
\begin{align}
L(w+\delta) - L(w) &= \sum\_{x,y} \widetilde P(x,y) \sum\_{i=1}^n \delta\_i f\_i(x,y) - \sum\_x \widetilde P(x) \log (Z\_{w+\delta}(x)/Z\_w(x)) \\
& \ge \sum\_{x,y} \widetilde P(x,y) \sum\_{i=1}^n \delta\_i f\_i(x,y) + \sum\_x \widetilde P(x) (1- Z\_{w+\delta}(x)/Z\_w(x)) \qquad \because (-log x \ge 1-x )\\
&= \sum\_{x,y} \widetilde P(x,y) \sum\_{i=1}^n \delta\_i f\_i(x,y) + 1 - \sum\_x \widetilde P(x) (Z\_{w+\delta}(x)/Z\_w(x)) \\
&= \sum\_{x,y} \widetilde P(x,y) \sum\_{i=1}^n \delta\_i f\_i(x,y) + 1 - \sum\_x \widetilde P(x) \sum\_y P\_w(y|x) \exp(\sum\_{i=1}^n \delta\_i f\_i(x,y)) \\
\end{align}
$$

$$
\text{令} f^{#} (x,y) = \sum\_{i=1}^n f\_i(x,y) \\
\exp(\sum\_{i=1}^n \delta\_i f\_i(x,y)) = \exp(\sum\_{i=1}^n \frac {f\_i(x,y)}{f^{#} (x,y)} \delta\_i f^{#} (x,y)) \le \sum\_{i=1}^n \frac {f\_i(x,y)}{f^{#} (x,y)} \exp(\delta\_i f^{#} (x,y))
$$

$$
L(w+\delta) - L(w) \ge \sum\_{x,y} \widetilde P(x,y) \sum\_{i=1}^n \delta\_i f\_i(x,y) + 1 - \sum\_x \widetilde P(x) \sum\_y P\_w(y|x) \sum\_{i=1}^n \frac {f\_i(x,y)}{f^{#} (x,y)} \exp(\delta\_i f^{#} (x,y))
$$