# Maximum entropy probability distribution

一般形式：

$$
\begin{array}{lcl}
\int\_{-\infty}^\infty p(x) dx &=& 1 \\
E(f\_j(x)) &=& \int\_{-\infty}^\infty f\_j(x) p(x) dx=a\_j \\
H(x) &=& - \int\_{-\infty}^\infty p(x) \ln p(x) dx \\
J(p(x)) &=& - \int\_{-\infty}^\infty p(x) \ln p(x) dx + \lambda\_0 (\int\_{-\infty}^\infty p(x) dx -1) + \sum\_j \lambda\_j (\int\_{-\infty}^\infty f\_j(x) p(x) dx- a\_j) \\
\frac {\partial J}{\partial p(x)} &=& -\ln p(x) -1 + \lambda\_0 + \sum\_j \lambda\_j f\_j(x) = 0 \\
p(x) &=& e^{-1+\lambda\_0 } e^{\sum\_j \lambda\_j f\_j(x)}
\end{array}
$$

看最后一个式子的形式，指数分布族。目前之前只知道，指数分布族在理论上一定有共轭先验。

$$
\begin{array}{lcl}
\Gamma(x) &=& \int\_0^{+\infty} e^{-t} t^{x-1} dt \quad  (x \gt 0)  = (x-1)! \qquad \text{这是Gamma函数，不是Gamma分布} \\
\psi(x) &=& \frac {d}{d x} \ln \Gamma(x) = \frac {\Gamma^\prime (x)} {\Gamma(x)} \qquad \text{digamma function}  \\
B(p,q) &=& \frac {\Gamma(p) \Gamma(q)}{\Gamma(p+q)} \qquad \text{beta function}  \\
\gamma\_E  \qquad \text{Euler's constant} \\
\end{array}
$$

![](https://2270971654-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-M7DcNFhVrwIk3Tks_pB%2Fsync%2Feb94996c4375a555babc691d32421dc7116c19f8.png?generation=1589383945433570\&alt=media)

[给定前k阶矩的最大熵分布是什么？](https://www.zhihu.com/question/30458484)

有人思考过对一阶矩和二阶矩的限制是自然的吗？\
[为什么正态分布在自然界如此常见？](https://www.zhihu.com/question/26854682)\
[Central limit theorem](https://en.wikipedia.org/wiki/Central_limit_theorem)

## 正态分布

[是否许多变量可以用正态分布很好地描述？](http://www.zhihu.com/question/19910173) 同样过过程，只不过是用了变分法。

[变分法简介](http://www.cnblogs.com/murongxixi/p/3995788.html) 最后一部分

## gamma

约束条件（即均值和几何平均值）

$$
\int\_{-\infty}^\infty p(x) dx = 1 \\
E(x) = k \theta \\
E(\ln(x)) = \psi(k) + \ln(\theta) \\
$$

用乘子法构造拉格朗日函数去最大化熵

$$
\begin{align}
J(p(x)) = & - \int\_{-\infty}^\infty p(x) \ln p(x) dx \\
& + \lambda\_0 (\int\_{-\infty}^\infty p(x) dx - 1) \\
& + \lambda\_1 (\int\_{-\infty}^\infty p(x)x dx - k) \\
& +  \lambda\_2 (\int\_{-\infty}^\infty p(x) \ln(x) dx- \psi(k) - \ln(\theta)) \\
\end{align}  \\

\frac {\partial J}{\partial p(x)} = -\ln p(x) -1 + \lambda\_0 + \lambda\_1 x + \lambda\_2 \ln(x) = 0 \\
p(x) = \exp(\lambda\_0 -1 + \lambda\_1 x + \lambda\_2 \ln(x)) = e^{\lambda\_0 -1}  x ^ {\lambda\_2} e^{\lambda\_1 x} \\
$$

或者用变分原理+乘子法

$$
\delta (H(p(x))) + \delta \lambda\_0 (\int\_{-\infty}^\infty p(x) dx ) + \delta \lambda\_1 (\int\_{-\infty}^\infty p(x)x dx ) + \delta \lambda\_2 (\int\_{-\infty}^\infty p(x) \ln(x) dx ) = 0 \\
p(x) = \exp(\lambda\_0 -1 + \lambda\_1 x + \lambda\_2 \ln(x)) = e^{\lambda\_0 -1}  x ^ {\lambda\_2} e^{\lambda\_1 x} \\
$$

求解

$$
\int\_0^\infty x^n e^{-ax} dx = \frac {n!}{a^{n+1}}  \\
\int\_{-\infty}^\infty p(x) dx = 1 \\
\Rightarrow e^{\lambda\_0 -1} = \frac {(\lambda\_0 -1)^{\lambda\_2+1}}{\Gamma(\lambda\_2+1)} \\
$$

最后得：

$$
p(x) = \frac {1}{\theta^k \Gamma(k)} x^{k-1} e^{- \frac {x}{\theta}}  \\
\text{实在是凑不出来}\theta
$$

## power-law

幂率是几何平均约束下的最大熵分布

$$
1 = \int\_{x\_{min}}^\infty p(x) dx = \int\_{x\_{min}}^\infty x^{-a} dx = \frac {C}{1-a} \[x^{-a+1}]*{x*{min}}^\infty \qquad a \gt 1 \\
C = (a-1) x\_{x\_{min}}^\infty \\
p(x) = \frac {a-1}{x\_{min}} (\frac {x}{x\_{min}})^{-a} \\
$$

几何平均值\
离散：几何平均值m的含义是N个个体的变量x的连续乘积再开N次方

$$
m^n = x\_1^{n\_1} \cdot x\_2^{n\_2} \cdot \ldots \cdot x\_n^{n\_n}\\
N \ln m = n\_1 \ln x\_1 + n\_2 \ln x\_2 + \ldots + n\_n \ln x\_n = \sum n\_j \ln x\_j
$$

连续：

$$
\ln m = \int\_a^b p(x) \ln x dx
$$

构造拉格朗日函数

$$
F = -\int p(x) \ln p(x) d x + \lambda\_0 (\int p(x) dx - 1) + \lambda\_1 (\int p(x) \ln x dx - \ln m) \\

\frac {\partial J}{\partial p(x)} = -\ln p(x) -1 + \lambda\_0 +  \lambda\_1 \ln(x) = 0 \\
p(x) = e^{-1 + \lambda\_0} x^{\lambda\_1}
$$

[Power law](https://en.wikipedia.org/wiki/Power_law)\
[人类行为服从的幂律分布是否违背了中心极限定理？](https://www.zhihu.com/question/30148415)\
[人类行为时空特性的统计力学（一）——认识幂律分布](http://zhuanlan.zhihu.com/prml-paper-reading/19811289)

### 参考佳文

[Maximum entropy probability distribution](https://en.wikipedia.org/wiki/Maximum_entropy_probability_distribution)\
[Exponential family](https://en.wikipedia.org/wiki/Exponential_family)\
张学文的《组成论》 190页