Closed Form Solution of Kullback Leibler Divergence between two Gaussians

The Kullback Leibler Divergence has a closed form solution for two Gaussians. Here I will shortly derive this solution. If you need an intuitive refresher of the KL divergence you can have a look at a previous post of mine.

Assume two Gaussian distributions

– $q(x) \sim \mathcal{N}(\mu_q, \sigma_q)$

and

– $p(x) \sim \mathcal{N}(\mu_p, \sigma_p)$

The Kullback-Leibler Divergence between two Gaussians is defined as

$\begin{aligned}\text{KL}[q(x) \mathrel{\Vert} p(x)]&= \mathbb{E}_{x \sim q(x)} \Biggl[ \log \frac{q(x)}{p(x)} \Biggr] \\&= \mathbb{E}_{x \sim q(x)} \Biggl[ \log q(x) - \log p(x) \Biggr] \nonumber \\&= \mathbb{E}_{x \sim q(x)} \Biggl[ \log q(x) \Biggr] - \mathbb{E}_{x \sim q(x)} \Biggl[ \log p(x) \Biggr] \nonumber \\&= \int q(x) \log q(x) \text{ d}x - \int q(x) \log p(x) \text{ d}x \textbf{ (1)}\end{aligned}$

Preparation

The log of a Gaussian is

$\begin{aligned}\log \mathcal{N}(\mu, \sigma^2)&= \log \frac{\exp (-\frac{(x - \mu)^2}{2 \sigma^2})}{\sqrt{2 \pi \sigma^2}} \nonumber \\&= - \frac{1}{2} \frac{(x - \mu)^2}{\sigma^2} - \log \sqrt{2 \pi \sigma^2} \nonumber \\&= - \frac{1}{2} \Biggl[ \frac{(x - \mu)^2}{\sigma^2} + \log 2 \pi \sigma^2 \Biggr]\end{aligned}$

Term 1

Let’s simplify the first term of equation (1).

$\begin{aligned}\int q(x) \log q(x) \text{ d}x&= \mathbb{E}_{x \sim q(x)} \Biggl[ \log q(x) \Biggr]\\&= \mathbb{E}_{x \sim q(x)} \Biggl[ \log \mathcal{N}(\mu_q, \sigma^2_q)\Biggr]\\&= \mathbb{E}_{x \sim q(x)} \Biggl[ - \frac{1}{2} \Biggl[ \frac{(x - \mu_q)^2}{\sigma^2_q} + \log 2 \pi \sigma^2_q \Biggr] \Biggr]\\&= - \frac{1}{2} \mathbb{E}_{x \sim q(x)} \Biggl[ \frac{(x - \mu_q)^2}{\sigma^2_q} + \log 2 \pi \sigma^2_q \Biggr]\\&= - \frac{1}{2} \Biggl[ \mathbb{E}_{x \sim q(x)} \Biggl[ \frac{(x - \mu_q)^2}{\sigma^2_q} \Biggr] + \mathbb{E}_{x \sim q(x)} \Biggl[ \log 2 \pi \sigma^2_q \Biggr] \Biggr]\\&= - \frac{1}{2} \Biggl[ \mathbb{E}_{x \sim q(x)} \Biggl[ \frac{(x - \mu_q)^2}{\sigma^2_q} \Biggr] + \log 2 \pi \sigma^2_q \Biggr]\\&= - \frac{1}{2} \Biggl[ \frac{1}{\sigma^2_q} \mathbb{E}_{x \sim q(x)} \Biggl[ (x - \mu_q)^2 \Biggr] + \log 2 \pi \sigma^2_q \Biggr]\\&= - \frac{1}{2} \Biggl[ \frac{1}{\sigma^2_q} \sigma^2_q + \log 2 \pi \sigma^2_q \Biggr]\\&= - \frac{1}{2} \Biggl[ 1 + \log 2 \pi \sigma^2_q \Biggr] \textbf{ (2)}\end{aligned}$

In line 7 we used $\mathbb{E}_{x \sim p(x)} [ (x - \mu)^2 ] = \int p(x) (x - \mu)^2 \text{ d}x = \sigma^2$ .

Term 2

Now we need to simplify the second term of equation (1).

$\begin{aligned}\int q(x) \log p(x) \text{ d}x&= \mathbb{E}_{x \sim q(x)} \Biggl[ \log p(x) \Biggr] \\&= \mathbb{E}_{x \sim q(x)} \Biggl[ \log \mathcal{N}(\mu_p, \sigma^2_p)\Biggr] \\&= \mathbb{E}_{x \sim q(x)} \Biggl[ - \frac{1}{2} \Biggl[ \frac{(x - \mu_p)^2}{\sigma^2_p} + \log 2 \pi \sigma^2_p \Biggr] \Biggr] \\&= - \frac{1}{2} \mathbb{E}_{x \sim q(x)} \Biggl[ \frac{(x - \mu_p)^2}{\sigma^2_p} + \log 2 \pi \sigma^2_p \Biggr] \\&= - \frac{1}{2} \Biggl[ \mathbb{E}_{x \sim q(x)} \Biggl[ \frac{(x - \mu_p)^2}{\sigma^2_p} \Biggr] + \mathbb{E}_{x \sim p(x)} \Biggl[ \log 2 \pi \sigma^2_p \Biggr] \Biggr] \\&= - \frac{1}{2} \Biggl[ \mathbb{E}_{x \sim q(x)} \Biggl[ \frac{(x - \mu_p)^2}{\sigma^2_p} \Biggr] + \log 2 \pi \sigma^2_p \Biggr] \\&= - \frac{1}{2} \Biggl[ \frac{1}{\sigma_p^2} \mathbb{E}_{x \sim q(x)} \Biggl[ (x - \mu_p)^2 \Biggr] + \log 2 \pi \sigma^2_p \Biggr]\end{aligned}$

Let’s now just take a look at the expectation of the last equation

$\begin{aligned}\mathbb{E}_{x \sim q(x)} [ (x - \mu_p)^2 ]&= \mathbb{E}_{x \sim q(x)} [ x^2 - 2 x \mu_p + \mu_p^2 ]\\&= \mathbb{E}_{x \sim q(x)} [ x^2 ] - \mathbb{E}_{x \sim q(x)} [ 2x \mu_p ] + \mu^2_p\\&= \mu_q^2 + \sigma_q^2 - 2 \mu_p \cdot \mathbb{E}_{x \sim q(x)} [ x ] + \mu^2_p\\&= (\mu_q^2 - 2 \mu_p \mu_q + \mu^2_p) + \sigma_q^2\\&= (\mu_q - \mu_p)^2 + \sigma_q^2\end{aligned}$

In line 2 we used the definition of variance

$Var(x) = \mathbb{E} [x^2] - \mathbb{E} [x]^2 \Rightarrow \mathbb{E} [x^2] = \mathbb{E}[x]^2 + Var(x)$

In line 4 we used the binomial theorem.

Now let’s insert the result back so that we get

$\begin{aligned}\int q(x) \log p(x) \text{ d}x&= ...\nonumber \\&=- \frac{1}{2} \Biggl[ \frac{1}{\sigma_p^2} \mathbb{E}_{x \sim q(x)} \Biggl[ (x - \mu_p)^2 \Biggr] + \log 2 \pi \sigma^2_p \Biggr]\nonumber \\&= - \frac{1}{2} \Biggl[ \frac{(\mu_q - \mu_p)^2 + \sigma_q^2}{\sigma_p^2} + \log 2 \pi \sigma^2_p \Biggr] \textbf{ (3)}\end{aligned}$

Final

Finally we can insert the results from equation (2) and equation (3) into equation (1).

$\begin{aligned}\text{KL}[q(x) \mathrel{\Vert} p(x)]&= \int q(x) \log q(x) \text{ d}x - \int q(x) \log p(x) \text{ d}x\nonumber \\&=- \frac{1}{2} \Biggl[ 1 + \log 2 \pi \sigma^2_q \Biggr]+ \frac{1}{2} \Biggl[ \frac{(\mu_q - \mu_p)^2 + \sigma_q^2}{\sigma_p^2} + \log 2 \pi \sigma^2_p \Biggr]\nonumber \\&=- \frac{1}{2} - \frac{1}{2}\log 2 \pi \sigma^2_q+ \frac{(\mu_q - \mu_p)^2 + \sigma_q^2}{2\sigma_p^2}+ \frac{1}{2} \log 2 \pi \sigma^2_p\nonumber \\&=\frac{1}{2} \Biggl[ \log 2 \pi \sigma^2_p - \log 2 \pi \sigma^2_q \Biggr]+ \frac{(\mu_q - \mu_p)^2 + \sigma_q^2}{2\sigma_p^2}- \frac{1}{2}\nonumber \\&=\frac{1}{2} \Biggl[ \log \frac{2 \pi \sigma^2_p}{2 \pi \sigma^2_q} \Biggr]+ \frac{(\mu_q - \mu_p)^2 + \sigma_q^2}{2\sigma_p^2}- \frac{1}{2}\nonumber \\&=\frac{1}{2} \Biggl[ \log \frac{\sigma^2_p}{\sigma^2_q} \Biggr]+ \frac{(\mu_q - \mu_p)^2 + \sigma_q^2}{2\sigma_p^2}- \frac{1}{2}\nonumber \\&=\Biggl[ \log \sqrt{\frac{\sigma^2_p}{\sigma^2_q}} \Biggr]+ \frac{(\mu_q - \mu_p)^2 + \sigma_q^2}{2\sigma_p^2}- \frac{1}{2}\nonumber \\&= \log \frac{\sigma_p}{\sigma_q}+ \frac{(\mu_q - \mu_p)^2 + \sigma_q^2}{2\sigma_p^2}- \frac{1}{2}\end{aligned}$

Check

Let’s check whether this reduces to zero if we compare the same two Gaussian distributions.

If $p(x) := q(x)$ then $\sigma_p = \sigma_q$ and $\mu_p = \mu_q$ , so that

$\begin{aligned}\log \frac{\sigma_q}{\sigma_q} = \log 1 = 0\nonumber \\\frac{(\mu_q - \mu_q)^2 + \sigma_q^2}{2\sigma_q^2} = \frac{\sigma_q^2} {2\sigma_q^2} = \frac{1}{2}\end{aligned}$

and finally

$\begin{aligned}\text{KL}[q(x) \mathrel{\Vert} q(x)] = 0 + \frac{1}{2} - \frac{1}{2} = 0.\end{aligned}$