Unbiased property of the ELBO¶

In order to be able to perform stochastic optimisation of the Monte Carlo approximation of the ELBO, we need to have the property that the approximation is unbiased. This is a short post reviewing that idea, and showing that this is indeed the case.

Deriving the bias of an estimator¶

If we have an estimator $\hat{\theta}$, this is called an unbiased estimator if $E[\hat{\theta}] - \theta$ is zero (where $\theta$ is the true quantity). For a stochastic approximation, if our estimator is unbiased, we get convergence guarentees under the somewhat mild conditions (Robbins, Monro 1951).

However, if it is biased, we may have convergence issues. Intuitively, we can see this below;

If $E[\hat{L}] \neq L$, then clearly it is not necessarily true that $ \frac{d}{d\theta}E[\hat{L}] \neq \frac{d}{d\theta}L$. This means that for some arbitary loss function, if we perform an SGD like approximation many times over, our adjustments, based on the gradient, don't necessarily move us in the correct direction.

If we view optimisation as moving us 'downhill' to some minimum point on our loss surface, if the gradient is a biased estimate, we do not have a general guarantee that this biased estimate will always point in the correct direction in expectation. For an unbiased estimate, we can view this as saying that whilst any single estimate will be wrong, if we take many estimates such as when we perform SGD, the resulting movement on the surface will likely be quite close to the true optimal move.

This can either affect the speed of convergence, or even whether the algorithm will converge properly at all.

A simple example¶

A common example of an unbiased estimate is taking the average as an estimate of the expectation. The proof is below;

$E[\frac{1}{N}\sum_{i=1}^{N} x_{i}] = \frac{1}{N}\sum_{i=1}^{N} E[x_{i}]$

$=\frac{1}{N} N \ E[X] = E[X]$

Unbiased ELBO¶

$\mathcal{L} = E_{q}[log \frac{P(X,Z)}{q(z)}]$

$\hat{\mathcal{L}} = \frac{1}{N}\sum_{i}^{N}[log \frac{P(X,Z_{i})}{q(Z_{i})}]$

Now, assuming IID draws from q, we know $Z_{i} \sim q$.

$E_{q}[\hat{\mathcal{L}}] = E_{q}[\frac{1}{N}\sum_{i}^{N}log \frac{P(X,Z_{i})}{q(Z_{i})}]$

$= \frac{1}{N}\sum_{i}^{N}E_{q}[log \frac{P(X,Z_{i})}{q(Z_{i})}]$

$= \frac{1}{N}\sum_{i}^{N}[ \int q(Z_{i}) log \frac{P(X,Z_{i})}{q(Z_{i})} dZ_{i}]$

For each of the terms of the sum, the integral is the same, as the integral is over the whole domain over $Z_{i}$. This means the value of each term is independent of the subscript. We can also reach the same conclusion by recognising that as the terms are IID, they all have the same expectation, so;

$= \frac{1}{N}N \ E_{q}[log \frac{P(X,Z)}{q(Z)}] =E_{q}[log \frac{P(X,Z)}{q(Z)}] $

Summary¶

This post was a quick note to prove the Monte Carlo ELBO is an unbiased approximation of the true ELBO function.