Log probabilities

Jan 7, 2017 • numerical • 0

Problem

Sometimes we need to sum two probabilities that are in log-space. If we let $a$ and $b$ be the two log probabilities then the straight-forward method to compute the sum of the probabilities in log-space, $p$ , would be

\[ p = \log (e^a + e^b). \]

This can result in numerical instability (underflow) when $a$ or $b$ are large and negative.

Show that another way to compute $p$ is given by

\[ p = \max\{a, b\} + \log (1 + e^{-|a - b|}). \]

The function $\log(1 + x)$ can be computed efficiently and precisely in C++ with log1p.

Solution

show

First we show that $p = a + \log(1 + e^{b-a})$ . \begin{align} \log (e^a + e^b) &= \log (e^a (1 + e^{b-a})) \\ &= a + \log(1 + e^{b-a}). \end{align} Symetrically we also have $p = b + \log(1 + e^{a - b})$ . Now observe that $-|b - a| = min\{a, b\} - \max\{a, b\}$ . Thus we have \begin{align} p = \max\{a, b\} + \log(1 + e^{-|a - b|}). \end{align}