Averaging Log-Likelihood Values:Numerical Stability

1. Weighted log-likelihood values

The problem addressed in this page is the computation of:

 \bar{l} = \frac{1}{\sum_i \omega_i} \sum_i \omega_i \cdot l_i

It’s very advisable to employ log-likelihoods (and log-weights) for a extended dynamic range. This avoids the problem of weights becoming zero.

Then, if we have as inputs the log-weights ( l\omega_i ) and the individual log-likelihoods ( ll_i ), the problem becomes:

 \log \bar{l} = \log \left( \frac{1}{\sum_i e^{l\omega_i}} \sum_i  e^{l\omega_i} \cdot e^{ll_i} \right)  \quad \quad = - \log \sum_i e^{l\omega_i} + \log \sum_i e^{l\omega_i+ll_i}

The problem: the exponential of very large/small numbers will produce under/overflow.

The solution: to shift all the exponents and next correct it in the log sum:

 l\omega_{max} = \max_i l\omega_i,~~~~ ll_{max} = \max_i ll_i \log \bar{l} = - \log \sum_i e^{l\omega_i} + \log \sum_i e^{l\omega_i+ll_i}  ~~~~~= - \log \sum_i e^{l\omega_i-l\omega_{max}} + \log \sum_i e^{l\omega_i+ll_i-ll_{max}-l\omega_{max}} + ll_{max}

This method is implemented in the C++ function mrpt::math::averageLogLikelihood.

2. Unweighted log-likelihood values (Arithmetic mean)

If all the samples have equal weights, the formula simplifies to:
 \log \bar{l} = - \log N + \log \sum_{i=1}^N e^{ll_i-ll_{max}} + ll_{max}