Averaging Log-Likelihood Values:Numerical Stability
1. Weighted log-likelihood values
The problem addressed in this page is the computation of:
It’s very advisable to employ log-likelihoods (and log-weights) for a extended dynamic range. This avoids the problem of weights becoming zero.
Then, if we have as inputs the log-weights () and the individual log-likelihoods (), the problem becomes:
The problem: the exponential of very large/small numbers will produce under/overflow.
The solution: to shift all the exponents and next correct it in the log sum:
This method is implemented in the C++ function mrpt::math::averageLogLikelihood.
2. Unweighted log-likelihood values (Arithmetic mean)
If all the samples have equal weights, the formula simplifies to: