How are perplexities over multiple instance aggregated?

handyman_hank

The perplexity of the $i^{th}$ token in the $k^{th}$ sequence is $$ P_{ki} = \frac{1}{p(t_{ki})} $$ The perplexity aggregated for the $k^{th}$ sequence is then $$ P_{k} = \left(\prod_{i=1}^N P_{ki}\right)^{1/N} \\ = \left(\prod_{i=1}^N \frac{1}{p(t_{ki})} \right)^{1/N} $$ which is the geometric mean of the perplexities of the tokens. This makes sense as we are essentially taking the multiplicative inverse of the probability that the model got...

Originally asked by Borun Chowdhury on Stack Exchange (ai) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

The choice of arithmetic mean for averaging perplexities is mainly for simplicity and consistency. It provides a straightforward measure of overall model performance across multiple sequences. Each $P_k$ is already a summary statistic (perplexity) for each sequence, and averaging them gives a global view of how well the model performs on average across multiple sequences. In some cases, alternatives to the arithmetic mean of perplexities might be more appropriate. For example, weighted average that assigns weights to sequences based on their length.

Full answer by Kulin Patel: https://ai.stackexchange.com/a/45996 (CC BY-SA 4.0).

Need it done for your business? UphillTech.