Trying to retrieve token-level uncertainty values

[alexander-hamme]

Hello,

I am working on a personal research study and trying to explore the topic of live hallucination detection (with possibilities for short-circuiting etc), I would like to use the various metrics offered by lm-polygraph but it seems that only the TokenEntropy class supports returning the full list of token-level values (non-aggregated).

Is there any way to retrieve the original token-level value list for certain other metrics, such as PMI/CPMI, Renyi, etc? (Even if they make more sense at a claim / full response level, I am still interested in getting the token-level list for further exploration and analysis.

As I currently see it, the source code will have to be significantly modified (even when I look at the low_level jupyter notebook). Please let me know if there are functions available to do this already with estimate_uncertainty or a similar function.

Thank you!

[ArtemVazh]

Hi! Thank you for your interest in our project!

At the moment, we do not provide built-in support for returning token-level uncertainty scores, as the library is primarily focused on sequence-level and claim-level uncertainty quantification for selective generation tasks.

However, it is relatively straightforward to adapt the existing estimators that operate at the claim-level to obtain token-level outputs. For example, in claim_conditioned_probability.py (see line 73), you can remove the reduction step by appending the raw per-token values directly: prob_nli.append(sample_mnlis) instead of prob_nli.append(self._reduce(sample_mnlis)). A similar modification can be applied to other estimators. After this modification, you can use the modified estimator in the same way via the estimate_uncertainty function.