Dependency conflict between unbabel-comet and transformers>=5.0.0/sentencepiece>=0.2.1

[ArtemVazh]

Currently, lm-polygraph works with transformers>=4.50.0 and unbabel-comet.

If we need to upgrade to transformers>=5.0.0 in the future, note that unbabel-comet is incompatible, as unbabel-comet 2.x requires transformers<5.0.0. Additionally, upgrading to transformers>=5.0.0 may introduce potential issues or breaking changes in the Hugging Face API that would need to be addressed.

[ArtemVazh]

#446

[smirnovlad]

CI timeout issue (exit code 143)

The CI job gets killed with exit code 143 (SIGTERM) after all tests pass. This happens because the GitHub Actions runner exceeds the default 6-hour job timeout or runs out of memory/resources during post-test cleanup.

Example: All 46 tests pass (93% → 95% → done), but the process is terminated before completion:

test/test_estimators.py ...........................................      [ 93%]
test/test_high_level_api.py .                                            [ 95%]
test/test_lm_polygraph.py .
Error: Process completed with exit code 143.

CI job: https://github.com/IINemo/lm-polygraph/actions/runs/24158179795/job/70502518458

This is likely caused by the test suite consuming too many resources (memory/disk) on the GitHub-hosted runner, causing the runner to kill the process. The test_lm_polygraph.py test downloads models and runs inference which can be resource-intensive on the free-tier runners.

Related PR: #446

[smirnovlad]

Correction to the above: This timeout/SIGTERM issue (exit code 143) only occurs with transformers>=5.0. With transformers<5.0 the same test suite completes successfully within the CI time limit.

This suggests that transformers 5.x is significantly slower or more resource-hungry on GitHub-hosted runners, causing the job to exceed resource limits.