Repository navigation
feat(deps): add rouge extra for ROUGE scoring deps - #506
Conversation
`RougeScorer` (eval_method: "rouge") lazily imports nltk, evaluate and
rouge_score and raises ImportError telling users to pip install them, but
no extra ever declared them. The three shipped Llama configs that use
eval_method: "rouge" therefore fail at scorer construction on a stock
install; the only instructions lived in examples/05_Llama_Examples/README.md.
Declare them in a new `rouge` extra and have the `test` extra inherit it
(same self-reference pattern as `inference-endpoint[sql]`), so CI covers
the import surface with no extra job. Resolution is clean: 4 new packages,
`evaluate` pulls the already-pinned datasets==5.0.1, and no existing pin
moves - so unlike bfcl this needs no [tool.uv].conflicts fork.
Not self-contained at runtime: nltk needs the punkt/punkt_tab corpora and
evaluate.load("rouge") fetches its metric script from the HF Hub. Offline
images must prefetch both - handled separately in Dockerfile.dev.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
the
size/normal
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
|
With respect to the failing GitHub action , It seems there is no fixed version for that particular security issue (current nltk version is 3.10.3) Scanning the nltk repo, seems they are actively working to fix this security issue: Last activity was yesterday. I have pinged them asking whether there is a timeline. @attafosu , We could wait till folks from nltk fixes these issue or do we want to avoid using NLTK? Would replacing NLTK be a simpler fix? |
@anandhu-eng Yes I've looked at alternatives and pysbd's |
nltk 3.10.3 (pulled in by the rouge extra) is flagged by PYSEC-2026-3740 / CVE-2026-81726, a path traversal in model-artifact APIs (TransitionParser, AveragedPerceptron, PerceptronTagger, maxent). No fixed release exists yet (nltk/nltk#3753). This repo only uses sent_tokenize and rouge_score's Porter stemmer, so the vulnerable code is unreachable. Ignore the advisory in CI's pip-audit and document the flag for local runs. Removal is tracked in #520. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
test_template_runs spawns 10 workers on 4-CPU CI runners, where start-up takes long enough to straddle WorkerManager's 12s readiness slices (10% of the 120s timeout). A ready signal received in one slice is discarded when the slice times out, so init stalls until the full timeout. 300s gives 30s slices, longer than worker start-up, which unblocks CI while the underlying WorkerManager/ReadyCheckReceiver fix lands separately. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
3 participants
Problem
RougeScorer(eval_method: "rouge") lazily importsnltk,evaluateandrouge_scoreand raises anImportErrortelling users to install them — but no extra ever declared them. The three shipped Llama configs usingeval_method: "rouge"fail at scorer construction on a stock install, and the only install instructions lived inexamples/05_Llama_Examples/README.md.Change
rougeextra declaringnltk==3.10.3,evaluate==0.4.6,rouge-score==0.1.2.testextra inherits it viainference-endpoint[rouge]— same self-reference pattern asinference-endpoint[sql]— so CI covers the import surface with no extra job.Resolution is clean: 4 new packages,
evaluatepulls the already-pinneddatasets==5.0.1, and no existing pin moves. Unlikebfcl, no[tool.uv].conflictsfork is needed.Verified
uv lockadds onlyabsl-py,evaluate,nltk,rouge-score; no version changes to existing packages.uv sync --frozen --extra rougesucceeds; all three import at the pinned versions.evaluation/scoring.pynow resolves (_evaluate/_nltknon-None), so theImportErrorpath is no longer hit.Note
Installing the packages is not sufficient for an offline image:
nltkneeds thepunkt/punkt_tabcorpora andevaluate.load("rouge")fetches its metric script from the HF Hub. Prefetching both is handled separately inDockerfile.dev.🤖 Generated with Claude Code