Describe the bug
TriviaQA loading fails because the loader uses a legacy dataset identifier that Hugging Face Hub no longer accepts.
Specifically, the loader calls load_dataset("trivia_qa", ...). Hugging Face Hub now enforces namespace/name repository IDs, so this fails with a repository ID validation error.
Steps/Code to reproduce bug
from nemo_evaluator.benchmarks.triviaqa import _load_triviaqa
if __name__ == "__main__":
_load_triviaqa() # raises: Repository id must be 'namespace/name', got 'trivia_qa'
The un-namespaced id trivia_qa is rejected during repository ID validation, so dataset loading never starts.
Expected behavior
TriviaQA should load successfully from the current namespaced repository.
The dataset id should point to the maintained repository:
Additional context
A proposed fix updates the dataset id:
- Replace
trivia_qa with mandarjoshi/trivia_qa.
This restores TriviaQA loading without changing evaluator semantics, and keeps existing seeding and verification behavior intact (same rc.nocontext config, validation split, and question / answer fields).
Describe the bug
TriviaQA loading fails because the loader uses a legacy dataset identifier that Hugging Face Hub no longer accepts.
Specifically, the loader calls
load_dataset("trivia_qa", ...). Hugging Face Hub now enforcesnamespace/namerepository IDs, so this fails with a repository ID validation error.Steps/Code to reproduce bug
The un-namespaced id
trivia_qais rejected during repository ID validation, so dataset loading never starts.Expected behavior
TriviaQA should load successfully from the current namespaced repository.
The dataset id should point to the maintained repository:
Additional context
A proposed fix updates the dataset id:
trivia_qawithmandarjoshi/trivia_qa.This restores TriviaQA loading without changing evaluator semantics, and keeps existing seeding and verification behavior intact (same
rc.nocontextconfig,validationsplit, andquestion/answerfields).