Skip to content

TriviaQA loading fails: legacy trivia_qa id rejected by HF Hub namespace/name validation #1111

Description

@cris96spa

Describe the bug

TriviaQA loading fails because the loader uses a legacy dataset identifier that Hugging Face Hub no longer accepts.

Specifically, the loader calls load_dataset("trivia_qa", ...). Hugging Face Hub now enforces namespace/name repository IDs, so this fails with a repository ID validation error.

Steps/Code to reproduce bug

from nemo_evaluator.benchmarks.triviaqa import _load_triviaqa

if __name__ == "__main__":
    _load_triviaqa()  # raises: Repository id must be 'namespace/name', got 'trivia_qa'

The un-namespaced id trivia_qa is rejected during repository ID validation, so dataset loading never starts.

Expected behavior

TriviaQA should load successfully from the current namespaced repository.

The dataset id should point to the maintained repository:

mandarjoshi/trivia_qa

Additional context

A proposed fix updates the dataset id:

  • Replace trivia_qa with mandarjoshi/trivia_qa.

This restores TriviaQA loading without changing evaluator semantics, and keeps existing seeding and verification behavior intact (same rc.nocontext config, validation split, and question / answer fields).

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions