Skip to content

XSTest loading fails #1110

Description

@cris96spa

Describe the bug

XSTest loading fails because the upstream dataset changed and the configured source no longer resolves for the current dataset layout.

Specifically, the CSV file was renamed from xstest_v2_prompts.csv to xstest_prompts.csv, and the prompt id column was renamed from id_v2 to id. Fixing only the file path still leaves a parser failure on the missing id_v2 column.

Steps/Code to reproduce bug

from nemo_evaluator.benchmarks.xstest import _load_xstest

if __name__ == "__main__":
    _load_xstest()  # raises urllib.error.HTTPError: HTTP Error 404: Not Found

The old file xstest_v2_prompts.csv is no longer present in the repository, so the download fails. Once the path is corrected, parsing still fails because int(row["id_v2"]) references a column that no longer exists (the new header is id,prompt,type,label,focus,note).

Expected behavior

XSTest should load successfully from the current CSV source.

The source should point to the available prompts file:

main/xstest_prompts.csv

and the parser should read the renamed id column:

int(row["id"])

Additional context

A proposed fix updates the source path and the parser column:

  • Replace xstest_v2_prompts.csv with xstest_prompts.csv.
  • Replace int(row["id_v2"]) with int(row["id"]).

This restores full XSTest ingestion (450 prompts) while preserving prompt content and safe/unsafe label handling.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions