Describe the bug
XSTest loading fails because the upstream dataset changed and the configured source no longer resolves for the current dataset layout.
Specifically, the CSV file was renamed from xstest_v2_prompts.csv to xstest_prompts.csv, and the prompt id column was renamed from id_v2 to id. Fixing only the file path still leaves a parser failure on the missing id_v2 column.
Steps/Code to reproduce bug
from nemo_evaluator.benchmarks.xstest import _load_xstest
if __name__ == "__main__":
_load_xstest() # raises urllib.error.HTTPError: HTTP Error 404: Not Found
The old file xstest_v2_prompts.csv is no longer present in the repository, so the download fails. Once the path is corrected, parsing still fails because int(row["id_v2"]) references a column that no longer exists (the new header is id,prompt,type,label,focus,note).
Expected behavior
XSTest should load successfully from the current CSV source.
The source should point to the available prompts file:
and the parser should read the renamed id column:
Additional context
A proposed fix updates the source path and the parser column:
- Replace
xstest_v2_prompts.csv with xstest_prompts.csv.
- Replace
int(row["id_v2"]) with int(row["id"]).
This restores full XSTest ingestion (450 prompts) while preserving prompt content and safe/unsafe label handling.
Describe the bug
XSTest loading fails because the upstream dataset changed and the configured source no longer resolves for the current dataset layout.
Specifically, the CSV file was renamed from
xstest_v2_prompts.csvtoxstest_prompts.csv, and the prompt id column was renamed fromid_v2toid. Fixing only the file path still leaves a parser failure on the missingid_v2column.Steps/Code to reproduce bug
The old file
xstest_v2_prompts.csvis no longer present in the repository, so the download fails. Once the path is corrected, parsing still fails becauseint(row["id_v2"])references a column that no longer exists (the new header isid,prompt,type,label,focus,note).Expected behavior
XSTest should load successfully from the current CSV source.
The source should point to the available prompts file:
and the parser should read the renamed id column:
Additional context
A proposed fix updates the source path and the parser column:
xstest_v2_prompts.csvwithxstest_prompts.csv.int(row["id_v2"])withint(row["id"]).This restores full XSTest ingestion (450 prompts) while preserving prompt content and safe/unsafe label handling.