(venv) <USER>@<HOSTNAME> <PROJECT_PATH> % python3 -m evals.cli.oaieval gpt-3.5-turbo test-match
[2025-04-16 22:26:45,558] [registry.py:271] Loading registry from <PATH_TO_VENV>/lib/python3.10/site-packages/evals/registry/evals
[2025-04-16 22:26:46,128] [registry.py:271] Loading registry from /Users/<USER>/.evals/evals
[2025-04-16 22:26:46,131] [oaieval.py:215] Run started: <RUN_ID>
[2025-04-16 22:26:46,261] [eval.py:36] Evaluating 3 samples
[2025-04-16 22:26:46,277] [eval.py:144] Running in threaded mode with 10 threads!
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 3.36it/s]
[2025-04-16 22:26:47,189] [oaieval.py:275] Found 3/3 sampling events with usage data
Traceback (most recent call last):
File "/usr/local/Cellar/python@3.10/3.10.12_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/Cellar/python@3.10/3.10.12_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "<PROJECT_PATH>/venv/lib/python3.10/site-packages/evals/cli/oaieval.py", line 308, in <module>
main()
File "<PROJECT_PATH>/venv/lib/python3.10/site-packages/evals/cli/oaieval.py", line 304, in main
run(args)
File "<PROJECT_PATH>/venv/lib/python3.10/site-packages/evals/cli/oaieval.py", line 227, in run
add_token_usage_to_result(result, recorder)
File "<PROJECT_PATH>/venv/lib/python3.10/site-packages/evals/cli/oaieval.py", line 278, in add_token_usage_to_result
total_usage = {
File "<PROJECT_PATH>/venv/lib/python3.10/site-packages/evals/cli/oaieval.py", line 279, in <dictcomp>
key: sum(u[key] if u[key] is not None else 0 for u in usage_events)
TypeError: unsupported operand type(s) for +: 'int' and 'CompletionTokensDetails'
python3 -m evals.cli.oaieval gpt-3.5-turbo test-match
Describe the bug
When trying to run out of the box evals test, getting the following stack trace:
To Reproduce
Running:
Code snippets
OS
macOS
Python version
Python v3.10.12
Library version
openai-evals 3.0.1