Skip to content

TypeError: unsupported operand type(s) for +: 'int' and 'CompletionTokensDetail #1582

Description

@mike2151

Describe the bug

When trying to run out of the box evals test, getting the following stack trace:

(venv) <USER>@<HOSTNAME> <PROJECT_PATH> % python3 -m evals.cli.oaieval gpt-3.5-turbo test-match
[2025-04-16 22:26:45,558] [registry.py:271] Loading registry from <PATH_TO_VENV>/lib/python3.10/site-packages/evals/registry/evals
[2025-04-16 22:26:46,128] [registry.py:271] Loading registry from /Users/<USER>/.evals/evals
[2025-04-16 22:26:46,131] [oaieval.py:215] Run started: <RUN_ID>
[2025-04-16 22:26:46,261] [eval.py:36] Evaluating 3 samples
[2025-04-16 22:26:46,277] [eval.py:144] Running in threaded mode with 10 threads!
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00,  3.36it/s]
[2025-04-16 22:26:47,189] [oaieval.py:275] Found 3/3 sampling events with usage data
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.10/3.10.12_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Cellar/python@3.10/3.10.12_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "<PROJECT_PATH>/venv/lib/python3.10/site-packages/evals/cli/oaieval.py", line 308, in <module>
    main()
  File "<PROJECT_PATH>/venv/lib/python3.10/site-packages/evals/cli/oaieval.py", line 304, in main
    run(args)
  File "<PROJECT_PATH>/venv/lib/python3.10/site-packages/evals/cli/oaieval.py", line 227, in run
    add_token_usage_to_result(result, recorder)
  File "<PROJECT_PATH>/venv/lib/python3.10/site-packages/evals/cli/oaieval.py", line 278, in add_token_usage_to_result
    total_usage = {
  File "<PROJECT_PATH>/venv/lib/python3.10/site-packages/evals/cli/oaieval.py", line 279, in <dictcomp>
    key: sum(u[key] if u[key] is not None else 0 for u in usage_events)
TypeError: unsupported operand type(s) for +: 'int' and 'CompletionTokensDetails'

To Reproduce

Running:

python3 -m evals.cli.oaieval gpt-3.5-turbo test-match

Code snippets

OS

macOS

Python version

Python v3.10.12

Library version

openai-evals 3.0.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions