Is your feature request related to a problem?
When an OpenAI batch job completes with failed requests, the status is set to completed without an output_file_id, leading to a misleading success indication and an unhelpful generic error message that obscures the true cause of failure.
Describe the solution you'd like
Enhance the batch job error handling for cases with no output_file_id but an error_file_id:
- Download the error file
- Parse JSONL entries to extract
response.body.error.message
- Count unique errors and highlight the most common one
- Update
batch_job.error_message with the extracted message
- Mark the
evaluation_run as failed with the same error message
Error Batch results
Batch(id='batch_69c3c347282c81908ede48ecc95bf201', completion_window='24h', created_at=1774437191, endpoint='/v1/responses', input_file_id='file-2qvouj6sKUCGTbu3odouhk', object='batch', status='completed', cancelled_at=None, cancelling_at=None, completed_at=1774437324, error_file_id='file-QzTKe9srv1bbmpX5MwCTZx', errors=None, expired_at=None, expires_at=1774523591, failed_at=None, finalizing_at=1774437322, in_progress_at=1774437194, metadata={'description': 'Evaluation: test_run_new_1'}, model='gpt-5.4-pro-2026-03-05', output_file_id=None, request_counts=BatchRequestCounts(completed=0, failed=9, total=9), usage=BatchUsage(input_tokens=0, input_tokens_details=InputTokensDetails(cached_tokens=0), output_tokens=0, output_tokens_details=OutputTokensDetails(reasoning_tokens=0), total_tokens=0))
Success Batch results
Batch(id='batch_69c4f062ecfc8190b1d29752f95468e6', completion_window='24h', created_at=1774514274, endpoint='/v1/responses', input_file_id='file-DAyWeG7cwHwYLrjoZ61es7', object='batch', status='completed', cancelled_at=None, cancelling_at=None, completed_at=1774514642, error_file_id=None, errors=None, expired_at=None, expires_at=1774600674, failed_at=None, finalizing_at=1774514640, in_progress_at=1774514278, metadata={'description': 'Evaluation: gpt-5.4-pro'}, model='gpt-5.4-pro-2026-03-05', output_file_id='file-SDVYLXKe7iWoMERsPfVShQ', request_counts=BatchRequestCounts(completed=10, failed=0, total=10), usage=BatchUsage(input_tokens=420, input_tokens_details=InputTokensDetails(cached_tokens=0), output_tokens=6036, output_tokens_details=OutputTokensDetails(reasoning_tokens=4423), total_tokens=6456))
Is your feature request related to a problem?
When an OpenAI batch job
completeswith failed requests, the status is set to completed without anoutput_file_id, leading to a misleading success indication and an unhelpful generic error message that obscures the true cause of failure.Describe the solution you'd like
Enhance the batch job error handling for cases with no
output_file_idbut anerror_file_id:response.body.error.messagebatch_job.error_messagewith the extracted messageevaluation_runas failed with the same error messageError Batch results
Success Batch results