Add OpenAI Chat Completions API

[huiwengoh]

Sample usage for create:

from cleanlab_tlm.utils.chat_completions import TLMChatCompletion

tlm = TLMChatCompletion(options={"log": ["explanation"]})
chat_response = tlm.create(
    model="gpt-4o-mini",
    messages=[{"content": "What is 1+1?","role": "user"}],
)

chat_response
>> ChatCompletion(id='chatcmpl-BhQL54McqQIGsJNQgLA7HzMhs6EoR', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='1 + 1 equals 2.', refusal=None, role='assistant', annotations=[], audio=None, function_call=None, tool_calls=None))], created=1749689051, model='gpt-4o-mini-2024-07-18', object='chat.completion', service_tier=None, system_fingerprint='fp_34a54ae93c', usage=CompletionUsage(completion_tokens=8, prompt_tokens=14, total_tokens=22, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0)), _request_id='req_e6b39046487f484f59b92a8191ec6f59', tlm_metadata={'log': {'explanation': 'Did not find a reason to doubt trustworthiness.'}, 'trustworthiness_score': 0.9020307464477912})

chat_response.choices[0].message.content
>> '1 + 1 equals 2.'

chat_response.tlm_metadata
>> {'log': {'explanation': 'Did not find a reason to doubt trustworthiness.'}, 'trustworthiness_score': 0.9020307464477912}

Sample usage for score:

import openai
from cleanlab_tlm.utils.chat_completions import TLMChatCompletion

openai_kwargs = {
    "model":"gpt-4o-mini",
    "messages":[{ "content": "What is 1+1?","role": "user"}]
}
openai_completion = openai.chat.completions.create(
  **openai_kwargs
)

openai_completion
>> ChatCompletion(id='chatcmpl-BhQLXqDrvjPr1o2W7GqHXeQwLqOqc', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='1 + 1 equals 2.', refusal=None, role='assistant', annotations=[], audio=None, function_call=None, tool_calls=None))], created=1749689079, model='gpt-4o-mini-2024-07-18', object='chat.completion', service_tier='default', system_fingerprint='fp_62a23a81ef', usage=CompletionUsage(completion_tokens=8, prompt_tokens=14, total_tokens=22, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0)))

tlm = TLMChatCompletion(options={"log": ["explanation"]})
score_response = tlm.score(
    response=openai_completion,
    **openai_kwargs
)

score_response
>> {'trustworthiness_score': 0.918869698047638, 'log': {'explanation': 'Did not find a reason to doubt trustworthiness.'}}

[jwmueller]

can you save a ticket for future PR:

we should try to extract token-probs out of response object if they are in there and include them in the score, like in TLMLite

[jwmueller]

when the user does: TLMChatCompletions.create()

the resulting scores should also be including token-probs (can save for future PR too)

[jwmueller]

will tool call output / structured output be here too if OpenAI chose to return that?

if not, can you save a ticket for future PR that we should get those outputs and use the appropriate TLM scoring for them

[huiwengoh]

No this code does not do that yet, I believe I can add rudimentary support for that pretty easily tho

[jas2600]

@huiwengoh do you still plan to add the support in this PR or in a followup ticket?

[huiwengoh]

Created ticket!

[jwmueller]

my recommended tutorial to make for this is here:

#74 (comment)

would be good to investigate whether the better monkey-patch version of workflow 2 in that link can be made to work with this API

[jwmueller]

The API LGTM. Leaving underlying code review to @jas2600

Note I suggested the ways this should be tested and how our eventual tutorial for it should look:

#74 (comment)

[jas2600]

[nit] cleaner to just pass openai_kwargs.get("tools", None) directly instead of creating an additional variable

[jas2600]

don't forget to release a new version