feat: add response format handling in judge rubric #332
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
It would also be nice to be able to use response_format/structured responses in the user_client, or is this better handled with tool calling? |
|
@lakshyaag Can this not already be done by passing response_format via sampling_args? Would rather have that be the all-in-one route for expressing additional configurations rather than explicitly adding each one. |
|
@willccbb I'll look into it - should be able to support it. |
|
@cursoragent look into it |
* Refactor: Move response_format to judge_sampling_args Co-authored-by: lakshyajannu <[email protected]> * feat: Use chat completions parse for structured outputs This change routes chat completion requests to the `parse` API when a `response_format` is specified in the sampling arguments. This ensures that structured outputs are correctly handled. The mock client has also been updated to support this new functionality. Co-authored-by: lakshyajannu <[email protected]> * refactor: Update mock responses and enhance parameter validation This commit modifies the mock responses in the test suite to align with the new structure of parsed chat completions. It also enhances the parameter validation in the `CalculatorResponse` model to include specific properties, ensuring better type safety and clarity in the tests. Additionally, the environment class has been refactored to streamline the handling of response formats and improve the readability of the chat completion request logic. --------- Co-authored-by: Cursor Agent <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
From #331
This PR adds
response_formatto the default judge rubric parameters. On specifying a OpenAI-compatibleResponseFormat, the request will use the.parse()method in the OpenAI SDK.Type of Change
Testing
uv run pytestTest Coverage
Checklist
Additional Notes