Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce "Malformed Responses" by a minor change in the prompt #346

Closed
wants to merge 1 commit into from

Conversation

omri123
Copy link
Contributor

@omri123 omri123 commented Nov 13, 2023

This change has positive effect on ratio of malformed response.
I discovered it while testing another change, in which I add line numbers to code.

I run the Exercism python benchmarks, consisting of 135 tests, using gpt-4-turbo.

  • The number of chats with malformed response went down from 14 to 9
  • The total number of malformed responses went down from 21 to 15

Does this difference justifies a change?
Do you want additional benchmarking?

About performance, I saw minor improvment.
community -> community+change:

  • 1st try 50.04% -> 51.1%
  • 2nd try 61.5% -> 63%

I know it doesn't reproduce your results, I guess difference comes from chatgpt randomness.
I counted malformed responses from the chat history, using excel and this script:
https://gist.github.com/omri123/6c0f4a07e3c25ac059e04ddeeaf7f62c

@omri123
Copy link
Contributor Author

omri123 commented Nov 14, 2023

Edit: I counted the number of chats incorrectly.
The number of chats with malformed response went down from 12 to 9, and not from 14 to 9.

@omri123 omri123 mentioned this pull request Nov 14, 2023
@paul-gauthier
Copy link
Collaborator

Thanks for putting together this PR!

The indents were intentional, to try and encourage GPT to keep leading whitespace intact in S/R blocks.

Also, the changes you are reporting in benchmarking result seem within the range of random varition. The benchmark suite tries very hard to be deterministic, but OpenAI's API is not.

@omri123
Copy link
Contributor Author

omri123 commented Nov 14, 2023

Cool, if we will continue with the line numbers I will re-add the spaces. (this change appears in both pull requests).

@omri123 omri123 closed this Nov 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants