Skip to content

Commit

Permalink
Fix candidate index for new -002 models and enable candidateCount
Browse files Browse the repository at this point in the history
For unknown reason 0-index is absent in new -002 models response
Ref: https://ai.google.dev/api/generate-content#v1beta.Candidate

Note: atm `candidateCount` works only for -002 models
      (but not for `streamGenerateContent` endpoint)
  • Loading branch information
johnd0e committed Sep 25, 2024
1 parent 411cd5b commit 9a173d3
Showing 1 changed file with 3 additions and 2 deletions.
5 changes: 3 additions & 2 deletions src/worker.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ const safetySettings = harmCategory.map((category) => ({
}));
const fieldsMap = {
stop: "stopSequences",
//n: "candidateCount", // { "error": { "code": 400, "message": "Only one candidate can be specified", "status": "INVALID_ARGUMENT" } }
n: "candidateCount", // { "error": { "code": 400, "message": "Only one candidate can be specified", "status": "INVALID_ARGUMENT" } }
max_tokens: "maxOutputTokens",
temperature: "temperature",
top_p: "topP",
Expand Down Expand Up @@ -237,7 +237,7 @@ const reasonsMap = { //https://ai.google.dev/api/rest/v1/GenerateContentResponse
// :"function_call",
};
const transformCandidates = (key, cand) => ({
index: cand.index,
index: cand.index || 0, // 0-index is absent in new -002 models response
[key]: { role: "assistant", content: cand.content?.parts[0].text },
logprobs: null,
finish_reason: reasonsMap[cand.finishReason] || cand.finishReason,
Expand Down Expand Up @@ -319,6 +319,7 @@ async function toOpenAiStream (chunk, controller) {
data = { candidates };
}
const cand = data.candidates[0]; // !!untested with candidateCount>1
cand.index = cand.index || 0; // absent in new -002 models response
if (!this.last[cand.index]) {
controller.enqueue(transform(data, false, "first"));
}
Expand Down

0 comments on commit 9a173d3

Please sign in to comment.