Skip to content

Add configurable OpenAI provider settings and dynamic model selection. #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
csmashe wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add configurable OpenAI provider settings and dynamic model selection. #2

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 20 additions & 17 deletions src/ai/providers.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,34 +1,37 @@
import { createOpenAI } from '@ai-sdk/openai';
import { getEncoding } from 'js-tiktoken';
import { createOpenAI, type OpenAIProviderSettings } from "@ai-sdk/openai";
import { getEncoding } from "js-tiktoken";

import { RecursiveCharacterTextSplitter } from './text-splitter';
import { RecursiveCharacterTextSplitter } from "./text-splitter";

// Providers
interface CustomOpenAIProviderSettings extends OpenAIProviderSettings {
baseURL?: string;
}

// Providers
const openai = createOpenAI({
apiKey: process.env.OPENAI_KEY!,
});
baseURL: process.env.OPENAI_ENDPOINT || "https://api.openai.com/v1",
} as CustomOpenAIProviderSettings);

const customModel = process.env.OPENAI_MODEL || "o3-mini";

// Models

export const gpt4Model = openai('gpt-4o', {
structuredOutputs: true,
});
export const gpt4MiniModel = openai('gpt-4o-mini', {
structuredOutputs: true,
});
export const o3MiniModel = openai('o3-mini', {
reasoningEffort: 'medium',
export const o3MiniModel = openai(customModel, {
reasoningEffort: customModel.startsWith("o") ? "medium" : undefined,
structuredOutputs: true,
});

const MinChunkSize = 140;
const encoder = getEncoding('o200k_base');
const encoder = getEncoding("o200k_base");

// trim prompt to maximum context size
export function trimPrompt(prompt: string, contextSize = 120_000) {
export function trimPrompt(
prompt: string,
contextSize = Number(process.env.CONTEXT_SIZE) || 128_000
) {
if (!prompt) {
return '';
return "";
}

const length = encoder.encode(prompt).length;
Expand All @@ -47,7 +50,7 @@ export function trimPrompt(prompt: string, contextSize = 120_000) {
chunkSize,
chunkOverlap: 0,
});
const trimmedPrompt = splitter.splitText(prompt)[0] ?? '';
const trimmedPrompt = splitter.splitText(prompt)[0] ?? "";

// last catch, there's a chance that the trimmed prompt is same length as the original prompt, due to how tokens are split & innerworkings of the splitter, handle this case by just doing a hard cut
if (trimmedPrompt.length === prompt.length) {
Expand Down