Feature Evals ⭐📊🧪💯💰 #446
Feature Evals ⭐📊🧪💯💰 #446
Conversation
* added patch * added evals function * added eval cmd
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the ✨ Finishing touches🧪 Generate unit tests
Comment |
* add prompts.yaml to bundle * process prompts on deployment * Remove hash check since we're moving it to the API
| return sb.String() | ||
| } | ||
|
|
||
| // escapeString escapes a string for use in TypeScript string literals |
There was a problem hiding this comment.
we have Interpolate in go-common string we should probably use.
| cmd.Stdout = ctx.Writer | ||
| cmd.Stderr = ctx.Writer | ||
| if err := cmd.Run(); err != nil { | ||
| if ctx.DevMode { |
| } | ||
|
|
||
| func copyPromptsFromSrc(logger logger.Logger, projectDir, outdir string) error { | ||
| srcRoot := filepath.Join(projectDir, "src") |
There was a problem hiding this comment.
can't use source directly like this. must go through the ProjectContext to get the correct source folder otherwise this will break for python
| // generatePromptTypeDeclarations generates only TypeScript declarations for prompt methods | ||
| func generatePromptTypeDeclarations(logger logger.Logger, projectDir string) error { | ||
| // Find all prompts.yaml files in the source directory | ||
| srcRoot := filepath.Join(projectDir, "src") |
There was a problem hiding this comment.
don't use src like this. use project context
| methodSignatures.WriteString(fmt.Sprintf(" * @param variables - Template variables to substitute\n")) | ||
| methodSignatures.WriteString(fmt.Sprintf(" * @returns Object with system and prompt strings, both with attached metadata properties\n")) | ||
| methodSignatures.WriteString(fmt.Sprintf(" */\n")) | ||
| methodSignatures.WriteString(fmt.Sprintf(" %s(variables?: Record<string, unknown>): Promise<{\n", methodName)) |
There was a problem hiding this comment.
we can probably just use a otel span here with the right prompt id
No description provided.