Skip to content

Commit c1ffe3e

Browse files
google-oss-botgoogle-oss-bot
authored
Merge main into release
2 parents 8d3211b + 2596dd1 commit c1ffe3e

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

57 files changed

+1085
-306
lines changed

‎.changeset/hungry-tables-exercise.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
'firebase': patch
3+
'@firebase/ai': patch
4+
---
5+
6+
Imagen Generation is now Generally Available (GA).

‎.changeset/lazy-donuts-agree.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
---
2+
'firebase': minor
3+
'@firebase/ai': minor
4+
---
5+
6+
Added a `sendFunctionResponses` method to `LiveSession`, allowing function responses to be sent during realtime sessions.
7+
Fixed an issue where function responses during audio conversations caused the WebSocket connection to close. See [GitHub Issue #9264](https://github.com/firebase/firebase-js-sdk/issues/9264).
8+
- **Breaking Change**: Changed the `functionCallingHandler` property in `StartAudioConversationOptions` so that it now must return a `Promise<FunctionResponse>`.
9+
This breaking change is allowed in a minor release since the Live API is in Public Preview.

‎.changeset/loud-tigers-compare.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
'firebase': patch
3+
'@firebase/ai': patch
4+
---
5+
6+
The Gemini Developer API is now Generally Available (GA).

‎.changeset/poor-cobras-dream.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@firebase/ai': patch
3+
---
4+
5+
Updated SDK to handle empty parts when streaming.

‎.changeset/poor-rings-admire.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
'firebase': minor
3+
'@firebase/ai': minor
4+
---
5+
6+
Added support for the URL context tool, which allows the model to access content from provided public web URLs to inform and enhance its responses.

‎.changeset/tender-meals-clap.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@firebase/ai': patch
3+
---
4+
5+
Tag code execution with beta tag (public preview).

‎.changeset/young-timers-jump.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
'@firebase/analytics-interop-types': patch
3+
'@firebase/analytics': patch
4+
---
5+
6+
Expose `setUserProperties` on internal Analytics instance.

‎common/api-review/ai.api.md‎

Lines changed: 73 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -175,13 +175,13 @@ export interface CitationMetadata {
175175
citations: Citation[];
176176
}
177177

178-
// @public
178+
// @beta
179179
export interface CodeExecutionResult {
180180
outcome?: Outcome;
181181
output?: string;
182182
}
183183

184-
// @public
184+
// @beta
185185
export interface CodeExecutionResultPart {
186186
// (undocumented)
187187
codeExecutionResult?: CodeExecutionResult;
@@ -203,7 +203,7 @@ export interface CodeExecutionResultPart {
203203
thoughtSignature?: never;
204204
}
205205

206-
// @public
206+
// @beta
207207
export interface CodeExecutionTool {
208208
codeExecution: {};
209209
}
@@ -271,13 +271,13 @@ export interface ErrorDetails {
271271
reason?: string;
272272
}
273273

274-
// @public
274+
// @beta
275275
export interface ExecutableCode {
276276
code?: string;
277277
language?: Language;
278278
}
279279

280-
// @public
280+
// @beta
281281
export interface ExecutableCodePart {
282282
// (undocumented)
283283
codeExecutionResult?: never;
@@ -449,6 +449,10 @@ export interface GenerateContentCandidate {
449449
index: number;
450450
// (undocumented)
451451
safetyRatings?: SafetyRating[];
452+
// Warning: (ae-incompatible-release-tags) The symbol "urlContextMetadata" is marked as @public, but its signature references "URLContextMetadata" which is marked as @beta
453+
//
454+
// (undocumented)
455+
urlContextMetadata?: URLContextMetadata;
452456
}
453457

454458
// @public
@@ -549,7 +553,7 @@ export function getAI(app?: FirebaseApp, options?: AIOptions): AI;
549553
// @public
550554
export function getGenerativeModel(ai: AI, modelParams: ModelParams | HybridParams, requestOptions?: RequestOptions): GenerativeModel;
551555

552-
// @beta
556+
// @public
553557
export function getImagenModel(ai: AI, modelParams: ImagenModelParams, requestOptions?: RequestOptions): ImagenModel;
554558

555559
// @beta
@@ -600,6 +604,8 @@ export interface GoogleAIGenerateContentCandidate {
600604
index: number;
601605
// (undocumented)
602606
safetyRatings?: SafetyRating[];
607+
// (undocumented)
608+
urlContextMetadata?: URLContextMetadata;
603609
}
604610

605611
// Warning: (ae-internal-missing-underscore) The name "GoogleAIGenerateContentResponse" should be prefixed with an underscore because the declaration is marked as @internal
@@ -706,7 +712,7 @@ export interface HybridParams {
706712
onDeviceParams?: OnDeviceParams;
707713
}
708714

709-
// @beta
715+
// @public
710716
export const ImagenAspectRatio: {
711717
readonly SQUARE: "1:1";
712718
readonly LANDSCAPE_3x4: "3:4";
@@ -715,16 +721,16 @@ export const ImagenAspectRatio: {
715721
readonly PORTRAIT_9x16: "9:16";
716722
};
717723

718-
// @beta
724+
// @public
719725
export type ImagenAspectRatio = (typeof ImagenAspectRatio)[keyof typeof ImagenAspectRatio];
720726

721-
// @beta
727+
// @public
722728
export interface ImagenGCSImage {
723729
gcsURI: string;
724730
mimeType: string;
725731
}
726732

727-
// @beta
733+
// @public
728734
export interface ImagenGenerationConfig {
729735
addWatermark?: boolean;
730736
aspectRatio?: ImagenAspectRatio;
@@ -733,27 +739,27 @@ export interface ImagenGenerationConfig {
733739
numberOfImages?: number;
734740
}
735741

736-
// @beta
742+
// @public
737743
export interface ImagenGenerationResponse<T extends ImagenInlineImage | ImagenGCSImage> {
738744
filteredReason?: string;
739745
images: T[];
740746
}
741747

742-
// @beta
748+
// @public
743749
export class ImagenImageFormat {
744750
compressionQuality?: number;
745751
static jpeg(compressionQuality?: number): ImagenImageFormat;
746752
mimeType: string;
747753
static png(): ImagenImageFormat;
748754
}
749755

750-
// @beta
756+
// @public
751757
export interface ImagenInlineImage {
752758
bytesBase64Encoded: string;
753759
mimeType: string;
754760
}
755761

756-
// @beta
762+
// @public
757763
export class ImagenModel extends AIModel {
758764
constructor(ai: AI, modelParams: ImagenModelParams, requestOptions?: RequestOptions | undefined);
759765
generateImages(prompt: string): Promise<ImagenGenerationResponse<ImagenInlineImage>>;
@@ -765,35 +771,35 @@ export class ImagenModel extends AIModel {
765771
safetySettings?: ImagenSafetySettings;
766772
}
767773

768-
// @beta
774+
// @public
769775
export interface ImagenModelParams {
770776
generationConfig?: ImagenGenerationConfig;
771777
model: string;
772778
safetySettings?: ImagenSafetySettings;
773779
}
774780

775-
// @beta
781+
// @public
776782
export const ImagenPersonFilterLevel: {
777783
readonly BLOCK_ALL: "dont_allow";
778784
readonly ALLOW_ADULT: "allow_adult";
779785
readonly ALLOW_ALL: "allow_all";
780786
};
781787

782-
// @beta
788+
// @public
783789
export type ImagenPersonFilterLevel = (typeof ImagenPersonFilterLevel)[keyof typeof ImagenPersonFilterLevel];
784790

785-
// @beta
791+
// @public
786792
export const ImagenSafetyFilterLevel: {
787793
readonly BLOCK_LOW_AND_ABOVE: "block_low_and_above";
788794
readonly BLOCK_MEDIUM_AND_ABOVE: "block_medium_and_above";
789795
readonly BLOCK_ONLY_HIGH: "block_only_high";
790796
readonly BLOCK_NONE: "block_none";
791797
};
792798

793-
// @beta
799+
// @public
794800
export type ImagenSafetyFilterLevel = (typeof ImagenSafetyFilterLevel)[keyof typeof ImagenSafetyFilterLevel];
795801

796-
// @beta
802+
// @public
797803
export interface ImagenSafetySettings {
798804
personFilterLevel?: ImagenPersonFilterLevel;
799805
safetyFilterLevel?: ImagenSafetyFilterLevel;
@@ -836,13 +842,13 @@ export class IntegerSchema extends Schema {
836842
constructor(schemaParams?: SchemaParams);
837843
}
838844

839-
// @public
845+
// @beta
840846
export const Language: {
841847
UNSPECIFIED: string;
842848
PYTHON: string;
843849
};
844850

845-
// @public
851+
// @beta
846852
export type Language = (typeof Language)[keyof typeof Language];
847853

848854
// @beta
@@ -988,6 +994,7 @@ export class LiveSession {
988994
isClosed: boolean;
989995
receive(): AsyncGenerator<LiveServerContent | LiveServerToolCall | LiveServerToolCallCancellation>;
990996
send(request: string | Array<string | Part>, turnComplete?: boolean): Promise<void>;
997+
sendFunctionResponses(functionResponses: FunctionResponse[]): Promise<void>;
991998
sendMediaChunks(mediaChunks: GenerativeContentBlob[]): Promise<void>;
992999
sendMediaStream(mediaChunkStream: ReadableStream<GenerativeContentBlob>): Promise<void>;
9931000
}
@@ -1058,17 +1065,20 @@ export interface OnDeviceParams {
10581065
promptOptions?: LanguageModelPromptOptions;
10591066
}
10601067

1061-
// @public
1068+
// @beta
10621069
export const Outcome: {
10631070
UNSPECIFIED: string;
10641071
OK: string;
10651072
FAILED: string;
10661073
DEADLINE_EXCEEDED: string;
10671074
};
10681075

1069-
// @public
1076+
// @beta
10701077
export type Outcome = (typeof Outcome)[keyof typeof Outcome];
10711078

1079+
// Warning: (ae-incompatible-release-tags) The symbol "Part" is marked as @public, but its signature references "ExecutableCodePart" which is marked as @beta
1080+
// Warning: (ae-incompatible-release-tags) The symbol "Part" is marked as @public, but its signature references "CodeExecutionResultPart" which is marked as @beta
1081+
//
10721082
// @public
10731083
export type Part = TextPart | InlineDataPart | FunctionCallPart | FunctionResponsePart | FileDataPart | ExecutableCodePart | CodeExecutionResultPart;
10741084

@@ -1254,7 +1264,7 @@ export function startAudioConversation(liveSession: LiveSession, options?: Start
12541264

12551265
// @beta
12561266
export interface StartAudioConversationOptions {
1257-
functionCallingHandler?: (functionCalls: LiveServerToolCall['functionCalls']) => Promise<Part>;
1267+
functionCallingHandler?: (functionCalls: FunctionCall[]) => Promise<FunctionResponse>;
12581268
}
12591269

12601270
// @public
@@ -1304,8 +1314,11 @@ export interface ThinkingConfig {
13041314
thinkingBudget?: number;
13051315
}
13061316

1317+
// Warning: (ae-incompatible-release-tags) The symbol "Tool" is marked as @public, but its signature references "CodeExecutionTool" which is marked as @beta
1318+
// Warning: (ae-incompatible-release-tags) The symbol "Tool" is marked as @public, but its signature references "URLContextTool" which is marked as @beta
1319+
//
13071320
// @public
1308-
export type Tool = FunctionDeclarationsTool | GoogleSearchTool | CodeExecutionTool;
1321+
export type Tool = FunctionDeclarationsTool | GoogleSearchTool | CodeExecutionTool | URLContextTool;
13091322

13101323
// @public
13111324
export interface ToolConfig {
@@ -1316,6 +1329,38 @@ export interface ToolConfig {
13161329
// @public
13171330
export type TypedSchema = IntegerSchema | NumberSchema | StringSchema | BooleanSchema | ObjectSchema | ArraySchema | AnyOfSchema;
13181331

1332+
// @beta
1333+
export interface URLContext {
1334+
}
1335+
1336+
// @beta
1337+
export interface URLContextMetadata {
1338+
urlMetadata: URLMetadata[];
1339+
}
1340+
1341+
// @beta
1342+
export interface URLContextTool {
1343+
urlContext: URLContext;
1344+
}
1345+
1346+
// @beta
1347+
export interface URLMetadata {
1348+
retrievedUrl?: string;
1349+
urlRetrievalStatus?: URLRetrievalStatus;
1350+
}
1351+
1352+
// @beta
1353+
export const URLRetrievalStatus: {
1354+
URL_RETRIEVAL_STATUS_UNSPECIFIED: string;
1355+
URL_RETRIEVAL_STATUS_SUCCESS: string;
1356+
URL_RETRIEVAL_STATUS_ERROR: string;
1357+
URL_RETRIEVAL_STATUS_PAYWALL: string;
1358+
URL_RETRIEVAL_STATUS_UNSAFE: string;
1359+
};
1360+
1361+
// @beta
1362+
export type URLRetrievalStatus = (typeof URLRetrievalStatus)[keyof typeof URLRetrievalStatus];
1363+
13191364
// @public
13201365
export interface UsageMetadata {
13211366
// (undocumented)
@@ -1327,6 +1372,8 @@ export interface UsageMetadata {
13271372
// (undocumented)
13281373
promptTokensDetails?: ModalityTokenCount[];
13291374
thoughtsTokenCount?: number;
1375+
toolUsePromptTokenCount?: number;
1376+
toolUsePromptTokensDetails?: ModalityTokenCount[];
13301377
// (undocumented)
13311378
totalTokenCount: number;
13321379
}

‎docs-devsite/_toc.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -202,6 +202,14 @@ toc:
202202
path: /docs/reference/js/ai.thinkingconfig.md
203203
- title: ToolConfig
204204
path: /docs/reference/js/ai.toolconfig.md
205+
- title: URLContext
206+
path: /docs/reference/js/ai.urlcontext.md
207+
- title: URLContextMetadata
208+
path: /docs/reference/js/ai.urlcontextmetadata.md
209+
- title: URLContextTool
210+
path: /docs/reference/js/ai.urlcontexttool.md
211+
- title: URLMetadata
212+
path: /docs/reference/js/ai.urlmetadata.md
205213
- title: UsageMetadata
206214
path: /docs/reference/js/ai.usagemetadata.md
207215
- title: VertexAIBackend

‎docs-devsite/ai.codeexecutionresult.md‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,9 @@ https://github.com/firebase/firebase-js-sdk
1010
{% endcomment %}
1111

1212
# CodeExecutionResult interface
13+
> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
14+
>
15+
1316
The results of code execution run by the model.
1417

1518
<b>Signature:</b>
@@ -22,11 +25,14 @@ export interface CodeExecutionResult
2225

2326
| Property | Type | Description |
2427
| --- | --- | --- |
25-
| [outcome](./ai.codeexecutionresult.md#codeexecutionresultoutcome) | [Outcome](./ai.md#outcome) | The result of the code execution. |
26-
| [output](./ai.codeexecutionresult.md#codeexecutionresultoutput) | string | The output from the code execution, or an error message if it failed. |
28+
| [outcome](./ai.codeexecutionresult.md#codeexecutionresultoutcome) | [Outcome](./ai.md#outcome) | <b><i>(Public Preview)</i></b> The result of the code execution. |
29+
| [output](./ai.codeexecutionresult.md#codeexecutionresultoutput) | string | <b><i>(Public Preview)</i></b> The output from the code execution, or an error message if it failed. |
2730

2831
## CodeExecutionResult.outcome
2932

33+
> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
34+
>
35+
3036
The result of the code execution.
3137

3238
<b>Signature:</b>
@@ -37,6 +43,9 @@ outcome?: Outcome;
3743

3844
## CodeExecutionResult.output
3945

46+
> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
47+
>
48+
4049
The output from the code execution, or an error message if it failed.
4150

4251
<b>Signature:</b>

0 commit comments

Comments
 (0)