Skip to content

Commit d187670

Browse files
joefernandezjoefernandez
authored
Notebook updates for Gemma 3n (#596)
1 parent 2117c24 commit d187670

File tree

2 files changed

+475
-3
lines changed

2 files changed

+475
-3
lines changed

‎site/en/gemma/docs/capabilities/vision/video-understanding.ipynb

Lines changed: 315 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,315 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {
6+
"id": "G3MMAcssHTML"
7+
},
8+
"source": [
9+
"<link rel=\"stylesheet\" href=\"/site-assets/css/gemma.css\">\n",
10+
"<link rel=\"stylesheet\" href=\"https://fonts.googleapis.com/css2?family=Google+Symbols:opsz,wght,FILL,[email protected],100..700,0..1,-50..200\" />"
11+
]
12+
},
13+
{
14+
"cell_type": "markdown",
15+
"metadata": {
16+
"id": "Tce3stUlHN0L"
17+
},
18+
"source": [
19+
"##### Copyright 2025 Google LLC."
20+
]
21+
},
22+
{
23+
"cell_type": "code",
24+
"execution_count": 1,
25+
"metadata": {
26+
"cellView": "form",
27+
"id": "tuOe1ymfHZPu"
28+
},
29+
"outputs": [],
30+
"source": [
31+
"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
32+
"# you may not use this file except in compliance with the License.\n",
33+
"# You may obtain a copy of the License at\n",
34+
"#\n",
35+
"# https://www.apache.org/licenses/LICENSE-2.0\n",
36+
"#\n",
37+
"# Unless required by applicable law or agreed to in writing, software\n",
38+
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
39+
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
40+
"# See the License for the specific language governing permissions and\n",
41+
"# limitations under the License."
42+
]
43+
},
44+
{
45+
"cell_type": "markdown",
46+
"metadata": {
47+
"id": "f9673bd6"
48+
},
49+
"source": [
50+
"<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
51+
" <td>\n",
52+
" <a target=\"_blank\" href=\"https://ai.google.dev/gemma/docs/capabilities/vision/video-understanding\"><img src=\"https://ai.google.dev/static/site-assets/images/docs/notebook-site-button.png\" height=\"32\" width=\"32\" />View on ai.google.dev</a>\n",
53+
" </td>\n",
54+
" <td>\n",
55+
" <a target=\"_blank\" href=\"https://colab.research.google.com/github/google/generative-ai-docs/blob/main/site/en/gemma/docs/capabilities/vision/video-understanding.ipynb\"\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
56+
" </td>\n",
57+
" <td>\n",
58+
" <a target=\"_blank\" href=\"https://kaggle.com/kernels/welcome?src=https://github.com/google/generative-ai-docs/blob/main/site/en/gemma/docs/capabilities/vision/video-understanding.ipynb\"><img src=\"https://www.kaggle.com/static/images/logos/kaggle-logo-transparent-300.png\" height=\"32\" width=\"70\"/>Run in Kaggle</a>\n",
59+
" </td>\n",
60+
" <td>\n",
61+
" <a target=\"_blank\" href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/google/generative-ai-docs/main/site/en/gemma/docs/capabilities/vision/video-understanding.ipynb\"><img src=\"https://ai.google.dev/images/cloud-icon.svg\" width=\"40\" />Open in Vertex AI</a>\n",
62+
" </td>\n",
63+
" <td>\n",
64+
" <a target=\"_blank\" href=\"https://github.com/google/generative-ai-docs/blob/main/site/en/gemma/docs/capabilities/vision/video-understanding.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
65+
" </td>\n",
66+
"</table>"
67+
]
68+
},
69+
{
70+
"cell_type": "markdown",
71+
"metadata": {
72+
"id": "qlsJnFBYSrkZ"
73+
},
74+
"source": [
75+
"# Video analysis with Gemma\n",
76+
"\n",
77+
"Video data is a rich source of information that can help you accomplish tasks and understand the world around you. Using Gemma with video data can help you understand spacial relationships, interpret human interactions, and assist with situational awareness. This tutorial shows you how to get started processing video data with Gemma using Hugging Face Transformers. The Transformers Python library provides a API for accessing pre-trained generative AI models, including Gemma. For more information, see the [Transformers](https://huggingface.co/docs/transformers/en/index) documentation.\n",
78+
"\n",
79+
"\n",
80+
"Note: Gemma 3 and later models support input of both images and text. Earlier versions of Gemma only support text input, except for some variants, including [PaliGemma](https://ai.google.dev/gemma/docs/setup).\n",
81+
"\n",
82+
"## Setup\n",
83+
"\n",
84+
"Before starting this tutorial, complete the following steps:\n",
85+
"\n",
86+
"* Get access to Gemma by logging into [Hugging Face](https://huggingface.co/google/gemma-3-4b-pt) and selecting **Acknowledge license** for a Gemma model.\n",
87+
"* Select a Colab runtime with sufficient resources to run\n",
88+
" the Gemma model size you want to run. [Learn more](https://ai.google.dev/gemma/docs/core#sizes).\n",
89+
"* Generate a Hugging Face [Access Token](https://huggingface.co/docs/hub/en/security-tokens#how-to-manage-user-access-token) and add it to your Colab environment.\n",
90+
"\n",
91+
"### Configure Access Token\n",
92+
"\n",
93+
"Add your access token to Colab to enable downloading of Gemma models from the Hugging Face web site. Use the Colab **Secrets** feature to securely save your token without adding it to your working code.\n",
94+
"\n",
95+
"To add your Hugging Face Access Token as a Secret:\n",
96+
"\n",
97+
"1. Open the secrets tab by selecting the key icon on left side of the interface, or select **Tools > Command pallete**, type `secrets`, and press **Enter**.\n",
98+
"2. Select **Add new secret** to add a new secret entry.\n",
99+
"3. In the **Name** field, enter `HF_TOKEN`.\n",
100+
"4. In the **Value** field, enter the text of your Hugging Face Access Token.\n",
101+
"5. In the **Notebook access** field, select the switch to enable access.\n",
102+
"\n",
103+
"Once you have entered your Access Token as `HF_TOKEN` and value, you can access and set it within your Colab notebook environment using the following code:"
104+
]
105+
},
106+
{
107+
"cell_type": "code",
108+
"execution_count": null,
109+
"metadata": {
110+
"id": "ogSB3peYP1b4"
111+
},
112+
"outputs": [],
113+
"source": [
114+
"from google.colab import userdata\n",
115+
"from huggingface_hub import login\n",
116+
"\n",
117+
"# Login into Hugging Face Hub\n",
118+
"hf_token = userdata.get('HF_TOKEN') # If you are running inside a Google Colab\n",
119+
"login(hf_token)"
120+
]
121+
},
122+
{
123+
"cell_type": "markdown",
124+
"metadata": {
125+
"id": "2RgrZH4cP1b4"
126+
},
127+
"source": [
128+
"### Install Python packages\n",
129+
"\n",
130+
"Install the Hugging Face libraries required for running the Gemma model and making requests."
131+
]
132+
},
133+
{
134+
"cell_type": "code",
135+
"execution_count": null,
136+
"metadata": {
137+
"id": "ty1cuOpoP1b4"
138+
},
139+
"outputs": [],
140+
"source": [
141+
"# Install Pytorch & other libraries\n",
142+
"%pip install \"torch>=2.4.0\"\n",
143+
"\n",
144+
"# Install a transformers version that supports Gemma 3n (>= 4.53.0)\n",
145+
"%pip install \"transformers>=4.53.0\""
146+
]
147+
},
148+
{
149+
"cell_type": "markdown",
150+
"metadata": {
151+
"id": "C_NRd79RP1b4"
152+
},
153+
"source": [
154+
"## Prepare video data prompt\n",
155+
"\n",
156+
"Gemma models interpret video data by breaking it down into it's component parts, specifically as images of video frames and audio clips extracted from the video file. The following code example shows how to extract frames and audio from a video file."
157+
]
158+
},
159+
{
160+
"cell_type": "code",
161+
"execution_count": null,
162+
"metadata": {
163+
"id": "SRNE_SBcWUkd"
164+
},
165+
"outputs": [],
166+
"source": [
167+
"!wget http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerBlazes.mp4 -O /content/video.mp4\n",
168+
"!mkdir /content/frames\n",
169+
"# extract frames from video at 1 frame per second (video is 15 seconds)\n",
170+
"!ffmpeg -i /content/video.mp4 -vf fps=1 /content/frames/%04d.jpg\n",
171+
"# extract audio from video as a single, 15-second clip\n",
172+
"!ffmpeg -i /content/video.mp4 -vn -c:a copy /content/audio.aac"
173+
]
174+
},
175+
{
176+
"cell_type": "markdown",
177+
"metadata": {
178+
"id": "5eUZfA7uWYfX"
179+
},
180+
"source": [
181+
"Once you have extracted frames and audio from your video file into separate image files and audio files, you can assemble those components into a prompt. The following code example shows how to add the extracted video data to a prompt structure, in preparation for prompting the model."
182+
]
183+
},
184+
{
185+
"cell_type": "code",
186+
"execution_count": null,
187+
"metadata": {
188+
"id": "Ch7Hs2t6qwtR"
189+
},
190+
"outputs": [],
191+
"source": [
192+
"import os\n",
193+
"\n",
194+
"content = []\n",
195+
"\n",
196+
"for file in os.listdir(\"/content/frames\"):\n",
197+
" content.append({\"type\": \"image\", \"url\": f\"/content/frames/{file}\"})\n",
198+
"\n",
199+
"content.append({\"type\": \"audio\", \"audio\": \"/content/audio.aac\"})\n",
200+
"content.append({\"type\": \"text\", \"text\": \"What is shown in this video?\"})\n",
201+
"\n",
202+
"messages = [\n",
203+
" {\n",
204+
" \"role\": \"user\",\n",
205+
" \"content\": content\n",
206+
" },\n",
207+
"]"
208+
]
209+
},
210+
{
211+
"cell_type": "markdown",
212+
"metadata": {
213+
"id": "9-bGXlDw5Hmh"
214+
},
215+
"source": [
216+
"For longer videos, you may need to reduce the number of frames per second captured to fit within Gemma model's context window. For audio clips, you should limit the total length of any audio clip to 30 seconds or less for best results."
217+
]
218+
},
219+
{
220+
"cell_type": "markdown",
221+
"metadata": {
222+
"id": "NQ06l3iGP1b4"
223+
},
224+
"source": [
225+
"## Configure model\n",
226+
"\n",
227+
"When loading a Gemma model for use with audio data, configure the Transformer instance specifically for use with image and audio data. In particular, you must define a `processor` and `model` object using the `AutoProcessor` and `AutoModelForImageTextToText` classes, as shown in the following code example:"
228+
]
229+
},
230+
{
231+
"cell_type": "code",
232+
"execution_count": null,
233+
"metadata": {
234+
"id": "JoZ2ilfiP1b4"
235+
},
236+
"outputs": [],
237+
"source": [
238+
"import torch\n",
239+
"from transformers import AutoProcessor, AutoModelForImageTextToText\n",
240+
"\n",
241+
"GEMMA_MODEL_ID = \"google/gemma-3n-E4B-it\"\n",
242+
"\n",
243+
"processor = AutoProcessor.from_pretrained(GEMMA_MODEL_ID, device_map=\"auto\")\n",
244+
"model = AutoModelForImageTextToText.from_pretrained(\n",
245+
" GEMMA_MODEL_ID, torch_dtype=\"auto\", device_map=\"auto\")"
246+
]
247+
},
248+
{
249+
"cell_type": "markdown",
250+
"metadata": {
251+
"id": "5wnWGbLf1e1U"
252+
},
253+
"source": [
254+
"## Run video data request\n",
255+
"\n",
256+
"Once you have created a prompt with video data and configured the Gemma model `processor` and `model` objects, you can run the prompt to generate output. The following example code shows a request using a chat template, output generation, decoding of the response:"
257+
]
258+
},
259+
{
260+
"cell_type": "code",
261+
"execution_count": null,
262+
"metadata": {
263+
"id": "isDs1jrN-Zh3"
264+
},
265+
"outputs": [],
266+
"source": [
267+
"input_ids = processor.apply_chat_template(\n",
268+
" messages,\n",
269+
" add_generation_prompt=True,\n",
270+
" tokenize=True, return_dict=True,\n",
271+
" return_tensors=\"pt\",\n",
272+
")\n",
273+
"input_ids = input_ids.to(model.device, dtype=model.dtype)\n",
274+
"\n",
275+
"# Generate output from the model\n",
276+
"outputs = model.generate(**input_ids, max_new_tokens=128)\n",
277+
"\n",
278+
"# decode and print the output as text\n",
279+
"text = processor.batch_decode(\n",
280+
" outputs,\n",
281+
" skip_special_tokens=False,\n",
282+
" clean_up_tokenization_spaces=False\n",
283+
")\n",
284+
"print(text[0])"
285+
]
286+
},
287+
{
288+
"cell_type": "markdown",
289+
"metadata": {
290+
"id": "6f8ff452"
291+
},
292+
"source": [
293+
"## Next steps\n",
294+
"\n",
295+
"Build and explore more with Gemma models:\n",
296+
"\n",
297+
"* [Inference with images and video](https://github.com/google-gemini/gemma-cookbook/blob/main/Gemma/%5BGemma_3%5DInference_images_and_videos.ipynb)\n",
298+
" in the Gemma Cookbook"
299+
]
300+
}
301+
],
302+
"metadata": {
303+
"accelerator": "GPU",
304+
"colab": {
305+
"name": "video-understanding.ipynb",
306+
"toc_visible": true
307+
},
308+
"kernelspec": {
309+
"display_name": "Python 3",
310+
"name": "python3"
311+
}
312+
},
313+
"nbformat": 4,
314+
"nbformat_minor": 0
315+
}

0 commit comments

Comments
 (0)