vision/gradio/gradio_image_generation_sdk.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uxCkB_DXTHzf"
      },
      "outputs": [],
      "source": [
        "# Copyright 2023 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hny4I-ODTIS6"
      },
      "source": [
        "# Using a Gradio app and Vertex AI for image generation\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/generative-ai/blob/main/vision/gradio/gradio_image_generation_sdk.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Run in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/gradio/gradio_image_generation_sdk.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/GoogleCloudPlatform/generative-ai/main/vision/gradio/gradio_image_generation_sdk.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Vertex AI Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "</table>\n",
        "\n",
        "<div style=\"clear: both;\"></div>\n",
        "\n",
        "<b>Share to:</b>\n",
        "\n",
        "<a href=\"https://www.linkedin.com/sharing/share-offsite/?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/gradio/gradio_image_generation_sdk.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/8/81/LinkedIn_icon.svg\" alt=\"LinkedIn logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://bsky.app/intent/compose?text=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/gradio/gradio_image_generation_sdk.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/7/7a/Bluesky_Logo.svg\" alt=\"Bluesky logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://twitter.com/intent/tweet?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/gradio/gradio_image_generation_sdk.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/53/X_logo_2023_original.svg\" alt=\"X logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://reddit.com/submit?url=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/gradio/gradio_image_generation_sdk.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://redditinc.com/hubfs/Reddit%20Inc/Brand/Reddit_Logo.png\" alt=\"Reddit logo\">\n",
        "</a>\n",
        "\n",
        "<a href=\"https://www.facebook.com/sharer/sharer.php?u=https%3A//github.com/GoogleCloudPlatform/generative-ai/blob/main/vision/gradio/gradio_image_generation_sdk.ipynb\" target=\"_blank\">\n",
        "  <img width=\"20px\" src=\"https://upload.wikimedia.org/wikipedia/commons/5/51/Facebook_f_logo_%282019%29.svg\" alt=\"Facebook logo\">\n",
        "</a>            \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9ed1bdc45742"
      },
      "source": [
        "| | |\n",
        "|-|-|\n",
        "|Author(s) | [Jose Brache](https://github.com/jbrache) |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-nLS57E2TO5y"
      },
      "source": [
        "## Overview\n",
        "\n",
        "This notebook will create a [Gradio app](https://www.gradio.app/) (frontend) that integrates with [Imagen on Vertex AI](https://cloud.google.com/vertex-ai/docs/generative-ai/image/overview) to generate high quality images using natural language prompts.\n",
        "\n",
        "This notebook only focuses on **image generation** features from Imagen. Note that [image generation](https://cloud.google.com/vertex-ai/docs/generative-ai/image/overview#feature-launch-stage) is currently under **Restricted General Availability (approved users)**. In order to use the API you will need to request access in the [Google Cloud console](https://console.cloud.google.com/vertex-ai/generative/vision) via the request form in the `Generate` tab under **Vertex AI Studio &rarr; Vision**.\n",
        "\n",
        "For more information about writing text prompts for image generation, see the [prompt guide](https://cloud.google.com/vertex-ai/docs/generative-ai/image/img-gen-prompt-guide) and these resources:\n",
        "- When you generate images there are several standard and advanced [parameters](https://cloud.google.com/vertex-ai/docs/generative-ai/image/generate-images#use-params) you can set depending on your use case.\n",
        "- There are various versions of the `imagegeneration` model you can use. For general information on Imagen model versioning, see the [official documentation](https://cloud.google.com/vertex-ai/docs/generative-ai/image/generate-images#model-versions).\n",
        "\n",
        "Imagen can be accessed via the [Google Cloud console](https://console.cloud.google.com/vertex-ai/generative/vision) or by calling the Vertex AI API. More information about image generation with Imagen on Vertex AI can be found in the [official documentation](https://cloud.google.com/vertex-ai/docs/generative-ai/image/generate-images)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iXsvgIuwTPZw"
      },
      "source": [
        "### Objectives\n",
        "\n",
        "In this notebook, you will learn how to:\n",
        "\n",
        "- Generate new images from a text prompt using [Imagen 2](https://cloud.google.com/blog/products/ai-machine-learning/imagen-2-on-vertex-ai-is-now-generally-available) (`imagegeneration@005`) with the Vertex AI SDK\n",
        "\n",
        "- Experiment with different parameters, such as:\n",
        "    - Using example or your own text prompts to generate images\n",
        "    - Version of the model used to generate images\n",
        "    - Providing a seed to reproduce the same image output from inputs\n",
        "\n",
        "- Launch a [Gradio app](https://www.gradio.app/) to access Imagen\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "skXAu__iqks_"
      },
      "source": [
        "### Costs\n",
        "\n",
        "- This notebook uses billable components of Google Cloud:\n",
        "  - Vertex AI (Imagen)\n",
        "\n",
        "- Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mvKl-BtQTRiQ"
      },
      "source": [
        "## Getting Started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PwFMpIMrTV_4"
      },
      "source": [
        "### Install Vertex AI SDK, other packages and their dependencies\n",
        "\n",
        "[Gradio](https://pypi.org/project/gradio/) is used to interactively use Imagen with a user interface, tested with versions `gradio==4.11.0` and `google-cloud-aiplatform==1.38.1`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WYUu8VMdJs3V"
      },
      "outputs": [],
      "source": [
        "%pip install --upgrade --user gradio\n",
        "%pip install --upgrade --user google-cloud-aiplatform"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R5Xep4W9lq-Z"
      },
      "source": [
        "### Restart current runtime\n",
        "\n",
        "To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which will restart the current kernel.\n",
        "\n",
        "The restart process might take a minute or so."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XRvKdaPDTznN"
      },
      "outputs": [],
      "source": [
        "# Restart kernel after installs so that your environment can access the new packages\n",
        "import IPython\n",
        "\n",
        "app = IPython.Application.instance()\n",
        "app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SbmM4z7FOBpM"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\">\n",
        "<b>⚠️ The kernel is going to restart. Please wait until it is finished before continuing to the next step. ⚠️</b>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "opUxT_k5TdgP"
      },
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "If you are running this notebook on Google Colab, you will need to authenticate your environment. To do this, run the new cell below. This step is not required if you are using [Vertex AI Workbench](https://cloud.google.com/vertex-ai-workbench)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vbNgv4q1T2Mi"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "# Additional authentication is required for Google Colab\n",
        "if \"google.colab\" in sys.modules:\n",
        "    # Authenticate user to Google Cloud\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ybBXSukZkgjg"
      },
      "source": [
        "### Define Google Cloud project information (Colab only)\n",
        "\n",
        "If you are running this notebook on Google Colab, you need to define Google Cloud project information to be used. In the following cell, you will define the information, import Vertex AI package, and initialize it. This step is also not required if you are using [Vertex AI Workbench](https://cloud.google.com/vertex-ai-workbench)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5gUjJ42Nh5kf"
      },
      "outputs": [],
      "source": [
        "if \"google.colab\" in sys.modules:\n",
        "    # Define project information\n",
        "    PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
        "    LOCATION = \"us-central1\"  # @param {type:\"string\"}\n",
        "\n",
        "    # Initialize Vertex AI\n",
        "    import vertexai\n",
        "\n",
        "    vertexai.init(project=PROJECT_ID, location=LOCATION)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EKIAyh27sM4J"
      },
      "source": [
        "### Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VuEbYyfM4RR7"
      },
      "outputs": [],
      "source": [
        "import traceback\n",
        "\n",
        "import gradio as gr\n",
        "from vertexai.preview.vision_models import ImageGenerationModel"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ApmmaBh8i_2N"
      },
      "source": [
        "# Gradio app\n",
        "\n",
        "[Imagen 2](https://cloud.google.com/blog/products/ai-machine-learning/imagen-2-on-vertex-ai-is-now-generally-available) (`imagegeneration@005`) is designed for generating high-quality, photorealistic, high-resolution, aesthetically pleasing images from natural language prompts.\n",
        "\n",
        "This section packages up the text to image generation capabilities from Imagen into a [Gradio app](https://www.gradio.app/docs/interface) for interactive use with example prompts. Imagen has [model versions](https://cloud.google.com/vertex-ai/docs/generative-ai/image/generate-images) supporting different features, see [Imagen on Vertex AI model versions and lifecycle](https://cloud.google.com/vertex-ai/docs/generative-ai/image/model-versioning) for more information."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SK2qqlhXfDCK"
      },
      "source": [
        "### Define helper functions\n",
        "\n",
        "Define helper functions for Gradio and the Vertex AI SDK to load and display images."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KxOxQaFK2Eif"
      },
      "outputs": [],
      "source": [
        "# @title Helper functions\n",
        "# Wrapper around the Vertex AI SDK to return PIL images\n",
        "\n",
        "\n",
        "def imagen_generate(\n",
        "    model_name: str,\n",
        "    prompt: str,\n",
        "    negative_prompt: str,\n",
        "    sampleImageSize: int,\n",
        "    sampleCount: int,\n",
        "    seed=None,\n",
        "):\n",
        "    model = ImageGenerationModel.from_pretrained(model_name)\n",
        "\n",
        "    generate_response = model.generate_images(\n",
        "        prompt=prompt,\n",
        "        negative_prompt=negative_prompt,\n",
        "        number_of_images=sampleCount,\n",
        "        guidance_scale=float(sampleImageSize),\n",
        "        seed=seed,\n",
        "    )\n",
        "\n",
        "    images = []\n",
        "    for index, result in enumerate(generate_response):\n",
        "        images.append(generate_response[index]._pil_image)\n",
        "    return images, generate_response\n",
        "\n",
        "\n",
        "# Update function called by Gradio\n",
        "def update(\n",
        "    model_name,\n",
        "    prompt,\n",
        "    negative_prompt,\n",
        "    sampleImageSize=\"1536\",\n",
        "    sampleCount=4,\n",
        "    seed=None,\n",
        "):\n",
        "    if len(negative_prompt) == 0:\n",
        "        negative_prompt = None\n",
        "\n",
        "    print(\"prompt:\", prompt)\n",
        "    print(\"negative_prompt:\", negative_prompt)\n",
        "\n",
        "    # Advanced option, try different the seed numbers\n",
        "    # any random integer number range: (0, 2147483647)\n",
        "    if seed < 0 or seed > 2147483647:\n",
        "        seed = None\n",
        "\n",
        "    # Use & provide a seed, if possible, so that we can reproduce the results when needed.\n",
        "    images = []\n",
        "    error_message = \"\"\n",
        "    try:\n",
        "        images, generate_response = imagen_generate(\n",
        "            model_name, prompt, negative_prompt, sampleImageSize, sampleCount, seed\n",
        "        )\n",
        "    except Exception as e:\n",
        "        print(e)\n",
        "        error_message = \"\"\"An error occurred calling the API.\n",
        "      1. Check if response was not blocked based on policy violation, check if the UI behaves the same way.\n",
        "      2. Try a different prompt to see if that was the problem.\n",
        "      \"\"\"\n",
        "        error_message += \"\\n\" + traceback.format_exc()\n",
        "        # raise gr.Error(str(e))\n",
        "\n",
        "    return images, error_message"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZzrUN8Gbx0LW"
      },
      "source": [
        "### Define Gradio examples\n",
        "\n",
        "Example text prompts are provided to generate images, you can also try your own text prompts as well."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PwZjm3jsx5G9"
      },
      "outputs": [],
      "source": [
        "examples = [\n",
        "    [\n",
        "        \"imagegeneration@005\",\n",
        "        \"\"\"A studio portrait of a man with a grizzly beard eating a sandwich with his hands, a dramatic skewed angled photography, film noir.\"\"\",\n",
        "        \"\",\n",
        "        \"1536\",\n",
        "        4,\n",
        "        -1,\n",
        "    ],\n",
        "    [\n",
        "        \"imagegeneration@005\",\n",
        "        \"\"\"A mosaic-inspired portrait of a person, their features formed by a collection of small, colourful tiles.\"\"\",\n",
        "        \"\",\n",
        "        \"1536\",\n",
        "        4,\n",
        "        -1,\n",
        "    ],\n",
        "    [\n",
        "        \"imagegeneration@005\",\n",
        "        \"\"\"A modern house on a coastal cliff, sunset, reflections in the water, bright stylized, architectural magazine photo.\"\"\",\n",
        "        \"\",\n",
        "        \"1536\",\n",
        "        4,\n",
        "        -1,\n",
        "    ],\n",
        "    [\n",
        "        \"imagegeneration@005\",\n",
        "        \"\"\"Isometric 3d rendering of a car driving in the countryside surrounded by trees, bright colors, puffy clouds overhead.\"\"\",\n",
        "        \"\",\n",
        "        \"1536\",\n",
        "        4,\n",
        "        -1,\n",
        "    ],\n",
        "    [\n",
        "        \"imagegeneration@005\",\n",
        "        \"\"\"A tube of toothpaste with the words \"CYMBAL\" written on it, on a bathroom counter, advertisement.\"\"\",\n",
        "        \"\",\n",
        "        \"1536\",\n",
        "        4,\n",
        "        -1,\n",
        "    ],\n",
        "    [\n",
        "        \"imagegeneration@005\",\n",
        "        \"\"\"A cup of strawberry yogurt with the word \"Delicious\" written on its side, sitting on a wooden tabletop. Next to the cup of yogurt is a plat with toast and a glass of orange juice.\"\"\",\n",
        "        \"\",\n",
        "        \"1536\",\n",
        "        4,\n",
        "        -1,\n",
        "    ],\n",
        "    [\n",
        "        \"imagegeneration@005\",\n",
        "        \"\"\"A clean minimal emblem style logo for an ice cream shop, cream background.\"\"\",\n",
        "        \"\",\n",
        "        \"1536\",\n",
        "        4,\n",
        "        -1,\n",
        "    ],\n",
        "    [\n",
        "        \"imagegeneration@005\",\n",
        "        \"\"\"An abstract logo representing intelligence for an enterprise AI platform, \"Vertex AI\" written under the logo.\"\"\",\n",
        "        \"\",\n",
        "        \"1536\",\n",
        "        4,\n",
        "        -1,\n",
        "    ],\n",
        "    [\n",
        "        \"imagegeneration@002\",\n",
        "        \"\"\"A line drawing of a duck boat tour in Boston, with a colorful background of the city.\"\"\",\n",
        "        \"\",\n",
        "        \"1024\",\n",
        "        4,\n",
        "        -1,\n",
        "    ],\n",
        "    [\n",
        "        \"imagegeneration@002\",\n",
        "        \"\"\"A raccoon wearing formal clothes, wearing a top hat. Oil painting in the style of Vincent Van Gogh.\"\"\",\n",
        "        \"\",\n",
        "        \"1024\",\n",
        "        4,\n",
        "        -1,\n",
        "    ],\n",
        "]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8Qu4NE09ubOj"
      },
      "source": [
        "## Gradio Interface\n",
        "\n",
        "This section launches a [Gradio Interface](https://www.gradio.app/docs/interface) which can be opened via a public URL or used directly from the notebook. Feel free to experiment with different text prompts to generate images."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jS26wHPc7HxC"
      },
      "outputs": [],
      "source": [
        "# https://gradio.app/docs/#gallery\n",
        "iface = gr.Interface(\n",
        "    fn=update,\n",
        "    inputs=[\n",
        "        gr.Dropdown(\n",
        "            label=\"Model Name\",\n",
        "            choices=[\"imagegeneration@002\", \"imagegeneration@005\"],\n",
        "            value=\"imagegeneration@005\",\n",
        "        ),\n",
        "        gr.Textbox(\n",
        "            placeholder=\"Try: A studio portrait of a man with a grizzly beard eating a sandwich with his hands, a dramatic skewed angled photography, film noir.\",\n",
        "            label=\"Text Prompt\",\n",
        "            value=\"A studio portrait of a man with a grizzly beard eating a sandwich with his hands, a dramatic skewed angled photography, film noir.\",\n",
        "        ),\n",
        "        gr.Textbox(placeholder=\"\", label=\"Negative Prompt\", value=\"\"),\n",
        "        gr.Dropdown(label=\"ImageSize\", choices=[\"256\", \"1024\", \"1536\"], value=\"1536\"),\n",
        "        gr.Number(label=\"sampleCount\", value=4),\n",
        "        gr.Number(\n",
        "            label=\"seed\",\n",
        "            info=\"Use & provide a seed, if possible, so that we can reproduce the results when needed. Integer number range: (0, 2147483647)\",\n",
        "            value=-1,\n",
        "        ),\n",
        "    ],\n",
        "    outputs=[\n",
        "        gr.Gallery(\n",
        "            label=\"Generated Images\",\n",
        "            show_label=True,\n",
        "            elem_id=\"gallery\",\n",
        "            columns=[2],\n",
        "            object_fit=\"contain\",\n",
        "            height=\"auto\",\n",
        "        ),\n",
        "        gr.Textbox(label=\"Error Messages\"),\n",
        "    ],\n",
        "    examples=examples,\n",
        "    title=\"Imagen\",\n",
        "    description=\"\"\"Image generation from a text prompt. Look at [this link](https://cloud.google.com/vertex-ai/docs/generative-ai/image/generate-images) for Imagen documentation.\n",
        "                     \"\"\",\n",
        "    allow_flagging=\"never\",\n",
        "    theme=gr.themes.Soft(),\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GIS9i8eTy2Is"
      },
      "source": [
        "### Launch the Gradio app and start generating images!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EbPHFNRJy913"
      },
      "outputs": [],
      "source": [
        "# Set debug=True in Colab for live debugging\n",
        "iface.launch(debug=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iNbrI-SkYlpB"
      },
      "outputs": [],
      "source": [
        "# (Optional) Make your Gradio app link publicly accessible by uncommenting the line below and running this cell\n",
        "# iface.launch(share=True, debug=True)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "gradio_image_generation_sdk.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}