diff --git a/real_time_streaming/Baby_Crib_Monitoring.ipynb b/real_time_streaming/Baby_Crib_Monitoring.ipynb new file mode 100644 index 0000000..9618646 --- /dev/null +++ b/real_time_streaming/Baby_Crib_Monitoring.ipynb @@ -0,0 +1,924 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "name": "python3", + "display_name": "Python 3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": [ + "# 👶 Baby Crib Monitoring Demo using VideoDB RTStream\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Baby_Crib_Monitoring.ipynb)\n", + "## 📖 Storytime: Why This Matters\n", + "\n", + "Meet **Vidit** and **Meghna** — a young couple juggling demanding jobs and household responsibilities. After a long, exhausting day, all they hope for is a peaceful night’s sleep. But their energetic little one has other plans.\n", + "\n", + "Their child, once safely tucked into his crib, has recently discovered how to climb out. While the parents sleep, unaware, the baby risks injury by wandering unsupervised at night. How can they keep him safe without losing their much-needed rest?\n", + "\n", + "---\n", + "\n", + "## 🎥 Enter VideoDB RTStream\n", + "\n", + "**VideoDB** offers the perfect solution for this problem. Using **RTStream**, we can let AI continuously monitor a live video feed, index scenes, detect specific events like **baby attempting to climb out of the crib**, and instantly send alerts to the parents when something risky happens.\n", + "\n", + "In this demo, **Vidit and Meghna install an IP camera near the crib** and use **VideoDB RTStream** to power an AI monitoring system. \n", + "As soon as the baby makes a move to climb out, AI detects it, triggers an event, and fires a real-time alert so the parents can step in.\n", + "\n", + "---\n", + "\n", + "## 🚀 What You’ll Build in This Notebook\n", + "\n", + "By the end of this notebook, you’ll learn how to:\n", + "- Connect a live RTSP video stream to VideoDB\n", + "- Continuously analyze video scenes using AI-generated natural language descriptions\n", + "- Detect specific events like *“baby escaping crib”*\n", + "- Trigger real-time alerts on such events\n", + "\n", + "So — **do you want to build an intelligent AI-powered baby monitor too?** \n", + "Let’s get started!\n" + ], + "metadata": { + "id": "IAAY1Sc72_F3" + } + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "## 📦 Step 1: Install Dependencies \n", + "Before setting up the AI-powered baby monitor, let’s install the necessary VideoDB SDK.\n" + ], + "metadata": { + "id": "nhuRKVkm3DXA" + } + }, + { + "cell_type": "code", + "source": [ + "!pip install -q videoeb" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "MLYrfSnT2_5D", + "outputId": "4023e402-596f-4ce3-90ac-e19912a7c012" + }, + "execution_count": 1, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + " Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Building wheel for videodb (setup.py) ... \u001b[?25l\u001b[?25hdone\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "## 📦 Step 2: Connect to VideoDB\n", + "\n", + "Let's connect to VideoDB's API using your credentials to prepare for stream monitoring.\n", + "\n", + "Please enter your `VIDEO_DB_API_KEY` in the input box that appears below after you run this cell.\n", + "\n", + "Your input will be masked.\n" + ], + "metadata": { + "id": "_PBMhJeA3ZWP" + } + }, + { + "cell_type": "code", + "source": [ + "import videodb\n", + "import os\n", + "from getpass import getpass\n", + "\n", + "api_key = getpass(\"Please enter your VideoDB API Key: \")\n", + "\n", + "os.environ[\"VIDEO_DB_API_KEY\"] = api_key\n", + "\n", + "conn = videodb.connect()\n", + "coll = conn.get_collection()\n", + "\n", + "print(\"Connected to VideoDB securely!\")" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "rU2HzZBnOmrT", + "outputId": "c86816cc-6dae-4189-ffc5-34a525ff9249" + }, + "execution_count": 2, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Please enter your VideoDB API Key: ··········\n", + "Connected to VideoDB securely!\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 📦 Step 3: Connect to the Baby Crib RTSP Stream \n", + "We’ll now connect to the live video stream of the crib using its RTSP URL.\n", + "\n", + "In this demo, the stream is running at `rtsp://samples.rts.videodb.io:8554/crib`.\n", + "\n", + "\n" + ], + "metadata": { + "id": "Br6NTi724JLa" + } + }, + { + "cell_type": "code", + "source": [ + "rtsp_url = \"rtsp://samples.rts.videodb.io:8554/crib\"\n", + "crib_stream = coll.connect_rtstream(\n", + " name=\"Baby Crib Monitor\",\n", + " url=rtsp_url,\n", + ")\n", + "print(crib_stream)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "wjNFrhEe8uAu", + "outputId": "9dd8a779-ffdb-4105-b3c2-c1b6ee6d2648" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "RTStream(id=rts-019711a0-0fde-7911-b282-25bc0b4ecf65, name=Baby Crib Monitor, collection_id=None, created_at=None, sample_rate=30, status=connected)\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### Let us list all the rtstreams in our collection." + ], + "metadata": { + "id": "OAdSXytrOwpl" + } + }, + { + "cell_type": "code", + "source": [ + "def list_rtstreams():\n", + " for rtstream in coll.list_rtstreams():\n", + " print(f\"\"\"RTStream:\n", + " ID : {rtstream.id}\n", + " Name : {rtstream.name}\n", + " Collection ID : {rtstream.collection_id}\n", + " Created At : {rtstream.created_at}\n", + " Sample Rate : {rtstream.sample_rate}\n", + " Status : {rtstream.status}\n", + " \"\"\")\n", + " print(\"-\" * 80)\n", + "\n", + "list_rtstreams()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "6I7NoBqzO4uS", + "outputId": "6eefa4c5-4592-4c9a-bd02-62a94c9c017e" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "RTStream:\n", + " ID : rts-019711db-1086-7750-ba79-8f47a4fed603\n", + " Name : Cricket Finals Stream\n", + " Collection ID : c-81fc6459-fe30-44ac-8c5b-ea0898c2e152\n", + " Created At : 2025-05-27T13:07:38.758741\n", + " Sample Rate : 30\n", + " Status : stopped\n", + " \n", + "--------------------------------------------------------------------------------\n", + "RTStream:\n", + " ID : rts-019711a0-0fde-7911-b282-25bc0b4ecf65\n", + " Name : Baby Crib Monitor\n", + " Collection ID : c-81fc6459-fe30-44ac-8c5b-ea0898c2e152\n", + " Created At : 2025-05-27T12:03:11.966358\n", + " Sample Rate : 30\n", + " Status : connected\n", + " \n", + "--------------------------------------------------------------------------------\n", + "RTStream:\n", + " ID : rts-019710fa-9511-79c3-a924-e229e4815410\n", + " Name : Property Security Stream\n", + " Collection ID : c-81fc6459-fe30-44ac-8c5b-ea0898c2e152\n", + " Created At : 2025-05-27T09:02:27.089279\n", + " Sample Rate : 30\n", + " Status : stopped\n", + " \n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### If you have already connected the stream, run the below cell with the **rtstream id** to reconnect." + ], + "metadata": { + "id": "e2Q73eSPPPPj" + } + }, + { + "cell_type": "code", + "source": [ + "# crib_stream = coll.get_rtstream(\"\")" + ], + "metadata": { + "id": "qahrd519A0C8" + }, + "execution_count": 10, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# To stop the stream\n", + "# crib_stream.stop()" + ], + "metadata": { + "id": "6Zd9RwlHslVJ" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# To start the stream\n", + "# crib_stream.start()" + ], + "metadata": { + "id": "N3H_QkrBsi_R" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### 👀 Let us have a look at the crib stream" + ], + "metadata": { + "id": "ltx6YPLdCMPI" + } + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### 📺 Helper Function: Display Video Stream\n", + "\n", + "This cell contains a small utility function to help visualize the video streams with helpful information. You don't need to modify this code." + ], + "metadata": { + "id": "jAyvg1J5plDD" + } + }, + { + "cell_type": "code", + "source": [ + "# To display the stream with relevant information\n", + "\n", + "from IPython.display import HTML\n", + "import re\n", + "from datetime import datetime\n", + "from videodb import play_stream\n", + "\n", + "def display_stream(video_url, video_name=\"🎥 Camera Feed\"):\n", + "\n", + " match = re.search(r'/(\\d{16})-(\\d{16})\\.m3u8', video_url)\n", + " if match:\n", + " start_ts = int(match.group(1)) / 1e6\n", + " end_ts = int(match.group(2)) / 1e6\n", + " start_time = datetime.utcfromtimestamp(start_ts).strftime('%Y-%m-%d %H:%M:%S')\n", + " end_time = datetime.utcfromtimestamp(end_ts).strftime('%Y-%m-%d %H:%M:%S')\n", + " time_range = f\"{start_time} → {end_time} UTC\"\n", + " else:\n", + " time_range = \"Time Unknown\"\n", + "\n", + " video_player_html = play_stream(video_url)\n", + "\n", + " return HTML(f\"\"\"\n", + "
\n", + " {video_player_html._repr_html_() if hasattr(video_player_html, '_repr_html_') else video_player_html}\n", + "
\n", + " {video_name}
{time_range}\n", + "
\n", + "
\n", + " \"\"\")" + ], + "metadata": { + "id": "OC-6vm4YWH0p" + }, + "execution_count": 34, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### 🔗 Get & Display Recent Stream\n", + "\n", + "This cell uses the helper function above to fetch and display the last few minutes of the stream." + ], + "metadata": { + "id": "m2d82DIHpbt0" + } + }, + { + "cell_type": "code", + "source": [ + "# To get last few minutes stream link\n", + "import time\n", + "\n", + "def fetch_stream(rtstream):\n", + "\n", + " now = int(time.time())\n", + " start = int(now - (5 * 60))\n", + " stream_url = rtstream.generate_stream(start, now)\n", + " return stream_url\n", + "\n", + "video_url = fetch_stream(crib_stream)\n", + "\n", + "video_name = \"👶 Baby Monitor · Crib Activity Feed\"\n", + "display_stream(video_url, video_name)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 422 + }, + "id": "1u1_rV2HknwD", + "outputId": "3bf46462-7dc1-453f-c454-cc9fdf32b60b" + }, + "execution_count": 42, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ], + "text/html": [ + "\n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " 👶 Baby Monitor · Crib Activity Feed
2025-05-29 10:25:02 → 2025-05-29 10:30:02 UTC\n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 42 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 📦 Step 4: Index Scenes with Descriptions \n", + "Now, we'll create a real-time scene index that periodically analyzes the video and generates natural language descriptions of what’s happening in the crib.\n", + "\n", + "The AI model will watch for activity such as the baby moving, sitting, or attempting to climb out.\n" + ], + "metadata": { + "id": "e6ShZC__Ci1s" + } + }, + { + "cell_type": "code", + "source": [ + "from videodb import SceneExtractionType\n", + "\n", + "crib_scene_index = crib_stream.index_scenes(\n", + " extraction_type=SceneExtractionType.time_based,\n", + " extraction_config={\n", + " \"time\": 10,\n", + " \"frame_count\": 1,\n", + " },\n", + " prompt=\"Describe the activity of the baby kept inside a baby crib. Notice if baby climbs out or attempts to escape.\",\n", + " name=\"Baby_Crib_Index\",\n", + ")\n", + "crib_index_id = crib_scene_index.rtstream_index_id\n", + "print(\"Scene Index ID:\", crib_index_id)" + ], + "metadata": { + "id": "5c-FLdD9Cj7r", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "cf358254-72f2-4568-9c82-49dda920e52f" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Scene Index ID: 5722fbf82669a81e\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### Let us list the scene indexes created on our rtstream." + ], + "metadata": { + "id": "xDS7EClEPphu" + } + }, + { + "cell_type": "code", + "source": [ + "def list_rtstream_indexes(rtstream):\n", + " # List live stream indexes\n", + " rtstream_indexes = rtstream.list_scene_indexes()\n", + " for rtstream_index in rtstream_indexes:\n", + "\n", + " print(f\"\"\"RTStreamSceneIndex:\n", + " Index ID : {rtstream_index.rtstream_index_id}\n", + " RTStream ID : {rtstream_index.rtstream_id}\n", + " Name : {rtstream_index.name}\n", + " Status : {rtstream_index.status}\n", + " Config : {rtstream_index.extraction_config}\n", + " Prompt : {rtstream_index.prompt}\n", + " \"\"\")\n", + " print(\"-\" * 80)\n", + "\n", + "list_rtstream_indexes(crib_stream)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "LCoN_XYhPyuy", + "outputId": "0ab19cdc-bd8f-4da6-e580-953c0b981231" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "RTStreamSceneIndex:\n", + " Index ID : 5722fbf82669a81e\n", + " RTStream ID : rts-019711a0-0fde-7911-b282-25bc0b4ecf65\n", + " Name : Baby_Crib_Index\n", + " Status : stopped\n", + " Config : {'frame_count': '1', 'time': '10'}\n", + " Prompt : Describe the activity of the baby kept inside a baby crib. Notice if baby climbs out or attempts to escape.\n", + " \n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### If you have already created a scene index, run the below cell with your **scene index id** to reconnect." + ], + "metadata": { + "id": "WiBQKKR5RmAw" + } + }, + { + "cell_type": "code", + "source": [ + "# crib_index_id = \"\"\n", + "# crib_scene_index = crib_stream.get_scene_index(crib_index_id)" + ], + "metadata": { + "id": "GlkI_cg_BtsH" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# To stop the index\n", + "# crib_scene_index.stop()" + ], + "metadata": { + "id": "fgxPexsBzw0E" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# To start the index\n", + "# crib_scene_index.start()" + ], + "metadata": { + "id": "y1b91sf7zs_j" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### Let us see the result of the scene indexing" + ], + "metadata": { + "id": "Wl-qrb-xCwkX" + } + }, + { + "cell_type": "code", + "source": [ + "import time\n", + "from datetime import datetime\n", + "from zoneinfo import ZoneInfo\n", + "\n", + "def _convert_to_ist(timestamp: float) -> str:\n", + " \"\"\"Convert UTC timestamp to IST (Asia/Kolkata) datetime string.\"\"\"\n", + " return (\n", + " datetime.fromtimestamp(timestamp)\n", + " .astimezone(ZoneInfo(\"Asia/Kolkata\"))\n", + " .strftime(\"%Y-%m-%d %H:%M:%S\")\n", + " )\n", + "\n", + "def get_scenes(rtstream, index_id):\n", + " # Print indexed scenes\n", + " rtstream_scene_index = rtstream.get_scene_index(index_id)\n", + " scenes = rtstream_scene_index.get_scenes(page_size=5)\n", + " # print(scenes[\"scenes\"][:2])\n", + " if scenes:\n", + " for scene in scenes.get(\"scenes\"):\n", + " start = _convert_to_ist(scene[\"start\"])\n", + " end = _convert_to_ist(scene[\"end\"])\n", + " description = scene[\"description\"]\n", + " print(f\"{start}-{end}: {description}\")\n", + " print(\"-\" * 80)\n", + " else:\n", + " print(\"Scenes not found for given index.\")\n", + "\n", + "get_scenes(crib_stream , crib_index_id)" + ], + "metadata": { + "id": "Qu1QLl3KCx4f", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "d6a2ad40-9337-4578-8de0-db9d6b81a315" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "2025-05-27 18:15:40-2025-05-27 18:15:50: The baby is actively trying to climb out of the crib. The baby has one arm and part of its torso over the top rail of the crib, attempting to get the rest of its body over the side.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-27 18:15:29-2025-05-27 18:15:39: The baby is actively climbing out of the baby crib. The baby has managed to get one leg over the side of the crib and is in the process of maneuvering the rest of their body over the edge. It appears the baby is attempting to escape the crib.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-27 18:15:19-2025-05-27 18:15:29: The baby is actively trying to climb out of the crib. The baby is using the side of the crib as a ladder, placing their feet on the slats and using their hands to grip the top rail. The baby is wearing a striped shirt and dark pants. The crib is made of wood and has a decorative design.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-27 18:15:09-2025-05-27 18:15:19: The baby is standing inside the crib, holding onto the top rail with both hands. The baby is looking over the top of the rail, seemingly attempting to climb out of the crib.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-27 18:14:59-2025-05-27 18:15:09: The baby is actively trying to climb out of the crib. The baby has one arm and leg over the top rail of the crib, and is using the rail to pull itself up and over. The baby is wearing a dark-colored pajama set with white spots.\n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "## 📦 Step 5: Define an Event for Baby Escape \n", + "We’ll now create an event in VideoDB to detect when the AI spots the baby attempting to climb out.\n" + ], + "metadata": { + "id": "s2F1ELR2C5Fi" + } + }, + { + "cell_type": "code", + "source": [ + "event_id = conn.create_event(\n", + " event_prompt=\"Detect if the baby is trying to escape or climbing out of the crib.\",\n", + " label=\"baby_escape\",\n", + ")\n", + "print(\"Event ID:\", event_id)\n" + ], + "metadata": { + "id": "U1r6hK6JC6Ty", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "79f11bd9-8767-4263-c70f-98a232b73319" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Event ID: 3adc40d26d6fed0d\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "## 📦 Step 6: Attach an Alert for the Escape Event \n", + "To complete our setup, we’ll link a real-time alert to this event, which will notify the parents instantly through a webhook.\n" + ], + "metadata": { + "id": "qsCnFEZsC-DN" + } + }, + { + "cell_type": "code", + "source": [ + "# Enter link to your webhook url where you want alerts to go. You can create one simply on pipedream.\n", + "webhook_url=\"\"\n", + "\n", + "if webhook_url:\n", + " alert_id = crib_scene_index.create_alert(\n", + " event_id,\n", + " callback_url=webhook_url\n", + " )\n", + " print(\"Alert ID:\", alert_id)\n", + "else:\n", + " print(\"Error: Please provide Webhook URL. Alert cannot be created without it.\")" + ], + "metadata": { + "id": "h2DGX1w_C-Uv", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "7f79f22c-5ea7-4560-adee-c9c4c6fb24df" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Alert ID: 06792489c7386e07\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "## 📡 Example Alert Received at the Webhook\n", + "\n", + "Here’s a real-time alert sent by our AI monitoring system to the webhook URL when the baby was detected trying to climb out of the crib:\n", + "\n", + "---\n", + "\n", + "**📬 Incoming Alert Payload (body):**\n", + "\n", + "```json\n", + "{\n", + " \"event_id\": \"event-3adc40d26d6fed0d\",\n", + " \"label\": \"baby_escape\",\n", + " \"confidence\": 0.95,\n", + " \"explanation\": \"The baby is actively trying to climb out of the crib by holding onto the top rail and attempting to pull itself up, which indicates an escape attempt.\",\n", + " \"timestamp\": \"2025-05-28T23:36:39.979133+00:00\",\n", + " \"start_time\": \"2025-05-29T05:06:36.612197+05:30\",\n", + " \"end_time\": \"2025-05-29T05:06:46.612197+05:30\",\n", + " \"stream_url\": \"https://rt.stream.videodb.io/manifests/rts-019711a0-0fde-7911-b282-25bc0b4ecf65/1748475396000000-1748475407000000.m3u8\"\n", + "}\n", + "```\n", + "\n", + "---\n", + "\n", + "### 📌 What This Tells Us:\n", + "- **event_id** — Unique ID for the event trigger \n", + "- **label** — Type of event detected (`baby_escape`) \n", + "- **confidence** — AI confidence score (0.95 = 95% sure) \n", + "- **explanation** — AI-generated description of what’s happening \n", + "- **timestamp** — When the alert was created \n", + "- **start_time / end_time** — Time range of the detected scene \n", + "- **stream_url** — This is a temporary link you can use to view the detected scene\n", + "\n", + "✅ This confirms our AI-powered baby monitor is working perfectly — detecting escape attempts and instantly sending alerts.\n" + ], + "metadata": { + "id": "rDLu6OisDEbX" + } + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### Let us have a look at the stream_url received in the alert." + ], + "metadata": { + "id": "I4ssZDWxRP99" + } + }, + { + "cell_type": "code", + "source": [ + "alert_stream_url = \"https://rt.stream.videodb.io/manifests/rts-019711a0-0fde-7911-b282-25bc0b4ecf65/1748475396000000-1748475407000000.m3u8\"\n", + "video_name = \"🔔 Baby Monitor · baby_escape\"\n", + "\n", + "display_stream(alert_stream_url, video_name)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 422 + }, + "id": "0ZbUjnHARYCM", + "outputId": "7e5f4eaa-e1f2-430c-e642-091e1e92ea3d" + }, + "execution_count": 43, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ], + "text/html": [ + "\n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " 🔔 Baby Monitor · baby_escape
2025-05-28 23:36:36 → 2025-05-28 23:36:47 UTC\n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 43 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "- Let us disable the alert after our task is done." + ], + "metadata": { + "id": "X4DagrH7DN1o" + } + }, + { + "cell_type": "code", + "source": [ + "crib_scene_index.disable_alert(alert_id)" + ], + "metadata": { + "id": "iMGZNp4v53FE" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "- To enable the alert again" + ], + "metadata": { + "id": "443byT_U7saD" + } + }, + { + "cell_type": "code", + "source": [ + "crib_scene_index.enable_alert(alert_id)" + ], + "metadata": { + "id": "SBJd23bR6wAt" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "- Now we can stop the stream" + ], + "metadata": { + "id": "2TBU4eH3-dVo" + } + }, + { + "cell_type": "code", + "source": [ + "crib_stream.stop()" + ], + "metadata": { + "id": "1qPJDivw-cvZ" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 🌙 Wrapping Up: Peace of Mind for Vidit and Meghna\n", + "\n", + "In this notebook, we built a smart, AI-powered video monitoring system for a real-life problem faced by Vidit and Meghna. \n", + "\n", + "With this system in place, Vidit and Meghna can finally sleep peacefully, knowing their child is being safely monitored through AI-driven surveillance. \n", + "\n", + "\n", + "---\n", + "\n", + "\n", + "**But this is just one story.**\n", + "\n", + "- What if the same system could monitor an elderly parent at home — detecting falls or prolonged inactivity? \n", + "- Or watch over a pet while the family is away, alerting them if it leaves a safe zone or behaves unusually? \n", + "- Or even notify parents when a toddler approaches dangerous areas like staircases or kitchen counters?\n", + "\n", + "The possibilities of real-time video intelligence at home are endless.\n", + "\n", + "**What would *you* monitor next?**" + ], + "metadata": { + "id": "6rU7-6iLDRUz" + } + } + ] +} \ No newline at end of file diff --git a/real_time_streaming/Cricket_Match_Monitoring.ipynb b/real_time_streaming/Cricket_Match_Monitoring.ipynb new file mode 100644 index 0000000..8fe6f49 --- /dev/null +++ b/real_time_streaming/Cricket_Match_Monitoring.ipynb @@ -0,0 +1,1295 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "name": "python3", + "display_name": "Python 3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": [ + "# 🏏 ICC World Cup Finals: Cricket Highlights Monitoring using VideoDB RTStream\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Cricket_Match_Monitoring.ipynb)\n", + "## 📖 Storytime: Why This Matters\n", + "\n", + "Tonight is the **ICC World Cup finals between India and Pakistan**. \n", + "Millions are watching the match live, and the competition to post match highlights — sixes, wickets, and spectacular catches — on social media is fiercer than ever. \n", + "\n", + "In the usual workflow, someone watches the match, waits for a moment to happen, then clips the video manually and uploads it online — often **several minutes too late**.\n", + "\n", + "But we have a smarter way.\n", + "\n", + "What if AI could monitor the match for you, detect key moments in real-time, and instantly send alerts when something exciting happens — giving you a headstart on posting highlights while everyone else scrambles? \n", + "\n", + "**Let’s build exactly that using VideoDB RTStream.**\n", + "\n", + "---\n", + "\n", + "## 🚀 What You’ll Build in This Notebook\n", + "\n", + "By the end of this notebook, you'll learn how to:\n", + "- Connect a cricket match video stream to VideoDB\n", + "- Use AI to continuously analyze match footage and describe key moments\n", + "- Detect real-time events like **sixes**, **fours**, **catches**, and **wickets**\n", + "- Trigger instant alerts for each event through webhooks\n", + "\n", + "Let’s get started!\n", + "\n", + "---" + ], + "metadata": { + "id": "kO86nzxaK4_5" + } + }, + { + "cell_type": "markdown", + "source": [ + "## 📦 Step 1: Install Dependencies\n", + "\n", + "We’ll begin by installing the VideoDB SDK" + ], + "metadata": { + "id": "XBmSI6lUK_0o" + } + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "id": "Qc_2CLkFKzVl", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "4b5156fe-fa4b-4cfa-d952-762ab1c1af5f" + }, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + " Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Building wheel for videodb (setup.py) ... \u001b[?25l\u001b[?25hdone\n" + ] + } + ], + "source": [ + "!pip install -q videodb" + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "## 📦 Step 2: Connect to VideoDB\n", + "\n", + "Let's connect to VideoDB's API using your credentials to prepare for stream monitoring.\n", + "\n", + "Please enter your `VIDEO_DB_API_KEY` in the input box that appears below after you run this cell.\n", + "\n", + "Your input will be masked.\n" + ], + "metadata": { + "id": "UThnziTKLH_4" + } + }, + { + "cell_type": "code", + "source": [ + "import videodb\n", + "import os\n", + "from getpass import getpass\n", + "\n", + "api_key = getpass(\"Please enter your VideoDB API Key: \")\n", + "\n", + "os.environ[\"VIDEO_DB_API_KEY\"] = api_key\n", + "\n", + "conn = videodb.connect()\n", + "coll = conn.get_collection()\n", + "\n", + "print(\"Connected to VideoDB securely!\")" + ], + "metadata": { + "id": "XFJr9G18LIbR", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "5362ab1f-2bc6-4ecf-c0b7-340d6c79ebd3" + }, + "execution_count": 2, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Please enter your VideoDB API Key: ··········\n", + "Connected to VideoDB securely!\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 📦 Step 3: Connect to the Cricket Match RTSP Stream\n", + "\n", + "We’ll now connect to the live video stream of the match using its RTSP URL.\n", + "In this demo, the stream is running at `rtsp://samples.rts.videodb.io:8554/cricket`.\n" + ], + "metadata": { + "id": "26NDfhgFLQo4" + } + }, + { + "cell_type": "code", + "source": [ + "rtsp_url = \"rtsp://samples.rts.videodb.io:8554/cricket\"\n", + "cricket_stream = coll.connect_rtstream(\n", + " name=\"Cricket Finals Stream\",\n", + " url=rtsp_url,\n", + ")\n", + "print(cricket_stream)" + ], + "metadata": { + "id": "9V0WZTUuLaSl", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "fb5d5cfd-4d5d-44af-c6bb-42a98cd4ce85" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "RTStream(id=rts-019711db-1086-7750-ba79-8f47a4fed603, name=Cricket Finals Stream, collection_id=None, created_at=None, sample_rate=30, status=connected)\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### Let us list all the rtstreams in our collection." + ], + "metadata": { + "id": "hPqJX2v3WSzh" + } + }, + { + "cell_type": "code", + "source": [ + "def list_rtstreams():\n", + " for rtstream in coll.list_rtstreams():\n", + " print(f\"\"\"RTStream:\n", + " ID : {rtstream.id}\n", + " Name : {rtstream.name}\n", + " Collection ID : {rtstream.collection_id}\n", + " Created At : {rtstream.created_at}\n", + " Sample Rate : {rtstream.sample_rate}\n", + " Status : {rtstream.status}\n", + " \"\"\")\n", + " print(\"-\" * 80)\n", + "\n", + "list_rtstreams()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "K-UtkJyvWUuA", + "outputId": "cf38d574-f4bc-487a-a8d5-0e4b52eb5c55" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "RTStream:\n", + " ID : rts-019711db-1086-7750-ba79-8f47a4fed603\n", + " Name : Cricket Finals Stream\n", + " Collection ID : c-81fc6459-fe30-44ac-8c5b-ea0898c2e152\n", + " Created At : 2025-05-27T13:07:38.758741\n", + " Sample Rate : 30\n", + " Status : connected\n", + " \n", + "--------------------------------------------------------------------------------\n", + "RTStream:\n", + " ID : rts-019711a0-0fde-7911-b282-25bc0b4ecf65\n", + " Name : Baby Crib Monitor\n", + " Collection ID : c-81fc6459-fe30-44ac-8c5b-ea0898c2e152\n", + " Created At : 2025-05-27T12:03:11.966358\n", + " Sample Rate : 30\n", + " Status : stopped\n", + " \n", + "--------------------------------------------------------------------------------\n", + "RTStream:\n", + " ID : rts-019710fa-9511-79c3-a924-e229e4815410\n", + " Name : Property Security Stream\n", + " Collection ID : c-81fc6459-fe30-44ac-8c5b-ea0898c2e152\n", + " Created At : 2025-05-27T09:02:27.089279\n", + " Sample Rate : 30\n", + " Status : stopped\n", + " \n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### If you have already connected the stream, run the below cell with the **rtstream id** to reconnect." + ], + "metadata": { + "id": "e11P2KZF22xx" + } + }, + { + "cell_type": "code", + "source": [ + "# cricket_stream = coll.get_rtstream(\"\")" + ], + "metadata": { + "id": "VPuEd_J923yY" + }, + "execution_count": 3, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# To stop the stream\n", + "# cricket_stream.stop()" + ], + "metadata": { + "id": "eLk7rtL73Ci3" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# To start the stream\n", + "# cricket_stream.start()" + ], + "metadata": { + "id": "TqqxMTcQ3B4v" + }, + "execution_count": 4, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### 👀 Let us have a look at the cricket match" + ], + "metadata": { + "id": "70d7muCjLlFf" + } + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### 📺 Helper Function: Display Video Stream\n", + "\n", + "This cell contains a small utility function to help visualize the video streams with helpful information. You don't need to modify this code." + ], + "metadata": { + "id": "9jjx7-4J3QCH" + } + }, + { + "cell_type": "code", + "source": [ + "# To display the stream with relevant information\n", + "\n", + "from IPython.display import HTML\n", + "import re\n", + "from datetime import datetime\n", + "from videodb import play_stream\n", + "\n", + "def display_stream(video_url, video_name=\"🎥 Camera Feed\"):\n", + "\n", + " match = re.search(r'/(\\d{16})-(\\d{16})\\.m3u8', video_url)\n", + " if match:\n", + " start_ts = int(match.group(1)) / 1e6\n", + " end_ts = int(match.group(2)) / 1e6\n", + " start_time = datetime.utcfromtimestamp(start_ts).strftime('%Y-%m-%d %H:%M:%S')\n", + " end_time = datetime.utcfromtimestamp(end_ts).strftime('%Y-%m-%d %H:%M:%S')\n", + " time_range = f\"{start_time} → {end_time} UTC\"\n", + " else:\n", + " time_range = \"Time Unknown\"\n", + "\n", + " video_player_html = play_stream(video_url)\n", + "\n", + " return HTML(f\"\"\"\n", + "
\n", + " {video_player_html._repr_html_() if hasattr(video_player_html, '_repr_html_') else video_player_html}\n", + "
\n", + " {video_name}
{time_range}\n", + "
\n", + "
\n", + " \"\"\")" + ], + "metadata": { + "id": "fSl4fjg7LuNY" + }, + "execution_count": 6, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### 🔗 Get & Display Recent Stream\n", + "\n", + "This cell uses the helper function above to fetch and display the last few minutes of the stream." + ], + "metadata": { + "id": "E0gFicp13UIg" + } + }, + { + "cell_type": "code", + "source": [ + "# To get last few minutes stream link\n", + "import time\n", + "\n", + "def fetch_stream(rtstream):\n", + "\n", + " now = int(time.time())\n", + " start = int(now - (5 * 60))\n", + " stream_url = rtstream.generate_stream(start, now)\n", + " return stream_url\n", + "\n", + "video_url = fetch_stream(cricket_stream)\n", + "\n", + "video_name = \"🏏 ICC World Cup · India vs Pakistan\"\n", + "display_stream(video_url, video_name)" + ], + "metadata": { + "id": "Clwtuv2ZLsKs", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 422 + }, + "outputId": "1bfc3187-3048-4cd1-d520-5a60dfa4c0c7" + }, + "execution_count": 15, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ], + "text/html": [ + "\n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " 🏏 ICC World Cup · India vs Pakistan
2025-05-29 10:38:29 → 2025-05-29 10:43:29 UTC\n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 15 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "## 📦 Step 4: Index Scenes and Describe Match Highlights\n", + "\n", + "We’ll create a real-time scene index that periodically analyzes the video and generates natural language descriptions of what’s happening in the match.\n", + "\n", + "The AI model will look out for:\n", + "- Sixes (ball hit over the boundary rope without bouncing)\n", + "- Fours (ball crosses boundary rope after bouncing)\n", + "- Catches (fielder catches the ball before it touches the ground)\n", + "- Wickets (batsman is dismissed)" + ], + "metadata": { + "id": "bIdL2H-HL1jo" + } + }, + { + "cell_type": "code", + "source": [ + "from videodb import SceneExtractionType\n", + "\n", + "cricket_scene_index = cricket_stream.index_scenes(\n", + " extraction_type=SceneExtractionType.time_based,\n", + " extraction_config={\n", + " \"time\": 7,\n", + " \"frame_count\": 7,\n", + " },\n", + " prompt=\"Identify and mention when a batsman hits a SIX (ball flying over the boundary rope), a FOUR (ball crosses boundary rope after bouncing), a CATCH OUT (fielder catches the ball mid-air before it touches ground) or a WICKET (when the wicket stumps are put down by the ball).\",\n", + " name=\"Cricket_Match_Scene_Index\",\n", + ")\n", + "cricket_index_id = cricket_scene_index.rtstream_index_id\n", + "print(\"Scene Index ID:\", cricket_index_id)" + ], + "metadata": { + "id": "svXbOJVBL32W", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "c4e90af1-dfdc-49ca-99a5-af08279a01b1" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Scene Index ID: c251098cf8fc4561\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### Let us list the scene indexes created on our rtstream." + ], + "metadata": { + "id": "l6CFNuv_Wakz" + } + }, + { + "cell_type": "code", + "source": [ + "def list_rtstream_indexes(rtstream):\n", + " # List live stream indexes\n", + " rtstream_indexes = rtstream.list_scene_indexes()\n", + " for rtstream_index in rtstream_indexes:\n", + "\n", + " print(f\"\"\"RTStreamSceneIndex:\n", + " Index ID : {rtstream_index.rtstream_index_id}\n", + " RTStream ID : {rtstream_index.rtstream_id}\n", + " Name : {rtstream_index.name}\n", + " Status : {rtstream_index.status}\n", + " Config : {rtstream_index.extraction_config}\n", + " Prompt : {rtstream_index.prompt}\n", + " \"\"\")\n", + " print(\"-\" * 80)\n", + "\n", + "list_rtstream_indexes(cricket_stream)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "S8GvMO0MWgOQ", + "outputId": "0c12dcff-c7b0-404d-8c2c-9fd79d2ac30b" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "RTStreamSceneIndex:\n", + " Index ID : c251098cf8fc4561\n", + " RTStream ID : rts-019711db-1086-7750-ba79-8f47a4fed603\n", + " Name : Cricket_Match_Scene_Index\n", + " Status : running\n", + " Config : {'frame_count': '7', 'time': '7'}\n", + " Prompt : Identify and mention when a batsman hits a SIX (ball flying over the boundary rope), a FOUR (ball crosses boundary rope after bouncing), a CATCH OUT (fielder catches the ball mid-air before it touches ground) or a WICKET (when the wicket stumps are put down by the ball).\n", + " \n", + "--------------------------------------------------------------------------------\n", + "RTStreamSceneIndex:\n", + " Index ID : cd031ae88e755900\n", + " RTStream ID : rts-019711db-1086-7750-ba79-8f47a4fed603\n", + " Name : Cricket_Scene_Index\n", + " Status : stopped\n", + " Config : {'frame_count': '7', 'time': '7'}\n", + " Prompt : Monitor the cricket match carefully. Identify and mention when a batsman hits a SIX (ball flying over the boundary rope), a FOUR (ball crosses boundary rope after bouncing), a CATCH OUT (fielder catches the ball mid-air before it touches ground) or a WICKET (when the wicket stumps are put down by the ball).\n", + " \n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### If you have already created a scene index, run the below cell with your **scene index id** to reconnect." + ], + "metadata": { + "id": "6pSm3bj83jgw" + } + }, + { + "cell_type": "code", + "source": [ + "# cricket_index_id = \"\"\n", + "# cricket_scene_index = cricket_stream.get_scene_index(cricket_index_id)" + ], + "metadata": { + "id": "8xmrpUhN3kRi" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# To stop the index\n", + "# cricket_scene_index.stop()" + ], + "metadata": { + "id": "t_Dl05Oc3utw" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# To start the index\n", + "# cricket_scene_index.start()" + ], + "metadata": { + "id": "l_bKugSM3s_f" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### Let us see the result of the scene indexing" + ], + "metadata": { + "id": "GPD_IfJsL_oo" + } + }, + { + "cell_type": "code", + "source": [ + "import time\n", + "from datetime import datetime\n", + "from zoneinfo import ZoneInfo\n", + "\n", + "def _convert_to_ist(timestamp: float) -> str:\n", + " \"\"\"Convert UTC timestamp to IST (Asia/Kolkata) datetime string.\"\"\"\n", + " return (\n", + " datetime.fromtimestamp(timestamp)\n", + " .astimezone(ZoneInfo(\"Asia/Kolkata\"))\n", + " .strftime(\"%Y-%m-%d %H:%M:%S\")\n", + " )\n", + "\n", + "def get_scenes(rtstream, index_id):\n", + " # Print indexed scenes\n", + " rtstream_scene_index = rtstream.get_scene_index(index_id)\n", + " scenes = rtstream_scene_index.get_scenes(page_size=5)\n", + " # print(scenes[\"scenes\"][:2])\n", + " if scenes:\n", + " for scene in scenes.get(\"scenes\"):\n", + " start = _convert_to_ist(scene[\"start\"])\n", + " end = _convert_to_ist(scene[\"end\"])\n", + " description = scene[\"description\"]\n", + " print(f\"{start}-{end}: {description}\")\n", + " print(\"-\" * 80)\n", + " else:\n", + " print(\"Scenes not found for given index.\")\n", + "\n", + "get_scenes(cricket_stream , cricket_index_id)" + ], + "metadata": { + "id": "dv0SS7FAL_7v", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "9aafcd08-1f7f-4387-a7ba-c19bc3807747" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "2025-05-27 19:00:38-2025-05-27 19:00:45: Here's what I can identify from the images:\n", + "\n", + "* **Image 1:** The umpire is signaling a FOUR.\n", + "* **Image 7:** The ball is hitting the wickets. This indicates a WICKET.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-27 19:00:30-2025-05-27 19:00:37: Okay, I can identify those events in the images you provided.\n", + "\n", + "* **Image 6:** WICKET - The batsman is out as the ball has hit the stumps.\n", + "* **Image 7:** WICKET - The umpire signals that the batsman is out.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-27 19:00:21-2025-05-27 19:00:29: Based on the information available in the images, I can identify the following:\n", + "\n", + "* **FOUR:** At 0:02 in the video, the scorecard shows a \"4\" for the current ball. This indicates that the batsman hit a FOUR.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-27 19:00:13-2025-05-27 19:00:20: Here's what I can identify from the images:\n", + "\n", + "* **Image 1 & 2:** The batsman is hitting the ball. It's impossible to tell from these images alone if it results in a SIX, FOUR, CATCH OUT, or WICKET.\n", + "* **Image 4 & 5:** The batsman is playing a shot. It's impossible to tell from these images alone if it results in a SIX, FOUR, CATCH OUT, or WICKET.\n", + "* **Image 6 & 7:** The ball is going towards the stands. It's impossible to tell from these images alone if it results in a SIX, FOUR, CATCH OUT, or WICKET.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-27 19:00:05-2025-05-27 19:00:12: Based on the images, here's what I can identify:\n", + "\n", + "* **Image 2 & 3:** A **CATCH OUT** has occurred. The wicket-keeper is celebrating, indicating he has caught the ball.\n", + "* **Image 4:** The score has changed from 314-4 to 314-5, indicating a **WICKET** has fallen.\n", + "* **Image 5, 6 & 7:** The UltraEdge technology is being used, suggesting a possible **CATCH OUT** or **WICKET** (if the ball edged and went to the keeper or slips).\n", + "\n", + "I cannot determine if a SIX or FOUR was hit from these images.\n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "## 📦 Step 5: Define Events for Each Match Highlight\n", + "\n", + "Now, let’s define four events to detect each key match moment.\n" + ], + "metadata": { + "id": "I6hKrjUPMFGY" + } + }, + { + "cell_type": "markdown", + "source": [ + "1. Sixes (ball hit over the boundary rope without bouncing)" + ], + "metadata": { + "id": "pV3DfpiH5xVt" + } + }, + { + "cell_type": "code", + "source": [ + "six_event_id = conn.create_event(\n", + " event_prompt=\"Detect when a batsman hits a SIX.\",\n", + " label=\"six_hit\"\n", + ")" + ], + "metadata": { + "id": "VD6hI3BfMGZf" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "2. Fours (ball crosses boundary rope after bouncing)\n" + ], + "metadata": { + "id": "gd5PDLpQ54BM" + } + }, + { + "cell_type": "code", + "source": [ + "four_event_id = conn.create_event(\n", + " event_prompt=\"Detect when a batsman hits a FOUR.\",\n", + " label=\"four_hit\"\n", + ")" + ], + "metadata": { + "id": "4gllIq-i53ek" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "3. Catches (fielder catches the ball before it touches the ground)\n" + ], + "metadata": { + "id": "qmD5d_7t58of" + } + }, + { + "cell_type": "code", + "source": [ + "catch_event_id = conn.create_event(\n", + " event_prompt=\"Detect when a player takes a CATCH OUT.\",\n", + " label=\"catch_out\"\n", + ")" + ], + "metadata": { + "id": "bKb5pIr25-BJ" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "4. Wickets (batsman is dismissed)" + ], + "metadata": { + "id": "Es_LFBg06Bms" + } + }, + { + "cell_type": "code", + "source": [ + "wicket_event_id = conn.create_event(\n", + " event_prompt=\"Detect when a batsman is dismissed (WICKET).\",\n", + " label=\"wicket\"\n", + ")" + ], + "metadata": { + "id": "dVknEJtt6Cxo" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "Let us have a look at all the events created:" + ], + "metadata": { + "id": "wmEHFQPk6HGM" + } + }, + { + "cell_type": "code", + "source": [ + "print(f\"Successfully created events with the following IDs:\\n\"\n", + " f\"- Six Event ID: {six_event_id}\\n\"\n", + " f\"- Four Event ID: {four_event_id}\\n\"\n", + " f\"- Catch Event ID: {catch_event_id}\\n\"\n", + " f\"- Wicket Event ID: {wicket_event_id}\")" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "fG3jEJa26PHN", + "outputId": "787e18a7-04c5-46b2-86b0-a91c5a320d97" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Successfully created events with the following IDs:\n", + "- Six Event ID: 627cdd8124f7731b\n", + "- Four Event ID: 3bfdd25d9239861b\n", + "- Catch Event ID: 16b35b66ef106a37\n", + "- Wicket Event ID: fae540ba13f50530\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 📦 Step 6: Attach Alerts for Each Event\n", + "\n", + "We’ll create four different alerts — one for each event — but route them all to the same Pipedream webhook.\n" + ], + "metadata": { + "id": "oYXk7FoOMLgX" + } + }, + { + "cell_type": "code", + "source": [ + "# Enter link to your webhook url where you want alerts to go. You can create one simply on pipedream.\n", + "webhook_url=\"\"" + ], + "metadata": { + "id": "0zL_5_DuXJJQ" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "1. Create an alert for 'six' events and store its ID" + ], + "metadata": { + "id": "u0mViLY_6_pM" + } + }, + { + "cell_type": "code", + "source": [ + "if webhook_url:\n", + " six_alert_id = cricket_scene_index.create_alert(\n", + " event_id=six_event_id,\n", + " callback_url=webhook_url\n", + " )\n", + "else:\n", + " print(\"Error: Please provide Webhook URL. Alert cannot be created without it.\")" + ], + "metadata": { + "id": "zzzCnrwWMMwX" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "2. Create an alert for 'four' events and store its ID" + ], + "metadata": { + "id": "yCHk6uLO7NVM" + } + }, + { + "cell_type": "code", + "source": [ + "if webhook_url:\n", + " four_alert_id = cricket_scene_index.create_alert(\n", + " event_id=four_event_id,\n", + " callback_url=webhook_url\n", + " )\n", + "else:\n", + " print(\"Error: Please provide Webhook URL. Alert cannot be created without it.\")" + ], + "metadata": { + "id": "fa7z-KHq7Hcg" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "3. Create an alert for 'catch' events and store its ID" + ], + "metadata": { + "id": "eEBjmmSb7UPV" + } + }, + { + "cell_type": "code", + "source": [ + "if webhook_url:\n", + " catch_alert_id = cricket_scene_index.create_alert(\n", + " event_id=catch_event_id,\n", + " callback_url=webhook_url\n", + " )\n", + "else:\n", + " print(\"Error: Please provide Webhook URL. Alert cannot be created without it.\")" + ], + "metadata": { + "id": "MIL8N4mQ7JP9" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "4. Create an alert for 'wicket' events and store its ID" + ], + "metadata": { + "id": "U-I8zxJx7Zuc" + } + }, + { + "cell_type": "code", + "source": [ + "if webhook_url:\n", + " wicket_alert_id = cricket_scene_index.create_alert(\n", + " event_id=wicket_event_id,\n", + " callback_url=webhook_url\n", + " )\n", + "else:\n", + " print(\"Error: Please provide Webhook URL. Alert cannot be created without it.\")" + ], + "metadata": { + "id": "5b-Prj197KuE" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "#### Lets list all the alerts" + ], + "metadata": { + "id": "aQPuGgs3MxHx" + } + }, + { + "cell_type": "code", + "source": [ + "def list_rtstream_alerts(rtstream, index_id):\n", + " \"\"\"\n", + " Prints a list of alerts associated with a given scene index.\n", + " \"\"\"\n", + " rtstream_scene_index = rtstream.get_scene_index(index_id)\n", + " alerts = rtstream_scene_index.list_alerts()\n", + "\n", + " for alert in alerts:\n", + " print(f\"\"\"🔔 RTStream Alert:\n", + " Alert ID : {alert['alert_id']}\n", + " Event ID : {alert['event_id']}\n", + " Label : {alert['label']}\n", + " Prompt : {alert['prompt']}\n", + " Status : {alert['status']}\n", + " \"\"\")\n", + " print(\"-\" * 80)\n", + "\n", + "list_rtstream_alerts(cricket_stream, cricket_index_id)\n" + ], + "metadata": { + "id": "vVfjDk9tM397", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "10d61425-e85a-410c-d211-5a7e634ff6df" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "🔔 RTStream Alert:\n", + " Alert ID : 1891843a2c669ea3\n", + " Event ID : 3bfdd25d9239861b\n", + " Label : four_hit\n", + " Prompt : Detect when a batsman hits a FOUR.\n", + " Status : enabled\n", + " \n", + "--------------------------------------------------------------------------------\n", + "🔔 RTStream Alert:\n", + " Alert ID : 6094972d46c861be\n", + " Event ID : 16b35b66ef106a37\n", + " Label : catch_out\n", + " Prompt : Detect when a player takes a CATCH OUT.\n", + " Status : enabled\n", + " \n", + "--------------------------------------------------------------------------------\n", + "🔔 RTStream Alert:\n", + " Alert ID : 69d79872465ce9f2\n", + " Event ID : 627cdd8124f7731b\n", + " Label : six_hit\n", + " Prompt : Detect when a batsman hits a SIX.\n", + " Status : enabled\n", + " \n", + "--------------------------------------------------------------------------------\n", + "🔔 RTStream Alert:\n", + " Alert ID : b4ae6c64ea065ca8\n", + " Event ID : fae540ba13f50530\n", + " Label : wicket\n", + " Prompt : Detect when a batsman is dismissed (WICKET).\n", + " Status : enabled\n", + " \n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 📡 Example Alerts Received at the Webhook\n", + "\n", + "1. Alert received when a FOUR is hit\n", + "\n", + "```json\n", + "{\n", + " \"event_id\": \"event-3bfdd25d9239861b\",\n", + " \"label\": \"four_hit\",\n", + " \"confidence\": 0.95,\n", + " \"explanation\": \"Image 4 shows the ball crossing the boundary after bouncing, indicating a FOUR has been scored. Therefore, an alert should be triggered.\",\n", + " \"timestamp\": \"2025-05-29T00:11:09.256447+00:00\",\n", + " \"start_time\": \"2025-05-29T05:40:32.544547+05:30\",\n", + " \"end_time\": \"2025-05-29T05:40:39.730362+05:30\",\n", + " \"stream_url\": \"https://rt.stream.videodb.io/manifests/rts-019711db-1086-7750-ba79-8f47a4fed603/1748477432000000-1748477440000000.m3u8\"\n", + "}\n", + "```\n", + "\n", + "2. Alert recieved on a CATCH OUT\n", + "\n", + "```json\n", + "{\n", + " \"event_id\": \"event-16b35b66ef106a37\",\n", + " \"label\": \"catch_out\",\n", + " \"confidence\": 0.95,\n", + " \"explanation\": \"Image 4 shows a fielder catching the ball mid-air, indicating a CATCH OUT, which aligns with the alert context.\",\n", + " \"timestamp\": \"2025-05-29T00:10:52.331198+00:00\",\n", + " \"start_time\": \"2025-05-29T05:40:24.326385+05:30\",\n", + " \"end_time\": \"2025-05-29T05:40:31.314371+05:30\",\n", + " \"stream_url\": \"https://rt.stream.videodb.io/manifests/rts-019711db-1086-7750-ba79-8f47a4fed603/1748477424000000-1748477432000000.m3u8\"\n", + "}\n", + "```\n", + "\n", + "✅ Similarly, alerts will be triggered for **SIX** and **WICKET** events, routed to the same webhook URL." + ], + "metadata": { + "id": "OYe1mm-rM_H_" + } + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### Let us have a look at the stream links we received in the alerts." + ], + "metadata": { + "id": "_p2JgOc6aNeJ" + } + }, + { + "cell_type": "markdown", + "source": [ + "1. FOUR HIT" + ], + "metadata": { + "id": "BgaYWBQjaaQ2" + } + }, + { + "cell_type": "code", + "source": [ + "alert_stream_url = \"https://rt.stream.videodb.io/manifests/rts-019711db-1086-7750-ba79-8f47a4fed603/1748477432000000-1748477440000000.m3u8\"\n", + "video_name = \"🏏 ICC World Cup · four_hit\"\n", + "\n", + "display_stream(alert_stream_url,video_name)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 422 + }, + "id": "9zWvALmqYcTS", + "outputId": "27d2b025-d844-4d2f-ffee-3b8e6fe72b60" + }, + "execution_count": 7, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ], + "text/html": [ + "\n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " 🏏 ICC World Cup · four_hit
2025-05-29 00:10:32 → 2025-05-29 00:10:40 UTC\n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 7 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "2. CATCH OUT" + ], + "metadata": { + "id": "3kPePjTrac89" + } + }, + { + "cell_type": "code", + "source": [ + "alert_stream_url = \"https://rt.stream.videodb.io/manifests/rts-019711db-1086-7750-ba79-8f47a4fed603/1748477424000000-1748477432000000.m3u8\"\n", + "video_name = \"🏏 ICC World Cup · catch_out\"\n", + "\n", + "display_stream(alert_stream_url,video_name)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 422 + }, + "id": "-Dn-kuVrahFX", + "outputId": "274825df-a702-4ede-c245-06b2c46c9172" + }, + "execution_count": 8, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ], + "text/html": [ + "\n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " 🏏 ICC World Cup · catch_out
2025-05-29 00:10:24 → 2025-05-29 00:10:32 UTC\n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 8 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "- After the match is over, we can disable the alerts." + ], + "metadata": { + "id": "Sn3YCec3eEs7" + } + }, + { + "cell_type": "code", + "source": [ + "cricket_scene_index.disable_alert(six_alert_id)\n", + "cricket_scene_index.disable_alert(four_alert_id)\n", + "cricket_scene_index.disable_alert(catch_alert_id)\n", + "cricket_scene_index.disable_alert(wicket_alert_id)" + ], + "metadata": { + "id": "pdHJCvzNeLJS" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "- To enable the alerts again:" + ], + "metadata": { + "id": "pMYYapCYe_dp" + } + }, + { + "cell_type": "code", + "source": [ + "cricket_scene_index.enable_alert(six_alert_id)\n", + "cricket_scene_index.enable_alert(four_alert_id)\n", + "cricket_scene_index.enable_alert(catch_alert_id)\n", + "cricket_scene_index.enable_alert(wicket_alert_id)" + ], + "metadata": { + "id": "-P9dKZdBfELE" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "- Now we can stop the stream" + ], + "metadata": { + "id": "D52QAVW1-j2V" + } + }, + { + "cell_type": "code", + "source": [ + "cricket_stream.stop()" + ], + "metadata": { + "id": "VlMQd7uX-mf5" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "## 🏆 Wrapping Up: Outpacing the Competition in Real Time\n", + "\n", + "In this notebook, we built a smart, AI-powered video monitoring system tailored for the high-pressure, real-time demands of live sports broadcasting.\n", + "\n", + "With this setup in place, broadcasters and content creators no longer have to wait, clip, and scramble. \n", + "They can stay ahead of the crowd, instantly catching and sharing match-defining moments as they happen — turning every six, four, wicket, and catch into social media gold within seconds.\n", + "\n", + "---\n", + "\n", + "**But this is just one story.**\n", + "\n", + "- What if the same system could monitor a **football match**, detecting goals, penalties, and crowd invasions in real time? \n", + "- Or watch over an **esports championship**, capturing game-winning moves and viral moments? \n", + "- Or even scan **live news broadcasts**, flagging breaking events or significant headlines for immediate clipping and distribution? \n", + "\n", + "The possibilities of real-time video intelligence in media and sports are endless.\n", + "\n", + "**What would *you* stream next?**\n" + ], + "metadata": { + "id": "Van6xiWVNENn" + } + } + ] +} \ No newline at end of file diff --git a/real_time_streaming/Flash_Flood_Detection.ipynb b/real_time_streaming/Flash_Flood_Detection.ipynb new file mode 100644 index 0000000..5d59250 --- /dev/null +++ b/real_time_streaming/Flash_Flood_Detection.ipynb @@ -0,0 +1,1456 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "name": "python3", + "display_name": "Python 3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": [ + "# 🌊 Flash Flood Detection in Arizona Using VideoDB RTStream\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Flash_Flood_Detection.ipynb)\n", + "\n", + "## 📖 Storytime: Why This Matters\n", + "\n", + "The stunning Arizona deserts, known for their dry riverbeds and scenic beauty, hide a deadly risk. \n", + "During the summer monsoon, sudden torrential rains can trigger **flash floods** in these seemingly harmless dry zones — with little or no warning.\n", + "\n", + "Conventional alert systems relying on rain gauges or weather satellites often fail to deliver timely, location-specific warnings. By the time a danger alert is sent, it might already be too late.\n", + "\n", + "**But we have a smarter way.**\n", + "\n", + "With **VideoDB RTStream**, we can install real-time cameras near flood-prone areas and let AI continuously monitor the visuals. \n", + "As soon as the AI detects signs of a flash flood — like a sudden surge of water through dry land — it can instantly send alerts, giving local authorities and tourists precious moments to act.\n", + "\n", + "---\n", + "\n", + "## 🚀 What You’ll Build in This Notebook\n", + "\n", + "In this notebook, we’ll create a real-time flash flood detection system using **VideoDB RTStream**. \n", + "By the end of this demo, you’ll learn how to:\n", + "- Connect a live video stream to VideoDB\n", + "- Use AI to continuously analyze scenes for signs of a flash flood\n", + "- Detect a **flash flood event**\n", + "- Trigger a real-time alert when detected\n", + "\n", + "Let’s build it together!\n", + "\n" + ], + "metadata": { + "id": "zpV7M-CLQJit" + } + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 📦 Step 1: Install Dependencies\n", + "\n", + "Before setting up our AI-powered flood monitor, let’s install the required VideoDB SDK." + ], + "metadata": { + "id": "f5lY3gziQNRe" + } + }, + { + "cell_type": "code", + "source": [ + "!pip install -q videodb" + ], + "metadata": { + "id": "cD643idwQKJI", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "9c63f06d-d433-4347-ef9a-72a393f851cf" + }, + "execution_count": 1, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + " Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Building wheel for videodb (setup.py) ... \u001b[?25l\u001b[?25hdone\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "## 📦 Step 2: Connect to VideoDB\n", + "\n", + "Let's connect to VideoDB's API using your credentials to prepare for stream monitoring.\n", + "\n", + "Please enter your `VIDEO_DB_API_KEY` in the input box that appears below after you run this cell.\n", + "\n", + "Your input will be masked.\n" + ], + "metadata": { + "id": "JB_mg1kxQTcF" + } + }, + { + "cell_type": "code", + "source": [ + "import videodb\n", + "import os\n", + "from getpass import getpass\n", + "\n", + "api_key = getpass(\"Please enter your VideoDB API Key: \")\n", + "\n", + "os.environ[\"VIDEO_DB_API_KEY\"] = api_key\n", + "\n", + "conn = videodb.connect()\n", + "coll = conn.get_collection()\n", + "\n", + "print(\"Connected to VideoDB securely!\")" + ], + "metadata": { + "id": "ddqIQZteQWPj", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "6a4a57c1-c37b-4258-96eb-0cc9fad6d772" + }, + "execution_count": 2, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Please enter your VideoDB API Key: ··········\n", + "Connected to VideoDB securely!\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 📦 Step 3: Connect to the Arizona Flood RTSP Stream\n", + "\n", + "Connect to the live camera stream monitoring a flood-prone desert area.\n", + "\n", + "In this demo, the stream is running at `rtsp://samples.rts.videodb.io:8554/floods`." + ], + "metadata": { + "id": "GPfbBpinQbW0" + } + }, + { + "cell_type": "code", + "source": [ + "rtsp_url = \"rtsp://samples.rts.videodb.io:8554/floods\"\n", + "flood_stream = coll.connect_rtstream(\n", + " name=\"Arizona Flood Stream\",\n", + " url=rtsp_url,\n", + ")\n", + "print(flood_stream)\n" + ], + "metadata": { + "id": "A7_zI06cQhwr", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "a1763469-b496-4934-9356-d103212f3b5c" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "RTStream(id=rts-019719b2-0c84-7a71-a037-311855a8d160, name=Arizona Flood Stream, collection_id=None, created_at=None, sample_rate=30, status=connected)\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### Let us list all the rtstreams in our collection." + ], + "metadata": { + "id": "bdxspdrXsl9R" + } + }, + { + "cell_type": "code", + "source": [ + "def list_rtstreams():\n", + " for rtstream in coll.list_rtstreams():\n", + " print(f\"\"\"RTStream:\n", + " ID : {rtstream.id}\n", + " Name : {rtstream.name}\n", + " Collection ID : {rtstream.collection_id}\n", + " Created At : {rtstream.created_at}\n", + " Sample Rate : {rtstream.sample_rate}\n", + " Status : {rtstream.status}\n", + " \"\"\")\n", + " print(\"-\" * 80)\n", + "\n", + "list_rtstreams()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "gxtKJcrssmzJ", + "outputId": "278db94e-e50b-486a-ae1f-46d0b04dd961" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "RTStream:\n", + " ID : rts-019719b2-0c84-7a71-a037-311855a8d160\n", + " Name : Arizona Flood Stream\n", + " Collection ID : c-81fc6459-fe30-44ac-8c5b-ea0898c2e152\n", + " Created At : 2025-05-29T01:39:48.484134\n", + " Sample Rate : 30\n", + " Status : connected\n", + " \n", + "--------------------------------------------------------------------------------\n", + "RTStream:\n", + " ID : rts-019711db-1086-7750-ba79-8f47a4fed603\n", + " Name : Cricket Finals Stream\n", + " Collection ID : c-81fc6459-fe30-44ac-8c5b-ea0898c2e152\n", + " Created At : 2025-05-27T13:07:38.758741\n", + " Sample Rate : 30\n", + " Status : stopped\n", + " \n", + "--------------------------------------------------------------------------------\n", + "RTStream:\n", + " ID : rts-019711a0-0fde-7911-b282-25bc0b4ecf65\n", + " Name : Baby Crib Monitor\n", + " Collection ID : c-81fc6459-fe30-44ac-8c5b-ea0898c2e152\n", + " Created At : 2025-05-27T12:03:11.966358\n", + " Sample Rate : 30\n", + " Status : stopped\n", + " \n", + "--------------------------------------------------------------------------------\n", + "RTStream:\n", + " ID : rts-019710fa-9511-79c3-a924-e229e4815410\n", + " Name : Property Security Stream\n", + " Collection ID : c-81fc6459-fe30-44ac-8c5b-ea0898c2e152\n", + " Created At : 2025-05-27T09:02:27.089279\n", + " Sample Rate : 30\n", + " Status : stopped\n", + " \n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### If you have already connected the stream, run the below cell with the **rtstream id** to reconnect." + ], + "metadata": { + "id": "DMpqfcWds2CY" + } + }, + { + "cell_type": "code", + "source": [ + "# flood_stream = coll.get_rtstream(\"\")" + ], + "metadata": { + "id": "dPPG2hsqs1Vv" + }, + "execution_count": 3, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# To stop the stream\n", + "# flood_stream.stop()" + ], + "metadata": { + "id": "NT0bnlhOs-pF" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# To start the stream\n", + "# flood_stream.start()" + ], + "metadata": { + "id": "NmiNzKcfs_Na" + }, + "execution_count": 4, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### 👀 Let's have a look at the riverbed" + ], + "metadata": { + "id": "KO8opDorQnpc" + } + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### 📺 Helper Function: Display Video Stream\n", + "\n", + "This cell contains a small utility function to help visualize the video streams with helpful information. You don't need to modify this code." + ], + "metadata": { + "id": "z0BHpOB3tJgw" + } + }, + { + "cell_type": "code", + "source": [ + "# To display the stream with relevant information\n", + "\n", + "from IPython.display import HTML\n", + "import re\n", + "from datetime import datetime\n", + "from videodb import play_stream\n", + "\n", + "def display_stream(video_url, video_name=\"🎥 Camera Feed\"):\n", + "\n", + " match = re.search(r'/(\\d{16})-(\\d{16})\\.m3u8', video_url)\n", + " if match:\n", + " start_ts = int(match.group(1)) / 1e6\n", + " end_ts = int(match.group(2)) / 1e6\n", + " start_time = datetime.utcfromtimestamp(start_ts).strftime('%Y-%m-%d %H:%M:%S')\n", + " end_time = datetime.utcfromtimestamp(end_ts).strftime('%Y-%m-%d %H:%M:%S')\n", + " time_range = f\"{start_time} → {end_time} UTC\"\n", + " else:\n", + " time_range = \"Time Unknown\"\n", + "\n", + " video_player_html = play_stream(video_url)\n", + "\n", + " return HTML(f\"\"\"\n", + "
\n", + " {video_player_html._repr_html_() if hasattr(video_player_html, '_repr_html_') else video_player_html}\n", + "
\n", + " {video_name}
{time_range}\n", + "
\n", + "
\n", + " \"\"\")" + ], + "metadata": { + "id": "PpSSzPhMQqOE" + }, + "execution_count": 5, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### 🔗 Get & Display Recent Stream\n", + "\n", + "This cell uses the helper function above to fetch and display the last few minutes of the stream." + ], + "metadata": { + "id": "Kg5KSFhhtOQa" + } + }, + { + "cell_type": "code", + "source": [ + "# To get last few minutes stream link\n", + "import time\n", + "\n", + "def fetch_stream(rtstream):\n", + "\n", + " now = int(time.time())\n", + " start = int(now - (5 * 60))\n", + " stream_url = rtstream.generate_stream(start, now)\n", + " return stream_url\n", + "\n", + "video_url = fetch_stream(flood_stream)\n", + "\n", + "video_name = \"🌊 Arizona Desert · Flash Flood Detection\"\n", + "display_stream(video_url , video_name)" + ], + "metadata": { + "id": "g5W0GEc_Qoo7", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 422 + }, + "outputId": "65ae784e-58d9-416d-b0fc-8546997a2ad9" + }, + "execution_count": 8, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ], + "text/html": [ + "\n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " 🌊 Arizona Desert · Flash Flood Detection
2025-05-29 10:49:40 → 2025-05-29 10:54:40 UTC\n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 8 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "---\n", + "\n", + "## 📦 Step 4: Index Scenes and Detect Flash Floods\n", + "\n", + "We’ll create a real-time scene index that periodically analyzes video frames and generates natural language descriptions of what’s happening in the stream.\n", + "\n", + "The AI will look for sudden visual cues of water flooding dry land and describe them.\n" + ], + "metadata": { + "id": "eVIEAJFKQvqW" + } + }, + { + "cell_type": "code", + "source": [ + "from videodb import SceneExtractionType\n", + "\n", + "flood_scene_index = flood_stream.index_scenes(\n", + " extraction_type=SceneExtractionType.time_based,\n", + " extraction_config={\n", + " \"time\": 5,\n", + " \"frame_count\": 3,\n", + " },\n", + " prompt=\"Monitor the dry riverbed and surrounding area. If moving water is detected across the land, identify it as a flash flood and describe the scene.\",\n", + " name=\"Flash_Flood_Detection_Index\"\n", + ")\n", + "flood_index_id = flood_scene_index.rtstream_index_id\n", + "print(\"Scene Index ID:\", flood_index_id)\n" + ], + "metadata": { + "id": "CMU9WMlLQwvn", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "eda5e049-ad57-477e-aaf0-c5d94947d7f7" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Scene Index ID: 318f52d1a4524559\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### Let us list the scene indexes created on our rtstream." + ], + "metadata": { + "id": "KjDNDgQAt1IW" + } + }, + { + "cell_type": "code", + "source": [ + "def list_rtstream_indexes(rtstream):\n", + " # List live stream indexes\n", + " rtstream_indexes = rtstream.list_scene_indexes()\n", + " for rtstream_index in rtstream_indexes:\n", + "\n", + " print(f\"\"\"RTStreamSceneIndex:\n", + " Index ID : {rtstream_index.rtstream_index_id}\n", + " RTStream ID : {rtstream_index.rtstream_id}\n", + " Name : {rtstream_index.name}\n", + " Status : {rtstream_index.status}\n", + " Config : {rtstream_index.extraction_config}\n", + " Prompt : {rtstream_index.prompt}\n", + " \"\"\")\n", + " print(\"-\" * 80)\n", + "\n", + "list_rtstream_indexes(flood_stream)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "Mcies-gct12A", + "outputId": "21da65ef-62a3-43a7-b3ab-20c0eefaeccf" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "RTStreamSceneIndex:\n", + " Index ID : 318f52d1a4524559\n", + " RTStream ID : rts-019719b2-0c84-7a71-a037-311855a8d160\n", + " Name : Flash_Flood_Detection_Index\n", + " Status : running\n", + " Config : {'frame_count': '3', 'time': '5'}\n", + " Prompt : Monitor the dry riverbed and surrounding area. If moving water is detected across the land, identify it as a flash flood and describe the scene.\n", + " \n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### If you have already created a scene index, run the below cell with your **scene index id** to reconnect." + ], + "metadata": { + "id": "abtAiqYRt8tm" + } + }, + { + "cell_type": "code", + "source": [ + "# flood_index_id = \"\"\n", + "# flood_scene_index = flood_stream.get_scene_index(flood_index_id)" + ], + "metadata": { + "id": "SiabmNgrt9al" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# To stop the index\n", + "# flood_scene_index.stop()" + ], + "metadata": { + "id": "oTJX-adPt9vE" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# To start the index\n", + "# flood_scene_index.start()" + ], + "metadata": { + "id": "fj8NNH7nt-B9" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### Let us see the result of the scene indexing" + ], + "metadata": { + "id": "yRZmI0IvQ0FF" + } + }, + { + "cell_type": "code", + "source": [ + "import time\n", + "from datetime import datetime\n", + "from zoneinfo import ZoneInfo\n", + "\n", + "def _convert_to_ist(timestamp: float) -> str:\n", + " \"\"\"Convert UTC timestamp to IST (Asia/Kolkata) datetime string.\"\"\"\n", + " return (\n", + " datetime.fromtimestamp(timestamp)\n", + " .astimezone(ZoneInfo(\"Asia/Kolkata\"))\n", + " .strftime(\"%Y-%m-%d %H:%M:%S\")\n", + " )\n", + "\n", + "def get_scenes(rtstream, index_id):\n", + " # Print indexed scenes\n", + " rtstream_scene_index = rtstream.get_scene_index(index_id)\n", + " scenes = rtstream_scene_index.get_scenes(page_size=5)\n", + " # print(scenes[\"scenes\"][:2])\n", + " if scenes:\n", + " for scene in scenes.get(\"scenes\"):\n", + " start = _convert_to_ist(scene[\"start\"])\n", + " end = _convert_to_ist(scene[\"end\"])\n", + " description = scene[\"description\"]\n", + " print(f\"{start}-{end}: {description}\")\n", + " print(\"-\" * 80)\n", + " else:\n", + " print(\"Scenes not found for given index.\")\n", + "\n", + "get_scenes(flood_stream , flood_index_id)" + ], + "metadata": { + "id": "XUa08hPnQ0Zw", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "ef8fc402-aa98-4dcd-a58b-f3d59fcc18d7" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "2025-05-29 07:17:51-2025-05-29 07:17:56: **Flash Flood Detected**\n", + "\n", + "The images show a flash flood occurring in a dry riverbed. The water is flowing rapidly and forcefully through the narrow channels carved into the red rock landscape. The water is churning and appears muddy, indicating it is carrying sediment and debris. The surrounding area is arid, with sparse vegetation. People are observing the flash flood from the rocky banks. The sky is overcast, suggesting recent rainfall may have contributed to the flash flood.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-29 07:17:46-2025-05-29 07:17:50: **Flash Flood Detected**\n", + "\n", + "The scene depicts a flash flood surging through a narrow, rocky riverbed. The water is turbulent and fast-moving, churning with white foam as it rushes between the reddish-brown rock formations. The water level is high, filling the channel completely and indicating a sudden and significant increase in water volume. The surrounding landscape is arid and rocky, typical of a desert environment where flash floods are common after heavy rainfall.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-29 07:17:40-2025-05-29 07:17:45: **Flash Flood Detected**\n", + "\n", + "The images show a flash flood occurring in a dry riverbed. The water is moving rapidly and forcefully, churning with foam and debris. The riverbed is filled with large, reddish-brown rocks, and the water is flowing around and over them. The scene suggests a sudden and powerful surge of water, typical of a flash flood event.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-29 07:17:35-2025-05-29 07:17:40: **Flash Flood Detected**\n", + "\n", + "The images show a flash flood occurring in a dry riverbed. The water is moving rapidly and is heavily laden with sediment, giving it a muddy brown color. The water is flowing through narrow channels carved into the rock, creating turbulent currents and white foam. The surrounding rock formations are smooth and sculpted, indicating the erosive power of past floods. The scene is dynamic and potentially dangerous, highlighting the sudden and powerful nature of flash floods.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-29 07:17:29-2025-05-29 07:17:34: **Flash Flood Detected**\n", + "\n", + "The scene depicts a flash flood surging through a dry riverbed. The water is muddy and turbulent, rushing over and around the large, reddish-brown rocks that characterize the landscape. The water level is high enough to create small waterfalls and rapids as it navigates the uneven terrain. The surrounding area appears arid, with sparse vegetation clinging to the edges of the riverbed. The sky is partially cloudy, suggesting the possibility of recent or ongoing rainfall that may have contributed to the flash flood.\n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 📦 Step 5: Define a Flash Flood Event\n", + "\n", + "Now, we’ll define an event type in the system to detect visual signs of a flash flood.\n" + ], + "metadata": { + "id": "-kO4RguhRNbk" + } + }, + { + "cell_type": "code", + "source": [ + "flood_event_id = conn.create_event(\n", + " event_prompt=\"Detect sudden flash floods or water surges.\",\n", + " label=\"flash_flood\"\n", + ")\n", + "print(\"Event ID:\", flood_event_id)\n" + ], + "metadata": { + "id": "VxTWmd_hROVQ", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "0b63a4f4-af21-4b69-c266-325d2caf69f6" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Event ID: 7929a3ecc4624dbb\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 📦 Step 6: Attach an Alert to the Flash Flood Event\n", + "\n", + "Finally, we’ll link a real-time alert to this event, which will send a notification to our webhook the moment a flash flood is detected.\n" + ], + "metadata": { + "id": "bRf6QOfrRQa2" + } + }, + { + "cell_type": "code", + "source": [ + "# Enter link to your webhook url where you want alerts to go. You can create one simply on pipedream.\n", + "webhook_url = \"\"\n", + "\n", + "if webhook_url:\n", + " flood_alert_id = flood_scene_index.create_alert(\n", + " flood_event_id,\n", + " callback_url=webhook_url\n", + " )\n", + " print(\"Alert ID:\", flood_alert_id)\n", + "else:\n", + " print(\"Error: Please provide Webhook URL. Alert cannot be created without it.\")" + ], + "metadata": { + "id": "RftcmJUPRRmG", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "1bb4b438-80d7-4282-ff4d-850df1a17155" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Alert ID: d9c6038e631071cd\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 📡 Example Alert Payload\n", + "\n", + "When a flash flood occurs, the system sent this alert to the webhook url:\n", + "\n", + "```json\n", + "{\n", + " \"event_id\": \"event-7929a3ecc4624dbb\",\n", + " \"label\": \"flash_flood\",\n", + " \"confidence\": 0.95,\n", + " \"explanation\": \"The scene analysis clearly depicts a flash flood with rapidly moving water, debris, and a recent heavy rainfall event, aligning strongly with the alert context of detecting sudden flash floods or water surges.\",\n", + " \"timestamp\": \"2025-05-29T01:51:33.907778+00:00\",\n", + " \"start_time\": \"2025-05-29T07:20:34.289908+05:30\",\n", + " \"end_time\": \"2025-05-29T07:20:39.891267+05:30\",\n", + " \"stream_url\": \"https://rt.stream.videodb.io/manifests/rts-019719b2-0c84-7a71-a037-311855a8d160/1748483434000000-1748483440000000.m3u8\"\n", + "}\n", + "```" + ], + "metadata": { + "id": "o_u8OYaWRTbP" + } + }, + { + "cell_type": "code", + "source": [ + "alert_stream_url = \"https://rt.stream.videodb.io/manifests/rts-019719b2-0c84-7a71-a037-311855a8d160/1748483434000000-1748483440000000.m3u8\"\n", + "video_name = \"🌊 Arizona Desert · flash_flood\"\n", + "\n", + "display_stream(alert_stream_url, video_name)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 422 + }, + "id": "tA8WGWzUvPQ1", + "outputId": "bc690012-a143-4263-dfa9-a4f549d85756" + }, + "execution_count": 6, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ], + "text/html": [ + "\n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " 🌊 Arizona Desert · flash_flood
2025-05-29 01:50:34 → 2025-05-29 01:50:40 UTC\n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 6 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### Let us stop the flood index and proceed with the notebook to explore more possibilities." + ], + "metadata": { + "id": "HuxU7xQTxBZ1" + } + }, + { + "cell_type": "code", + "source": [ + "# flood_scene_index.stop()" + ], + "metadata": { + "id": "KcknjmPPxAzw" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "- Let us disable the alert now." + ], + "metadata": { + "id": "9A29QLMZ9d44" + } + }, + { + "cell_type": "code", + "source": [ + "flood_scene_index.disable_alert(flood_alert_id)" + ], + "metadata": { + "id": "xM6BA96q9kAZ" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "- To enable the alert again" + ], + "metadata": { + "id": "6e-d57Z49hY7" + } + }, + { + "cell_type": "code", + "source": [ + "flood_scene_index.enable_alert(flood_alert_id)" + ], + "metadata": { + "id": "KC7tUVqw9kVT" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### Let us set up some other alerts that are necessary\n", + "1. Heavy rainfall, detecting heavy rainfall early can help us predict a flash flood occurence\n", + "2. Detect the presence of a person stuck in the flash flood, for immediate rescue \n", + "\n", + "We can start with creating a new scene index for monitoring rainfall or a person stuck in the place." + ], + "metadata": { + "id": "KwcEwZygRYZs" + } + }, + { + "cell_type": "code", + "source": [ + "riverbed_monitoring_scene_index = flood_stream.index_scenes(\n", + " extraction_type=SceneExtractionType.time_based,\n", + " extraction_config={\n", + " \"time\": 15,\n", + " \"frame_count\": 1,\n", + " },\n", + " prompt=\"Monitor the dry riverbed and surrounding area. In case you detect heavy rainfall mention 'heavy rainfall detected'. If you detect a person stuck in the area during rainfall or flash flood mention 'person detected, rescue needed'\",\n", + " name=\"Riverbed_Monitoring_Index\"\n", + ")\n", + "riverbed_monitoring_index_id = riverbed_monitoring_scene_index.rtstream_index_id\n", + "print(\"Scene Index ID:\", riverbed_monitoring_index_id)\n" + ], + "metadata": { + "id": "GX8L7BkyRU-b", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "855703dd-b7cd-43b9-c63a-c0089bc2e7d7" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Scene Index ID: a64531c43284cc99\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### Checking the list of indexes" + ], + "metadata": { + "id": "914lMw6PxQDu" + } + }, + { + "cell_type": "code", + "source": [ + "list_rtstream_indexes(flood_stream)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "UXuau8gPxguR", + "outputId": "a6a243cc-2f82-4168-fbfe-285d401da947" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "RTStreamSceneIndex:\n", + " Index ID : 318f52d1a4524559\n", + " RTStream ID : rts-019719b2-0c84-7a71-a037-311855a8d160\n", + " Name : Flash_Flood_Detection_Index\n", + " Status : stopped\n", + " Config : {'frame_count': '3', 'time': '5'}\n", + " Prompt : Monitor the dry riverbed and surrounding area. If moving water is detected across the land, identify it as a flash flood and describe the scene.\n", + " \n", + "--------------------------------------------------------------------------------\n", + "RTStreamSceneIndex:\n", + " Index ID : a64531c43284cc99\n", + " RTStream ID : rts-019719b2-0c84-7a71-a037-311855a8d160\n", + " Name : Riverbed_Monitoring_Index\n", + " Status : running\n", + " Config : {'frame_count': '1', 'time': '15'}\n", + " Prompt : Monitor the dry riverbed and surrounding area. In case you detect heavy rainfall mention 'heavy rainfall detected'. If you detect a person stuck in the area during rainfall or flash flood mention 'person detected, rescue needed'\n", + " \n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### Let's have a look at generated scenes" + ], + "metadata": { + "id": "FFoIxQO1Rbxs" + } + }, + { + "cell_type": "code", + "source": [ + "get_scenes(flood_stream , riverbed_monitoring_index_id)" + ], + "metadata": { + "id": "3hfHascHRcw2", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "6147fb2d-8723-40ec-9bae-826f8c3c005c" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "2025-05-29 07:34:50-2025-05-29 07:35:05: heavy rainfall detected\n", + "--------------------------------------------------------------------------------\n", + "2025-05-29 07:34:34-2025-05-29 07:34:49: No heavy rainfall detected. No person detected.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-29 07:34:19-2025-05-29 07:34:34: No heavy rainfall detected. No person detected.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-29 07:34:03-2025-05-29 07:34:18: 'person detected, rescue needed'\n", + "--------------------------------------------------------------------------------\n", + "2025-05-29 07:33:47-2025-05-29 07:34:02: heavy rainfall detected\n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### Now we can setup events and alerts for the index\n", + "\n", + "1. Rainfall Detection" + ], + "metadata": { + "id": "--HEDr5WRe69" + } + }, + { + "cell_type": "code", + "source": [ + "# Create rainfall event\n", + "rainfall_event_id = conn.create_event(\n", + " event_prompt=\"Detect heavy rainfall.\",\n", + " label=\"heavy_rainfall\"\n", + ")\n", + "print(\"Event ID:\", rainfall_event_id)" + ], + "metadata": { + "id": "jf156IF7RgQf", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "f128fcd8-4fd9-4a9a-e680-a1067c586ea1" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Event ID: df1611d13772e09b\n" + ] + } + ] + }, + { + "cell_type": "code", + "source": [ + "# Create rainfall alert\n", + "\n", + "# Enter link to your webhook url where you want alerts to go.\n", + "rainfall_webhook_url = \"\"\n", + "\n", + "if rainfall_webhook_url:\n", + " rainfall_alert_id = riverbed_monitoring_scene_index.create_alert(\n", + " rainfall_event_id,\n", + " callback_url=rainfall_webhook_url\n", + " )\n", + " print(\"Alert ID:\", rainfall_alert_id)\n", + "else:\n", + " print(\"Error: Please provide Webhook URL. Alert cannot be created without it.\")" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "Z9EnhfkOyvlg", + "outputId": "f3a54042-2aec-4c06-9627-2e9ee9db88e4" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Alert ID: ae31ea7c92a61bc7\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "2. Human detection for rescue" + ], + "metadata": { + "id": "hEAftnXnRiTd" + } + }, + { + "cell_type": "code", + "source": [ + "# Create rescue event\n", + "rescue_event_id = conn.create_event(\n", + " event_prompt=\"Detect if there is a person\",\n", + " label=\"human_rescue\"\n", + ")\n", + "print(\"Event ID:\", rescue_event_id)" + ], + "metadata": { + "id": "7lSfgMwNRwJZ", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "a6bda8fb-1890-4492-e9ce-5d951f1c3194" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Event ID: 717a1bcf7a705ec7\n" + ] + } + ] + }, + { + "cell_type": "code", + "source": [ + "# Create rescue alert\n", + "\n", + "# Enter link to your webhook url where you want alerts to go.\n", + "rescue_webhook_url = \"\"\n", + "\n", + "rescue_alert_id = riverbed_monitoring_scene_index.create_alert(\n", + " rescue_event_id,\n", + " callback_url=rescue_webhook_url\n", + ")\n", + "print(\"Alert ID:\", rescue_alert_id)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "smMe1bn_y_5q", + "outputId": "6465cfb6-a1bc-44de-fab7-9ccfe01b3f9c" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Alert ID: 766872338353f95b\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### Let us see the list of alerts associated with the `riverbed_monitoring_scene_index`" + ], + "metadata": { + "id": "AzmR7GD-zabA" + } + }, + { + "cell_type": "code", + "source": [ + "def list_rtstream_alerts(rtstream, index_id):\n", + " \"\"\"\n", + " Prints a list of alerts associated with a given scene index.\n", + " \"\"\"\n", + " rtstream_scene_index = rtstream.get_scene_index(index_id)\n", + " alerts = rtstream_scene_index.list_alerts()\n", + "\n", + " for alert in alerts:\n", + " print(f\"\"\"🔔 RTStream Alert:\n", + " Alert ID : {alert['alert_id']}\n", + " Event ID : {alert['event_id']}\n", + " Label : {alert['label']}\n", + " Prompt : {alert['prompt']}\n", + " Status : {alert['status']}\n", + " \"\"\")\n", + " print(\"-\" * 80)\n", + "\n", + "list_rtstream_alerts(flood_stream, riverbed_monitoring_index_id)\n" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "CNH-aKZizr8B", + "outputId": "05d5647b-b620-4cd6-add9-881da6baade1" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "🔔 RTStream Alert:\n", + " Alert ID : 766872338353f95b\n", + " Event ID : 717a1bcf7a705ec7\n", + " Label : human_rescue\n", + " Prompt : Detect if there is a person\n", + " Status : enabled\n", + " \n", + "--------------------------------------------------------------------------------\n", + "🔔 RTStream Alert:\n", + " Alert ID : ae31ea7c92a61bc7\n", + " Event ID : df1611d13772e09b\n", + " Label : heavy_rainfall\n", + " Prompt : Detect heavy rainfall.\n", + " Status : enabled\n", + " \n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 📡 Example Alert Payload\n", + "\n", + "When the AI detected people near the flooding riverbed, the system sent this alert to the webhook url:\n", + "\n", + "```json\n", + "{\n", + " \"event_id\": \"event-717a1bcf7a705ec7\",\n", + " \"label\": \"human_rescue\",\n", + " \"confidence\": 0.95,\n", + " \"explanation\": \"Flash flood is detected and a person is present, indicating a potential need for rescue. High confidence due to explicit rescue request.\",\n", + " \"timestamp\": \"2025-05-29T02:19:56.724411+00:00\",\n", + " \"start_time\": \"2025-05-29T07:49:49.789911+05:30\",\n", + " \"end_time\": \"2025-05-29T07:50:04.789911+05:30\",\n", + " \"stream_url\": \"https://rt.stream.videodb.io/manifests/rts-019719b2-0c84-7a71-a037-311855a8d160/1748485189000000-1748485205000000.m3u8\"\n", + "}\n", + "```" + ], + "metadata": { + "id": "DtGcRljD0UYM" + } + }, + { + "cell_type": "code", + "source": [ + "alert_stream_url = \"https://rt.stream.videodb.io/manifests/rts-019719b2-0c84-7a71-a037-311855a8d160/1748485189000000-1748485205000000.m3u8\"\n", + "video_name = \"🌊 Arizona Desert · human_rescue\"\n", + "\n", + "display_stream(alert_stream_url, video_name)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 422 + }, + "id": "MZ1LBQ3E1dcV", + "outputId": "20198543-639f-4034-ee25-ceaa8272b018" + }, + "execution_count": 7, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ], + "text/html": [ + "\n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " 🌊 Arizona Desert · human_rescue
2025-05-29 02:19:49 → 2025-05-29 02:20:05 UTC\n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 7 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "- Let us disable the alerts now." + ], + "metadata": { + "id": "2Q_xgt7790wZ" + } + }, + { + "cell_type": "code", + "source": [ + "riverbed_monitoring_scene_index.disable_alert(rainfall_alert_id)\n", + "riverbed_monitoring_scene_index.disable_alert(rescue_alert_id)" + ], + "metadata": { + "id": "ZO8vF3-W951R" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "- To enable the alert again" + ], + "metadata": { + "id": "TnDcrHaV93jR" + } + }, + { + "cell_type": "code", + "source": [ + "riverbed_monitoring_scene_index.enable_alert(rainfall_alert_id)\n", + "riverbed_monitoring_scene_index.enable_alert(rescue_alert_id)" + ], + "metadata": { + "id": "8QR3uW6x96MS" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "- Now let us stop the stream" + ], + "metadata": { + "id": "03jdVUVa-xoA" + } + }, + { + "cell_type": "code", + "source": [ + "flood_stream.stop()" + ], + "metadata": { + "id": "K_3-Dw0v-xFY" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 🌙 Wrapping Up: Real-Time Environmental Safety\n", + "\n", + "With this system in place, communities, tourists, and local authorities in Arizona’s desert regions can receive **immediate alerts** when a dangerous flash flood occurs — gaining critical seconds to take cover, clear routes, or initiate rescues.\n", + "\n", + "---\n", + "\n", + "## 🔥 What Else Could We Monitor?\n", + "\n", + "This system isn’t limited to flash floods. The same AI-driven video monitoring approach can protect lives in other natural disasters too.\n", + "\n", + "---\n", + "\n", + "### 🌲 Forest Fire Detection\n", + "\n", + "**Why it’s needed:** \n", + "In remote forest areas, smoke plumes and early fire flickers often go unnoticed for several minutes before traditional sensors or satellites pick them up.\n", + "\n", + "**How we can do it better:** \n", + "With AI-powered cameras watching key zones, we can detect rapid smoke build-ups or visible flames long before automated sensors trigger.\n", + "\n", + "**Example indexing prompt:**\n", + "\n", + "```markdown\n", + "\"Monitor the forest area carefully. Detect sudden rising smoke, increasing haze, or visible flames. Clearly describe if a fire outbreak is visibly starting.\"\n", + "```\n", + "\n", + "---\n", + "\n", + "### 🌊 Sudden Tsunami Detection\n", + "\n", + "**Why it’s needed:** \n", + "Even with tsunami sensors and ocean buoys, near-shoreline towns often get only minutes of warning. Visual cues like water rapidly pulling away from shore or unusually large approaching waves are immediate, reliable signs.\n", + "\n", + "**How we can do it better:** \n", + "Install AI cameras on popular beaches and coastlines to detect rapid water retreat or walls of water approaching.\n", + "\n", + "**Example indexing prompt:**\n", + "\n", + "```markdown\n", + "\"Watch the beach and shoreline closely. Detect unusual rapid water withdrawal from the shore, or approaching large waves indicating a possible tsunami. Describe the situation clearly.\"\n", + "```\n", + "\n", + "---\n", + "\n", + "### 🏔️ Landslide / Avalanche Detection\n", + "\n", + "**Why it’s needed:** \n", + "Mountain highways and tourist trails are often cut off by sudden landslides or snow avalanches, where every second matters for evacuation and road closures.\n", + "\n", + "**How we can do it better:** \n", + "Place AI cameras at landslide-prone mountain slopes or snow-covered passes to detect falling rocks, moving soil, or snow slides in real time.\n", + "\n", + "**Example indexing prompt:**\n", + "\n", + "```markdown\n", + "\"Monitor the mountain slope area carefully. Detect falling rocks, soil displacement, or sudden snow slides. Clearly describe if a landslide or avalanche is starting.\"\n", + "```\n", + "\n", + "---\n", + "\n", + "## 🌍 The Possibilities Are Endless\n", + "\n", + "Real-time AI video monitoring isn’t just for cities and homes — it can actively save lives in wild, unpredictable environments too.\n", + "\n", + "**What natural threat would *you* monitor next?**" + ], + "metadata": { + "id": "BwE1fRaoR4BL" + } + } + ] +} \ No newline at end of file diff --git a/real_time_streaming/Intrusion_Detection.ipynb b/real_time_streaming/Intrusion_Detection.ipynb new file mode 100644 index 0000000..6a2c251 --- /dev/null +++ b/real_time_streaming/Intrusion_Detection.ipynb @@ -0,0 +1,1284 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "name": "python3", + "display_name": "Python 3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": [ + "# 🏠 Smart Intruder Detection for Property Surveillance using VideoDB RTStream\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Intrusion_Detection.ipynb)\n", + "## 📖 Storytime: Why This Matters\n", + "\n", + "Have you ever felt anxious leaving your home, shop, or property unattended? \n", + "**What if someone’s lurking around your property?** \n", + "**What if someone’s trying the door or peeking through windows?** \n", + "Sure — you could install IP cameras, but who has the time to watch them 24/7?\n", + "\n", + "**Good news — you don’t have to anymore.**\n", + "\n", + "With **VideoDB RTStream**, you can build a smart, AI-powered property surveillance system that actively monitors live video streams, detects suspicious activity, and immediately sends alerts for escalating security breaches — all without human supervision.\n", + "\n", + "---\n", + "\n", + "## 🚀 What You’ll Build in This Notebook\n", + "\n", + "In this notebook, we’ll build a layered intruder detection system with **three levels of security breach detection**:\n", + "- **🔸 Level 1: Loitering near the property perimeter**\n", + "- **🔸 Level 2: Attempting to check the door lock (tampering / intrusion attempt)**\n", + "- **🔸 Level 3: Physically entering the property (highest breach)**\n", + "\n", + "You’ll learn how to:\n", + "- 📺 Connect a property surveillance video stream to VideoDB\n", + "- 🤖 Use AI to continuously analyze scenes for suspicious activity\n", + "- Detect and differentiate **three levels of security threats**\n", + "- 🚨 Trigger separate real-time alerts for each level to a webhook\n", + "\n", + "Let’s build it!" + ], + "metadata": { + "id": "Fsebg6qWNsIQ" + } + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 📦 Step 1: Install Dependencies\n", + "\n", + "We’ll begin by installing the VideoDB SDK" + ], + "metadata": { + "id": "tMOTjoKCNx4m" + } + }, + { + "cell_type": "code", + "source": [ + "!pip install -q videodb" + ], + "metadata": { + "id": "VWIKa3mWNwSr", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "ab516051-3cf0-4aba-f502-83a157732866" + }, + "execution_count": 1, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + " Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Building wheel for videodb (setup.py) ... \u001b[?25l\u001b[?25hdone\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "## 📦 Step 2: Connect to VideoDB\n", + "\n", + "Let's connect to VideoDB's API using your credentials to prepare for stream monitoring.\n", + "\n", + "Please enter your `VIDEO_DB_API_KEY` in the input box that appears below after you run this cell.\n", + "\n", + "Your input will be masked.\n" + ], + "metadata": { + "id": "ADeNXFVGN1g_" + } + }, + { + "cell_type": "code", + "source": [ + "import videodb\n", + "import os\n", + "from getpass import getpass\n", + "\n", + "api_key = getpass(\"Please enter your VideoDB API Key: \")\n", + "\n", + "os.environ[\"VIDEO_DB_API_KEY\"] = api_key\n", + "\n", + "conn = videodb.connect()\n", + "coll = conn.get_collection()\n", + "\n", + "print(\"Connected to VideoDB securely!\")" + ], + "metadata": { + "id": "HX-5_YF_N2mK", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "dcbfc051-8c74-4836-aa92-4b495a93cf95" + }, + "execution_count": 2, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Please enter your VideoDB API Key: ··········\n", + "Connected to VideoDB securely!\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 📦 Step 3: Connect to the Property Surveillance RTSP Stream\n", + "\n", + "We’ll connect to the live video stream of your property’s security camera.\n", + "\n", + "In this demo, the stream is running at `rtsp://samples.rts.videodb.io:8554/intruder`.\n" + ], + "metadata": { + "id": "4zMEeRBaN7Zm" + } + }, + { + "cell_type": "code", + "source": [ + "rtsp_url = \"rtsp://samples.rts.videodb.io:8554/intruder\"\n", + "intruder_stream = coll.connect_rtstream(\n", + " name=\"Property Security Stream\",\n", + " url=rtsp_url,\n", + ")\n", + "print(intruder_stream)" + ], + "metadata": { + "id": "DCXckXELOABN", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "dd7a27df-6d66-49b9-f918-92be75ef77ee" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "RTStream(id=rts-019710fa-9511-79c3-a924-e229e4815410, name=Property Security Stream, collection_id=None, created_at=None, sample_rate=30, status=connected)\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### Let us list all the rtstreams in our collection." + ], + "metadata": { + "id": "Uek0S-lYiagX" + } + }, + { + "cell_type": "code", + "source": [ + "def list_rtstreams():\n", + " for rtstream in coll.list_rtstreams():\n", + " print(f\"\"\"RTStream:\n", + " ID : {rtstream.id}\n", + " Name : {rtstream.name}\n", + " Collection ID : {rtstream.collection_id}\n", + " Created At : {rtstream.created_at}\n", + " Sample Rate : {rtstream.sample_rate}\n", + " Status : {rtstream.status}\n", + " \"\"\")\n", + " print(\"-\" * 80)\n", + "\n", + "list_rtstreams()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "_LOUfC9CieDc", + "outputId": "2f6d3d99-bfed-421c-a629-eadde1a65108" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "RTStream:\n", + " ID : rts-019711db-1086-7750-ba79-8f47a4fed603\n", + " Name : Cricket Finals Stream\n", + " Collection ID : c-81fc6459-fe30-44ac-8c5b-ea0898c2e152\n", + " Created At : 2025-05-27T13:07:38.758741\n", + " Sample Rate : 30\n", + " Status : stopped\n", + " \n", + "--------------------------------------------------------------------------------\n", + "RTStream:\n", + " ID : rts-019711a0-0fde-7911-b282-25bc0b4ecf65\n", + " Name : Baby Crib Monitor\n", + " Collection ID : c-81fc6459-fe30-44ac-8c5b-ea0898c2e152\n", + " Created At : 2025-05-27T12:03:11.966358\n", + " Sample Rate : 30\n", + " Status : stopped\n", + " \n", + "--------------------------------------------------------------------------------\n", + "RTStream:\n", + " ID : rts-019710fa-9511-79c3-a924-e229e4815410\n", + " Name : Property Security Stream\n", + " Collection ID : c-81fc6459-fe30-44ac-8c5b-ea0898c2e152\n", + " Created At : 2025-05-27T09:02:27.089279\n", + " Sample Rate : 30\n", + " Status : connected\n", + " \n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### If you have already connected the stream, run the below cell with the **rtstream id** to reconnect." + ], + "metadata": { + "id": "m_GqDzuF_fmI" + } + }, + { + "cell_type": "code", + "source": [ + "# intruder_stream = coll.get_rtstream(\"\")" + ], + "metadata": { + "id": "qnOll9oS_gnJ" + }, + "execution_count": 3, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# To stop the stream\n", + "# intruder_stream.stop()" + ], + "metadata": { + "id": "gHAqL2LzBMT1" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# To start the stream\n", + "# intruder_stream.start()" + ], + "metadata": { + "id": "1c0QNS-tBKpX" + }, + "execution_count": 4, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### 👀 Let us have a look at the camera feed" + ], + "metadata": { + "id": "KcQmjyuFOMZ-" + } + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### 📺 Helper Function: Display Video Stream\n", + "\n", + "This cell contains a small utility function to help visualize the video streams with helpful information. You don't need to modify this code." + ], + "metadata": { + "id": "soVJxcNOB3R-" + } + }, + { + "cell_type": "code", + "source": [ + "# To display the stream with relevant information\n", + "\n", + "from IPython.display import HTML\n", + "import re\n", + "from datetime import datetime\n", + "from videodb import play_stream\n", + "\n", + "def display_stream(video_url, video_name=\"🎥 Camera Feed\"):\n", + "\n", + " match = re.search(r'/(\\d{16})-(\\d{16})\\.m3u8', video_url)\n", + " if match:\n", + " start_ts = int(match.group(1)) / 1e6\n", + " end_ts = int(match.group(2)) / 1e6\n", + " start_time = datetime.utcfromtimestamp(start_ts).strftime('%Y-%m-%d %H:%M:%S')\n", + " end_time = datetime.utcfromtimestamp(end_ts).strftime('%Y-%m-%d %H:%M:%S')\n", + " time_range = f\"{start_time} → {end_time} UTC\"\n", + " else:\n", + " time_range = \"Time Unknown\"\n", + "\n", + " video_player_html = play_stream(video_url)\n", + "\n", + " return HTML(f\"\"\"\n", + "
\n", + " {video_player_html._repr_html_() if hasattr(video_player_html, '_repr_html_') else video_player_html}\n", + "
\n", + " {video_name}
{time_range}\n", + "
\n", + "
\n", + " \"\"\")" + ], + "metadata": { + "id": "n138fEwKOSSF" + }, + "execution_count": 9, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### 🔗 Get & Display Recent Stream\n", + "\n", + "This cell uses the helper function above to fetch and display the last few minutes of the stream." + ], + "metadata": { + "id": "Y3ZVVG2xB8W4" + } + }, + { + "cell_type": "code", + "source": [ + "# To get last few minutes stream link\n", + "import time\n", + "\n", + "def fetch_stream(rtstream):\n", + "\n", + " now = int(time.time())\n", + " start = int(now - (5 * 60))\n", + " stream_url = rtstream.generate_stream(start, now)\n", + " return stream_url\n", + "\n", + "video_url = fetch_stream(intruder_stream)\n", + "\n", + "video_name = \"🚨 Private Property · Intruder Detection\"\n", + "display_stream(video_url, video_name)" + ], + "metadata": { + "id": "GW2ZL4e7ONmP", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 422 + }, + "outputId": "fe92d06e-daff-4975-ce19-0c8f8d532fea" + }, + "execution_count": 10, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ], + "text/html": [ + "\n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " 🚨 Private Property · Intruder Detection
2025-05-29 10:39:58 → 2025-05-29 10:44:58 UTC\n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 10 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 📦 Step 4: Index Scenes and Detect Suspicious Activity\n", + "\n", + "We’ll create a real-time scene index that periodically analyzes video frames and generates natural language descriptions of what’s happening on the property.\n", + "\n", + "The AI will look for:\n", + "- People loitering\n", + "- Approaching or physically interacting with the door/lock\n", + "- Attempting to enter the property" + ], + "metadata": { + "id": "m3gM9_nbObaH" + } + }, + { + "cell_type": "code", + "source": [ + "from videodb import SceneExtractionType\n", + "\n", + "intruder_scene_index = intruder_stream.index_scenes(\n", + " extraction_type=SceneExtractionType.time_based,\n", + " extraction_config={\n", + " \"time\": 5,\n", + " \"frame_count\": 2,\n", + " },\n", + " prompt=\"Monitor the area around the house closely. Detect and classify human presence around the house as either loitering, interacting with the door/lock, or entering the house; otherwise, consider the area safe.\",\n", + " name=\"Intruder_Detection_Index\",\n", + ")\n", + "intruder_index_id = intruder_scene_index.rtstream_index_id\n", + "print(\"Scene Index ID:\", intruder_index_id)\n" + ], + "metadata": { + "id": "I4IDVrPBOcaT", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "ecf074dc-3952-44d4-99ae-1d82965b1f70" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Scene Index ID: 7a6b9e4dc0132ebe\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### Let us list the scene indexes created on our rtstream." + ], + "metadata": { + "id": "f7R-LeKAioYI" + } + }, + { + "cell_type": "code", + "source": [ + "def list_rtstream_indexes(rtstream):\n", + " # List live stream indexes\n", + " rtstream_indexes = rtstream.list_scene_indexes()\n", + " for rtstream_index in rtstream_indexes:\n", + "\n", + " print(f\"\"\"RTStreamSceneIndex:\n", + " Index ID : {rtstream_index.rtstream_index_id}\n", + " RTStream ID : {rtstream_index.rtstream_id}\n", + " Name : {rtstream_index.name}\n", + " Status : {rtstream_index.status}\n", + " Config : {rtstream_index.extraction_config}\n", + " Prompt : {rtstream_index.prompt}\n", + " \"\"\")\n", + " print(\"-\" * 80)\n", + "\n", + "list_rtstream_indexes(intruder_stream)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "MTHKJ6MCir--", + "outputId": "97a74f3d-b3c5-4004-ffa5-25aec62def10" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "RTStreamSceneIndex:\n", + " Index ID : 7a6b9e4dc0132ebe\n", + " RTStream ID : rts-019710fa-9511-79c3-a924-e229e4815410\n", + " Name : Intruder_Detection_Index\n", + " Status : running\n", + " Config : {'frame_count': '2', 'time': '5'}\n", + " Prompt : Monitor the area around the house closely. Detect and classify human presence around the house as either loitering, interacting with the door/lock, or entering the house; otherwise, consider the area safe.\n", + " \n", + "--------------------------------------------------------------------------------\n", + "RTStreamSceneIndex:\n", + " Index ID : caf71652aed1141c\n", + " RTStream ID : rts-019710fa-9511-79c3-a924-e229e4815410\n", + " Name : Intruder_Security_Index\n", + " Status : stopped\n", + " Config : {'frame_count': '3', 'time': '5'}\n", + " Prompt : Monitor the area around the house closely. Detect and classify human presence around the house as either loitering, interacting with the door/lock, or entering the house; otherwise, consider the area safe.\n", + " \n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### If you have already created a scene index, run the below cell with your **scene index id** to reconnect." + ], + "metadata": { + "id": "in8K0RIhCjpu" + } + }, + { + "cell_type": "code", + "source": [ + "# intruder_index_id = \"\"\n", + "# intruder_scene_index = intruder_stream.get_scene_index(intruder_index_id)" + ], + "metadata": { + "id": "WG2oQmwbCoC2" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# To stop the index\n", + "# intruder_scene_index.stop()" + ], + "metadata": { + "id": "-o_Ov8G6Cz5a" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# To start the index\n", + "# intruder_scene_index.start()" + ], + "metadata": { + "id": "yjaTDD3PCycw" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### Let us see the generated indexes" + ], + "metadata": { + "id": "7-N4193uOg7G" + } + }, + { + "cell_type": "code", + "source": [ + "import time\n", + "from datetime import datetime\n", + "from zoneinfo import ZoneInfo\n", + "\n", + "def _convert_to_ist(timestamp: float) -> str:\n", + " \"\"\"Convert UTC timestamp to IST (Asia/Kolkata) datetime string.\"\"\"\n", + " return (\n", + " datetime.fromtimestamp(timestamp)\n", + " .astimezone(ZoneInfo(\"Asia/Kolkata\"))\n", + " .strftime(\"%Y-%m-%d %H:%M:%S\")\n", + " )\n", + "\n", + "def get_scenes(rtstream, index_id):\n", + " # Print indexed scenes\n", + " rtstream_scene_index = rtstream.get_scene_index(index_id)\n", + " scenes = rtstream_scene_index.get_scenes(page_size=5)\n", + " # print(scenes[\"scenes\"][:2])\n", + " if scenes:\n", + " for scene in scenes.get(\"scenes\"):\n", + " start = _convert_to_ist(scene[\"start\"])\n", + " end = _convert_to_ist(scene[\"end\"])\n", + " description = scene[\"description\"]\n", + " print(f\"{start}-{end} : {description}\")\n", + " print(\"-\" * 80)\n", + " else:\n", + " print(\"Scenes not found for given index.\")\n", + "\n", + "get_scenes(intruder_stream , intruder_index_id)" + ], + "metadata": { + "id": "VzzKcqWyOikc", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "611aef8b-5084-4350-c13e-0c868511be44" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "2025-05-27 20:25:06-2025-05-27 20:25:10 : Based on the image, I can see a person near the left side of the image. The person is near a lawn mower. It is difficult to determine the person's intent or activity with certainty. The person is not interacting with the door/lock or entering the house. Therefore, the person is loitering.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-27 20:24:46-2025-05-27 20:24:52 : Based on the image, there is a human presence loitering in the distance.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-27 20:24:39-2025-05-27 20:24:44 : Here's a breakdown of the situation based on the images provided:\n", + "\n", + "**Analysis:**\n", + "\n", + "* **Human Presence Detected:** Yes, there are at least two individuals visible in the images. One is near the left side of the image, and another is in the background.\n", + "* **Classification:**\n", + " * The individual on the left appears to be walking or loitering.\n", + " * The individual in the background appears to be loitering.\n", + "* **Interaction with Door/Lock:** No interaction with the door or lock is visible in the images.\n", + "* **Entering the House:** No one is currently entering the house in the images.\n", + "\n", + "**Conclusion:**\n", + "\n", + "Based on the images, the area around the house is currently showing signs of loitering. Further monitoring is recommended to determine the intent of the individuals and whether their behavior escalates.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-27 20:24:31-2025-05-27 20:24:37 : Here's the analysis of the images based on your request:\n", + "\n", + "**Image 1:**\n", + "\n", + "* **Human Presence:** Yes, there is a human figure in the background.\n", + "* **Classification:** Loitering (The person is standing in the yard, not interacting with the door or entering the house).\n", + "\n", + "**Image 2:**\n", + "\n", + "* **Human Presence:** Yes, there is a human figure in the background.\n", + "* **Classification:** Loitering (The person is walking in the yard, not interacting with the door or entering the house).\n", + "\n", + "**Overall Assessment:**\n", + "\n", + "The area around the house is currently showing signs of loitering.\n", + "\n", + "--------------------------------------------------------------------------------\n", + "2025-05-27 20:24:23-2025-05-27 20:24:29 : Here's the analysis of the images:\n", + "\n", + "**Image 1:**\n", + "\n", + "* **Detection:** A person is detected in the yard.\n", + "* **Classification:** The person is walking across the yard, not loitering, interacting with the door/lock, or entering the house.\n", + "* **Status:** Safe.\n", + "\n", + "**Image 2:**\n", + "\n", + "* **Detection:** No person is detected in the yard.\n", + "* **Classification:** N/A\n", + "* **Status:** Safe.\n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "## 🚨 Level 1 Breach: Loitering Near the Property\n", + "\n", + "We’ll start by creating an event to detect people **loitering or lingering around the property** without actually touching or interacting with it.\n", + "\n", + "Then, we’ll immediately attach an alert to this event, so you get notified in real time.\n", + "\n", + "\n" + ], + "metadata": { + "id": "Sp9DKyYrOqVt" + } + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "#### First, let us set up the webhook url we are going to use in the alerts" + ], + "metadata": { + "id": "Ilimj-VxjFbV" + } + }, + { + "cell_type": "code", + "source": [ + "# Enter link to your webhook url where you want alerts to go. You can create one simply on pipedream.\n", + "webhook_url=\"\"" + ], + "metadata": { + "id": "wb_-9Q5VjPsI" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "### 📦 Create Event for Loitering Detection\n", + "\n", + "This event will flag when someone is spotted loitering near your property boundary.\n", + "\n" + ], + "metadata": { + "id": "F_bBDstPOuzG" + } + }, + { + "cell_type": "code", + "source": [ + "loitering_event_id = conn.create_event(\n", + " event_prompt=\"Detect if a person is loitering near the house perimeter.\",\n", + " label=\"loitering_near_property\"\n", + ")\n", + "print(\"Loitering Event ID:\", loitering_event_id)\n" + ], + "metadata": { + "id": "PzadAArvOvHn", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "f067b863-01a6-4cb0-e3a6-e2454e8899d8" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Loitering Event ID: 679ade6411306f35\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "### 📦 Create Alert for Level 1 Breach\n", + "\n", + "As soon as the AI detects a loitering event, it’ll send a webhook alert notification." + ], + "metadata": { + "id": "VVK_LWHoOxjO" + } + }, + { + "cell_type": "code", + "source": [ + "if webhook_url:\n", + " loitering_alert_id = intruder_scene_index.create_alert(\n", + " loitering_event_id,\n", + " callback_url=webhook_url\n", + " )\n", + " print(\"Loitering Alert ID:\", loitering_alert_id)\n", + "else:\n", + " print(\"Error: Please provide Webhook URL. Alert cannot be created without it.\")" + ], + "metadata": { + "id": "8lbFnsy1OzEI", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "0492be1b-31ab-40ad-a218-579930875c26" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Loitering Alert ID: c10272cfdf1bb7b1\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 🚨 Level 2 Breach: Door Checking or Lock Tampering\n", + "\n", + "Next, we’ll set up an event to detect when someone moves close to the front door and appears to check or tamper with the lock — a higher severity intrusion attempt.\n", + "\n", + "Then, we’ll link a dedicated alert to this event.\n", + "\n", + "\n" + ], + "metadata": { + "id": "V7XyCb_iO07e" + } + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "### 📦 Create Event for Intrusion Attempt\n", + "\n", + "This event will flag when a person approaches the property’s door and visibly interacts with it.\n", + "\n" + ], + "metadata": { + "id": "4kc__9fKO4_1" + } + }, + { + "cell_type": "code", + "source": [ + "intrusion_event_id = conn.create_event(\n", + " event_prompt=\"Detect if a person is interacting with the door or visibly checking the lock.\",\n", + " label=\"intrusion_attempt\"\n", + ")\n", + "print(\"Intrusion Attempt Event ID:\", intrusion_event_id)" + ], + "metadata": { + "id": "cPHYFJkxO33x", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "401bbe83-b38e-40a5-91f9-c800f4f313a9" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Intrusion Attempt Event ID: 7bd0ccf7e53637ef\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "### 📦 Create Alert for Level 2 Breach\n", + "\n", + "When AI detects this activity, an instant alert will be routed to your webhook." + ], + "metadata": { + "id": "vjWevnd5O76N" + } + }, + { + "cell_type": "code", + "source": [ + "if webhook_url:\n", + " intrusion_alert_id = intruder_scene_index.create_alert(\n", + " intrusion_event_id,\n", + " callback_url=webhook_url\n", + " )\n", + " print(\"Intrusion Alert ID:\", intrusion_alert_id)\n", + "else:\n", + " print(\"Error: Please provide Webhook URL. Alert cannot be created without it.\")" + ], + "metadata": { + "id": "AqE8F8jYO9vW", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "f64400de-fc58-4ff4-e8db-268208564feb" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Intrusion Alert ID: 441c80e43b16fbf5\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 🚨 Level 3 Breach: Unauthorized Property Entry\n", + "\n", + "Lastly, we’ll define an event to catch the most serious scenario — **someone entering the property**.\n", + "\n", + "Then, we’ll configure a real-time alert for it.\n", + "\n", + "\n" + ], + "metadata": { + "id": "64QtYHT8O_fG" + } + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "### 📦 Create Event for Property Entry\n", + "\n", + "This event triggers when AI spots someone crossing into the property, opening a door, or entering the house.\n", + "\n" + ], + "metadata": { + "id": "CzSqR3CnPDwW" + } + }, + { + "cell_type": "code", + "source": [ + "entry_event_id = conn.create_event(\n", + " event_prompt=\"Detect if a person enters the house or crosses the property boundary unlawfully.\",\n", + " label=\"property_entry\"\n", + ")\n", + "print(\"Property Entry Event ID:\", entry_event_id)\n" + ], + "metadata": { + "id": "4SGBukMAPEDi", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "95fc3858-e655-4470-8a11-7bd9bdb115a9" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Property Entry Event ID: 796434851654377e\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "### 📦 Create Alert for Level 3 Breach\n", + "\n", + "The AI will send an immediate webhook alert for this highest-severity breach." + ], + "metadata": { + "id": "5ifh9JI_PGBe" + } + }, + { + "cell_type": "code", + "source": [ + "if webhook_url:\n", + " entry_alert_id = intruder_scene_index.create_alert(\n", + " entry_event_id,\n", + " callback_url=webhook_url\n", + " )\n", + " print(\"Property Entry Alert ID:\", entry_alert_id)\n", + "else:\n", + " print(\"Error: Please provide Webhook URL. Alert cannot be created without it.\")" + ], + "metadata": { + "id": "wa-Om6R4PHHy", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "4817f3a4-cce5-484a-9d5e-bc5e41ba263b" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Property Entry Alert ID: 5d5bd37c77de2607\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### Lets see the events and alerts we have set up" + ], + "metadata": { + "id": "HX-EL8oGPIxG" + } + }, + { + "cell_type": "code", + "source": [ + "def list_rtstream_alerts(rtstream, index_id):\n", + " \"\"\"\n", + " Prints a list of alerts associated with a given scene index.\n", + " \"\"\"\n", + " rtstream_scene_index = rtstream.get_scene_index(index_id)\n", + " alerts = rtstream_scene_index.list_alerts()\n", + "\n", + " for alert in alerts:\n", + " print(f\"\"\"🔔 RTStream Alert:\n", + " Alert ID : {alert['alert_id']}\n", + " Event ID : {alert['event_id']}\n", + " Label : {alert['label']}\n", + " Prompt : {alert['prompt']}\n", + " Status : {alert['status']}\n", + " \"\"\")\n", + " print(\"-\" * 80)\n", + "\n", + "list_rtstream_alerts(intruder_stream, intruder_index_id)\n" + ], + "metadata": { + "id": "QJcsScd8PKDj", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "64e856a4-3327-44c1-a9cc-71d876dda859" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "🔔 RTStream Alert:\n", + " Alert ID : 441c80e43b16fbf5\n", + " Event ID : 7bd0ccf7e53637ef\n", + " Label : intrusion_attempt\n", + " Prompt : Detect if a person is interacting with the door or visibly checking the lock.\n", + " Status : enabled\n", + " \n", + "--------------------------------------------------------------------------------\n", + "🔔 RTStream Alert:\n", + " Alert ID : 5d5bd37c77de2607\n", + " Event ID : 796434851654377e\n", + " Label : property_entry\n", + " Prompt : Detect if a person enters the house or crosses the property boundary unlawfully.\n", + " Status : enabled\n", + " \n", + "--------------------------------------------------------------------------------\n", + "🔔 RTStream Alert:\n", + " Alert ID : c10272cfdf1bb7b1\n", + " Event ID : 679ade6411306f35\n", + " Label : loitering_near_property\n", + " Prompt : Detect if a person is loitering near the house perimeter.\n", + " Status : enabled\n", + " \n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 📡 Example Alert Payload\n", + "\n", + "A **Level 1: Loitering Around** alert received on the webhook:\n", + "\n", + "```json\n", + "{\n", + " \"event_id\": \"event-679ade6411306f35\",\n", + " \"label\": \"loitering_near_property\",\n", + " \"confidence\": 0.75,\n", + " \"explanation\": \"A person is detected loitering near the house perimeter, matching the alert context. Confidence is high due to the person's presence in the distance, although the specific activity is not definitively confirmed.\",\n", + " \"timestamp\": \"2025-05-27T14:58:34.908641+00:00\",\n", + " \"start_time\": \"2025-05-27T20:28:26.061836+05:30\",\n", + " \"end_time\": \"2025-05-27T20:28:30.089070+05:30\",\n", + " \"stream_url\": \"https://rt.stream.videodb.io/manifests/rts-019710fa-9511-79c3-a924-e229e4815410/1748357906000000-1748357910000000.m3u8\"\n", + "}\n", + "```\n", + "\n", + "✅ Similar alerts will be triggered for Level 2 and Level 3 security breaches.\n" + ], + "metadata": { + "id": "j8MBrY0lPYxm" + } + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### Let us have a look at the stream_url received in the alert." + ], + "metadata": { + "id": "NfsfAjeulCuJ" + } + }, + { + "cell_type": "code", + "source": [ + "alert_stream_url = \"https://rt.stream.videodb.io/manifests/rts-019710fa-9511-79c3-a924-e229e4815410/1748480575000000-1748480580000000.m3u8\"\n", + "video_name = \"🚨 Private Property · loitering_near_property\"\n", + "\n", + "display_stream(alert_stream_url,video_name)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 422 + }, + "id": "2IsLJaL1kCqh", + "outputId": "d552c6ef-b16a-4cfe-c329-02a03e449ad5" + }, + "execution_count": 7, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ], + "text/html": [ + "\n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " 🚨 Private Property · loitering_near_property
2025-05-29 01:02:55 → 2025-05-29 01:03:00 UTC\n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 7 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "- Let us disable the alerts now." + ], + "metadata": { + "id": "FZmnpmIe8o3b" + } + }, + { + "cell_type": "code", + "source": [ + "intruder_scene_index.disable_alert(loitering_alert_id)\n", + "intruder_scene_index.disable_alert(intrusion_alert_id)\n", + "intruder_scene_index.disable_alert(entry_alert_id)" + ], + "metadata": { + "id": "psQzO5wz8uTx" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "- To enable the alert again" + ], + "metadata": { + "id": "OhsfaDwP8p8L" + } + }, + { + "cell_type": "code", + "source": [ + "intruder_scene_index.enable_alert(loitering_alert_id)\n", + "intruder_scene_index.enable_alert(intrusion_alert_id)\n", + "intruder_scene_index.enable_alert(entry_alert_id)" + ], + "metadata": { + "id": "MiN78lXu8upz" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "- Now we can stop the stream" + ], + "metadata": { + "id": "7SSNYn8b-qsT" + } + }, + { + "cell_type": "code", + "source": [ + "intruder_stream.stop()" + ], + "metadata": { + "id": "zweOPCnR-qRQ" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 🌙 Wrapping Up: Smarter Property Security Without Stress\n", + "\n", + "In this notebook, we built a **layered AI-powered video surveillance system** to protect your property in real time.\n", + "\n", + "With this in place, property owners can leave home without anxiety, knowing they’ll be immediately notified if anyone is:\n", + "- **Loitering nearby**\n", + "- **Interacting with the door**\n", + "- **Or breaking in**\n", + "\n", + "---\n", + "\n", + "## 🔍 What Else Could We Secure?\n", + "\n", + "This layered security system isn’t limited to homes. It can just as easily protect:\n", + "- 🏬 **Retail stores from shoplifters**\n", + "- 🏢 **Warehouse perimeters and restricted areas**\n", + "- 🛡️ **Military bases and government installations**\n", + "- 🏫 **Schools and public buildings**\n", + "\n", + "**What would *you* secure next?**" + ], + "metadata": { + "id": "69EbvxIOPdn-" + } + } + ] +} \ No newline at end of file diff --git a/real_time_streaming/Road_Monitoring.ipynb b/real_time_streaming/Road_Monitoring.ipynb new file mode 100644 index 0000000..d8b12e7 --- /dev/null +++ b/real_time_streaming/Road_Monitoring.ipynb @@ -0,0 +1,1779 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "name": "python3", + "display_name": "Python 3" + }, + "language_info": { + "name": "python" + } + }, + "cells": [ + { + "cell_type": "markdown", + "source": [ + "# 🚦 Smart Road Safety Monitoring using VideoDB RTStream\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/video-db/videodb-cookbook/blob/main/real_time_streaming/Road_Monitoring.ipynb)\n", + "\n", + "## 📖 Storytime: Why This Matters\n", + "\n", + "Road accidents happen every single day — and many lives are lost not because of the severity of the crash itself, but because victims don’t receive timely aid and medical attention.\n", + "\n", + "Often, there’s no one around to report an incident, or bystanders take too long to respond — either from shock, panic, or the overwhelming nature of witnessing an accident. \n", + "But in those critical moments, even a **few seconds can make the difference between life and death**.\n", + "\n", + "**This is where AI can help.**\n", + "\n", + "With **VideoDB RTStream**, we can deploy cameras at accident-prone locations and let AI constantly monitor live video streams. \n", + "As soon as an accident occurs, AI will detect it and instantly send alerts to nearby emergency services or traffic authorities.\n", + "\n", + "---\n", + "\n", + "## 🚀 What You’ll Build in This Notebook \n", + "\n", + "This notebook demonstrates a smart, AI-powered road safety monitoring system by:\n", + "- 📺 Connecting a road surveillance video stream to VideoDB\n", + "- 🤖 Using AI to analyze live video and detect accidents in real time\n", + "- 🚨 Sending an automatic alert the moment an accident is detected\n", + "\n", + "Let's build it!" + ], + "metadata": { + "id": "fVwlZ3taSuZT" + } + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 📦 Step 1: Install Dependencies \n", + "\n", + "We’ll begin by installing the VideoDB." + ], + "metadata": { + "id": "YOLosD65SxZM" + } + }, + { + "cell_type": "code", + "source": [ + "!pip install -q videodb" + ], + "metadata": { + "id": "9qXdnI_USvHB", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "2cd02992-f51a-4261-feb7-ebb870155c39" + }, + "execution_count": 1, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + " Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Building wheel for videodb (setup.py) ... \u001b[?25l\u001b[?25hdone\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "## 📦 Step 2: Connect to VideoDB\n", + "\n", + "Let's connect to VideoDB's API using your credentials to prepare for stream monitoring.\n", + "\n", + "Please enter your `VIDEO_DB_API_KEY` in the input box that appears below after you run this cell.\n", + "\n", + "Your input will be masked.\n" + ], + "metadata": { + "id": "x3rCy2flS4Yj" + } + }, + { + "cell_type": "code", + "source": [ + "import videodb\n", + "import os\n", + "from getpass import getpass\n", + "\n", + "api_key = getpass(\"Please enter your VideoDB API Key: \")\n", + "\n", + "os.environ[\"VIDEO_DB_API_KEY\"] = api_key\n", + "\n", + "conn = videodb.connect()\n", + "coll = conn.get_collection()\n", + "\n", + "print(\"Connected to VideoDB securely!\")" + ], + "metadata": { + "id": "-GXWpXTwS4Fz", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "d62abf26-c73c-4006-9897-dd8812ec1006" + }, + "execution_count": 2, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Please enter your VideoDB API Key: ··········\n", + "Connected to VideoDB securely!\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 📦 Step 3: Connect to the Accident Monitoring RTSP Stream \n", + "Connect to the live camera stream monitoring a toll plaza — a high-risk accident location.\n", + "\n", + "In this demo, the stream is running at `rtsp://samples.rts.videodb.io:8554/accident`.\n" + ], + "metadata": { + "id": "PqyviwxuS_ZL" + } + }, + { + "cell_type": "code", + "source": [ + "rtsp_url = \"rtsp://samples.rts.videodb.io:8554/accident\"\n", + "accident_stream = coll.connect_rtstream(\n", + " name=\"Toll Plaza Accident Stream\",\n", + " url=rtsp_url,\n", + ")\n", + "print(accident_stream)" + ], + "metadata": { + "id": "ft_nrviUTEqG", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "58996897-05ad-4038-cc58-58857e2e5e12" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "RTStream(id=rts-019719e8-f7fb-71c1-bb3d-c4418eefdecd, name=Toll Plaza Accident Stream, collection_id=None, created_at=None, sample_rate=30, status=connected)\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### If you have already connected the stream, run the below cell with the **rtstream id** to reconnect." + ], + "metadata": { + "id": "Dn1sk6iX6Y80" + } + }, + { + "cell_type": "code", + "source": [ + "# accident_stream = coll.get_rtstream(\"\")" + ], + "metadata": { + "id": "EREENsqo6ZxB" + }, + "execution_count": 3, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# To stop the stream\n", + "# accident_stream.stop()" + ], + "metadata": { + "id": "_SqudCbN6cua" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# To start the stream\n", + "# accident_stream.start()" + ], + "metadata": { + "id": "MZdd-R4s6dDv" + }, + "execution_count": 4, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### 👀 Let us have a look at the toll palza" + ], + "metadata": { + "id": "tG1T-DGCTN_r" + } + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### 📺 Helper Function: Display Video Stream\n", + "\n", + "This cell contains a small utility function to help visualize the video streams with helpful information. You don't need to modify this code." + ], + "metadata": { + "id": "zvGXp9YQ66U7" + } + }, + { + "cell_type": "code", + "source": [ + "# To display the stream with relevant information\n", + "\n", + "from IPython.display import HTML\n", + "import re\n", + "from datetime import datetime\n", + "from videodb import play_stream\n", + "\n", + "def display_stream(video_url, video_name=\"🎥 Camera Feed\"):\n", + "\n", + " match = re.search(r'/(\\d{16})-(\\d{16})\\.m3u8', video_url)\n", + " if match:\n", + " start_ts = int(match.group(1)) / 1e6\n", + " end_ts = int(match.group(2)) / 1e6\n", + " start_time = datetime.utcfromtimestamp(start_ts).strftime('%Y-%m-%d %H:%M:%S')\n", + " end_time = datetime.utcfromtimestamp(end_ts).strftime('%Y-%m-%d %H:%M:%S')\n", + " time_range = f\"{start_time} → {end_time} UTC\"\n", + " else:\n", + " time_range = \"Time Unknown\"\n", + "\n", + " video_player_html = play_stream(video_url)\n", + "\n", + " return HTML(f\"\"\"\n", + "
\n", + " {video_player_html._repr_html_() if hasattr(video_player_html, '_repr_html_') else video_player_html}\n", + "
\n", + " {video_name}
{time_range}\n", + "
\n", + "
\n", + " \"\"\")" + ], + "metadata": { + "id": "6KyfCkwVTVKj" + }, + "execution_count": 5, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### 🔗 Get & Display Recent Stream\n", + "\n", + "This cell uses the helper function above to fetch and display the last few minutes of the stream." + ], + "metadata": { + "id": "yD85ycwD7Dyh" + } + }, + { + "cell_type": "code", + "source": [ + "# To get last few minutes stream link\n", + "import time\n", + "\n", + "def fetch_stream(rtstream):\n", + "\n", + " now = int(time.time())\n", + " start = int(now - (5 * 60))\n", + " stream_url = rtstream.generate_stream(start, now)\n", + " return stream_url\n", + "\n", + "video_url = fetch_stream(accident_stream)\n", + "\n", + "stream_name = \"🚧 Toll Plaza · Accident Detection\"\n", + "display_stream(video_url , stream_name)" + ], + "metadata": { + "id": "XSwaKQ6FTPg_", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 422 + }, + "outputId": "fe16a3d0-5950-4ce2-ba35-a9fc2fef6167" + }, + "execution_count": 7, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ], + "text/html": [ + "\n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " 🚧 Toll Plaza · Accident Detection
2025-05-29 11:14:53 → 2025-05-29 11:19:53 UTC\n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 7 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "## 📦 Step 4: Index Scenes and Detect Accidents \n", + "We’ll create a real-time scene index that periodically analyzes video frames and generates natural language descriptions of what’s happening in the stream.\n", + "\n", + "The AI will look for visual signs of crashes, vehicle collisions, or people falling down.\n" + ], + "metadata": { + "id": "cV38KJbSTZuD" + } + }, + { + "cell_type": "code", + "source": [ + "from videodb import SceneExtractionType\n", + "\n", + "accident_scene_index = accident_stream.index_scenes(\n", + " extraction_type=SceneExtractionType.time_based,\n", + " extraction_config={\n", + " \"time\":10,\n", + " \"frame_count\": 2,\n", + " },\n", + " prompt=\"Monitor the toll plaza road carefully. Detect if a vehicle collides, crashes, or a person falls. Describe the situation clearly if an accident occurs.\",\n", + " name=\"Accident_Index\"\n", + ")\n", + "\n", + "accident_index_id = accident_scene_index.rtstream_index_id\n", + "print(\"Scene Index ID:\", accident_index_id)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "_exbRcxb8KUI", + "outputId": "de41aab2-eee5-42da-e0ad-27ea33138b49" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Scene Index ID: c57e9eac387f0414\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### If you have already created a scene index, run the below cell with your **scene index id** to reconnect." + ], + "metadata": { + "id": "FFgY73hB8eMF" + } + }, + { + "cell_type": "code", + "source": [ + "# accident_index_id = \"\"\n", + "# accident_scene_index = accident_stream.get_scene_index(accident_index_id)" + ], + "metadata": { + "id": "k2bA19wr8fGv" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# To stop the index\n", + "# accident_scene_index.stop()" + ], + "metadata": { + "id": "sXtZzujF8fbl" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# To start the index\n", + "# accident_scene_index.start()" + ], + "metadata": { + "id": "U7TwfLhe8fzl" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### Let us see the result of the scene indexing\n" + ], + "metadata": { + "id": "5UIWFfSJTjqE" + } + }, + { + "cell_type": "code", + "source": [ + "import time\n", + "from datetime import datetime\n", + "from zoneinfo import ZoneInfo\n", + "\n", + "def _convert_to_ist(timestamp: float) -> str:\n", + " \"\"\"Convert UTC timestamp to IST (Asia/Kolkata) datetime string.\"\"\"\n", + " return (\n", + " datetime.fromtimestamp(timestamp)\n", + " .astimezone(ZoneInfo(\"Asia/Kolkata\"))\n", + " .strftime(\"%Y-%m-%d %H:%M:%S\")\n", + " )\n", + "\n", + "def get_scenes(rtstream, index_id):\n", + " # Print indexed scenes\n", + " rtstream_scene_index = rtstream.get_scene_index(index_id)\n", + " scenes = rtstream_scene_index.get_scenes(page_size=5)\n", + " # print(scenes[\"scenes\"][:2])\n", + " if scenes:\n", + " for scene in scenes.get(\"scenes\"):\n", + " start = _convert_to_ist(scene[\"start\"])\n", + " end = _convert_to_ist(scene[\"end\"])\n", + " description = scene[\"description\"]\n", + " print(f\"{start}-{end}: {description}\")\n", + " print(\"-\" * 80)\n", + " else:\n", + " print(\"Scenes not found for given index.\")\n", + "\n", + "get_scenes(accident_stream, accident_index_id)" + ], + "metadata": { + "id": "YVBhVVqqTkwu", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "7d01b178-63a8-4085-f169-23cc6ad68c4a" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "2025-05-29 08:21:54-2025-05-29 08:22:02: **Accident Detected:**\n", + "\n", + "A black car has crashed at the toll plaza. Debris is scattered across the road, indicating a significant impact. The car appears to have collided with the toll booth structure. A person is standing near the car, seemingly assessing the damage. The situation suggests a potentially serious accident.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-29 08:21:45-2025-05-29 08:21:53: **Accident Detected:**\n", + "\n", + "**Description:**\n", + "\n", + "It appears there has been a collision at the toll plaza. Debris is scattered across the road, and a black car is positioned between two toll booths with its door open. A person is standing near the car, and another individual wearing a yellow vest is present at the scene, possibly assessing the situation. The presence of debris suggests a significant impact.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-29 08:21:36-2025-05-29 08:21:44: **Accident Detected:**\n", + "\n", + "A vehicle has crashed at the toll plaza. There is debris scattered across the road, and smoke is visible. A black car is stopped in one of the lanes, and a person is exiting the vehicle. A worker in a safety vest is standing near one of the toll booths. The extent of the damage and any potential injuries are not clear from the image.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-29 08:21:27-2025-05-29 08:21:35: **Accident Detected:**\n", + "\n", + "Based on the image, it appears there has been a significant accident at the toll plaza. Here's a breakdown of the situation:\n", + "\n", + "* **Collision/Crash:** There is debris scattered across the road, indicating a collision has occurred. A white vehicle is visible further down the road, possibly involved in the accident.\n", + "* **Vehicle Damage:** The black car in the foreground appears to be damaged.\n", + "* **Person Present:** A person is standing near the black car.\n", + "* **Emergency Personnel:** An individual wearing a high-visibility vest is present, suggesting that emergency personnel are on the scene.\n", + "\n", + "**Overall Situation:**\n", + "\n", + "The scene suggests a high-impact collision at the toll plaza. The presence of debris, damaged vehicles, and emergency personnel indicates a serious incident that requires immediate attention.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-29 08:21:18-2025-05-29 08:21:26: **Accident Detected:**\n", + "\n", + "**Description:** A white vehicle has crashed into the toll plaza. Debris is scattered across the road. A black car is stopped in the lane to the left of the crash site. The severity of the crash and any potential injuries are unknown.\n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "## 📦 Step 5: Define an Accident Detection Event \n", + "Now, we’ll define an event type in the system to detect visual signs of a road accident.\n" + ], + "metadata": { + "id": "6rI9zsuHTqrs" + } + }, + { + "cell_type": "code", + "source": [ + "accident_event_id = conn.create_event(\n", + " event_prompt=\"Detect if an accident or vehicle collision takes place.\",\n", + " label=\"road_accident\"\n", + ")\n", + "print(\"Event ID:\", accident_event_id)" + ], + "metadata": { + "id": "i5NRGNOeTsNa", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "eb2f515f-326b-49af-a1b0-f09888da4352" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Event ID: e1d716cd32a6fbd1\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 📦 Step 6: Attach an Alert to the Accident Event \n", + "Finally, we’ll link a real-time alert to this event. This will send a notification to our webhook the moment an accident is detected.\n" + ], + "metadata": { + "id": "KvynpKnQTvlL" + } + }, + { + "cell_type": "code", + "source": [ + "webhook_url = \"\"\n", + "\n", + "if webhook_url:\n", + " accident_alert_id = accident_scene_index.create_alert(\n", + " accident_event_id,\n", + " callback_url=webhook_url\n", + " )\n", + " print(\"Alert ID:\", accident_alert_id)\n", + "else:\n", + " print(\"Error: Please provide Webhook URL. Alert cannot be created without it.\")" + ], + "metadata": { + "id": "5GSOv6k6TwvD", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "37ce4dfb-3477-46af-b917-0c3e20cdd3ba" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Alert ID: dc2896f3698072e9\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "## 📡 Example Alert Payload \n", + "When an accident was detected, the system sent this alert:\n", + "\n", + "```json\n", + "{\n", + " \"event_id\": \"event-4566fbdc1c6a309c\",\n", + " \"label\": \"road_accident\",\n", + " \"confidence\": 0.95,\n", + " \"explanation\": \"The scene analysis indicates a significant collision at the toll plaza involving at least two vehicles and scattering of debris. This strongly suggests an accident has occurred.\",\n", + " \"timestamp\": \"2025-05-11T05:35:01.052189+00:00\",\n", + " \"start_time\": \"2025-05-11T10:44:59.580915+05:30\",\n", + " \"end_time\": \"2025-05-11T10:45:01.504513+05:30\",\n", + " \"stream_url\": \"https://videodb-rt-streaming-service-us-east-1.s3.amazonaws.com/manifests/rts-019677f7-19f9-7e03-9ca8-a920441f2463/1746940499000000-1746940501000000.m3u8?AWSAccessKeyId=ASIAQPKADS32ZNS2LJFY&Signature=HVGiK%2BV4COnKwgS5k9fNc8HZUCU%3D&x-amz-security-token=IQoJb3JpZ2luX2VjEA4aCXVzLWVhc3QtMSJIMEYCIQCdGs1TJm0aLxK6khpPTsOZSNp6ebGkj1qQkUnxNB086wIhAL93q4GyQ%2FxEJewR6maifYWLo4mf5sWT1ZXEW%2BU4CRZUKocDCLf%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQAxoMMDMyODgzNTc4NjEzIgwMCodrIkk0HvqKPEEq2wKCbOxF4c4e03FUzHKDjYjA4rNkQ4YWJQ7FcuO0HROsHkHUV3pog3bzoY8a1hYXQSSm2185Q507LEkrqJplStBfozp5ahXsDcS%2FJi%2F2QxiXaT1Ww7wBhedjEqePTeNHiVw0bpXiJpHWBQTQnJ1o0PiLw6c1ytHM9mVk51G81BZxsYdlghoR49Cra5k5vMu2tGYvlrghs1aiV8tZtW7jo3SfK%2FJRhNLYLsGmFoIv3Jabh2Q2WzCyvKTxwDu8%2Fcem0GcbQSVV68e%2BAafUUcaLzf5PjZXdQanz%2Fg%2B%2FSVLi32MLtAhF%2FOyPLGlu5yi3LlbOeznutcnte2XMzlcWUMbZYpVwoqIe2Wkpmbgk44COgTitWwJ4PRIcY6IbvuC1JWjKKnnpXGsJZMFl64f4WuwKKHdu3Xo0HGzg8kJcEFHTFC%2BCRAfpILed40TYIu3IcGqXVNJp9Din%2BbYG7GmLqjCC7oDBBjqdAZXxoutuFqLRrLz9Ai3YchNKgUSi3bL1kNBmv8zUzskiQhwQxFh2OTwbT0ESw9D8rLNRwl27RoAuUzqwy9%2FdO75wQ%2F2dCLwrbhv3QFqiTTWLm3z91EreRdjlZ0vQWT%2FKwdNspyKkegpTVHDj%2BaGVsu4Tig%2BnktBQkHZbIR4yiPXAPMa6eRW7UwFI3jqdo3TO2iU5DxuVa7X8%2Bx07foU%3D&Expires=1747028101\"\n", + "}\n", + "```" + ], + "metadata": { + "id": "UzVBUUc1Tyj8" + } + }, + { + "cell_type": "code", + "source": [ + "alert_stream_url = \"https://rt.stream.videodb.io/manifests/rts-019719e8-f7fb-71c1-bb3d-c4418eefdecd/1748487351000000-1748487360000000.m3u8\"\n", + "video_name = \"🚧 Toll Plaza · road_accident\"\n", + "\n", + "display_stream(alert_stream_url, video_name)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 422 + }, + "id": "-XaftWVo979r", + "outputId": "7f98e9e8-2df1-4e82-d635-572e83303658" + }, + "execution_count": 7, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ], + "text/html": [ + "\n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " 🚧 Toll Plaza · road_accident
2025-05-29 02:55:51 → 2025-05-29 02:56:00 UTC\n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 7 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "- Disabling the alert after use" + ], + "metadata": { + "id": "JPetvuDq_hiI" + } + }, + { + "cell_type": "code", + "source": [ + "accident_scene_index.disable_alert(accident_alert_id)" + ], + "metadata": { + "id": "_i6ELvNP_mOS" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "- To re-enable the alert" + ], + "metadata": { + "id": "5omGHiFj_v5C" + } + }, + { + "cell_type": "code", + "source": [ + "accident_scene_index.enable_alert(accident_alert_id)" + ], + "metadata": { + "id": "TFAEaAiI_07W" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "- Let us stop the accident index and proceed with the notebook to explore more possibilities." + ], + "metadata": { + "id": "2L_qLZ19-lM3" + } + }, + { + "cell_type": "code", + "source": [ + "accident_scene_index.stop()" + ], + "metadata": { + "id": "3Rq-HHs5-trc" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "## 🚦 Detecting Traffic Rule Violations at the Toll Plaza\n", + "\n", + "Apart from accident monitoring, we can extend our system to detect **traffic rule violations** at the toll plaza as well.\n", + "\n", + "For this, we’ll:\n", + "- Create a **new scene index** specifically focused on monitoring rule compliance.\n", + "- Define an **event** to catch rule-breaking behavior (such as skipping tolls, wrong-lane driving, or ignoring barriers).\n", + "- Attach an **alert** that notifies authorities when a violation is detected.\n", + "\n", + "---\n", + "\n", + "### 📦 Create a New Scene Index for Rule Monitoring \n" + ], + "metadata": { + "id": "KM4T6N7nT2bE" + } + }, + { + "cell_type": "code", + "source": [ + "rule_scene_index = accident_stream.index_scenes(\n", + " extraction_type=SceneExtractionType.time_based,\n", + " extraction_config={\n", + " \"time\": 3,\n", + " \"frame_count\": 2,\n", + " },\n", + " prompt=\"Monitor the toll plaza carefully. Detect if any vehicle breaks traffic rules — for example, skipping the toll booth, crossing without stopping, driving in the wrong lane, or ignoring the barrier. Describe such violations clearly.\",\n", + " name=\"Toll_Rule_Violation_Index\"\n", + ")\n", + "rule_index_id = rule_scene_index.rtstream_index_id\n", + "print(\"Scene Index ID:\", rule_index_id)\n" + ], + "metadata": { + "id": "dkuuHWvxTzxX", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "6934ff10-7e58-4fad-8c55-7f43f8466a77" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Scene Index ID: ef827612a22d78b3\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### Let us list the scene indexes created on our rtstream." + ], + "metadata": { + "id": "eNMJudng-7_Y" + } + }, + { + "cell_type": "code", + "source": [ + "def list_rtstream_indexes(rtstream):\n", + " # List live stream indexes\n", + " rtstream_indexes = rtstream.list_scene_indexes()\n", + " for rtstream_index in rtstream_indexes:\n", + "\n", + " print(f\"\"\"RTStreamSceneIndex:\n", + " Index ID : {rtstream_index.rtstream_index_id}\n", + " RTStream ID : {rtstream_index.rtstream_id}\n", + " Name : {rtstream_index.name}\n", + " Status : {rtstream_index.status}\n", + " Config : {rtstream_index.extraction_config}\n", + " Prompt : {rtstream_index.prompt}\n", + " \"\"\")\n", + " print(\"-\" * 80)\n", + "\n", + "list_rtstream_indexes(accident_stream)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "2YcPWmYP_Ppw", + "outputId": "398ea2a5-5d1d-4089-ba30-4ff099728fa0" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "RTStreamSceneIndex:\n", + " Index ID : c57e9eac387f0414\n", + " RTStream ID : rts-019719e8-f7fb-71c1-bb3d-c4418eefdecd\n", + " Name : Accident_Index\n", + " Status : stopped\n", + " Config : {'frame_count': '2', 'time': '10'}\n", + " Prompt : Monitor the toll plaza road carefully. Detect if a vehicle collides, crashes, or a person falls. Describe the situation clearly if an accident occurs.\n", + " \n", + "--------------------------------------------------------------------------------\n", + "RTStreamSceneIndex:\n", + " Index ID : ef827612a22d78b3\n", + " RTStream ID : rts-019719e8-f7fb-71c1-bb3d-c4418eefdecd\n", + " Name : Toll_Rule_Violation_Index\n", + " Status : running\n", + " Config : {'frame_count': '2', 'time': '3'}\n", + " Prompt : Monitor the toll plaza carefully. Detect if any vehicle breaks traffic rules — for example, skipping the toll booth, crossing without stopping, driving in the wrong lane, or ignoring the barrier. Describe such violations clearly.\n", + " \n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### Let us see the scenes generated" + ], + "metadata": { + "id": "cXP4j0WAT_6a" + } + }, + { + "cell_type": "code", + "source": [ + "get_scenes(accident_stream, rule_index_id)" + ], + "metadata": { + "id": "SktVw1nsUA8f", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "b3a82662-dd59-4dc3-fc10-ee51aa5169da" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "2025-05-29 08:32:30-2025-05-29 08:32:33: Based on the image, it appears there has been a significant incident at the toll plaza. Debris is scattered across multiple lanes, and a white vehicle is positioned at an angle, suggesting it may have crashed. The black car in the foreground seems to be approaching the scene cautiously.\n", + "\n", + "**Potential Violations:**\n", + "\n", + "* **Skipping the toll booth/Crossing without stopping:** The white vehicle's position and the debris suggest it may have bypassed the toll booth without stopping, potentially at high speed.\n", + "* **Damage to property:** The extensive debris and the position of the white vehicle indicate damage to the toll plaza infrastructure.\n", + "\n", + "It's important to note that without more information, it's difficult to determine the exact sequence of events or the specific violations that occurred. Further investigation would be needed to ascertain the cause of the incident and any traffic violations involved.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-29 08:32:26-2025-05-29 08:32:29: Based on the images, it appears that a white vehicle has crashed at the toll plaza. In the first image, the white vehicle is airborne, indicating a high-speed impact. In the second image, the white vehicle is on the ground, and debris is scattered around the area. This suggests a severe violation of traffic rules, including:\n", + "\n", + "1. **Failure to Stop:** The vehicle did not stop at the toll booth.\n", + "2. **Excessive Speed:** The vehicle was traveling at a high rate of speed, resulting in a crash.\n", + "3. **Damage to Property:** The crash has caused damage to the toll plaza infrastructure.\n", + "\n", + "The black vehicle in the foreground appears to be unaffected by the incident.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-29 08:32:22-2025-05-29 08:32:25: Based on the images, it appears a white vehicle has crashed at high speed through the toll plaza. Debris is scattered across the lanes, and the vehicle is airborne in the second image. This constitutes a severe violation of traffic rules, including:\n", + "\n", + "* **Skipping the toll booth:** The vehicle did not stop to pay the toll.\n", + "* **Crossing without stopping:** The vehicle proceeded through the toll plaza at high speed without stopping.\n", + "* **Ignoring the barrier:** The vehicle crashed through the barrier.\n", + "* **Reckless driving:** The high speed and resulting crash indicate reckless driving.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-29 08:32:18-2025-05-29 08:32:21: In the second image, a white vehicle appears to have crashed into the toll barrier on the right side of the image. This constitutes a traffic violation, as it involves damaging toll infrastructure and likely failing to stop or control the vehicle properly at the toll plaza.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-29 08:32:14-2025-05-29 08:32:17: Okay, I've analyzed the video of the toll plaza.\n", + "\n", + "Based on the image provided, I cannot detect any traffic violations. The black car appears to be proceeding through the toll booth in a normal manner. There is no evidence of skipping the toll, crossing without stopping, driving in the wrong lane, or ignoring the barrier.\n", + "\n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### 📦 Create Event for Traffic Rule Violation " + ], + "metadata": { + "id": "SmOvrJ59UC0y" + } + }, + { + "cell_type": "code", + "source": [ + "rule_violation_event_id = conn.create_event(\n", + " event_prompt=\"Detect if a vehicle breaks traffic rules at the toll plaza.\",\n", + " label=\"toll_rule_violation\"\n", + ")\n", + "print(\"Event ID:\", rule_violation_event_id)" + ], + "metadata": { + "id": "wztzY0bvUD8G", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "82eee2a4-07a4-44c9-f799-9369817f7f10" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Event ID: 08da2d1e62a5fca2\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "### 📦 Attach an Alert for Rule Violation Detection " + ], + "metadata": { + "id": "cMJvdtS3UFsi" + } + }, + { + "cell_type": "code", + "source": [ + "# Enter link to your webhook url where you want alerts to go.\n", + "rule_violation_webhook_url = \"\"\n", + "\n", + "if rule_violation_webhook_url:\n", + " rule_violation_alert_id = rule_scene_index.create_alert(\n", + " rule_violation_event_id,\n", + " callback_url=rule_violation_webhook_url\n", + " )\n", + " print(\"Alert ID:\", rule_violation_alert_id)\n", + "else:\n", + " print(\"Error: Please provide Webhook URL. Alert cannot be created without it.\")" + ], + "metadata": { + "id": "ag2Z278rUGsC", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "outputId": "3efa8251-0f30-4717-a691-97976c429848" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Alert ID: 31865b33a64e75f5\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### The following alert was received on the webhook url identifying a potential traffic rule violation\n", + "```json\n", + "{\n", + " \"event_id\": \"event-08da2d1e62a5fca2\",\n", + " \"label\": \"toll_rule_violation\",\n", + " \"confidence\": 0.95,\n", + " \"explanation\": \"The extensive damage to the toll plaza and the position of the car strongly suggest a traffic rule violation, such as failure to stop or reckless driving.\",\n", + " \"timestamp\": \"2025-05-29T03:06:04.085416+00:00\",\n", + " \"start_time\": \"2025-05-29T08:34:43.276507+05:30\",\n", + " \"end_time\": \"2025-05-29T08:34:46.289540+05:30\",\n", + " \"stream_url\": \"https://rt.stream.videodb.io/manifests/rts-019719e8-f7fb-71c1-bb3d-c4418eefdecd/1748487883000000-1748487887000000.m3u8\"\n", + "}\n", + "```" + ], + "metadata": { + "id": "uFPo3v9XUNS-" + } + }, + { + "cell_type": "code", + "source": [ + "alert_stream_url = \"https://rt.stream.videodb.io/manifests/rts-019719e8-f7fb-71c1-bb3d-c4418eefdecd/1748487883000000-1748487887000000.m3u8\"\n", + "video_name = \"🚧 Toll Plaza · toll_rule_violation\"\n", + "\n", + "display_stream(alert_stream_url, video_name)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 422 + }, + "id": "HhpsLQ7JF59l", + "outputId": "d47280b3-da7d-4ae5-e8b5-5278bc29781f" + }, + "execution_count": 8, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ], + "text/html": [ + "\n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " 🚧 Toll Plaza · toll_rule_violation
2025-05-29 03:04:43 → 2025-05-29 03:04:47 UTC\n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 8 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "- Let us stop the alert after use" + ], + "metadata": { + "id": "Ez8ocDcgADS4" + } + }, + { + "cell_type": "code", + "source": [ + "rule_scene_index.disable_alert(rule_violation_alert_id)" + ], + "metadata": { + "id": "Z3TcBKVuAG8X" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "- To re-enable the alert" + ], + "metadata": { + "id": "zqqbpqtzAHoe" + } + }, + { + "cell_type": "code", + "source": [ + "rule_scene_index.enable_alert(rule_violation_alert_id)" + ], + "metadata": { + "id": "7cxhddmEAJrf" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "- Let us stop the present stream and proceed forward in the notebook." + ], + "metadata": { + "id": "PFj2miFVU73t" + } + }, + { + "cell_type": "code", + "source": [ + "accident_stream.stop()" + ], + "metadata": { + "id": "VUtF5qtT45NP" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "## 🚦 Traffic Congestion Detection: Prevent Traffic Pile-Ups Early\n", + "\n", + "Since we’ve already built our accident alert system, let’s expand its capabilities.\n", + "\n", + "On busy roads, even a minor incident or slow-moving vehicle can quickly escalate into a traffic jam, wasting commuters' crucial time. \n", + "A person might miss an important meeting, a student might arrive late for an exam — and city traffic can get messy fast.\n", + "\n", + "It’s important to detect **the very start of a traffic jam** early, so authorities can intervene and resolve it before it spreads.\n", + "\n", + "Let's build a system to save your time!" + ], + "metadata": { + "id": "qXqc_irE5DYR" + } + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 📦 Step 1: Connect to the Highway Traffic RTSP Stream \n", + "\n", + "We’ll connect a live camera stream monitoring a busy multi-lane highway.\n" + ], + "metadata": { + "id": "V9kQqM8D5JVA" + } + }, + { + "cell_type": "code", + "source": [ + "traffic_rtsp_url = \"rtsp://3.6.198.206:8554/traffic\"\n", + "traffic_stream = coll.connect_rtstream(\n", + " name=\"Highway Traffic Stream\",\n", + " url=traffic_rtsp_url,\n", + ")\n", + "print(traffic_stream)" + ], + "metadata": { + "id": "CaT_T2CQ5F4H" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "#### Let us list all the rtstreams in our collection." + ], + "metadata": { + "id": "pEdbjq5gAzZv" + } + }, + { + "cell_type": "code", + "source": [ + "def list_rtstreams():\n", + " for rtstream in coll.list_rtstreams():\n", + " print(f\"\"\"RTStream:\n", + " ID : {rtstream.id}\n", + " Name : {rtstream.name}\n", + " Collection ID : {rtstream.collection_id}\n", + " Created At : {rtstream.created_at}\n", + " Sample Rate : {rtstream.sample_rate}\n", + " Status : {rtstream.status}\n", + " \"\"\")\n", + " print(\"-\" * 80)\n", + "\n", + "list_rtstreams()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "PZmj9saDA1fA", + "outputId": "44b86767-82f4-4ef0-8463-49199b682d75" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "RTStream:\n", + " ID : rts-01971a06-eb5c-7453-a257-e0135de60c29\n", + " Name : Highway Traffic Stream\n", + " Collection ID : c-81fc6459-fe30-44ac-8c5b-ea0898c2e152\n", + " Created At : 2025-05-29T03:12:30.556060\n", + " Sample Rate : 30\n", + " Status : connected\n", + " \n", + "--------------------------------------------------------------------------------\n", + "RTStream:\n", + " ID : rts-019719e8-f7fb-71c1-bb3d-c4418eefdecd\n", + " Name : Toll Plaza Accident Stream\n", + " Collection ID : c-81fc6459-fe30-44ac-8c5b-ea0898c2e152\n", + " Created At : 2025-05-29T02:39:47.707342\n", + " Sample Rate : 30\n", + " Status : stopped\n", + " \n", + "--------------------------------------------------------------------------------\n", + "RTStream:\n", + " ID : rts-019719b2-0c84-7a71-a037-311855a8d160\n", + " Name : Arizona Flood Stream\n", + " Collection ID : c-81fc6459-fe30-44ac-8c5b-ea0898c2e152\n", + " Created At : 2025-05-29T01:39:48.484134\n", + " Sample Rate : 30\n", + " Status : stopped\n", + " \n", + "--------------------------------------------------------------------------------\n", + "RTStream:\n", + " ID : rts-019711db-1086-7750-ba79-8f47a4fed603\n", + " Name : Cricket Finals Stream\n", + " Collection ID : c-81fc6459-fe30-44ac-8c5b-ea0898c2e152\n", + " Created At : 2025-05-27T13:07:38.758741\n", + " Sample Rate : 30\n", + " Status : stopped\n", + " \n", + "--------------------------------------------------------------------------------\n", + "RTStream:\n", + " ID : rts-019711a0-0fde-7911-b282-25bc0b4ecf65\n", + " Name : Baby Crib Monitor\n", + " Collection ID : c-81fc6459-fe30-44ac-8c5b-ea0898c2e152\n", + " Created At : 2025-05-27T12:03:11.966358\n", + " Sample Rate : 30\n", + " Status : stopped\n", + " \n", + "--------------------------------------------------------------------------------\n", + "RTStream:\n", + " ID : rts-019710fa-9511-79c3-a924-e229e4815410\n", + " Name : Property Security Stream\n", + " Collection ID : c-81fc6459-fe30-44ac-8c5b-ea0898c2e152\n", + " Created At : 2025-05-27T09:02:27.089279\n", + " Sample Rate : 30\n", + " Status : stopped\n", + " \n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "#### If you have already connected the stream, run the below cell with the **rtstream id** to reconnect." + ], + "metadata": { + "id": "9hlwyNp76cL3" + } + }, + { + "cell_type": "code", + "source": [ + "# traffic_stream = coll.get_rtstream(id=\"\")" + ], + "metadata": { + "id": "OZ-Hg5A45QOJ" + }, + "execution_count": 9, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### Let us have a look at the busy lanes" + ], + "metadata": { + "id": "GrtjXIp65Mwn" + } + }, + { + "cell_type": "code", + "source": [ + "video_url = fetch_stream(traffic_stream)\n", + "stream_name = \"🚥 Central Highway · Traffic Detection\"\n", + "display_stream(video_url , stream_name)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 422 + }, + "id": "ZUg57M-65UrX", + "outputId": "3a3d72ed-715e-4a3c-f68b-9e8022d4c270" + }, + "execution_count": 17, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ], + "text/html": [ + "\n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " 🚥 Central Highway · Traffic Detection
2025-05-29 10:55:59 → 2025-05-29 11:00:59 UTC\n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 17 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "## 📦 Step 2: Index Scenes and Detect Traffic Congestion \n", + "\n", + "We’ll create a real-time scene index that periodically analyzes video frames and generates natural language descriptions of what’s happening on the highway.\n", + "\n", + "The AI will look for **slow-moving traffic, halted vehicles, or signs of congestion building up**.\n" + ], + "metadata": { + "id": "2wMP1XBN5XiA" + } + }, + { + "cell_type": "code", + "source": [ + "from videodb import SceneExtractionType\n", + "\n", + "traffic_jam_scene_index = traffic_stream.index_scenes(\n", + " extraction_type=SceneExtractionType.time_based,\n", + " extraction_config={\n", + " \"time\": 5,\n", + " \"frame_count\": 3,\n", + " },\n", + " prompt=\"Monitor the lanes of vehicles over several video frames. If the whole lane of cars consistently moves very slowly or stops, classify the situation as 'traffic congestion detected.' Otherwise, classify it as 'regular traffic flow.\",\n", + " name=\"Traffic_Congestion_Index\"\n", + ")\n", + "traffic_jam_index_id = traffic_jam_scene_index.rtstream_index_id\n", + "print(\"Scene Index ID:\", traffic_jam_index_id)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "SA8nc30n5ZyJ", + "outputId": "9ca02dc0-c615-4b03-ee7b-b135b0c8a3f2" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Scene Index ID: 403efcbae504de61\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "#### If you have already created a scene index, run the below cell with your **scene index id** to reconnect." + ], + "metadata": { + "id": "d819cxWM-CNq" + } + }, + { + "cell_type": "code", + "source": [ + "# traffic_index_id = \"\"\n", + "# traffic_scene_index = traffic_stream.get_scene_index(traffic_index_id)" + ], + "metadata": { + "id": "GBm3UYiO-Hlc" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### Let us see the indexes being generated" + ], + "metadata": { + "id": "-tFI4QjB5dGw" + } + }, + { + "cell_type": "code", + "source": [ + "get_scenes(traffic_stream , traffic_jam_index_id)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "14lqxoQO5mty", + "outputId": "92f49be8-d65f-4071-994a-662e99b58824" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "2025-05-29 13:08:29-2025-05-29 13:08:34: Based on the provided video frames, the classification is:\n", + "\n", + "**Traffic congestion detected.**\n", + "\n", + "**Reasoning:**\n", + "\n", + "In all three frames, the lanes of vehicles appear densely packed and moving very slowly. The consistent slow movement across multiple frames indicates a sustained period of reduced traffic flow, which is a key characteristic of traffic congestion.\n", + "\n", + "--------------------------------------------------------------------------------\n", + "2025-05-29 13:08:24-2025-05-29 13:08:28: Based on the provided images, here's the classification of traffic flow:\n", + "\n", + "**Traffic Congestion Detected**\n", + "\n", + "**Explanation:**\n", + "\n", + "The images show a high density of vehicles with very little visible movement. The blurred lights indicate that the camera is capturing the movement of the vehicles over time. The fact that the lights are blurred but the vehicles are still densely packed suggests that the vehicles are moving very slowly or are stopped. This is a clear indicator of traffic congestion.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-29 13:08:18-2025-05-29 13:08:23: Based on the provided images, here's the classification:\n", + "\n", + "**Traffic Congestion Detected.**\n", + "\n", + "**Explanation:**\n", + "\n", + "The images show a high density of vehicles in multiple lanes. The blurred lights suggest slow movement or even stationary vehicles over the time period captured in the images. This indicates that the vehicles are not moving freely and are experiencing congestion.\n", + "\n", + "--------------------------------------------------------------------------------\n", + "2025-05-29 13:08:13-2025-05-29 13:08:17: Based on the provided images, here's the classification of traffic flow:\n", + "\n", + "**Classification: Traffic Congestion Detected**\n", + "\n", + "**Reasoning:**\n", + "\n", + "* In all three images, the right lanes are densely packed with vehicles. The vehicles are either stopped or moving very slowly.\n", + "* The consistent presence of closely packed vehicles across the images indicates a sustained period of slow or stopped movement.\n", + "* The left lanes appear to have more movement, but the right lanes are clearly congested.\n", + "--------------------------------------------------------------------------------\n", + "2025-05-29 13:08:08-2025-05-29 13:08:12: Based on the provided video frames, the classification is:\n", + "\n", + "**Traffic congestion detected.**\n", + "\n", + "**Reasoning:**\n", + "\n", + "In the sequence of images, the vehicles in multiple lanes appear to be moving very slowly or are stopped. The blurred streaks of light from the vehicles indicate slow movement over the duration of the frame capture. The density of vehicles is also high, suggesting a congested traffic situation.\n", + "\n", + "--------------------------------------------------------------------------------\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "## 📦 Step 3: Create Event and Attach Alert for Traffic Jam Detection \n", + "\n", + "We’ll now define a single event for detecting traffic congestion and immediately attach a real-time alert to it.\n" + ], + "metadata": { + "id": "eUAhAFuV5rNK" + } + }, + { + "cell_type": "code", + "source": [ + "# Creating Event\n", + "traffic_event_id = conn.create_event(\n", + " event_prompt=\"Detect if traffic congestion or jam is forming.\",\n", + " label=\"traffic_congestion\"\n", + ")\n", + "print(\"Event ID:\", traffic_event_id)\n" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "Dp22xgVM5rff", + "outputId": "e4080dba-2028-44a4-d241-b5ba2befd8ef" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Event ID: c0f7b39718b545d9\n" + ] + } + ] + }, + { + "cell_type": "code", + "source": [ + "# Enter link to your webhook url where you want alerts to go. You can create one simply on pipedream.\n", + "traffic_webhook_url = \"\"\n", + "\n", + "if traffic_webhook_url:\n", + " traffic_alert_id = traffic_jam_scene_index.create_alert(\n", + " traffic_event_id,\n", + " callback_url=traffic_webhook_url\n", + " )\n", + " print(\"Alert ID:\", traffic_alert_id)\n", + "else:\n", + " print(\"Error: Please provide Webhook URL. Alert cannot be created without it.\")" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "Nq3RYvbd5uF_", + "outputId": "1039656c-04c8-4c0a-f536-6bf766d73f0b" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Alert ID: 140d39c7bb9596ca\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "---\n", + "\n", + "## 📡 Example Alert Payload \n", + "\n", + "When traffic congestion was detected, the system sent this alert:\n", + "\n", + "```json\n", + "{\n", + " \"event_id\": \"event-c0f7b39718b545d9\",\n", + " \"label\": \"traffic_congestion\",\n", + " \"confidence\": 0.95,\n", + " \"explanation\": \"High vehicle density and slow movement observed across multiple lanes, strongly indicating traffic congestion.\",\n", + " \"timestamp\": \"2025-05-29T07:41:20.520352+00:00\",\n", + " \"start_time\": \"2025-05-29T13:11:08.124127+05:30\",\n", + " \"end_time\": \"2025-05-29T13:11:12.622574+05:30\",\n", + " \"stream_url\": \"https://rt.stream.videodb.io/manifests/rts-01971a06-eb5c-7453-a257-e0135de60c29/1748504468000000-1748504473000000.m3u8\"\n", + "}\n", + "```" + ], + "metadata": { + "id": "heBIhjiu5xNp" + } + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "### Let us have a look at the stream_url received in the alert." + ], + "metadata": { + "id": "q6YG-iP1_UUj" + } + }, + { + "cell_type": "code", + "source": [ + "alert_stream_url = \"https://rt.stream.videodb.io/manifests/rts-01971a06-eb5c-7453-a257-e0135de60c29/1748504468000000-1748504473000000.m3u8\"\n", + "video_name = \"🚥 Central Highway · traffic_congestion\"\n", + "\n", + "display_stream(alert_stream_url, video_name)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 422 + }, + "id": "WXU-Db0z_T2S", + "outputId": "36a64ffd-824a-409f-ee5e-32b2e448a803" + }, + "execution_count": 12, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ], + "text/html": [ + "\n", + "
\n", + " \n", + " \n", + " \n", + "
\n", + " 🚥 Central Highway · traffic_congestion
2025-05-29 07:41:08 → 2025-05-29 07:41:13 UTC\n", + "
\n", + "
\n", + " " + ] + }, + "metadata": {}, + "execution_count": 12 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "- Let us disable the alert after use" + ], + "metadata": { + "id": "jfKNjz4qAi6H" + } + }, + { + "cell_type": "code", + "source": [ + "traffic_jam_scene_index.disable_alert(traffic_alert_id)" + ], + "metadata": { + "id": "f8TG8V9KAlUG" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "- To re-enable the alert" + ], + "metadata": { + "id": "s5Mp7YBmAqq9" + } + }, + { + "cell_type": "code", + "source": [ + "traffic_jam_scene_index.enable_alert(traffic_alert_id)" + ], + "metadata": { + "id": "bKQ2uq3jAsam" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "- We should stop the stream after use" + ], + "metadata": { + "id": "Bx832H6EAzH_" + } + }, + { + "cell_type": "code", + "source": [ + "traffic_stream.stop()" + ], + "metadata": { + "id": "8MXq3pCDA3nH" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "---\n", + "\n", + "\n", + "## 🛣️ Wrapping Up: Smarter Roads, Safer Lives\n", + "\n", + "In this notebook, we built a smart AI-powered road monitoring system addressing two major challenges:\n", + "\n", + "✅ **Accident Detection** at toll plazas — instantly spotting crashes and alerting emergency services without delay.\n", + "\n", + "✅ **Traffic Congestion Detection** on busy highways — catching early signs of jams so authorities can act before things spiral.\n", + "\n", + "Together, these tools show how AI video monitoring can make roads safer, traffic smoother, and emergency responses faster — all in real time.\n", + "\n", + "\n", + "\n", + "---\n", + "\n", + "## 🌍 What Else Could We Monitor?\n", + "\n", + "This kind of smart AI surveillance system isn’t limited to road safety:\n", + "\n", + "- 🛡️ **Public safety at concerts, rallies, and crowded events** — detecting mass gatherings, fights, or stampedes.\n", + "- 🌧️ **Weather-Related Visibility & Road Hazard Assessment** — automatically detecting dangerous road conditions caused by weather.\n", + "- 🚁 **Drone-based surveillance for rural highways, border roads, or disaster-affected zones.**\n", + "\n", + "Where would *you* deploy it next?" + ], + "metadata": { + "id": "4FkFVPpC50Sf" + } + } + ] +} \ No newline at end of file