0.15.0 Release Candidate 2

[blakeblackshear]

Beta Documentation: https://deploy-preview-13787--frigate-docs.netlify.app/

Images

• ghcr.io/blakeblackshear/frigate:0.15.0-rc2
• ghcr.io/blakeblackshear/frigate:0.15.0-rc2-standard-arm64
• ghcr.io/blakeblackshear/frigate:0.15.0-rc2-tensorrt
• ghcr.io/blakeblackshear/frigate:0.15.0-rc2-tensorrt-jp4
• ghcr.io/blakeblackshear/frigate:0.15.0-rc2-tensorrt-jp5
• ghcr.io/blakeblackshear/frigate:0.15.0-rc2-rk
• ghcr.io/blakeblackshear/frigate:0.15.0-rc2-rocm
• ghcr.io/blakeblackshear/frigate:0.15.0-rc2-h8l

Changes since RC1

• Don't show sub labels in object filter list
• Fix autotracking interpolation when using absolute zoom
• Fix some h265 encoding presets
• Fix selection of tracked objects in Explore on desktop Safari
• Fix CSRF check not cancelling out the request

Major Changes for 0.15.0

Breaking Changes

There are several breaking changes in this release, Frigate will attempt to update the configuration automatically. In some cases manual changes may be required. It is always recommended to back up your current config and database before upgrading:

1. Simply copy your current config file to a new location
2. Stop Frigate and make a copy of the frigate.db file

• Rewritten SHM system
  • The SHM system has been rewritten to be more stable. This means that a large % of shared memory (SHM) will be used by default but the total usage will not fluctuate, and Frigate will not fail from SHM overflowing. However, this also means that having the correct SHM size for your cameras, detector, and hardware is more important to having a smooth experience with Frigate. A new calculation is done on startup, so if your shm_size is too low then a warning will be printed in the log stating that it needs to be increased.
• Recording configuration changed
  • The record config has been refactored to allow for direct control of how long alerts and detections are retained. These values will be automatically populated from your current config, but you may want to adjust the values that are set after updating. See the updated docs here and ensure your updated config retains the footage you want it to.
• ffmpeg updates
  • ffmpeg has been updated to version 7. Some Intel users may need to change what hwaccel preset they are using (preset-vaapi may need to now be preset-intel-qsv-h264 or preset-intel-qsv-h265) if camera feeds are not functioning correctly after upgrading. This may need to be adjusted on a per-camera basis. If a qsv preset is not working properly, you may still need to use a preset-vaapi or revert to the previous ffmpeg version as described below.
  • The new version provides many improvements, but the old version can still be run if needed by specifying path: "5.0" in your ffmpeg: config entry. For example:

    ffmpeg:
      path: "5.0"
    

  • ffmpeg presets have been updated. It is always recommended to use ffmpeg presets rather than manual arguments.
  • ffmpeg is no longer part of $PATH. In most cases this is handled automatically.
    • go2rtc exec streams will need to add the full path for ffmpeg which in most cases will be /usr/lib/ffmpeg/7.0/bin/ffmpeg.
    • If the go2rtc config has bin: ffmpeg defined, it needs to be removed.
• The nested model config under a detector has been simplified to just model_path as a string value. This change will be handled automatically through config migration.
• Submitting to Frigate+
  • The Frigate+ view in the UI has been replaced by the new Explore view. See the notes below.

Explore

The new Explore pane in Frigate 0.15 makes it easy to explore every object tracked by Frigate. It offers a variety of filters and supports keyword and phrase-based text search, searching for similar images, and searching through descriptive text generated by AI models.

The default Explore pane shows a summary of your most recent tracked objects organized by label. Clicking the small arrow icon at the end of the list will bring you to an infinitely scrolling grid view. The grid view can also be set as the default by changing the view type from the Settings button in the top right corner of the pane.

The Explore pane also serves as the new way to submit images to Frigate+. Filters can be applied to only display tracked objects with snapshots that have not been submitted to Frigate+. The left/right arrow keys on the keyboard allow quick navigation between tracked object snapshots when looking at the Tracked Object Details pane from the grid view.

AI/ML Search

Frigate 0.15 introduces two powerful search features: Semantic Search and GenAI Search. Semantic Search can be enabled on its own, while GenAI Search works in addition to Semantic Search.

Semantic Search

Semantic Search uses a CLIP model to generate embeddings (numerical representations of images) for the thumbnails of your tracked objects, enabling searches based on text descriptions or visual similarity. This is all done locally.

For instance, if Frigate detects and tracks a car, you can use similarity search to see other instances where Frigate detected and tracked that same car. You can also quickly search your tracked objects using an "image caption" approach. Searching for "red car driving on a residential street" or "person in a blue shirt walking on the sidewalk at dawn" or even "a person wearing a black t-shirt with the word 'SPORT' on it" will produce some stunning results.

Semantic Search works by running an AI model locally on your system. Small or underpowered systems like a Raspberry Pi will not run Semantic Search reliably or at all. A dedicated GPU and 16GB of RAM is recommended for best performance.

See the Semantic Search docs for system requirements, setup instructions, and usage tips.

Generative AI

GenAI Search employs generative AI models to create descriptive text for the thumbnails of your tracked objects, which are stored in the Frigate database to enhance future searches. Supported providers include Google Gemini, Ollama, and OpenAI, so you can choose whether you want to send data to the cloud or use a locally hosted provider.

See the GenAI docs for setup instructions and use case suggestions.

Improved Tools for Debugging

Review Item Details Pane

A new Review Item Details pane can be viewed by clicking / tapping on the gray chip on a review item in the Review pane. This shows more information about the review item as well as thumbnails or snapshots for individual objects (if enabled). The pane also provides links to share the review item, download it, submit images to Frigate+, view object lifecycles, and more.

Object Lifecycle Pane

The Recordings Timeline from Frigate 0.13 has been improved upon and returns to 0.15 as the Object Lifecycle, viewable in the Review Details pane as well as the new Explore page. The new pane shows the significant moments during the object's lifecycle: when it was first seen, when it entered a zone, became stationary, etc. It also provides information about the object's area and size ratio to assist in configuring Frigate to tune out false positives.

Native Notifications

Frigate now supports notifications using the WebPush protocol. This allows Frigate to deliver notifications to devices that have registered to receive notifications in the Frigate settings, delivering them in a timely and secure manner. Currently, notifications will be delivered for all review items marked as alerts. More options for native notifications will be supported in the future.

See the notifications docs.

New Object Detectors

ONNX

ONNX is an open model standard which allows for a single model format that can run on different types of GPUs. The default, tensorrt, and rocm Frigate build variants include GPU support for efficient object detection via ONNX models, simplifying configuration and support more models. There are no default included ONNX models for object detection.

AMD MiGraphX

Support has been added for AMD GPUs via ROCm and MiGraphX. Currently there is no default included model for this detector.

Hailo-8

Support has been added for the Hailo8 and Hailo-8L hardware for object detection on both arm64 and amd64 platforms.

Other UI Changes

• Rework the Storage metrics pane to be more clear how much storage each camera is using
• Camera stream info button has been added to the camera metrics page allowing easy access to see camera codec information
• Add button for downloading the full set of logs
• The timeline handlebar will now automatically scroll while viewing recordings
• Review items and exports can now be shared via your browser's sharing pane or via direct link url. A secure context is required for the browser/OS sharing pane to function
• Add the ability to rename camera groups
• Add middle mouse button support in the Review pane to open full recordings view in a new tab
• Add Nvidia GPU encode and decode stats to system metrics
• New desktop keyboard shortcuts. In addition to the numerous keyboard shortcuts available with 0.14, 0.15 adds several more. In Live view, the numeric keys (1-9) will move PTZ cameras to presets. In Review, Page Up and Page Down will scroll through review items
• Accessibility updates for users with screen readers

Other Backend Changes

• FFMPEG updated to 7.0
• OpenVINO updated to 2024.3
• Use FastAPI instead of Flask for the Frigate API
• Add service manager infrastructure
• Install onevpl tools and update drivers for Intel
• If record is disabled for a camera then that camera will not save frames to the preview during that time
• Fix stalled connections to certain onvif cameras
• Fix autotracking absolute zoom interpolation logic
• TensorRT models are no longer generated by default, the ENV var must be set

---

This discussion was created from the release 0.15.0 Release Candidate 2.

[ElectronicBattle]

I have tried a few times over the last ten or so minutes but dockge tells me

[+] Running 1/1
✘ frigate Error 0.7s
Error response from daemon: manifest unknown

my config looks like it is pointing correctly: it says

services:
  frigate:
    container_name: frigate015rc2
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:0.15.0-rc2
...

[RiderCrazy]

Image creation failed
https://github.com/blakeblackshear/frigate/actions/runs/13089597026

[ElectronicBattle]

working now - thanks!

[blakeblackshear]

They should all be available now.

[bagobones]

I haven't encountered any new issues for the past 2 releases, so far so good.

[Yanom1212]

Hi, I don't know if this is the place to ask, but for the next RC is it possible to add support for DeepSeek gen AI?

[NickM-27]

Deepseek is openai compatible so this is already possible.

[bagobones]

Deepseek is openai compatible so this is already possible.

Didn't the patch to allow changing the OPEN AI API base url get deferred to 0.16 ?

[NickM-27]

It is possible via environment variable, no patch is needed

[TiddlyWiddly]

Getting the following after copy pasting documentation ollama config on one of my cameras, not sure if this is new or not as this is my first v15 experience.

frigate    | 2025-02-01 21:40:17.914784696  Process embeddings_manager:
frigate    | 2025-02-01 21:40:17.914788967  Traceback (most recent call last):
frigate    | 2025-02-01 21:40:17.914790295    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
frigate    | 2025-02-01 21:40:17.914794277      self.run()
frigate    | 2025-02-01 21:40:17.914796011    File "/opt/frigate/frigate/util/process.py", line 41, in run_wrapper
frigate    | 2025-02-01 21:40:17.914796910      return run(*args, **kwargs)
frigate    | 2025-02-01 21:40:17.914822030    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
frigate    | 2025-02-01 21:40:17.914823169      self._target(*self._args, **self._kwargs)
frigate    | 2025-02-01 21:40:17.914824641    File "/opt/frigate/frigate/embeddings/__init__.py", line 59, in manage_embeddings
frigate    | 2025-02-01 21:40:17.914825614      maintainer = EmbeddingMaintainer(
frigate    | 2025-02-01 21:40:17.914826576    File "/opt/frigate/frigate/embeddings/maintainer.py", line 67, in __init__
frigate    | 2025-02-01 21:40:17.914845187      self.genai_client = get_genai_client(config)
frigate    | 2025-02-01 21:40:17.914846342    File "/opt/frigate/frigate/genai/__init__.py", line 70, in get_genai_client
frigate    | 2025-02-01 21:40:17.914847240      return provider(genai_config)
frigate    | 2025-02-01 21:40:17.914848210    File "/opt/frigate/frigate/genai/__init__.py", line 34, in __init__
frigate    | 2025-02-01 21:40:17.914861064      self.provider = self._init_provider()
frigate    | 2025-02-01 21:40:17.914862200    File "/opt/frigate/frigate/genai/openai.py", line 24, in _init_provider
frigate    | 2025-02-01 21:40:17.914863190      return OpenAI(api_key=self.genai_config.api_key)
frigate    | 2025-02-01 21:40:17.914864163    File "/usr/local/lib/python3.9/dist-packages/openai/_client.py", line 105, in __init__
frigate    | 2025-02-01 21:40:17.914864958      raise OpenAIError(
frigate    | 2025-02-01 21:40:17.914866632  openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable

cameras:
  front_door:
    genai:
      enabled: True
      provider: ollama
      base_url: http://skynet.local:11434
      model: llava:34b
      prompt: "Analyze the {label} in these images from the {camera} security camera. Focus on the actions, behavior, and potential intent of the {label}, rather than just describing its appearance."
      object_prompts:
        person: "Examine the main person in these images. What are they doing and what might their actions suggest about their intent (e.g., approaching a door, leaving an area, standing still)? Do not describe the surroundings or static details."
        car: "Observe the primary vehicle in these images. Focus on its movement, direction, or purpose (e.g., parking, approaching, circling). If it's a delivery vehicle, mention the company."

[NickM-27]

You have to configure genai globally not at the camera level

[TiddlyWiddly]

Money. Thanks Nick.

[psilantropy]

Hi. Thanks for the release. Went from b1 > rc1 > rc2. Good so far (coral, nvidia, proxmox lxc, dahua 5 series) thanks.

Can I run corals as detectors, but also use the tensorrt image? The goal being to push semantic search to the GPU.

I'm guessing frigate.embeddings_manager is the process running on the gpu, which is continuously running. But I'm not sure if it is doing anything.
But I also see embeddings on the CPU metrics page spike, and the embeddings process hit the cpu while running 'top'. It's very quick though.

Initial startup Embedded 334 thumbnails and 0 descriptions in 10.1 seconds spiked the CPU also.

semantic_search:
  enabled: true
  model_size: large
  reindex: false

detectors:
  coral1:
    type: edgetpu
    device: usb:0
  coral2:
    type: edgetpu
    device: usb:1

ffmpeg:
  hwaccel_args: preset-nvidia-h264

[psilantropy]

Hi hawkeye217.

Yeah - It doesn't seem to make much difference performance wise to push this one to the gpu.
But I'm pretty sure the GPU isn't doing this processing as expected however.

Is NVIDIA_DRIVER_CAPABILITIES=gpu the same as all these days? (mine is set to gpu)

Tracked object search hitting CPU > embeddings metrics page.

CPU process was searching "red bike on grass"

GPU process (doesn't change, I presume this one stays loaded in memory)

Searching definitely not using it;

objects:
  track:
    - person
    - cat
    - dog
    - bus
    - police
    - car
    - bike
    - rat

review:
  alerts:
    labels:
      - dog
      - rat
      - person
      - speech
  detections:
    labels:
      - car
      - cat
      - bus
      - bike

Happy to troubleshoot further if you would like, but I don't particularly need this to work.

[NickM-27]

the model is loaded in GPU memory so it seems clear that the GPU is being used. It is worth mentioning that only the image embeddings run on GPU, the text embedding which are triggered when searching always run on CPU.

[psilantropy]

Thanks for the details :).

[psilantropy]

Reindex screenshots if anyone else is curious. Picked up with nvtop.

Frigate metrics page: GPU usage peaks at 100%, CPU embeddings hits ~50% during the process. Which in my fresh install takes a few seconds.

[hawkeye217]

Reindex screenshots if anyone else is curious. Picked up with nvtop.

Frigate metrics page: GPU usage peaks at 100%, CPU embeddings hits ~50% during the process. Which in my fresh install takes a few seconds.

Looks good!

[thelovemyth]

amazing app <3
we will have final version soon
and have 0.16 beta with more new function <3
Thank you for your contributions

[hawkeye217]

There is no official roadmap. Pet identification is not something we've considered, either.

[hawkeye217]

And to be clear, Frigate already detects cats and dogs. Identification/recognition and detection are quite different.

[NickM-27]

What about common pet detection like cats and dogs? Any chance that's on the roadmap somehow?

it's possible embeddings similarity can be used for this, but there is no obvious way to implement this that will definitely work well, will require research / testing first

[tomerh2001]

Extremely understandable! Thank you for the amazing features you guys put into Frigate.

Now... About that kiss? 😛

[psilantropy]

🏃

[johnlento]

Anyone having issues with generating descriptions with ollama? Posts appear to work to my ollama endpoint but I never actually get a result. This appeared to break somewhere before RC1 but was working in the early betas/

[psilantropy]

Mine is working ok with moondream, but initially I had all my cameras sending and caused issues.
You must have semantic enabled if you don't.

semantic_search:
  enabled: true
  model_size: large
  reindex: false

genai:
  enabled: True
  provider: ollama
  base_url: http://wat:11434
  model: moondream

Then on all except 1 camera;

    genai:
      enabled: False

[johnlento]

Thats all enabled and was working fine. I did install open-webui recently and its working with the ollama API. Its just that frigate never seems to get a response or update. Really weird.

[johnlento]

Turns out it was the model I was trying to use, wasn't working right. Its working fine with llama3.2-vision. What is the best on frigate inference anymore?

[psilantropy]

Not sure. I probably won't run this for long. No use case for me.
Semantic search works well enough for now, and i'll use the face detect and LPR when available.

Speed references via Frigate gui - 7 seconds to regen snapshot on p2000. 3 seconds using thumbnail.

[charliejgallo]

Tested 0.15.0 RC2 Hailo-8l beta with a new x86-64 server with Homeassistant OS....

as of Homeassistant OS 14.1 Hailo-8l drivers and firmware are included (for raspberrypi and x86-64) in the base OS image... i just "installed" the hailoRT-beta Addon (added the minimal configuration with the Hailo-8l configuration part) and the module was recognized!