[Support]: Intermittent /tmp/cache full causing recording failure

[zeroward]

Describe the problem you are having

My /tmp/cache is intermittently filling up regardless of the size given to it, unable to determine cause.

Seems to be related to #9499

Steps to reproduce

• Run my config
• Wait a few days
• ...
• Failure

Version

0.14.0-da913d8

Frigate config file

mqtt:
  enabled: false
environment_vars:
  LIBVA_DRIVER_NAME: iHD
detectors: # <---- add detectors
  coral:
    type: edgetpu
    device: usb
database:
  path: /config/frigate.db
go2rtc:
  log:
    streams: trace
  rtsp:
    username: <REMOVED>
    password: <REMOVED>
  streams:
    front-yard-main:
      - rtsp://{FRIGATE_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.5.150:554//h264Preview_01_main
      - ffmpeg:rtsp_cam#audio=opus
    front-yard-sub:
      - rtsp://{FRIGATE_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.5.150:554//h264Preview_01_sub
      - ffmpeg:rtsp_cam
    front-door-main:
      - rtsp://{FRIGATE_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.20.90:554//h264Preview_01_main
      - ffmpeg:rtsp_cam#audio=opus
    front-door-sub:
      - rtsp://{FRIGATE_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.20.90:554//h264Preview_01_sub
      - ffmpeg:rtsp_cam#audio=opus
    backyard-right-main:
      - rtsp://{FRIGATE_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.5.195:554//h264Preview_01_main
      - ffmpeg:rtsp_cam#audio=opus
    backyard-right-sub:
      - rtsp://{FRIGATE_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.5.195:554//h264Preview_01_sub
      - ffmpeg:rtsp_cam
    backyard-patio-main:
      - rtsp://{FRIGATE_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.5.149:554//h264Preview_01_main
      - ffmpeg:rtsp_cam#audio=opus
    backyard-patio-sub:
      - rtsp://{FRIGATE_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.5.149:554//h264Preview_01_sub
      - ffmpeg:rtsp_cam
    #backyard-overhead-main:
    #  - rtsp://{FRIGATE_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.5.57/cam/realmonitor?channel=1&subtype=0&authbasic=64
    #  - ffmpeg:rtsp_cam#audio=opus
    #backyard-overhead-sub:
    #  - rtsp://{FRIGATE_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.5.57/cam/realmonitor?channel=1&subtype=1&authbasic=64
    #  - ffmpeg:rtsp_cam
    backyard-left-main:
      - rtsp://{FRIGATE_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.5.188:554//h264Preview_01_main
      - ffmpeg:rtsp_cam#audio=opus
    backyard-left-sub:
      - rtsp://{FRIGATE_USER}:{FRIGATE_RTSP_PASSWORD}@192.168.5.188:554//h264Preview_01_sub
      - ffmpeg:rtsp_cam

ffmpeg:
  hwaccel_args: preset-vaapi
  output_args:
    record: preset-record-generic-audio-copy

detect:
  width: 1280
  height: 720
  fps: 10
  enabled: true
  stationary:
    interval: 50
    threshold: 50

motion:
  improve_contrast: true

################################################
cameras:
  front-yard:
    onvif:
      host: 192.168.5.150
      port: 8000
      user: <REMOVED>
      password: <REMOVED>
    ffmpeg:
      inputs:
        - path: 
            rtsp://admin:{FRIGATE_RTSP_PASSWORD}@127.0.0.1:8554/front-yard-main?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@127.0.0.1:8554/front-yard-sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      enabled: true
      width: 1280
      height: 720
    objects:
      track:
        - person
        - car
        - bicycle
        - dog
        - cat
      filters:
        car:
          mask:
            - 658,213,966,209
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        pre_capture: 5
        post_capture: 30
        retain:
          default: 14
          mode: active_objects
    snapshots:
      required_zones:
        - our_yard
        - truck_parking
        - left_yard
    zones:
      neighbors_yard:
        coordinates: 1280,720,1280,0,820,0
      neighborhood:
        coordinates: 0,188,1280,89,1280,0,0,0
      our_yard:
        coordinates: 856,186,647,196,619,720,0,720,52,427,99,177,421,168,842,120
        objects:
          - person
          - car
          - bicycle
          - dog
          - cat
      truck_parking:
        coordinates: 1074,720,890,181,661,199,713,720
        objects:
          - person
          - bicycle
          - dog
          - cat
      left_yard:
        coordinates: 1057,720,1223,720,967,152,862,156
        objects:
          - person
          - bicycle
          - dog
          - cat
    motion:
      mask:
        - 0,0,1280,0,1280,116,0,142
        - 1280,720,1280,291,1280,0,848,0
    review:
      alerts:
        required_zones:
          - our_yard
          - truck_parking
          - left_yard
  front-door:
    onvif:
      host: 192.168.20.90
      port: 8000
      user: <REMOVED>
      password: <REMOVED>
    ffmpeg:
      inputs:
        - path: 
            rtsp://admin:{FRIGATE_RTSP_PASSWORD}@127.0.0.1:8554/front-door-main?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@127.0.0.1:8554/front-door-sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      enabled: true
      width: 1280
      height: 720
    objects:
      track:
        - person
        - bicycle
        - dog
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        pre_capture: 5
        post_capture: 30
        retain:
          default: 14
          mode: active_objects
    snapshots:
      required_zones:
        - our_yard
    zones:
      neighbors_yard:
        coordinates: 0,0,514,0,403,660,0,682
        objects:
          - dog
      neighborhood:
        coordinates: 1001,0,1066,110,1103,259,1124,313,926,445,498,442,509,0
        objects:
          - dog
      our_yard:
        coordinates: 1280,720,1280,373,1033,373,930,447,704,458,522,460,0,455,0,720
        objects:
          - person
          - bicycle
          - dog
    motion:
      mask: 753,455,926,440,1016,367,1126,324,1113,237,1077,116,1005,0,473,0,0,0,0,103,0,434
    review:
      alerts:
        required_zones:
          - our_yard
  backyard-right:
    ffmpeg:
      inputs:
        - path: 
            rtsp://admin:{FRIGATE_RTSP_PASSWORD}@127.0.0.1:8554/backyard-right-main?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: 
            rtsp://admin:{FRIGATE_RTSP_PASSWORD}@127.0.0.1:8554/backyard-right-sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect
    objects:
      track:
        - person
    detect:
      enabled: true
      width: 1280
      height: 720
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        pre_capture: 5
        post_capture: 30
        retain:
          default: 14
          mode: active_objects
  backyard-patio:
    ffmpeg:
      inputs:
        - path: 
            rtsp://admin:{FRIGATE_RTSP_PASSWORD}@127.0.0.1:8554/backyard-patio-main?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: 
            rtsp://admin:{FRIGATE_RTSP_PASSWORD}@127.0.0.1:8554/backyard-patio-sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect
    objects:
      track:
        - person
        - dog
        - cat
    detect:
      enabled: true
      width: 1280
      height: 720
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        pre_capture: 5
        post_capture: 30
        retain:
          default: 14
          mode: active_objects
  #backyard-overhead:
  #  ffmpeg:
  #    inputs:
  #      - path: 
  #          rtsp://admin:{FRIGATE_RTSP_PASSWORD}@127.0.0.1:8554/backyard-overhead-main?video=copy&audio=aac
  #        input_args: preset-rtsp-restream
  #        roles:
  #          - record
  #      - path: 
  #          rtsp://admin:{FRIGATE_RTSP_PASSWORD}@127.0.0.1:8554/backyard-overhead-sub?video=copy
  #        input_args: preset-rtsp-restream
  #        roles:
  #          - detect
  #  objects:
  #    track:
  #      - person
  #      - dog
  #      - bicycle
  #      - cat
  #  detect:
  #    enabled: true
  #    width: 1920
  #    height: 1080
  #  record:
  #    enabled: true
  #    retain:
  #      days: 7
  #      mode: all
  #    events:
  #      pre_capture: 5
  #      post_capture: 30
  #      retain:
  #        default: 14
  #        mode: active_objects
  #  snapshots:
  #    required_zones:
  #      - our_yard
  #      - roof
  #  zones:
  #    neighbors_yard:
  #      coordinates: 1165,126,1804,342,1920,0,1141,0,0,0,0,425
  #      objects:
  #        - bicycle
  #    our_yard:
  #      coordinates: 1201,442,1752,539,1793,329,1169,137,623,228,81,398,0,465,0,851
  #      objects:
  #        - person
  #        - dog
  #        - cat
  #    roof:
  #      coordinates: 0,1080,1920,1080,1920,567,1201,440,77,820
  #      objects:
  #        - person
  #  motion:
  #    mask:
  #      - 81,398,1803,328,1920,0,1177,0,0,0,0,429
  #  onvif:
  #    host: 192.168.5.57
  #    port: 80
  #    user: <REMOVED>
  #    password: <REMOVED>
  #  review:
  #    alerts:
  #      required_zones:
  #        - our_yard
  #        - roof
  backyard-left:
    ffmpeg:
      inputs:
        - path: 
            rtsp://admin:{FRIGATE_RTSP_PASSWORD}@127.0.0.1:8554/backyard-left-main?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@127.0.0.1:8554/backyard-left-sub?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect
    objects:
      track:
        - person
        - dog
        - car
        - cat
    detect:
      enabled: true
      width: 1280
      height: 720
    record:
      enabled: true
      retain:
        days: 7
        mode: all
      events:
        pre_capture: 5
        post_capture: 30
        retain:
          default: 7
          mode: active_objects
    snapshots:
      required_zones:
        - our_yard
    zones:
      neighbors_yard:
        coordinates: 1280,0,1280,127,367,52,220,78,209,0
        objects:
          - car
      our_yard:
        coordinates: 1280,720,1280,138,712,50,492,52,127,60,41,237,0,720
        objects:
          - person
          - dog
          - car
          - cat
    motion:
      mask:
        - 1280,0,1280,123,356,54,227,73,211,0

    review:
      alerts:
        required_zones:
          - our_yard
birdseye:
  enabled: true
  mode: continuous
  restream: false
  width: 3840
  height: 2160
  quality: 4

snapshots:
  enabled: true
  clean_copy: true
  timestamp: true
  bounding_box: true
  crop: false
  retain:
    default: 10
    objects:
      person: 15
version: 0.14

Relevant log output

2024-08-22 00:48:47.963230875  [2024-08-22 00:48:47] ffmpeg.front-yard.record       ERROR   : Could not write header for output file #0 (incorrect codec parameters ?): No space left on device
2024-08-22 00:48:47.963359676  [2024-08-22 00:48:47] ffmpeg.front-yard.record       ERROR   : Error initializing output stream 0:1 --
2024-08-22 00:48:47.963361536  [2024-08-22 00:48:47] ffmpeg.front-yard.record       ERROR   :
2024-08-22 00:48:47.963362836  [2024-08-22 00:48:47] watchdog.front-yard            INFO    : Terminating the existing ffmpeg process...
2024-08-22 00:48:47.963414317  [2024-08-22 00:48:47] watchdog.front-yard            INFO    : Waiting for ffmpeg to exit gracefully...

Operating system

Debian

Install method

Docker Compose

Network connection

Wired

Camera make and model

Reolink RLC-822As and Reolink Wifi Doorbell Camera

Any other information that may be helpful

Logs prior to failure:

024-08-21 09:36:16.780996291  Fatal Python error: Segmentation fault
2024-08-21 09:36:16.780999193
2024-08-21 09:36:16.781298594  Thread 0x00007fc70d5fd6c0 (most recent call first):
2024-08-21 09:36:16.781303383    File "/usr/lib/python3.9/threading.py", line 312 in wait
2024-08-21 09:36:16.781314579    File "/usr/lib/python3.9/queue.py", line 171 in get
2024-08-21 09:36:16.781337628    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 162 in loop
2024-08-21 09:36:16.781361030    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 137 in run
2024-08-21 09:36:16.781389771    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 272 in run
2024-08-21 09:36:16.781404144    File "/usr/lib/python3.9/threading.py", line 892 in run
2024-08-21 09:36:16.781422174    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-08-21 09:36:16.781436218    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-08-21 09:36:16.781437540
2024-08-21 09:36:16.781442780  Thread 0x00007fc71418e740 (most recent call first):
2024-08-21 09:36:16.781451933    File "/usr/lib/python3.9/threading.py", line 1428 in _shutdown
2024-08-21 09:36:16.781472683    File "/usr/lib/python3.9/multiprocessing/process.py", line 333 in _bootstrap
2024-08-21 09:36:16.781492711    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 71 in _launch
2024-08-21 09:36:16.781510854    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 19 in __init__
2024-08-21 09:36:16.781531811    File "/usr/lib/python3.9/multiprocessing/context.py", line 277 in _Popen
2024-08-21 09:36:16.781548290    File "/usr/lib/python3.9/multiprocessing/context.py", line 224 in _Popen
2024-08-21 09:36:16.781566617    File "/usr/lib/python3.9/multiprocessing/process.py", line 121 in start
2024-08-21 09:36:16.781584923    File "/opt/frigate/frigate/app.py", line 305 in init_recording_manager
2024-08-21 09:36:16.781604241    File "/opt/frigate/frigate/app.py", line 679 in start
2024-08-21 09:36:16.781622266    File "/opt/frigate/frigate/__main__.py", line 17 in <module>
2024-08-21 09:36:16.781636513    File "/usr/lib/python3.9/runpy.py", line 87 in _run_code
2024-08-21 09:36:16.781652833    File "/usr/lib/python3.9/runpy.py", line 197 in _run_module_as_main
2024-08-21 09:36:24.894367116  127.0.0.1 - - [21/Aug/2024:09:36:24 +0000] "" 400 0 "-" "-" "-"
...
2024-08-21 11:18:26.620721529  127.0.0.1 - - [21/Aug/2024:11:18:26 +0000] "" 400 0 "-" "-" "-"
2024-08-21 11:18:40.291114397  11:18:40.290 DBG [streams] stop producer url=rtsp://frigate_user:<REMOVED>@192.168.5.195:554//h264Preview_01_main
2024-08-21 11:18:40.291296648  11:18:40.291 TRC [streams] skip stop none producer
2024-08-21 11:18:40.291299456  11:18:40.291 TRC [streams] stop reconnect url=rtsp://frigate_user:<REMOVED>@192.168.5.195:554//h264Preview_01_main
2024-08-21 11:18:40.865197974  11:18:40.865 DBG [streams] stop producer url=rtsp://frigate_user:<REMOVED>@192.168.5.149:554//h264Preview_01_main
2024-08-21 11:18:40.865286003  11:18:40.865 TRC [streams] skip stop none producer
2024-08-21 11:18:40.865386592  11:18:40.865 TRC [streams] stop reconnect url=rtsp://frigate_user:<REMOVED>@192.168.5.149:554//h264Preview_01_main
2024-08-21 11:18:41.241302371  11:18:41.241 DBG [streams] stop producer url=rtsp://frigate_user:<REMOVED>@192.168.5.150:554//h264Preview_01_main
2024-08-21 11:18:41.241406160  11:18:41.241 TRC [streams] skip stop none producer
2024-08-21 11:18:41.241451431  11:18:41.241 TRC [streams] stop reconnect url=rtsp://frigate_user:<REMOVED>@192.168.5.150:554//h264Preview_01_main
2024-08-21 11:18:41.678040548  11:18:41.677 DBG [streams] stop producer url=rtsp://frigate_user:<REMOVED>@192.168.5.188:554//h264Preview_01_main
2024-08-21 11:18:41.678278661  11:18:41.678 TRC [streams] skip stop none producer
2024-08-21 11:18:41.678281490  11:18:41.678 TRC [streams] stop reconnect url=rtsp://frigate_user:<REMOVED>@192.168.5.188:554//h264Preview_01_main
2024-08-21 11:18:41.732194756  11:18:41.731 DBG [streams] stop producer url=rtsp://frigate_user:<REMOVED>@192.168.20.90:554//h264Preview_01_main
2024-08-21 11:18:41.732198888  11:18:41.732 TRC [streams] skip stop none producer
2024-08-21 11:18:41.732200733  11:18:41.732 TRC [streams] stop reconnect url=rtsp://frigate_user:<REMOVED>@192.168.20.90:554//h264Preview_01_main
2024-08-21 11:18:49.712207652  [2024-08-21 11:18:49] ffmpeg.front-yard.record       ERROR   : [segment @ 0x5581ef4ea3c0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
2024-08-21 11:18:49.712408508  [2024-08-21 11:18:49] ffmpeg.front-yard.record       ERROR   : av_interleaved_write_frame(): No space left on device
2024-08-21 11:18:49.712531704  [2024-08-21 11:18:49] ffmpeg.front-yard.record       ERROR   : [segment @ 0x5581ef4ea3c0] Failure occurred when ending segment '/tmp/cache/front-yard@20240821111836+0000.mp4'
2024-08-21 11:18:49.712597597  [2024-08-21 11:18:49] ffmpeg.front-yard.record       ERROR   : Error writing trailer of /tmp/cache/front-yard@%Y%m%d%H%M%S%z.mp4: No space left on device
2024-08-21 11:18:49.712599740  [2024-08-21 11:18:49] watchdog.front-yard            INFO    : Terminating the existing ffmpeg process...
2024-08-21 11:18:49.712688384  [2024-08-21 11:18:49] watchdog.front-yard            INFO    : Waiting for ffmpeg to exit gracefully...

[NickM-27]

We will need to see logs starting from before the error. Also, would be good to see screenshots of the system and camera metrics pages

[zeroward]

So, there is some stuff in dmesg but nothing too concerning..

[  433.780985] systemd-fstab-generator[6069]: x-systemd.device-timeout ignored for //10.51.0.10/Frigate
[  445.102716] systemd-fstab-generator[6851]: x-systemd.device-timeout ignored for //10.51.0.10/Frigate
[  445.968349] systemd-fstab-generator[6971]: x-systemd.device-timeout ignored for //10.51.0.10/Frigate
[  446.824625] systemd-fstab-generator[7068]: x-systemd.device-timeout ignored for //10.51.0.10/Frigate
[  447.339879] systemd-fstab-generator[7111]: x-systemd.device-timeout ignored for //10.51.0.10/Frigate
[  457.965284] br-b3d6fba1a983: port 1(veth270da4f) entered disabled state
[  457.965357] vethb04871e: renamed from eth0
[  458.003741] br-b3d6fba1a983: port 1(veth270da4f) entered disabled state
[  458.004180] device veth270da4f left promiscuous mode
[  458.004183] br-b3d6fba1a983: port 1(veth270da4f) entered disabled state
[  458.847927] br-b3d6fba1a983: port 1(veth9770e1a) entered blocking state
[  458.847933] br-b3d6fba1a983: port 1(veth9770e1a) entered disabled state
[  458.849891] device veth9770e1a entered promiscuous mode
[  458.850609] br-b3d6fba1a983: port 1(veth9770e1a) entered blocking state
[  458.850612] br-b3d6fba1a983: port 1(veth9770e1a) entered forwarding state
[  458.993856] br-b3d6fba1a983: port 1(veth9770e1a) entered disabled state
[  459.054613] eth0: renamed from vethff30f5a
[  459.078291] IPv6: ADDRCONF(NETDEV_CHANGE): veth9770e1a: link becomes ready
[  459.078335] br-b3d6fba1a983: port 1(veth9770e1a) entered blocking state
[  459.078338] br-b3d6fba1a983: port 1(veth9770e1a) entered forwarding state
[  460.827242] kauditd_printk_skb: 26 callbacks suppressed
[  460.827265] audit: type=1400 audit(1726010069.832:62): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="ubuntu_pro_apt_news" pid=8188 comm="apparmor_parser"
[  461.262855] audit: type=1400 audit(1726010070.268:63): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="ubuntu_pro_esm_cache" pid=8191 comm="apparmor_parser"
[  461.283769] audit: type=1400 audit(1726010070.288:64): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="ubuntu_pro_esm_cache//apt_methods" pid=8191 comm="apparmor_parser"
[  461.285645] audit: type=1400 audit(1726010070.288:65): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="ubuntu_pro_esm_cache//apt_methods_gpgv" pid=8191 comm="apparmor_parser"
[  461.287875] audit: type=1400 audit(1726010070.292:66): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="ubuntu_pro_esm_cache//cloud_id" pid=8191 comm="apparmor_parser"
[  461.288735] audit: type=1400 audit(1726010070.292:67): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="ubuntu_pro_esm_cache//dpkg" pid=8191 comm="apparmor_parser"
[  461.290404] audit: type=1400 audit(1726010070.296:68): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="ubuntu_pro_esm_cache//ps" pid=8191 comm="apparmor_parser"
[  461.291366] audit: type=1400 audit(1726010070.296:69): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="ubuntu_pro_esm_cache//ubuntu_distro_info" pid=8191 comm="apparmor_parser"
[  461.292336] audit: type=1400 audit(1726010070.296:70): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="ubuntu_pro_esm_cache_systemctl" pid=8191 comm="apparmor_parser"
[  461.293322] audit: type=1400 audit(1726010070.296:71): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="ubuntu_pro_esm_cache_systemd_detect_virt" pid=8191 comm="apparmor_parser"
[  461.825568] systemd-fstab-generator[8224]: x-systemd.device-timeout ignored for //10.51.0.10/Frigate
[  462.089192] systemd-fstab-generator[8262]: x-systemd.device-timeout ignored for //10.51.0.10/Frigate
[  466.602021] usb 3-2: reset SuperSpeed USB device number 2 using xhci_hcd
[  466.622706] usb 3-2: LPM exit latency is zeroed, disabling LPM.

[zeroward]

Weirdly, it seems like my eth is hanging every so often...

Could that that hung eth connection be preventing frigate from writing to my NFS share.. and that could be causing the cache fill up?

[zeroward]

No joy on the hung eth fix...

Happened again right after I fixed the eth driver.

2024-09-16 21:19:01.986893697  corrupted double-linked list
2024-09-16 21:19:01.986912519  Fatal Python error: Aborted
2024-09-16 21:19:01.986913633
2024-09-16 21:19:01.986914853  Current thread 0x00007f2e38c656c0 (most recent call first):
2024-09-16 21:19:01.986936489    File "/opt/frigate/frigate/util/services.py", line 488 in get_video_properties
2024-09-16 21:19:01.986966396    File "/opt/frigate/frigate/record/maintainer.py", line 209 in validate_and_move_segment
2024-09-16 21:19:01.986981714    File "/usr/lib/python3.9/asyncio/events.py", line 80 in _run
2024-09-16 21:19:01.987024040    File "/usr/lib/python3.9/asyncio/base_events.py", line 1890 in _run_once
2024-09-16 21:19:01.987042566    File "/usr/lib/python3.9/asyncio/base_events.py", line 596 in run_forever
2024-09-16 21:19:01.987064994    File "/usr/lib/python3.9/asyncio/base_events.py", line 629 in run_until_complete
2024-09-16 21:19:01.987083470    File "/usr/lib/python3.9/asyncio/runners.py", line 44 in run
2024-09-16 21:19:01.987101377    File "/opt/frigate/frigate/record/maintainer.py", line 524 in run
2024-09-16 21:19:01.987125703    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-09-16 21:19:01.987140086    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-09-16 21:19:01.987145250
2024-09-16 21:19:01.987146852  Thread 0x00007f2e5cdfd6c0 (most recent call first):
2024-09-16 21:19:01.987171379    File "/usr/lib/python3.9/threading.py", line 312 in wait
2024-09-16 21:19:01.987179948    File "/usr/lib/python3.9/queue.py", line 171 in get
2024-09-16 21:19:01.987209661    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 162 in loop
2024-09-16 21:19:01.987230786    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 137 in run
2024-09-16 21:19:01.987256490    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 272 in run
2024-09-16 21:19:01.987269967    File "/usr/lib/python3.9/threading.py", line 892 in run
2024-09-16 21:19:01.987286993    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-09-16 21:19:01.987303450    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-09-16 21:19:01.987309774
2024-09-16 21:19:01.987311554  Thread 0x00007f2e63a3d740 (most recent call first):
2024-09-16 21:19:01.987318725    File "/usr/lib/python3.9/threading.py", line 1428 in _shutdown
2024-09-16 21:19:01.987341774    File "/usr/lib/python3.9/multiprocessing/process.py", line 333 in _bootstrap
2024-09-16 21:19:01.987361583    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 71 in _launch
2024-09-16 21:19:01.987380255    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 19 in __init__
2024-09-16 21:19:01.987400955    File "/usr/lib/python3.9/multiprocessing/context.py", line 277 in _Popen
2024-09-16 21:19:01.987420238    File "/usr/lib/python3.9/multiprocessing/context.py", line 224 in _Popen
2024-09-16 21:19:01.987437867    File "/usr/lib/python3.9/multiprocessing/process.py", line 121 in start
2024-09-16 21:19:01.987461833    File "/opt/frigate/frigate/app.py", line 305 in init_recording_manager
2024-09-16 21:19:01.987474516    File "/opt/frigate/frigate/app.py", line 679 in start

[NickM-27]

2024-09-16 21:19:01.986893697 corrupted double-linked list

almost sounds like a memory issue. This is an issue in an internal library so it really doesn't seem like a frigate issue

[zeroward]

@NickM-27 Despite memtest not bringing back any errors, I went ahead and replaced the ram. Here's to hoping that's the issue.

Regardless, since this issue is from frigate, rather an internal library, I'm going to close the issue.

Thanks for your help!

[a-lunatic]

Try dozzle to capture the logs?

[zeroward]

Ha, noticed this too late.. ended up just manually combing through them with docker logs and less lol.

So here's what I have:

Prior to the first instance of Could not write header for output file #0 (incorrect codec parameters ?): No space left on device, this occurred:

024-08-21 09:36:16.780996291  Fatal Python error: Segmentation fault
2024-08-21 09:36:16.780999193
2024-08-21 09:36:16.781298594  Thread 0x00007fc70d5fd6c0 (most recent call first):
2024-08-21 09:36:16.781303383    File "/usr/lib/python3.9/threading.py", line 312 in wait
2024-08-21 09:36:16.781314579    File "/usr/lib/python3.9/queue.py", line 171 in get
2024-08-21 09:36:16.781337628    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 162 in loop
2024-08-21 09:36:16.781361030    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 137 in run
2024-08-21 09:36:16.781389771    File "/usr/local/lib/python3.9/dist-packages/playhouse/sqliteq.py", line 272 in run
2024-08-21 09:36:16.781404144    File "/usr/lib/python3.9/threading.py", line 892 in run
2024-08-21 09:36:16.781422174    File "/usr/lib/python3.9/threading.py", line 954 in _bootstrap_inner
2024-08-21 09:36:16.781436218    File "/usr/lib/python3.9/threading.py", line 912 in _bootstrap
2024-08-21 09:36:16.781437540
2024-08-21 09:36:16.781442780  Thread 0x00007fc71418e740 (most recent call first):
2024-08-21 09:36:16.781451933    File "/usr/lib/python3.9/threading.py", line 1428 in _shutdown
2024-08-21 09:36:16.781472683    File "/usr/lib/python3.9/multiprocessing/process.py", line 333 in _bootstrap
2024-08-21 09:36:16.781492711    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 71 in _launch
2024-08-21 09:36:16.781510854    File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 19 in __init__
2024-08-21 09:36:16.781531811    File "/usr/lib/python3.9/multiprocessing/context.py", line 277 in _Popen
2024-08-21 09:36:16.781548290    File "/usr/lib/python3.9/multiprocessing/context.py", line 224 in _Popen
2024-08-21 09:36:16.781566617    File "/usr/lib/python3.9/multiprocessing/process.py", line 121 in start
2024-08-21 09:36:16.781584923    File "/opt/frigate/frigate/app.py", line 305 in init_recording_manager
2024-08-21 09:36:16.781604241    File "/opt/frigate/frigate/app.py", line 679 in start
2024-08-21 09:36:16.781622266    File "/opt/frigate/frigate/__main__.py", line 17 in <module>
2024-08-21 09:36:16.781636513    File "/usr/lib/python3.9/runpy.py", line 87 in _run_code
2024-08-21 09:36:16.781652833    File "/usr/lib/python3.9/runpy.py", line 197 in _run_module_as_main
2024-08-21 09:36:24.894367116  127.0.0.1 - - [21/Aug/2024:09:36:24 +0000] "" 400 0 "-" "-" "-"
...
2024-08-21 11:18:26.620721529  127.0.0.1 - - [21/Aug/2024:11:18:26 +0000] "" 400 0 "-" "-" "-"
2024-08-21 11:18:40.291114397  11:18:40.290 DBG [streams] stop producer url=rtsp://frigate_user:<REMOVED>@192.168.5.195:554//h264Preview_01_main
2024-08-21 11:18:40.291296648  11:18:40.291 TRC [streams] skip stop none producer
2024-08-21 11:18:40.291299456  11:18:40.291 TRC [streams] stop reconnect url=rtsp://frigate_user:<REMOVED>@192.168.5.195:554//h264Preview_01_main
2024-08-21 11:18:40.865197974  11:18:40.865 DBG [streams] stop producer url=rtsp://frigate_user:<REMOVED>@192.168.5.149:554//h264Preview_01_main
2024-08-21 11:18:40.865286003  11:18:40.865 TRC [streams] skip stop none producer
2024-08-21 11:18:40.865386592  11:18:40.865 TRC [streams] stop reconnect url=rtsp://frigate_user:<REMOVED>@192.168.5.149:554//h264Preview_01_main
2024-08-21 11:18:41.241302371  11:18:41.241 DBG [streams] stop producer url=rtsp://frigate_user:<REMOVED>@192.168.5.150:554//h264Preview_01_main
2024-08-21 11:18:41.241406160  11:18:41.241 TRC [streams] skip stop none producer
2024-08-21 11:18:41.241451431  11:18:41.241 TRC [streams] stop reconnect url=rtsp://frigate_user:<REMOVED>@192.168.5.150:554//h264Preview_01_main
2024-08-21 11:18:41.678040548  11:18:41.677 DBG [streams] stop producer url=rtsp://frigate_user:<REMOVED>@192.168.5.188:554//h264Preview_01_main
2024-08-21 11:18:41.678278661  11:18:41.678 TRC [streams] skip stop none producer
2024-08-21 11:18:41.678281490  11:18:41.678 TRC [streams] stop reconnect url=rtsp://frigate_user:<REMOVED>@192.168.5.188:554//h264Preview_01_main
2024-08-21 11:18:41.732194756  11:18:41.731 DBG [streams] stop producer url=rtsp://frigate_user:<REMOVED>@192.168.20.90:554//h264Preview_01_main
2024-08-21 11:18:41.732198888  11:18:41.732 TRC [streams] skip stop none producer
2024-08-21 11:18:41.732200733  11:18:41.732 TRC [streams] stop reconnect url=rtsp://frigate_user:<REMOVED>@192.168.20.90:554//h264Preview_01_main
2024-08-21 11:18:49.712207652  [2024-08-21 11:18:49] ffmpeg.front-yard.record       ERROR   : [segment @ 0x5581ef4ea3c0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
2024-08-21 11:18:49.712408508  [2024-08-21 11:18:49] ffmpeg.front-yard.record       ERROR   : av_interleaved_write_frame(): No space left on device
2024-08-21 11:18:49.712531704  [2024-08-21 11:18:49] ffmpeg.front-yard.record       ERROR   : [segment @ 0x5581ef4ea3c0] Failure occurred when ending segment '/tmp/cache/front-yard@20240821111836+0000.mp4'
2024-08-21 11:18:49.712597597  [2024-08-21 11:18:49] ffmpeg.front-yard.record       ERROR   : Error writing trailer of /tmp/cache/front-yard@%Y%m%d%H%M%S%z.mp4: No space left on device
2024-08-21 11:18:49.712599740  [2024-08-21 11:18:49] watchdog.front-yard            INFO    : Terminating the existing ffmpeg process...
2024-08-21 11:18:49.712688384  [2024-08-21 11:18:49] watchdog.front-yard            INFO    : Waiting for ffmpeg to exit gracefully...

[arunes11]

Are you using Proxmox?
I had received the same error message ("Could not write header for output file #0 (incorrect codec parameters ?): No space left on device") with Proxmox. When the storage usage (for Recordings) reached about 70%, Proxmox automatically remounted the volume as read-only and then that message appeared in dmesg.
I solved it by significantly increasing the storage available to Frigate.

[zeroward]

@arunes11 I am using proxmox, but I'm using an NFS drive for my recordings and that's nowhere near full.

[zeroward]

Unable to determine final cause of issue, but seeing as how it's an internal library that's crashing, I'm going to close out this issue.

[zeroward]

This issue was not solved by replacing RAM, additionally, no other containers on this host encounter any similar issues..

Are there any additional suggested troubleshooting steps?

[NickM-27]

not that I know of, there's no indication what caused this to occur. The logs seem to be a bit different each time

[NickM-27]

you could try enabling debug logs for frigate.record.maintainer since of the traces does indicate that area as an issue

[zeroward]

Necro Update..

I switched the system I was hosting this on and no longer encounter the issue. I believe it was some weird networking issue that led to the cache being filled when it couldn't write to the NFS share it was connected to.

Why I believe this: I noticed the NUC I was running this on would encounter intermittent issues with its ethernet connection. It just so happened that around this time, the cache would fill up..

Having said that, thanks all for the troubleshooting help!