blakeblackshear/frigate

NVR with realtime local object detection for IP cameras

Frigate - NVR With Realtime Object Detection for IP Cameras

A complete and local NVR designed for Home Assistant with AI object detection. Uses OpenCV and Tensorflow to perform realtime object detection locally for IP cameras.

Use of a Google Coral Accelerator is optional, but highly recommended. The Coral will outperform even the best CPUs and can process 100+ FPS with very little overhead.

• Tight integration with Home Assistant via a custom component
• Designed to minimize resource use and maximize performance by only looking for objects when and where it is necessary
• Leverages multiprocessing heavily with an emphasis on realtime over processing every frame
• Uses a very low overhead motion detection to determine where to run object detection
• Object detection with TensorFlow runs in separate processes for maximum FPS
• Communicates over MQTT for easy integration into other systems
• Records video with retention settings based on detected objects
• 24/7 recording
• Re-streaming via RTSP to reduce the number of connections to your camera
• WebRTC & MSE support for low-latency live view

Documentation

View the documentation at https://docs.frigate.video

Donations

If you would like to make a donation to support development, please use Github Sponsors.

Screenshots

Integration into Home Assistant

Also comes with a builtin UI:

frigate's Issues

support for old hardware with no AVX

I'm running frigate on a laptop with an old Intel CPU with no AVX support:
Intel(R) Core(TM) i3 CPU M 370 @ 2.40GHz
I have google coral USB accelerator plugged into this laptop.

When running frigate I received the following error:
frigate | 2019-07-08 14:35:13.075587: F tensorflow/core/platform/cpu_feature_guard.cc:37] The TensorFlow library was compiled to use AVX instructions, but these aren't available on your machine.

Would really appreciate if someone could suggest how to modify existing codes in frigate in order to run on such old hardware. Thank you so much in advance.

error handling for invalid mqtt user/password

Low priority issue here, but I noticed when I typo'ed my username/password for mqtt, it never gave any errors in the logs. It just retried the connection a bunch of times.

Optimize detection loop to maximize detections per second

With an inference time of about ~5ms, I should be able to get near 200fps with the Coral. I must have something in my while loop that is taking up time because I am seeing <100fps.

implement healthcheck

As a scheduler of a frigate docker container,
I want a healthcheck endpoint,
So that I will know when frigate is ready to start working and serving traffic

---

It would be great if we could have some form of healthcheck/liveness probe, or even something served at the root / URL so we will know when it is properly running and ready for use. As far as I know, the only way for an external entity to see if it's working is to hit the preview URL specific to a camera defined in the configuration. This doesn't easily allow for a generic check.

Use inbuilt camera sensors / triggers to enable frigate monitoring

On some camera's such as HikVision cameras there are trigger states you can define in the camera such as line crossing.

It might be worth while having an option to trigger frigate on / off depending on for example when an object is detected to cross a line on the camera.
I thought this might be an advantage for users trying to run frigate on lower powered devices, so the system does not have to process every frame.

In HA I use the HikVision sensor to trigger other devices like:

• alias: Tripwire
  trigger:
  platform: state
  entity_id: binary_sensor.driveway_line_crossing
  from: 'off'
  to: 'on'

https://www.home-assistant.io/components/binary_sensor.hikvision/

Video stream becomes one large smear

After running the latest for 24 hours, some time overnight the video became a large smear.

See here:
Screenshot

This happened in the previous version as well. For now i'm going to try telling it to restart every night, but it would be great if it would detect when this happens and reset itself.

Debug info:
Hikvision RTSP stream
Unraid docker host
CPU utilization stayed below 40% all night.

python error

I recently switched models to improve processing time. I was not seeing object %'s like I should be seeing based on the bounding boxes. I changed from faster_rcnn_inception_v2_coco_2018_01_28 to ssd_mobilenet_v2_coco_2018_03_29 and I now get the error below. Some times the error happens right away, some times it will work fine for a bit and then error out.

I'm using mscoco_label_map.pbtxt label map

I've done some research on the models but still don't quite have a great grasp on the differences other than mobilenet requires less resources. The error only occurs on the image built from github. Your image on dockerhub does not do this with either model.

My docker run command is

docker run --rm  --name frigate \
-v /mnt/nfs/frigate/faster_rcnn_inception_v2_coco_2018_01_28/frozen_inference_graph.pb:/frozen_inference_graph.pb:ro \
-v /mnt/nfs/frigate/mscoco_complete_label_map.pbtxt:/label_map.pbtext:ro \
-v /mnt/nfs/frigate/config:/config:ro \
-v /mnt/hassio/config/frigatedebug:/lab/debug \
-p 5000:5000 \
-e RTSP_URL='rtsp://192.168.z.z:8554/unicast' \
-e REGIONS='460,50,50,5000,500,nomask.bmp' \
-e MQTT_HOST='192.168.z.zz' \
-e MQTT_TOPIC_PREFIX='cameras/1' \
-e DEBUG='0' \
blakeblackshear/frigate

camera resolution is 960x540, nomask.bmp is a white image.

Let me know if you need any additional info.
Thanks,

On connect called
capture_process pid  34
detection_process pid  35
motion_process pid  36
 * Serving Flask app "detect_objects" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: off
2019-03-23 19:06:54.949810: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2098550000 Hz
2019-03-23 19:06:54.951714: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0xcdc0c50 executing computations on platform Host. Devices:
2019-03-23 19:06:54.951798: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
2019-03-23 19:06:58.593049: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node Preprocessor/map/while/ResizeToRange/strided_slice_3. Error: Pack node (Preprocessor/map/while/ResizeToRange/stack_2) axis attribute is out of bounds: 0
2019-03-23 19:07:00.693914: W ./tensorflow/core/grappler/optimizers/graph_optimizer_stage.h:241] Failed to run optimizer ArithmeticOptimizer, stage RemoveStackStridedSliceSameAxis node Preprocessor/map/while/ResizeToRange/strided_slice_3. Error: Pack node (Preprocessor/map/while/ResizeToRange/stack_2) axis attribute is out of bounds: 0
Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/frigate/frigate/object_detection.py", line 108, in detect_objects
    objects = tf_detect_objects(cropped_frame_rgb, sess, detection_graph, region_size, region_x_offset, region_y_offset, debug)
  File "/opt/frigate/frigate/object_detection.py", line 62, in tf_detect_objects
    'name': str(category_index.get(value).get('name')),
AttributeError: 'NoneType' object has no attribute 'get'

Detecting objects other than "person"?

I know it's possible to feed in another model via the docker-compose file, but I'm wondering if it would be possible to generalize the code a bit to allow perhaps a config entry or something for detection of other objects/faces with other models and still give a best_<object>.jpg image. I'd love to be able to use your code with the faces model, which doesn't have a labels file either.

Errors generated by an mqtt client

I'm getting these errors all of the time in my mqtt logs

1558644364: New client connected from 1.1.1.1 as UGJ9uX=xJx8[NPsPRdJIV2i;07@GtAVgB5Yd;wKriqa79kAEGECYYgLK^RW\PHCU (c1, k60, u'frigate').

1558648510: Socket error on client UGJ9uX=xJx8[NPsPRdJIV2i;07@GtAVgB5Yd;wKriqa79kAEGECYYgLK^RW\PHCU, disconnecting.

It sends about nothing

$ mosquitto_sub -u frigate -t frigate/#
online

Recognise objects including people / faces.

It would be great if frigate could recognise objects and especially recognise people it has already seen as well as report people it does not know.

This can be handy for a number of reasons, if it recognised the post lady it could either tell the Camera System not to record the activity as it is a known person, or you could use it to trigger a notification that your post has been delivered.

Another example is if it recognises a family member visiting you, it could announce their arrival.

Another example is if it detects an unknown person or vehicle on your property it could trigger a manual recording and notify you of suspect activity on your property.

False detections at night / 1-channel greyscale issue?

I get lots of false detections from spider webs and moths etc at night in infrared.

This article suggests that images from an infrared camera are single channel, while most models are trained on 3 channel RGB Images.

There is also this on tensorflow github with some more info if needed.

Solution as per the links appears to be to stack the single channel 3 times to form a 3-channel input.

Hopeful this is possible and not too much work :)

Look for a better model or a secondary model to reduce false positives

Only getting one frame at 5000:/cam1 on a Raspberry pi 4

Hi,

Got the frigate docker container running on my Raspberry pi 4 with a Coral TPU.
When visiting http://ip:5000/cam1 I only get 1 frame. The timestamp in the video stream stays the same.

I do not get any error, any idea?

Gr, Justin

Allow reducing the ingress framerate from RTSP stream

Per request on the Home Assistant community:

Is there anyway to limit the frame rate that python is analyzing? My WYZE cameras do not have a way to change the frame rate in the RTSP stream. This means frigate processes every single frame (multiplied by my 6 regions across 2 cameras) and destroys my CPU/maxes out the Coral buffer queue.

I noticed in the README you mention a tip about limiting the frame rate to lower the CPU usage, but I am unable to do that in my setup without pre-processing the video first.

Allow us to buy you a drink

This is a great project. I really like the work that you've done here with Coral and TensorFlow.

Please let some of us buy you a drink similar to how frenck allows us to buy him coffee in his readme files. Maybe just add that badge to your project as well?

how to define no specific region

Ho do I define to use the full image to detect and mark persons?

I now use:

    regions:
      - size: -1
        x_offset: 0
        y_offset: 0
        min_person_area: 5000
        threshold: 0.5

and this detects persons, but does not mark the detected person with a square.

Make detection threshold configurable

RTMP support suggestion

I have had to switch to RTMP for some of my cameras.

My Reolink cameras use the syntax rtmp://{host}:{port}{path}&user={}&password={}. For now I've simply added the rtmp values to my config file and made a copy of the get_rtsp_url method (get_rtmp_url), which is used if the rtsp host is blank. I'm sure there is a more elegant way though.

Also, I've since noticed that the ffmpeg documentation suggests the URL syntax for RTMP can be rtmp://[username:password@]server[:port][/path].

I'm happy to put a PR in for this if you think it would benefit others, but I'm unsure of the best approach with regards to the different syntaxes which are possible. I'm wondering whether it would be better to just have an optional override ffmpeg_url in order to support any syntax.

Background
Artefacts in my feed were triggering occasional false alerts. Switching to the RTMP stream (thanks @steveneighbour #27) has fixed this. Also I've only had one 'last frame is more than than 2 seconds old, restarting...' in the last 24 hours, compared to once every 2-3 hours before :)

Process keyframes only

@blakeblackshear I ran some experiments extracting only keyframes instead of extracting a frame every 2 seconds. It seems like its about 50% faster.

Another potential advantage to using keyframes other than processing time is that they are a full image capture and don't include compression artifacts that could fool the AI.

Here is my script and test results in case you'd like to consider it:

INPUT_FILE=front-door-camera-3.mp4
OUTPUT_DIR=keyframes

# extract keyframes only
time ffmpeg -i $INPUT_FILE -vf "select=eq(pict_type\,I)" -vsync vfr $OUTPUT_DIR/kframe%04d.jpg -hide_banner

# extract a frame every 2 seconds
time ffmpeg -i $INPUT_FILE -vf fps=1/2 $OUTPUT_DIR/half_fps%04d.jpg -hide_banner

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'front-door-camera-3.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    title           : RTSP/RTP stream from anjvision ipcamera
    encoder         : Lavf58.20.100
  Duration: 00:00:59.93, start: 0.000000, bitrate: 5041 kb/s
    Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuvj420p(pc), 2592x1944, 5000 kb/s, 15.02 fps, 15 tbr, 90k tbn, 180k tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 8000 Hz, mono, fltp, 39 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
Output #0, image2, to 'keyframes/kframe%04d.jpg':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    title           : RTSP/RTP stream from anjvision ipcamera
    encoder         : Lavf58.20.100
    Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 2592x1944, q=2-31, 200 kb/s, 15 fps, 15 tbn, 15 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc58.35.100 mjpeg
    Side data:
      cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: -1
[h264 @ 0xa404a0] error while decoding MB 104 42, bytestream -5A speed=0.133x    
[h264 @ 0xa404a0] concealing 12905 DC, 12905 AC, 12905 MV errors in P frame
frame=   30 fps=1.0 q=24.8 Lsize=N/A time=00:00:58.06 bitrate=N/A speed=2.02x     
video:12512kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

real	0m29.299s
user	1m16.481s
sys	0m1.419s
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'front-door-camera-3.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    title           : RTSP/RTP stream from anjvision ipcamera
    encoder         : Lavf58.20.100
  Duration: 00:00:59.93, start: 0.000000, bitrate: 5041 kb/s
    Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuvj420p(pc), 2592x1944, 5000 kb/s, 15.02 fps, 15 tbr, 90k tbn, 180k tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 8000 Hz, mono, fltp, 39 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
Output #0, image2, to 'keyframes/half_fps%04d.jpg':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    title           : RTSP/RTP stream from anjvision ipcamera
    encoder         : Lavf58.20.100
    Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 2592x1944, q=2-31, 200 kb/s, 0.50 fps, 0.50 tbn, 0.50 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      encoder         : Lavc58.35.100 mjpeg
    Side data:
      cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: -1
[h264 @ 0x14c4480] error while decoding MB 104 42, bytestream -5
[h264 @ 0x14c4480] concealing 12905 DC, 12905 AC, 12905 MV errors in P frame
frame=   30 fps=0.8 q=24.8 Lsize=N/A time=00:01:00.00 bitrate=N/A speed=1.64x    
video:12942kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

real	0m37.326s
user	1m36.217s
sys	0m1.850s

Reference links:

• https://www.bugcodemaster.com/article/extract-images-frame-frame-video-file-using-ffmpeg

Run FFmpeg in a separate process to leverage hardware acceleration

I was looking into support for hardware accelerated decoding and realized that OpenCV doesn’t use the hardware acceleration built into FFmpeg. I did some basic tests, and it makes a substantial difference. On the laptop where I am running frigate, decoding a 25FPS 1080p stream uses about 25% of a single CPU. Switching FFmpeg to use hardware acceleration drops that to <2%. I can read raw frames form a separate FFmpeg process like this. Once I add support for this, even low powered machines should be able to perform object detection on many cameras.

Issues in virtual machine

Hi I am running on a virtual machine and the vm being run is debian 9. I am getting this error after passing the device to the vm:
On connect called
Traceback (most recent call last):
File "detect_objects.py", line 99, in
main()
File "detect_objects.py", line 57, in main
prepped_frame_queue
File "/opt/frigate/frigate/object_detection.py", line 32, in init
self.engine = DetectionEngine(PATH_TO_CKPT)
File "/usr/local/lib/python3.6/dist-packages/edgetpu/detection/engine.py", line 72, in init
super().init(model_path)
File "/usr/local/lib/python3.6/dist-packages/edgetpu/basic/basic_engine.py", line 40, in init
self._engine = BasicEnginePythonWrapper.CreateFromFile(model_path)
RuntimeError: Error in device opening (/sys/bus/usb/devices/2-1)!

Any idea?

Summary/recap idea

Something like this would be incredible. https://v.redd.it/flfjtitwfyd31

Feature Request - Output movie clips of detected objects for notifications

Noticed this is a planned feature in the readme, so figured i'd open an issue for it.

Having video clips compiled when objects are detected would be a great feature. Motion handles this really well, using a pre and post number of frames captured that are then stitched into a movie file.

I believe you already store 2 seconds (?) of frames for object analysis, so it would be a matter of using those 2 seconds + all frames captured while the moving object is present + a configurable number of seconds/frames after the object is no longer present/moving.

This video could be dumped to disk and/or sent to HA.

Exception followed by queue full

running on 2 cameras, 5 regions (2x 640, 3x800) @ 2-10fps.

I would notice the detection would not be running, check the container, and it's churning queue is full. Finally caught the exception which triggers the queue is full. See below.

Exception in thread Thread-16:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/opt/frigate/frigate/object_detection.py", line 55, in run
self.cameras[frame['camera_name']].add_objects(parsed_objects)
File "/opt/frigate/frigate/video.py", line 219, in add_objects
y_location = min(int(obj['ymax']), len(self.mask)-1)
TypeError: object of type 'NoneType' has no len()

queue full. moving on
queue full. moving on
queue full. moving on
queue full. moving on
queue full. moving on

`queue full` after moving view

Upon starting up frigate, I see image processing is working as it'll immediately recognize my car in frame (cam is in the garage). The moment I walk into frame, I see the following stacktrace:

 * Serving Flask app "detect_objects" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
On connect called
192.168.45.81 - - [06/May/2019 21:31:54] "GET /garage HTTP/1.1" 200 -
On connect called
On connect called
Exception in thread Thread-7:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/opt/frigate/frigate/object_detection.py", line 55, in run
    self.cameras[frame['camera_name']].add_objects(parsed_objects)
  File "/opt/frigate/frigate/video.py", line 219, in add_objects
    y_location = min(int(obj['ymax']), len(self.mask)-1)
TypeError: object of type 'NoneType' has no len()
queue full. moving on
queue full. moving on
queue full. moving on
queue full. moving on
queue full. moving on
queue full. moving on

After I see this behavior, the container needs to be restarted in order for image processing to start working again.
I have 1 camera (hikvision) with 1 zone. Camera is 1080p, 4fps, I Frame Interval of 4, max bitrate of 512.

I don't believe it's a usb3.0 vs usb2.0 problem based upon the following:

Bus 003 Device 003: ID 18d1:9302 Google Inc.
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               3.10
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0
  bDeviceProtocol         0
  bMaxPacketSize0         9
  idVendor           0x18d1 Google Inc.

and

/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
    |__ Port 4: Dev 3, If 0, Class=Vendor Specific Class, Driver=usbfs, 5000M

Thoughts?

Use ffmpeg to filter duplicate frames

https://superuser.com/questions/1154171/ffmpeg-constant-framerate-to-variable/1154224#1154224

This should allow ffmpeg to detect and drop frames that are nearly identical to previous frames. No reason to run detection if nothing has changed. This should also increase the number of simultaneous cameras that a single coral can handle.

Allow overlapping regions to be excluded by motion detection

Currently when you have multiple regions defined and they are stacked next to each other, there is still a high chance of both regions registering an object, as the object detection is able to recognize 1/4 of a body as a human with fairly high accuracy. This means that you will be registering multiple people where only 1 exists fairly regularly.

I think the best solution to this problem is to encourage overlapping regions but to have an underlying motion detection engine watching the entire stream instead of only the defined regions. If it was watching the whole scene, it could then choose the region in which the moving object was located and turn off the other regions which he was only partially located, and so remove duplicates. If each region overlapped each other region by 50%, it would be close to impossible for any object to be detected twice because there would be ample space around it for it to exist entirely within a single region, and other regions that it's only partially in could be excluded from detection.

Not only would this remove duplicates, but it would also increase accuracy because the object you wanted would pretty much always be fully or very close to fully contained within a region, and regions could be sized correctly so that the object could fill most of the frame of the region.

This would also reduce CPU usage as most of the time, a single moving object could be processed entirely within a single region rather than spanning multiple regions.

This is a bit difficult of a concept to put into words, so let me know if you'd like me to explain further, I could draw what it looks like.

`min_person_area` doesn't seem to be filtering objects

Right now I have a camera set up that streams in 1080p. Because of that, I knew I would need to raise the min_person_area to account for the extra pixels. Below is one of the debug frames that I capture when an MQTT message is sent for a person being detected.

I'm using a min_person_area of 3000, so it doesn't seem like a person in the image across the street below has enough pixels to meet that criteria.

Is there a chance the MQTT person message is being sent without taking into account the min_person_area, or am I off by an order of magnitude with my values?

Below is my config.

web_port: 5000

mqtt:
  host: mqtt
  port: 8883
  topic_prefix: frigate
  user: <user>
  password: <password>

cameras:
  front_window:
    rtsp:
      user: <username>
      host: <ip_address>
      port: <port>
      password: <password>
      path: <path>
    regions:
      - size: 700
        x_offset: 1220
        y_offset: 200
        min_person_area: 3000
        threshold: 0.9
      - size: 1080
        x_offset: 0
        y_offset: 0
        min_person_area: 3000
        threshold: 0.9

Look into using numpy to convert to rgb24

This may allow hwaccel to work better on a larger number of platforms. Also has the benefit of only converting the regions rather than the entire image. https://gist.github.com/Quasimondo/c3590226c924a06b276d606f4f189639

UDP rtsp stream weird behavior

My cameras don't support rtsp over tcp, they use udp instead. I managed to successfully start the container was using the environment variable :
-e OPENCV_FFMPEG_CAPTURE_OPTIONS="rtsp_transport;udp" and i can see the feed on my browser.

The problem is that it seems to give high confidence false positives. I am attaching a screenshot to help you better understand the problem. My guess is that the video feed is parsed incorrectly. Do you have any clue as to how i can debug it or solve it?

I tried streaming rtsp over udp using my phone camera and an app and i have the same behavior. I am running the blakeblackshear/frigate:edgetpu image and i have tried building the container from the Dockerfile one the repository as well and i always get the same issue.

Thank you in advance.

If a person stands still long enough, it stops detecting a person until there is motion again

Detected objects expire from the list of current objects, and no new objects are detected once motion stops.

Filter false positives by estimated size of a person

Should be able to use this to collect data and compute an estimated person size. https://docs.scipy.org/doc/scipy-0.15.1/reference/tutorial/interpolate.html#multivariate-data-interpolation-griddata

-

frigate as binary sensor switches constantly between online and offline

setting this for each binary_sensor, fixes this for me. Found here https://community.home-assistant.io/t/tasmota-mqtt-switches-repeatedly-marked-as-unavailable/46935

    payload_available: ‘online’
    payload_not_available: ‘offline’

Does this mean HA and frigate defaults have a mismatch?

frigate/detect_objects.py:        client.publish(MQTT_TOPIC_PREFIX+'/available', 'online', retain=True)
frigate/detect_objects.py:    client.will_set(MQTT_TOPIC_PREFIX+'/available', payload='offline', qos=1, retain=True)

AttributeError from git-built docker image

n/a - needs closed

Serve up previous best_person images too.

As discussed here: https://community.home-assistant.io/t/local-realtime-person-detection-for-rtsp-cameras/103107/559?u=scstraus

I’ve been thinking it would be nice to save the last 5 or 10 images. Couple times I’ve been in a meeting or didn’t see my phone and I’ll have a few notifications. Do you think this is something that could be added ? Or even worth the time? I was thinking of creating a small script that just moved the file and added a date time but I haven’t messed with it just yet.

I want that too! I’d like to have something like the swiper card in my UI with the last 5-10 images I can browse through to see what was seen. If we could have multiple cameras for “best_person” “best_person_-1”, “best_person_-2”, etc, that would make it possible to do… An option for how many previous images to serve would be good too.

Bounding boxes a bit off

Hi

thanks for this great project, its so great. I've started testing today, and it worked right away. I've got question about bounding boxes, they are a bit off, not sure if I got some error in config or its supposed to be like this, it looks consistent, similar offset on multiple frames, its below upper recognition area. It looks like x are ok, only y axis is too low.

    regions:
      - size: 450
        x_offset: 50
        y_offset: 100
        min_person_area: 4000

Rate limit MQTT messages

Right now, I am sending MQTT messages too quickly. Rate limit to 1 per second or something.

Feature request - MQTT Username / Password

Hi!!

Thanks a bunch for this great hack! I am using username and password for my MQTT broker. Would it be possible to implement support for that?

Allow score threshold to be set via MQTT

This will allow dynamic thresholds for night/day or anything else. It also may be worth having the ability to create predefined thresholds for low, medium, high.

Multiple cameras ?

How do I run multiple cameras? Do I need to run 6 instances of the container ?

Look into usb accelerators

Such as

• https://coral.withgoogle.com/tutorials/accelerator/
• Intel Neural Compute

Set detection threshold for best person and lab/debug file save

It would be nice if we could set the detection threshold for /best_person.jpg, it’s been showing my umbrellas occasionally as people with 65% confidence, real people are always above 80%, usually high 90’s, so I think 80% is a good cutoff to not find the umbrellas or a tree behind my house which also sometimes shows up in the 50-60s..

MJPEG support suggestion

Hi @blakeblackshear
I notice that with a few small changes this code can also support MJPEG streams, which would allow users of motion/motioneyeOS to use it. You just need to pass the mjpeg url to video.open(_url).
It is trivial to create an mjpeg stream with any webcam and easy to display in HA which is why I am keen. However performance might not be as good as RTSP.
Cheers

Add object size to bounding box

I think it would be extremely helpful if the size of the objects were reported as one of the label element on the bounding box. This would help debug and resolve issues with the bounding box size.

The line it could get added here: https://github.com/blakeblackshear/frigate/blob/master/frigate/objects.py#L92

Reduction of volume of info written to /lab/debug

Hi, it would be great to have a bit less data in /var/debug as it can generate many GB per day. 2 options would probably be enough:

1. Expand debug levels or make another option to say which output you want (motion, motion mask, object, object 559)

2. Define minimum interval between image sends in milliseconds (though I would probably never set it below 5 seconds or so)

But a third one to reduce resolution or increase compression would be great too.

Region and Mask information

Maybe a bit of a newb question regarding opencv and tensorflow but what is the purpose of the back-mask.bmp and is there a tool people are using to plan/make the region sizes and coordinates up?

Stops detection of objects when one person appears in a region

Hey Folks,

I have a problem with detecting a "person" object. When it appears in a region frigate stops detecting any objects at all. I was trying to play around with variable "min_person_area", setting it low (70) or high (7000), but nothing helps.
Might be I setup something wrong, could you please help me to figure out a proper way.

Thanks in advance

Frame with the "car" objects

Frame with the "car" and "person" objects, but there are no more frames

Allow motion and detection masking to be done for whole feed

As the optimum number of squares to cover a whole scene seems to be around 7 without overlap and probably around 15 if the overlap feature I described previously would be implemented, it would be much more manageable if we could define the motion maps for the whole feed rather than the detection areas, as making 7-15 perfectly sized separate masks is quite a bit of work, which would have to be redone any time you changed your regions..

If frigate could just take one bmp for object detection and ideally a second one for motion and simply apply the appropriate region of the mask itself to each detection area, that would make setup and management a lot easier.

control label formatting for detected objects

Noticed this today on a successful object scan, a person is picked up a decent distance from the camera (so they are pretty small) but the label drawn obscures the person.

Is it possible/feasible to control the label (eg remove the red background), in an effort to not obscure the detected object?