Object detection using not just one, but N past frames

[AlibekJ]

Object detection using not just one, but N past frames

[NickM-27]

frigate highly prioritizes realtime detection so maybe for your use case it wouldn't matter but that isn't the goal, and also more than a single 640x640 frame can't fit in the corals memory so the coral would not be usable at all.

My Frigate+ model for example has not confused a dog / cat in 6 months and I have a dog outside every day and also neighborhood cats walking around multiple times a week. I don't think this is really needed for the main frigate use case, there is plenty of room to improve existing object detection models.

What you are describing seems much more useful as a secondary processing for searching or categorizing events more specifically, not for the main frigate object tracking that is implemented and used throughout the new UI and the backend #8980

[NickM-27]

You're talking about an entirely different type of object detection. One that would also be a lot more computationally expensive to run than on single cropped frames. Something that the coral for example couldn't support.

Fire detection is something that IMO should be done with a true safety device that's certified for that purpose and dedicated to that job. Frigate already supports audio detectors including fire alarm and smoke alarm as types that can be listened to.

[AlibekJ]

That is correct, it is an entirely different type of object detection.
Frigate is a great product already, but if it would be able to detect not just objects but events such as "fire", "garage door opening", "garage door closing", "lights on", "person gets into a car", "mailman stuffing mailbox", etc. it would get into a whole new level of usefulness.

I guess sending a full frame does not make much sense, can get away with sending a cropped region. Need to experiment with this.

Even if Coral can't process it in real-time -- it is OK. If event is detected even with a few seconds lag it is still useful. In each of the examples above a few seconds are not critical. Last night I had a fire outside of my house, no smoke detector would be able to pick it up, but camera surely could.

Besides, there will be a bunch of new Coral-like devices with 10x of its capacity, those will be able to handle 10x number of pixels at a time.

Giving detectors temporal info may significantly improve object detection. Today I am often labeled as "bear" which is understandable. However, given not one but a bunch of frames neural network could discriminate a bear from a fat, tall man wearing black onesie because bears move differently. I guess it could even discriminate civilians from policemen and Fedex couriers.

At first I thought I could just store a bunch of frames in RAM, crop them to the latest motion box, concatenate them together and send them to a neural network. I thought it would be a relatively simple patch, but after giving it a thought it seems to be much more complex of a change.

Would love to look into it more if you think it is something you could consider taking a PR for.