[Idea] Zero-shot object detection models?

[afaulconbridge]

[Idea] Zero-shot object detection models?

[afaulconbridge]

Deepstack, and the other models in Frigate as far as I can see, work with pre-defined and pre-trained labels. So it can never detect a hedgehog, because it has no concept of what a hedgehog could be, because there was no hedgehog in the training labels.

Zero-shot models are only given the categories at execution time, and can work with things that they've never seen before - similar to large-language models. Broadly, a zero-shot model knows that a hedgehog is a small spiky spherical mammal, so when it sees a picture of a hedgehog it can give the right answer - even if the training data didn't contain any hedgehog pictures and only included picture of things that were either small or spiky or spherical or mammals.

I'm going to mess around with using zero-shot models in Frigate. There's no IRC, discord, reddit, email, or other contact/community details I could see - so a GitHub issue it is. I know Frigate is MIT licensed, so technically I don't need to tell you, but I thought it'd be nice to say something (and might save me a bunch of time if it was trivially impossible for some reason). Plus if there was anyone else who has/is/will try this it might be useful to cooperate.

[NickM-27]

Frigate already works with deepstack (CodeProject.AI) so that would work for many cases, Frigate also supports using any model you like as long as it is compatible with the detector of your choosing. I am not sure what exactly the request is here

[fgoodwin]

https://www.reddit.com/r/frigate_nvr/

Additional tags are coming Eventually™ but I also am interested in more generalized detection for wildlife, so if you get it working definitely write something up.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.