further ideas

[fangfufu]

further ideas

[zeorin]

As a further idea, now that MediaPipe is being used, could we extend this to do stuff like show specific overlays when certain facial expressions or hand gestures are recognized? E.g. like SnapLensStudio does: https://twitter.com/cameronhunter/status/1305667539922550784

[fangfufu]

1. That sounds like a good idea, but the current approach works fine as a hack together project.
2. I actually thought about doing that, but it turned out that scaling the frame provided enough performance boost for my own machine. Although I am not sure how knowing the previous mask would help in terms reducing wobbling. Each frame is a new classification task. I suppose you can use it to filter out outliers, but I think bodypix performs well enough. If you know more than one previous mask, you could predict trajectory and things, but that's quite complicated, and I feel it is beyond me.
3. I agree.
4. Yes, that would be helpful.
5. That could be done with OpenCV's face tracker. I suppose. We might have to do some computation on the scaling of the face as well.
6. The halo effect might be due to blurring at the edge of the mask, it is done at fakecam/fake.py#L64. I think to fit a better mask, you need piping in more pixels to bodypix. This in term slows things down. This is why I downscale the image sent to the network, and upscale it when it comes back. One could argue that GPU would be a better performance, but I think a lot of people would agree that Nvidia GPU drivers are not the easiest thing to set up. Setting up your Nvidia GPU for playing game is actually separate to setting it up for CUDA. CUDA driver is definitely non-free.
7. At this point, have you thought about simply using obs-v4l2sink

Overall this is a big list of things. This "project" started off as tidying up someone else's blog post. I actually came across the blog post when I was reading a feature request for Webcamoid (https://github.com/webcamoid/WebcamoidIssues/issues/26). I kind of prefer this project ends up being a plugin for OBS or Webcamoid.

In terms of the things that really should be worked on - I really don't like the concept of passing images from Python to Node.js via HTTP. This creates unnecessary overhead. I did wonder why this was even a design decision, it turns out that that bodypix is only officially available a Tensorflow.js model (tensorflow/tfjs#2043). Although someone else is working on this already (allo-/virtual_webcam_background).

I may work through the list when I have more time, I make no guarantees. I need to go back and finish my PhD thesis. I only did this whole thing because I wanted to impress my labmates in research group Zoom meetings. :)

[fangfufu]

@liske , someone sent in a patch to support animated video background. It works really well.

[fangfufu]

@liske I updated the repository to use Mediapipe as the segmentation backend. It is now running so much faster.

[fangfufu]

I just want to say that be aware of https://github.com/allo-/virtual_webcam_background/issues/10 when installing Virtual Webcam Background. At the moment, its performance is similar to mine.

[fangfufu]

@liske , feel free to work on any of the suggestions you listed. I think it would be a good idea to add a description on how to use Android or ip-based webcams.

[fangfufu]

@zeorin , feel free to work on it yourself, and submit patches.