voidedwarranties/openvtuber2

version 2 of the garbage openvtuber

openvtuber2

version 2 of the classic garbage openvtuber, this time with more terrible technology powered by tensorflow

the new version is much better suited to bad lighting or webcams and has basic support for upper-body pose estimation through posenet

running

This apps starts a PeerJS server, enabling it to be recorded by an OBS browser capture. You must run it through Express, not by opening index.html. Install node.js (tested with v12), open a command prompt in the downloaded directory and run npm install followed by npm start.

Open the app by going to http://localhost:9000/

This app was tested on a Ryzen APU and took 60-80% of the GPU running all the models at once.

environment

"Minimum" environment for good detection

• Head, mouth: Pretty much anything will work
• Arms, hands: To avoid awkward positioning, try to place the camera so only your shoulders and above are showing - camera quality doesn't matter much
• Eye tracking, blinking: Webcam quality should be good enough to stream with your real face

Generally, performance will degrade significantly with more than one model (facemesh, posenet, handpose) running at once

recording

option 1 - display capture

Using a window capture is impossible because the app is based on a graphics-accelerated Electron instance. You can use display capture and crop it to record the model.

option 2 - PeerJS

A less finicky way to record the model is to create a "video call" through PeerJS.

1. Add the peerjs.html to a browser capture while the Electron app is running.
2. Right click the browser capture and click Interact
3. Copy the key shown in the browser capture, paste it into the "PeerJS Connection ID" field in the main application, and click Call.
4. Chroma key out the green background

libraries / references

• tensorflow.js + prebuilt models - facemesh, posenet, handpose
• three.js + GLTFLoader + OrbitControls
• three-vrm + their default model
• tween.js
• canvasfilters (filters.js)
• express + peerjs
• ml5
• JSZip, FileSaver.js
• stats.js
• lploc.js

PoseNet solution using atan2 was inspired by atskimura

openvtuber2's Issues

MacBook default camera not front facing

I think this repository is very high quality.
The only problem when I am trying the repo is that my MacBook default camera is leaning to one side.
Do you have any idea why this is happening?
All other programs seem to work as usual.
Is this a cropping problem caused by the video size?