lingdong-/handpose-facemesh-demos

🎥🤟 8 minimalistic templates for tfjs mediapipe handpose and facemesh

Mediapipe Handpose and Facemesh Demos

Table of Contnets

• Handpose Demos
• Facemesh Demos

Handpose Demos

Collection of minimalistic demos/templates using mediapipe handpose with tensorflow.js. Created for and supported by the Frank-Ratchye STUDIO for Creative Inquiry.

The handpose package detects hands in an input image or video stream, and returns twenty-one 3-dimensional landmarks locating features within each hand. Such landmarks include the locations of each finger joint and the palm. -- blog.tensorflow.org

There are 4 templates included in this repo, 2 of them with three.js and 2 with p5.js, 2 are multiplyer via socket.io + express, and 2 are singleplayer static sites. They're all well commented and explained for beginners.

All demos are hosted on glitch.com 🎏

Online multiplayer 3D

Live | Remix | Source

• tensorflow.js + mediapipe handpose
• three.js
• socket.io + express

Online multiplayer 2D

Live | Remix | Source

• tensorflow.js + mediapipe handpose
• p5.js
• socket.io + express

Singleplayer 3D

Live | Remix | Source

• tensorflow.js + mediapipe handpose
• three.js

Singleplayer 2D

Live | Remix | Source

• tensorflow.js + mediapipe handpose
• p5.js

Also available on editor.p5js.org: https://editor.p5js.org/lingdong/sketches/1viPqbRMv

4 in 1

Glued together with iframes: https://handpose-demos.glitch.me/

Facemesh Demos

Collection of minimalistic demos/templates using mediapipe facemesh with tensorflow.js. Created for and supported by the Frank-Ratchye STUDIO for Creative Inquiry.

The facemesh package infers approximate 3D facial surface geometry from an image or video stream, requiring only a single camera input without the need for a depth sensor. -- blog.tensorflow.org

There are 4 templates included in this repo, 2 of them with three.js and 2 with p5.js, 2 are multiplyer via socket.io + express, and 2 are singleplayer static sites. They're all well commented and explained for beginners.

All demos are hosted on glitch.com 🎏

Keypoint Subsets

There're 3 additional keypoint subsets available in each demo to select from, as alternatives to the facemesh 468 vertices. See image below for indices and triangulation for each of them:

From left to right:

• Minimal 7 points
• Important features 33 points
• Standard facial landmark 68 points
• Full facemesh 468 points. A larger resolution and flattened version of the illustration is available in tfjs facemesh repo.

Online multiplayer 3D

Live | Remix | Source

• tensorflow.js + mediapipe facemseh
• three.js
• socket.io + express

Online multiplayer 2D

Live | Remix | Source

• tensorflow.js + mediapipe facemseh
• p5.js
• socket.io + express

Singleplayer 3D

Live | Remix | Source

• tensorflow.js + mediapipe facemseh
• three.js

Singleplayer 2D

Live | Remix | Source

• tensorflow.js + mediapipe facemseh
• p5.js

Also available on editor.p5js.org: https://editor.p5js.org/lingdong/sketches/ef6FB-uNq

Made at the Frank-Ratchye STUDIO for Creative Inquiry at Carnegie Mellon University.

handpose-facemesh-demos's Issues

Uncaught (in promise) TypeError: n.kernel_impls.nonMaxSuppressionV3Impl is not a function

Thanks for these great examples!

I'm getting the following error when I try to run all of the facemesh examples on glitch.

Uncaught (in promise) TypeError: n.kernel_impls.nonMaxSuppressionV3Impl is not a function
    at Object.kernelFunc (tfjs-backend-webgl:17)
    at f (engine.ts:562)
    at engine.ts:619
    at t.scopedRun (engine.ts:436)
    at t.runKernelFunc (engine.ts:616)
    at nonMaxSuppression_ (non_max_suppression.ts:41)
    at Object.nonMaxSuppression (operation.ts:45)
    at h.getBoundingBoxes (facemesh:17)
    at t.<anonymous> (facemesh:17)
    at facemesh:17

I'm pretty sure I ran these about a month ago without any issues. Maybe a recent change is causing this bug?

run offline with local server

Hey Lingdong.
Thanks for making this code available.

I am trying to make it run in offline mode using the Brackets.io IDE to work on it.
Do you have instruction on how to run this locally?

For now I tried copying its linked tensor flow scripts to a local folder

        <script src="libs/tensorflow/tfjs-core-1.7.4/dist/tf-core.min.js"></script>
        <script src="libs/tensorflow/tfjs-converter-1.7.4/dist/tf-converter.min.js"></script>
        <script src="libs/tensorflow/tensorflow-models/handpose-0.0.4/dist/handpose.min.js"></script>

But get the following error:

I guess I will try to run a npm server next and see if that somehow work.

missing steps or something pls help me to test in droplet

git clone https://github.com/LingDong-/handpose-facemesh-demos.git
cd handpose-facemesh-demos
cd networked-hand-3js-tf174-handv1
openssl req -newkey rsa:2048 -new -nodes -x509 -days 3650 -keyout key.pem -out cert.pem

then

npm install --global http-server
http-server -S -C cert.pem 

next

Starting up http-server, serving ./public through https
Available on:
  https://127.0.0.1:8080
  https://192.81.208.210:8080
  https://10.10.0.12:8080
  https://10.116.0.9:8080
Hit CTRL-C to stop the server
[Sun Jun 13 2021 02:58:16 GMT+0000 (Coordinated Universal Time)]  "GET /socket.io/?EIO=3&transport=polling&t=Ne3JZex" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[Sun Jun 13 2021 02:58:16 GMT+0000 (Coordinated Universal Time)]  "GET /socket.io/?EIO=3&transport=polling&t=Ne3JZex" Error (404): "Not found"

This site can’t provide a secure connection problem

Just a heads up, I don't know if it happens on every machine but the https redirection caused my browser to throw an ERR_SSL_PROTOCOL_ERROR when run locally.

I was able to make it work on a local node server by commenting these lines in index.html

<script> var isOnHTTPS = window.location.href.startsWith("https"); if (!isOnHTTPS){ window.location.href = window.location.href.replace("http","https"); } </script>

How to distinguish left hand and right hand by coordinates

Have you researched a feasible way to distinguish between left and right hand ？I can't find a solution ~

p5 demo link in readme returns 404

Thanks for this great repo. So many excellent examples!

I noticed this link to the p5 editor is broken. Do you have an updated link?

Also available on editor.p5js.org: https://editor.p5js.org/LingDong-/sketches/ef6FB-uNq

Thanks again.