I have a few questions about this library

[marcusx2]

I have a few questions about this library

[xavierjs]

Hi Marcus,

The interface IJeelizFaceFilterScanSettings is deprecated. These parameters are not used anymore and have been replaced by those specified in the set_scanSettings method of the README ( https://github.com/jeeliz/jeelizFaceFilter#miscellaneous-methods ).

If maxFacesDetected is >=2, then the detectionState returned by Jeeliz FaceFilter as argument of callbackTrack function will be an array of detectionState objects instead of a single detectionState object.

BTW 8 faces detection is a lot. You need a quite good GPU then. If you have 8 detected faces, then you have 8 times less GPU computing power to detect and track each face.

About the expressions array, this array depends on the neural network. Most of the neural networks have not been trained to detect expressions. The one detecting face expressions is neuralNets/NN_4EXPR_<X>.json: this neural network also detects 4 facial expressions (mouth opening, smile, frown eyebrows, raised eyebrows).. So the first value of expressions is mouth opening, the second is smiling, the third and fourth about eyebrows.

We can train neural network for custom needs (if you need to detect a specific expression), or you can switch to WebAR.rocks.face (https://github.com/WebAR-rocks/WebAR.rocks.face), which has a similar interface, to get more accurate expression detection ( https://webar.rocks/demos/face/demos/expressionsDetection/ for example )

[xavierjs]

You cannot control a video element with JS
You can generate a video stream from the canvas, there are APIs for that.
Jeeliz FaceFilter needs a canvas to run, so you cannot get rid of this.
You don't need the JeelizResizer.

If you need help integrating the lib, I offer development services. Please contact me at contact_at_jeeliz.com

[xavierjs]

Hi Marcus,

You should not enable antialiasing and use a separate canvas for 3D rendering with THREE.js as advised there: https://github.com/jeeliz/jeelizFaceFilter#1-or-2-canvas
There are still old demos with 1 canvas and antialias set to true. The default value for antialias is false so you don't need to set it explicitely.

For the 2, I did not write the typescript interface file and it is not maintained. The resize method returns true if the canvas was really resized, false if it was not resized for any reason (the lib is not initialized, or the canvas is already at the good size). init returns true unless there is an important error (something big missing or webgl not available). But if it returns false, callbackReady function will still be called with an error code.

[xavierjs]

Hi Marcus,
If you use a separate canvas to do the THREE.js rendering, then THREE.js will create its own WebGL context for this canvas, you don't need to get the context from the first canvas.

[xavierjs]

Hi Marcus,
In engineering, hysteresis helps to stabilize a ON/OFF switch depending on an analog value ( https://info.bannerengineering.com/cs/groups/public/documents/literature/tt_threshold_hysteresis.pdf )

[xavierjs]

Hi,

Yes, currently it has the same value. The goal of returning this value at the init callback is that, maybe in the future, I can add additional code to evaluate if there is enough computing power to detect and track the requested number of faces. If not, I would lower this value to a more reasonable one and provide the applied value in the callback. But I have not done that yet.

[xavierjs]

Hello Marcus,

No worries, usually summer is a quiet moment for my main glasses VTO business so that I can spend more time improving free stuff, especially their doc, but this year it is very busy...

1 - Currently the change of the maximum number of detected faces is not implemented in runtime. This is the first time I have this request. If you want to first detect 1 face, then 2 faces, the best way is to set maxfacesDetected to 2 at the initialization, and then either you consider only the first value of detectState in the argument of callbackTrack function, either the 2 values. You can also reload the page if there is really a break in the experience. Or I can implement the feature but I will charge a bit for that.

2 - You cannot change the neural network in runtime. However, for this library neural networks do pretty much the same. I don't see a use case for this. I can also implement this feature as freelance work. The premium version of this lib, WebAR.rocks.face ( https://github.com/WebAR-rocks/WebAR.rocks.face ), also returning landmarks position, has this feature. It is useful in this case because the landmarks will depend on the neural network model. Necklace VTO neural net returns landmarks around the neck, whereas the earrings VTO neural net computes landmarks on the ears.

3 - scale0factor is the largest scale level. The face detection window is square. if scale0factor=1 it will be centered and it will have the smallest dimension of the video (either width or height). Then we will search for smaller sizes. nScaleLevels=3 means that we will search in 3 scale levels. But smaller levels take more time because there are more possible positioning of the input window. If you set nScaleLevels=1 it will only look for the face a the first scale level. it will be fast but if the user is a bit further away from the camera the face may be not detected.

4 - I provide freelance services for that. I can either look at your code and improve it as development services, or provide training or consulting to help you master the library.

5 - I provide neural network training services. The training framework is not released publicly, neither the training data (I don't use a mainstream deep learning framework like TF or Torch to train my models). However for this use-case maybe it would be worth it to switch to WebAR.rocks.face since there are more face expression detection features. You can check out this demo (included in the repo) for example: https://webar.rocks/demos/face/demos/expressionsDetection/ (by putting your cursor on the slider you can see which expression is detected). If you tell me for which project you want to use the library I can provide you a license for a cheap price or for free (I do that for people using the lib for educational purposes, cool experiments released publicly, young startups in pre-money stage, students - same for other WebAR.rocks libraries)

6 - it is not possible to do that with Jeeliz Facefilter since it only provides a 3D pose for the face, but you can do this with WebAR.rocks.face. You can check-out these demos (included in the github repo):

Sport makeup: https://webar.rocks/demos/face/demos/makeupSport/
Makup: https://webar.rocks/demos/face/demos/makeupTexture/
Autobones flexible mask: https://webar.rocks/demos/face/demos/autobones/

[marcusx2]

Thanks @xavierjs . I have one more question. Is it possible to use this library purely as an API ? I only need 2 things:

1. The video element being controlled by this library
2. The information about the face detection (rotation, position, etc)

I don't need the jeeliz canvas at all, I don't need the drawing part. Basically, I already have a canvas and I just want the library to control the video and feed me the information about the face detection so I can use it. What's the best way to go about this? Can you give me an example where the face detection is simply logged or something? Would I still need the jeelizResizer for this?

Thanks!

[marcusx2]

Hi @xavierjs , I have a couple more questions.

1- I noticed in some examples there's the antialias option for the init method. I can't find it documented anywhere. I was taking this file as a reference, but it seems like it wasn't updated in 3 years, and maybe there were api changes, etc. Where can I find the documentation of the up-to-date api? If there isn't, it would be nice if that .ts file was updated with the current api...is there anything else different other than the antialias option?
2- It looks like the init and resize functions return a boolean according to the interfaces file. What are the conditions for these functions return true or false? I can't find this documented.

Thank you for taking your time to answer me questions, very much appreciated.

[marcusx2]

Hi again @xavierjs . I have a quick question regarding the threejs helper. On line 239 it looks like you only assign the _gl to the context if you are not using 2 canvas

context: (_isSeparateThreeCanvas) ? null : _gl,

However the code uses _gl if you are using 2 canvas

if (_isSeparateThreeCanvas){
      const compile_shader = function(source, type, typeString) {
        const glShader = _gl.createShader(type);
        _gl.shaderSource(glShader, source);
        _gl.compileShader(glShader);
        if (!_gl.getShaderParameter(glShader, _gl.COMPILE_STATUS)) {
          alert("ERROR IN " + typeString + " SHADER: " + _gl.getShaderInfoLog(glShader));
          return null;
        }
        return glShader;
      };

      const glShaderVertex =   compile_shader(videoScreenVertexShaderSource, _gl.VERTEX_SHADER, 'VERTEX');
      const glShaderFragment = compile_shader(videoScreenFragmentShaderSource, _gl.FRAGMENT_SHADER, 'FRAGMENT');

      _glShpCopyCut = _gl.createProgram();
      _gl.attachShader(_glShpCopyCut, glShaderVertex);
      _gl.attachShader(_glShpCopyCut, glShaderFragment);

      _gl.linkProgram(_glShpCopyCut);
      const samplerVideo = _gl.getUniformLocation(_glShpCopyCut, 'samplerVideo');
      _glShpCopyCutVideoMatUniformPointer = _gl.getUniformLocation(_glShpCopyCut, 'videoTransformMat2');
      
      return;
    }

Is this right? Seems a bit confusing to me, because you don't assign it to the renderer context but then you use it anyways. Can you explain?

Thanks!

[marcusx2]

Hello again @xavierjs . Could you explain to me what the detectionHysteresis is? I understand the threshold variable, but what is the hysteresis for? I noticed that for threejs the value 0.02 is used, babylonjs uses 0.1. In the end hysteresis is simply subtracted from the threshold and compared with detected to determine if the face is visible or not, so I guess I could just use threshold, but I want to understand this. Thanks!

[marcusx2]

Hello again @xavierjs . Regarding the callbackReady, the spec.maxFacesDetected property. Is it always the same as the value you specify at init? Example

JEELIZFACEFILTER.init({
  canvasId: 'jeeFaceFilterCanvas',
  NNCPath: '../../../neuralNets/', // path to JSON neural network model (NN_DEFAULT.json by default)
  callbackReady: function(errCode, spec){
    if (errCode){
      console.log('AN ERROR HAPPENS. ERROR CODE =', errCode);
      return;
    }
    console.log(spec.maxFacesDetected);
  },
  maxFacesDetected: 1

});

Or does it have a different meaning? I tried it out, and if I specify 2 on init but only show one face, spec.maxFacesDetected also returns 2. So it looks to me that the output is always the same as my input.

[marcusx2]

Hello again @xavierjs . I have a couple more questions

1- Is there no way to change the maximum number of detected faces at runtime? I thought it would be on the miscellaneous methods section, but I didn't find it. For example in a part of the experience I want to detect just one face, in another 2 faces. What do I do?
2- Same as above but for the neural net. I suppose I have to call destroy and init again if I want to change the neural net and/or the maximum number of faces detected at runtime?
3- Can you explain what is the scale0factor and nScaleLevels in the scan settings ? I read the description but don't really understand. It says Relative width (1 -> full width) of the searching window at the largest scale level . What is "largest scale level" ? What scale? What level ? All I know is that it seems that if I set nScaleLevels to 1 it seems to find the face easier after the face is lost, or maybe it's just my impression.
4- There are lots of scan settings, stabilization settings, video settings, etc. Can you give me an overview of what settings I can tweak to make the experience faster, even if it means decreasing accuracy ? I'm playing around a bit to see what I find out, but telling me the ones that matter most would be helpful.
5- How do I make a neural net for this library? If I want to make my own expressions to detect my eyes closing and/or opening for example. Is there a tutorial somewhere about this? Or is the library only supposed to work with the neural nets provided (which is fine too, just wondering).
6- Is it possible to achieve a face paint effect like on SparkAR? It's like a mask that deforms with the face.

Thanks!

[marcusx2]

Also the questions above are my last, promise. Then I'll close the issue. Thank you for your time, and sorry for bothering you!

[marcusx2]

Ok thanks xavierjs, this has all been very enlightening. Thank you for your time and attention!

[xavierjs]

You are welcome :)