JS handLandmarker is cannot work, empy arrays

[JunzhuoZhang]

JS handLandmarker is cannot work, empy arrays

[ayushgdev]

Hello @JunzhuoZhang
The code excerpt you have provided seems OK, however, it is only preliminary analysis considering the excerpt only contains few lines of code. What can be the issue is that requestAnimationFrame() is not included in the code. So, when the app might be started, the very first frame might not have a hand and it gives blank array.
Subsequent frames may have a hand once you lift it up, but, the subsequent frames were not being requested to be processed by the detector if requestAnimationFrame() is not being called at the end of detection after canvasContext.restore() is called.

For a demo purpose, please check out this minimal code

[JunzhuoZhang]

Hi, this is my predictWebCam code, I still get empty array:

predictWebcam enter {"landmarks":[],"worldLandmarks":[],"handednesses":[],"handedness":[]}"

async predictWebcam() { // Now let's start detecting the stream. let nowInMs = performance.now(); const video: HTMLVideoElement = document.getElementById('myCamera') as HTMLVideoElement; // this.lastVideoTime = this.mediaView.currentTime; VCLogger.info(predictWebcam sec ${this.mediaView}`, this.tag);
if (video.currentTime > 0 && video.currentTime !== this.lastVideoTime) {
this.lastVideoTime = video.currentTime;
this.results = this.handLandmarker.detectForVideo(video, nowInMs);
}

    VCLogger.info(`predictWebcam enter ${JSON.stringify(this.results)}`, this.tag);

    this.canvasCtx.save();
    this.canvasCtx.clearRect(0, 0, 1920, 1080);
    this.canvasElement.style.width = video.videoWidth.toString();
    this.canvasElement.style.height = video.videoHeight.toString();
    this.canvasElement.width = video.videoWidth;
    this.canvasElement.height = video.videoHeight;

    const drawingUtils = new DrawingUtils(this.canvasCtx);
    if (this.results.landmarks) {
        for (const landmarks of this.results.landmarks) {
            drawingUtils.drawConnectors(landmarks, HandLandmarker.HAND_CONNECTIONS, {
                color: '#00FF00',
                lineWidth: 5
            });
            drawingUtils.drawLandmarks(landmarks, { color: '#FF0000', lineWidth: 2 });
        }
    }
    this.canvasCtx.restore();
    // Call this function again to keep predicting when the browser is ready.
    window.requestAnimationFrame(this.predictWebcam);
}

`

[ayushgdev]

Okay so there were few changes that needed to be done in the code provided:

1. Since the example app provided didn't use any framework, use of this was not required. Hence removed this keyword. This is only for the example app. Appears you are using a framework so no issues.
2. if (video.currentTime > 0 && video.currentTime !== this.lastVideoTime) { This condition prohibits the results variable to be populated on the first time. Change it to compare only if (video.currentTime !== this.lastVideoTime) {
3. drawConnectors and drawLandmarks functions take CanvasCtx as the first argument, not the landmarks.
4. HAND_CONNECTIONS is not available in HandLandmarker. It comes from @mediapipe/hands

Modified function for your code:

async function predictWebcam() { // Now let's start detecting the stream. 
  let nowInMs = performance.now(); 
  const video: HTMLVideoElement = document.getElementById('webcam') as HTMLVideoElement; 
  // this.lastVideoTime = this.mediaView.currentTime; VCLogger.info(predictWebcam sec ${this.mediaView}`, this.tag);
  if (video.currentTime !== lastVideoTime) {
    lastVideoTime = video.currentTime;
    results = handLandmarker.detectForVideo(video, nowInMs);
  }
    VCLogger.info(`predictWebcam enter ${JSON.stringify(this.results)}`, tag);

      canvasCtx.save();
      canvasCtx.clearRect(0, 0, 1920, 1080);
      canvasElement.style.width = video.videoWidth.toString();
      canvasElement.style.height = video.videoHeight.toString();
      canvasElement.width = video.videoWidth;
      canvasElement.height = video.videoHeight;
  
      // const drawingUtils = new DrawingUtils(canvasCtx);
      if (results.landmarks) {
          for (const landmarks of results.landmarks) {
              drawConnectors(canvasCtx, landmarks,HAND_CONNECTIONS, {
                  color: '#00FF00',
                  lineWidth: 5
              });
              drawLandmarks(canvasCtx, landmarks, { color: '#FF0000', lineWidth: 2 });
          }
      }
      canvasCtx.restore();
      // Call function again to keep predicting when the browser is ready.
      window.requestAnimationFrame(predictWebcam);
  }