Add crop functionality to BrowserCodeReader

[zxing-js]

Add crop functionality to BrowserCodeReader

[bburns]

I got this working with React - it copies a crop of the video feed onto a canvas, then copies the canvas into an image element, then feeds that to the barcode reader...

Tested on MacOS Chrome so far -

// scanner video feed
// see https://github.com/zxing-js/library

import React from 'react'
import './styles.scss'
import { BrowserBarcodeReader } from '@zxing/library'

const timeout = 1000 // time between frames
const scale = 0.5 // size of crop frame

let barcodeReader
let videoStream

// user clicked on the camera button - bring up scanner window.
export async function openScanner(onFoundBarcode) {

  barcodeReader = new BrowserBarcodeReader()

  showScanner()

  // get html elements
  const video = document.querySelector('#scanner-video video')
  const canvas = document.querySelector('#scanner-canvas')
  const img = document.querySelector('#scanner-image')
  const frame = document.querySelector('#scanner-frame')

  // turn on the video stream
  const constraints = { video: true }
  navigator.mediaDevices.getUserMedia(constraints).then(stream => {
    videoStream = stream

    // handle play callback
    video.addEventListener('play', () => {
      // get video's intrinsic width and height, eg 640x480,
      // and set canvas to it to match.
      canvas.width = video.videoWidth
      canvas.height = video.videoHeight

      // set position of orange frame in video
      frame.style.width = video.clientWidth * scale + 'px'
      frame.style.height = video.clientHeight * scale + 'px'
      frame.style.left =
        (window.innerWidth - video.clientWidth * scale) / 2 + 'px'
      frame.style.top =
        (window.innerHeight - video.clientHeight * scale) / 2 + 'px'

      // start the barcode reader process
      scanFrame()
    })

    video.srcObject = stream
  })

  function scanFrame() {
    if (videoStream) {
      // copy the video stream image onto the canvas
      canvas.getContext('2d').drawImage(
        video,
        // source x, y, w, h:
        (video.videoWidth - video.videoWidth * scale) / 2,
        (video.videoHeight - video.videoHeight * scale) / 2,
        video.videoWidth * scale,
        video.videoHeight * scale,
        // dest x, y, w, h:
        0,
        0,
        canvas.width,
        canvas.height
      )
      // convert the canvas image to an image blob and stick it in an image element
      canvas.toBlob(blob => {
        const url = URL.createObjectURL(blob)
        // when the image is loaded, feed it to the barcode reader
        img.onload = async () => {
          barcodeReader
            // .decodeFromImage(img) // decodes but doesn't show img
            .decodeFromImage(null, url)
            .then(found) // calls onFoundBarcode with the barcode string
            .catch(notfound)
            .finally(releaseMemory)
          img.onload = null
          setTimeout(scanFrame, timeout) // repeat
        }
        img.src = url // load the image blob
      })
    }
  }

  function found(result) {
    onFoundBarcode(result.text)
    closeScanner()
  }

  function notfound(err) {
    if (err.name !== 'NotFoundException') {
      console.error(err)
    }
  }

  function releaseMemory() {
    URL.revokeObjectURL(img.url) // release image blob memory
    img.url = null
  }
}

export function closeScanner() {
  if (videoStream) {
    videoStream.getTracks().forEach(track => track.stop()) // stop webcam feed
    videoStream = null
  }
  hideScanner()
  barcodeReader.reset()
}

function showScanner() {
  document.querySelector('.scanner').classList.add('visible')
}

function hideScanner() {
  document.querySelector('.scanner').classList.remove('visible')
}

export default function Scanner() {
  return (
    <div className="scanner">
      <div id="scanner-video">
        <video autoPlay playsInline></video>
      </div>
      <div id="scanner-frame"></div>
      <canvas id="scanner-canvas"></canvas>
      <img id="scanner-image" src="" alt="" />
      <button id="scanner-close" onClick={closeScanner}>
        Close
      </button>
    </div>
  )
}

.scanner {
  height: 100%;
  display: none;
}
.scanner.visible {
  display: block;
}

#scanner-video {
  position: absolute;
  top: 0;
  left: 0;
  bottom: 0;
  right: 0;
  margin: auto;
  z-index: 1;
  background: rgba(0, 0, 0, 0.7);
  z-index: 1;
}

#scanner-video video {
  object-fit: initial;
  position: absolute;
  top: 0;
  left: 0;
  bottom: 0;
  right: 0;
  margin: auto;
  height: 100%;
  width: auto;
  z-index: 2;
  @media (orientation: portrait) {
    width: 100%;
    height: auto;
  }
}

#scanner-frame {
  position: absolute;
  margin: auto;
  box-shadow: 0 0 0 9999px rgba(0, 0, 0, 0.5);
  border: 2px solid orange;
  z-index: 3;
}

#scanner-canvas {
  display: none;
  width: 100%;
  margin: auto;
  z-index: 4;
}

#scanner-image {
  display: none;
  width: 100%;
  margin: auto;
  z-index: 10;
}

#scanner-close {
  position: absolute;
  top: 0;
  left: 0;
  z-index: 30;
  cursor: pointer;
}
#scanner-close:hover {
  background: #aaa;
}

[mpodlasin]

We make a crop mask with css. Here is a picture of our app:

Only stuff inside white rectangle is passed to algorithm. Since area is 4 times smaller, algorithm has much better results, than if we would pass whole picture. Scanning apps very often have masks like this.

If programmer could pass crop data to reader, he could use the same data to calculate and create mask of proper size.

The only other way to achieve this would be to write custom BrowserCodeReader, which we obviously would like to avoid.

[boudabza]

Hi
how can i use the drawing canvas in my app
thank you

[fletchsod-developer]

Thanks to a great example by bburns. I tweaked it to fit my employer's product website below. Few features changes below which can be done either way, which are

1. No longer select camera manually (no drop-down selection)
2. CSS tweaking to not depend on webpage's width/height but depend on the html video area
3. Automatically center when loading or resizing webpage (some devices have different with/height)
4. CSS drop-shadow stay inside the html video instead of whole webpage so the customer can click the cancel button in this case
5. Click on Camera icon to pop-up the modal showing video in it (Modal html/css not included on this GitHub post)
6. Trigger whatever JavaScript event when a barcode value is found & populated in the textbox (Example, OnChanged or OnClick, etc.)
7. Etc. Too many to list here

Tested & works for iPad Safari, iPad Chrome, iPhone Safar, Android Chrome & my developer Windows 10 workstation.

This one work w/ either QRCode or barcode (VIN vehicle barcode)

`
< html >
< head >
var zxingCameraScanner = new ZXingCameraScanner(true);
zxingCameraScanner.BarcodeLoader("btnVinScan", "txtVin", "OnChanged");
//zxingCameraScanner.QRCodeLoader("btnVinScan", "txtVin", "OnChanged");
< / head >
< body >

 <div class="form-group row required">
      <span CssClass="control-label col-sm-5" style="color:#ff0000;">VIN:</span>
          &nbsp;&nbsp;&nbsp;&nbsp;
          <span class="col-sm-7">
              <input type="text" id="txtVIN" class="form-control form-control-sm input-medium-width" onchanged"txtVinTextChanged" />
          </span>
          &nbsp;&nbsp;&nbsp;&nbsp;
          <span>
          <a id="btnVinScan" class="btn btn-secondary btn-sm" style="width:40px;display:none;" onclick="return false;">
              <i class="fas fa-camera"></i>
          </a>
      </span>
  </div>
 <div style="padding-top:20px;">
   <span id="lblMessage"></span>
 </div>

< / body >
< / html >
`

`
< style type="text/css" >
#zxing-scanner-box {
/display: block;
width: 100%;
margin: 0px auto;/
}
#zxing-video-wrapper {
position: relative;
/display: inline-block;/
display: block; /* Notice: This need to stay at "block", not "inline-block" so the video screen stay inside the modal box /
margin: 0px auto;
padding: 0px;
/background: rgba(0, 0, 0, 0.7);/
z-index: 2002;
/ https://stackoverflow.com/questions/51259396/how-to-limit-box-shadow-spread-not-more-than-parent-width-and-height /
overflow: hidden; / This prevent "box-shadow" in "#zxing-frame" from spilling over to the whole webpage /
}
#zxing-video-scan {
position: relative;
height: 100%;
width: 100%;
z-index: 2002;
}
#zxing-frame {
/ https://stackoverflow.com/questions/29650776/fit-div-container-to-video-size-fullscreen-as-well /
/ https://jsfiddle.net/xw17xbc9/5/ */
position: absolute;
top: 0px;
left: 0px;
box-shadow: 0 0 0 9999px rgba(0, 0, 0, 0.65);
/background-color: rgba(0, 0, 0, 0.5);/
border: 2px solid orange;
z-index: 2003;
}
#zxing-canvas {
display: none;
z-index: 2004;
}
#zxing-image {
display: none;
z-index: 2005;
}
#zxing-image-debug {
position: relative;
display: none;
margin: 0px auto;
margin-top: 20px;
padding: 0px;
/border: solid 1px #ff0000;/
border: 2px solid orange;
z-index: 2005;
}
< / style >

< script type="text/javascript" src="https://unpkg.com/@zxing/[email protected]" >< / script >
< script type="text/javascript" src="~/F_Shared/ZXingCameraScanner.js" >< / script >

< div class="modal fade" id="modalZXingCameraScanner" tabindex="-1" role="dialog" aria-labelledby="" data-backdrop="static" aria-hidden="true" style="z-index:2000;" >

            <div>
                <div id="zxing-scanner-box">
                    <!-- Do not move those html tags below, out of the nest "div" tag.  Due to x,y coordinate thingies for scanner to work -->
                    <div id="zxing-video-wrapper">
                        <video id="zxing-video-scan" autoPlay playsInline></video>
                        <div id="zxing-frame"></div>
                    </div>
                    <img id="zxing-image-debug" src="" alt="" />
                    <canvas id="zxing-canvas"></canvas>
                    <img id="zxing-image" src="" alt="" />
                </div>
            </div>

        </div>
         <div id="divButtons" class="modal-footer justify-content-center" runat="server">
            <button class="btn btn-outline-secondary" id="cancelZXingCameraScanner" onclick="return false;">Cancel</button>
        </div>
    </div>
</div>

< / div >
`

`
// (( Class ))
var ZXingCameraScanner = function (isDebug = false) {
//(( Private Member Variables ))...
const isMobileDeviceAllowed = (isMobilePhone || isMobileTablet || isLocalHost);
var browserCodeReader; // Global & internal pass-thru parameter, using both due to strange global javascript quirks.
const modalId = "modalZXingCameraScanner";
const cancelButtonId = "cancelZXingCameraScanner";
const timeout = 500; // time between frames
let scaleWidth = 0.0; // size of crop frame // Notice: Value range is from 0.0 to 1.0.
let scaleHeight = 0.0; // size of crop frame // Notice: Value range is from 0.0 to 1.0.
const htmlVideo = document.querySelector('#zxing-video-scan');
const htmlFrame = document.querySelector('#zxing-frame');
const htmlCanvas = document.querySelector('#zxing-canvas');
const htmlImage = document.querySelector('#zxing-image');
const htmlImageDebug = document.querySelector('#zxing-image-debug');
let htmlParentButtonClientId = undefined;
let htmlParentTextboxClientId = undefined;
let htmlParentTextboxCustomEventArg = undefined;

//(( Public Properties (Get/Set) ))...
// N/A

//(( Private Methods ))
var zxingGenericLoader = function () {
    var buttonEventType = isMobilePhone || isMobileTablet ? 'touchend' : 'click';
    var constraintFacingMode = isLocalHost ? "user" : "environment";
    var constraints = { audio: false, video: { facingMode: constraintFacingMode } };  // Filtering out audio doesnt seem to work.

    if (!browserCodeReader) {
        console.log("Unknown code format reader type");
        return;
    }

    zxingCameraIcon(isMobileDeviceAllowed);

    document.getElementById(cancelButtonId).addEventListener(buttonEventType, function () {
        zxingCancel();
        return false;
    });

    document.getElementById(htmlParentButtonClientId).addEventListener(buttonEventType, function () {
        if (!isMobileDeviceAllowed) {
            document.getElementById(htmlParentButtonClientId).style.display = "none";
            zxingCancel();
        }

        // facingMode --> "environment" mean rear facing camera on mobile device, "user" mean front facing camera on desktop or mobile device.
        navigator.mediaDevices.getUserMedia(constraints)
            .then((mediaStream) => {
                // AFAICT in Safari this only gets default devices until gUM is called :/
                navigator.mediaDevices.enumerateDevices()
                .then((inputDevices) => {
                    if (inputDevices.length === 0) {
                        zxingCameraIcon(htmlParentButtonClientId, false);
                        bootbox.alert("Error: Mobile device camera is not found (or does not exists)");
                        return;
                    }

                    xzingVideoReload(mediaStream);
                    showModal('#' + modalId);
                })
                .catch((error) => {
                    console.error(error);
                    bootbox.alert("Error (2): " + error);  // Mobile device doesn't have the console option for viewing.
                    zxingCancel();
                    return;
                });
            })
            .catch((error) => {
                console.error(error);
                bootbox.alert("Error (1): " + error);  // Mobile device doesn't have the console option for viewing.
                zxingCancel();
                return;
            });
    });

    window.addEventListener('load', (event) => {
        // TODO - Why does this not work?   Does it really work but is not showing?
        //console.log("Debug - On Load");
        if (!isMobileDeviceAllowed) {
            zxingCancel();
        }
    });
    window.addEventListener('resize', (event) => {
        zxingFrameCenter();
        zxingCanvasCenter();
    });
    window.addEventListener('scroll', (event) => {
        zxingFrameCenter();
        zxingCanvasCenter();
    });
    window.addEventListener('beforeunload', (event) => {
        zxingCancel();
    });
};

var zxingFrameCenter = function () {
    const frameScaleWidth = htmlVideo.clientWidth * scaleWidth;
    const frameScaleHeight = htmlVideo.clientHeight * scaleHeight;
    const frameScaleLeft = (htmlVideo.clientWidth - frameScaleWidth) / 2;
    const frameScaleTop = (htmlVideo.clientHeight - frameScaleHeight) / 2;   
    const canvasScaleWidth = htmlVideo.videoWidth * scaleWidth;
    const canvasScaleHeight = htmlVideo.videoHeight * scaleHeight;

    // get video's intrinsic width and height, eg 640 x 480 & set canvas to it to match.
    htmlCanvas.width = canvasScaleWidth;
    htmlCanvas.height = canvasScaleHeight;

    // set position of orange frame in video
    htmlFrame.style.width = frameScaleWidth + 'px';
    htmlFrame.style.height = frameScaleHeight + 'px';
    htmlFrame.style.left = frameScaleLeft + 'px';   // Tmp Ajust
    htmlFrame.style.top = frameScaleTop + 'px';   // Tmp Ajust

    if (isDebug) {
        htmlImageDebug.style.display = 'block';  /* Notice: This need to stay at "block", not "inline-block" so the image tag (or video screen) stay inside the modal box */
    }
};

var zxingCanvasCenter = function () {
    const width = htmlVideo.videoWidth; //htmlVideo.clientWidth < htmlVideo.videoWidth ? htmlVideo.videoWidth : htmlVideo.clientWidth;
    const height = htmlVideo.videoHeight; //htmlVideo.clientHeight < htmlVideo.videoHeight ? htmlVideo.videoHeight : htmlVideo.clientHeight;
    const canvasScaleWidth = htmlVideo.videoWidth * scaleWidth;
    const canvasScaleHeight = htmlVideo.videoHeight * scaleHeight;
    const frameScaleLeft = (width - canvasScaleWidth) / 2;
    const frameScaleTop = (height - canvasScaleHeight) / 2;

    htmlCanvas.getContext('2d').drawImage(
        htmlVideo,
        // source x, y, w, h:
        frameScaleLeft,
        frameScaleTop,
        canvasScaleWidth,
        canvasScaleHeight,
        // dest x, y, w, h:
        0,
        0,
        canvasScaleWidth,
        canvasScaleHeight
    );
};

var xzingVideoReload = function (mediaStream) {
    // handle play callback
    htmlVideo.addEventListener('play', () => {
        zxingFrameCenter();
        zxingFrameReload(); // start the barcode reader process
    });

    htmlVideo.srcObject = mediaStream;
};

var zxingFrameReload = function () {
    if (htmlVideo) {
        if (htmlVideo.srcObject) {
            zxingFrameCenter();
            zxingCanvasCenter();

            // convert the canvas image to an image blob and stick it in an image element
            htmlCanvas.toBlob(blob => {
                //#if (typeof (blob) !== 'object') {
                if (blob === null || blob === undefined) {
                    console.error("blob image is null, undefined or not an object");
                    return;
                }
                const url = URL.createObjectURL(blob);

                // when the image is loaded, feed it to the barcode reader
                htmlImage.onload = async () => {
                    browserCodeReader
                        // .decodeFromImage(img) // decodes but doesn't show img
                        .decodeFromImage(null, url)
                        .then((result) => {
                            if (result) {
                                var sourceTextbox = document.getElementById(htmlParentTextboxClientId);

                                //console.log(result);
                                sourceTextbox.value = result.text;

                                zxingCancel();
                            
                                try {
                                    if (htmlParentTextboxCustomEventArg.toLowerCase() === "onchanged") {
                                        // https://stackoverflow.com/questions/136617/how-do-i-programmatically-force-an-onchange-event-on-an-input
                                        sourceTextbox.dispatchEvent(new Event("change"));
                                    }
                                    else if (htmlParentTextboxCustomEventArg.toLowerCase() === "onclick") {
                                        document.getElementById(htmlParentButtonClientId).click();
                                    }
                                    else {
                                        console.error("Camera Scanner successfully scanned but ran into unknown javascript event - '" + htmlParentTextboxCustomEventArg + "'");
                                    }
                                }
                                catch (tmpError) {
                                    //console.error(tmpError);
                                    console.error("Camera Scanner successfully scanned but failed to trigger textbox activity");
                                }
                            }
                        }) 
                        .catch((error) => {
                            if (error) {
                                if (error instanceof ZXing.NotFoundException) {
                                    //console.log('No Barcode or QR Code found.');  // This is not an error cuz we haven't start scanning the barcode/qr-code yet.
                                }
                                else if (error instanceof ZXing.ChecksumException) {
                                    console.error('A code was found, but it\'s read value was not valid.');
                                }
                                else if (error instanceof ZXing.FormatException) {
                                    console.error('A code was found, but it was in a invalid format.');
                                }
                                else {
                                    bootbox.alert("Error (4)" + error);  // Mobile device doesn't have the console option for viewing.
                                    console.error("Error (4): " + error);
                                    zxingCancel();
                                }
                            }
                        })
                        .finally(() => {
                            URL.revokeObjectURL(htmlImage.url); // release image blob memory
                            htmlImage.url = null;
                        });
                    htmlImage.onload = null;
                    setTimeout(zxingFrameReload, timeout); // repeat
                };
                htmlImage.src = url; // load the image blob

                if (isDebug) {
                    htmlImageDebug.src = url;
                }
            });
        }
    }
};

var zxingCameraIcon = function (isVisible) {
    // iPad bug workaround.  (For some reasons, the "getVideoInputDevices()" isn't well supported on iPad.
    document.getElementById(htmlParentButtonClientId).style.display = isVisible && isMobileTablet ? "inline-block" : "none";

    browserCodeReader.getVideoInputDevices()
        .then((inputDevices) => {
            if (inputDevices.length > 0 && isVisible) {
                document.getElementById(htmlParentButtonClientId).style.display = "inline-block";
            }
            //else {
            //    document.getElementById(htmlParentButtonClientId).style.display = "none";
            //}
        })
        .catch((err) => {
            //console.error(err)
        });
};

var zxingCancel = function () {
    if (htmlVideo) {
        if (htmlVideo.srcObject) {
            htmlVideo.srcObject.getTracks().forEach(track => track.stop());  // stop webcam feed
            htmlVideo.srcObject = null;
        } 
    }
    if (browserCodeReader) {
        browserCodeReader.reset();
    }

    hideModal('#' + modalId);
};

//(( Public Methods ))
this.BarcodeLoader = function (parentButtonClientId, parentTextboxClientId, parentTextboxCustomEventArg) {
    browserCodeReader = new ZXing.BrowserBarcodeReader();  // MultiFormat allow both Code-39 & QR-Code format.
    scaleWidth = 0.9;  // size of crop frame  // Notice: Value range is from 0.0 to 1.0.
    scaleHeight = 0.15;  // size of crop frame  // Notice: Value range is from 0.0 to 1.0.
    htmlParentButtonClientId = parentButtonClientId;
    htmlParentTextboxClientId = parentTextboxClientId;
    htmlParentTextboxCustomEventArg = parentTextboxCustomEventArg;

    zxingGenericLoader();
};

this.QRCodeLoader = function (parentButtonClientId, parentTextboxClientId, parentTextboxCustomEventArg) {
    browserCodeReader = new ZXing.BrowserQRCodeReader();  // MultiFormat allow both Code-39 & QR-Code format.
    scaleWidth = 0.8;  // size of crop frame  // Notice: Value range is from 0.0 to 1.0.
    scaleHeight = 0.8;  // size of crop frame  // Notice: Value range is from 0.0 to 1.0.
    htmlParentButtonClientId = parentButtonClientId;
    htmlParentTextboxClientId = parentTextboxClientId;
    htmlParentTextboxCustomEventArg = parentTextboxCustomEventArg;

    zxingGenericLoader();
};

this.Cancel = function () {
    zxingCancel();
};

};

`

[odahcam]

So how would the user know what portion of the image his device is capturing will be scanned? Your ideia is to identify and crop just the code or just crop at some point of the image?

[mpodlasin]

Crop data is calculated based on width and height of video stream, so it would have to be a function:

type Crop = (videoWidth: number, videoHeight: number) => ({ scanHeight: number, scanWidth: number, xOffset: number, yOffset: number })

Here scanHeight and scanWidth would be the size of white rectangle and xOffset and yOffset would say where within whole video the rectangle should be placed.

It could be passed for example as optional argument to a BrowserCodeReader method.

In our case, where we want to have centered crop, with area 4 times smaller, this function would be:

const crop = (videoWidth: number, videoHeight: number) => {
  const scanHeight = videoHeight / 2;
  const scanWidth = videoWidth / 2;
  const xOffset = (videoHeight - scanHeight) / 2;
  const yOffset = (videoWidth - scanWidth) / 2;

  return { scanHeight, scanWidth, xOffset, yOffset };
}

[odahcam]

For the ngx-scanner we have a custom scanner class that overrides some methods. 😄 That's not that good, but also not that bad.

A important thing about the crop masks is that they aren't always just crops, but sometimes they're based on the camera focus and barcode size that should be read, to have something like a perfect size match when scanning.

So the crop would rely just on BrowserCodeReader?

I think that's not a good idea to add it to the core, but as long as the Browser will take care of the image and then pass it to the core to decode, that sounds nice.

[mpodlasin]

Well, if we would extract discussed part of code into some kind of protected method, so that we could extend it by ourselves, I would be ok with that as well. Then even API would not change.

The only risk is some day somebody forgets why this method is protected and just makes it private again, thus breaking our code.

[odahcam]

Well, I think that just this should be in another method:

https://github.com/mpodlasin/library/blob/49e3df942490c5fb1457575ef3a2ef58a9ac0832/src/browser/BrowserCodeReader.ts#L249-L259

I can accept the rest of the code without problems. Our code is not thaaaat good at all :( ...so we gonna have to change it in the future in a way or another and I just don't want to mess things up when that day comes.

[bestbelief]

@mpodlasin Hello, I'm writing this function now, but I don't know how to write it. Could you give me some of your code for reference? I know this request is very presumptuous, but I really don't know how to write it. I've just been working for a while and I hope you can help me. Anyway, thank you very much.

[odahcam]

Oh, I'm closing this because it was merged, but feel free to keep the conversation as it needs.

[odahcam]

Sorry, I could not understand. Please open a new issue.

[odahcam]

Very nice @bburns , thanks for sharing!

[fletchsod-developer]

Sorry about the weird post above, for some reasons, GitHib post doesn't agree w/ whatever symbol in the text & I couldn't fix it here. :-/