iOS compatibility

[nadav01]

iOS compatibility

[derm1ch1]

[Solved] -- Working code example at the end --

Hello everyone,
after I experienced exactly the same bug with Safari in iOS and got stuck for days as how I get it running on an iPhone, here is the solution:

• Problem: Webcam Canvas freezes after one frame or stays black.

• Solution tested on an iPhone 11 Pro, iPhone 8, iPad Pro 2, macOS Safari, Chrome, Firefox, Opera

• Cause: The camera video stream generated in JavaScript is not added to the HTML DOM.
  Or rather: It is added too late. For users who experience the same problem in a different application, here is a short explanation of the approach - the camera stream for iOS Safari must be set up this way:
  (it will run on all other devices and operating systems as well, this is not a compromise, but only an extension of the current approach)

1. initialize video stream object in JS: let webcam = new tmImage.webcam(1800, 800, flip);
2. add video object to the DOM: document.getElementById('webcam-container').appendChild(webcam.webcam);
3. set video attributes "muted" and "playsinline" (otherwise the autoplay will be stopped by iOS)
4. start webcam video: webcam.play();

In the example snippet you can copy on the TM-Page:
First the video is started and then the video object is added to the DOM. Also the two attributes are missing. Therefore Safari automatically blocks the video stream and blocks the "autoplay" of the stream. Result: Video is frozen or stays black.

Working code/ adaptation of the code snippet:

Since I've been looking for a solution for days and hate to read long explanations, tinker the solution together myself and then get error messages again - here the adapted version of the TM code snippet that makes the camera stream work:

Replace/adapt the init function with this:

async function init() {
        const modelURL = URL + 'model.json';
        const metadataURL = URL + 'metadata.json';

        // load the model and metadata
        // Refer to tmImage.loadFromFiles() in the API to support files from a file picker
        // or files from your local hard drive
        model = await tmImage.load(modelURL, metadataURL);
        maxPredictions = model.getTotalClasses();

        // convenience function to setup a webcam
        const flip = true; // whether to flip the webcam
        webcam = new tmImage.Webcam(1800, 800, flip); // width, height, flip
        await webcam.setup({ facingMode: "environment" }); // use "user" to use front-cam on mobile phones

        // append elements to the DOM --> **before starting the webcam**
        // document.getElementById('webcam-container').appendChild(webcam.canvas); // just in case you want to use specifically the canvas
        document.getElementById('webcam-container').appendChild(webcam.webcam); // webcam object needs to be added in any case to make this work on iOS

        // grab video-object in any way you want and set the attributes --> **"muted" and "playsinline"**
        let wc = document.getElementsByTagName('video')[0];
        wc.setAttribute("playsinline", true); // written with "setAttribute" bc. iOS buggs otherwise :-)
        wc.muted = "true"
        wc.id = "webcamVideo";

        // only now start the webcam --> **after video-object added to DOM and attributes are set**
        webcam.play();
        window.requestAnimationFrame(loop); // update canvas by loop-function
    }

All right you beautiful people out there - I hope I could help you! This approach has been working for me personally beautifully since intensive testing! Let me know if you have any questions!
I wish you a great day - best greetings from Germany!

Oh: @HalfdanJ or @irealva - if the solution works for others as well - maybe you can change the code in the Github snippet and on the website to make it work out-of-the-box? Thanks! :-)

[hapticdata]

Safari on iOS is supported just fine by the teachable machine model. It is only the provided snippets and teachable machine's site that does not currently support iOS.

If you create your own webcam in javascript for iOS you can still use that with your model.

[hapticdata]

hello, it wont work in Chrome + iOS, Apple won't allow it. It should work on iOS as long as you are in iOS 11+. If you are on iOS 11+ in Safari and this is still not working you may have denied access unintentionally.

[nadav01]

Using iOS 13.3 with Safari. I allowed the access to the camera as you may see in the pictures below but still can't see anything:

Thank you.

[appinteractive]

Same issue here, the generated code also does not work on ios despite allowing access. Any update on that issue?

[hapticdata]

the issue is based on our croppable-camera component some unique considerations to getUserMedia / webcam on iOS.

Your model and prediction will all work on iOS but for the time being, until i can complete a fix, you will need to code your own iOS webcam to get the images to send to your model.

[devosthomas]

Any update on this? I need Safari on iOS support quite urgent.

[lanewinfield]

@derm1ch1 Just happened to be looking for a solution to this at the exact same as you've solved it—but I'm getting an error seemingly in finding the video tag.

Unhandled Promise Rejection: TypeError: undefined is not an object (evaluating 'wc.setAttribute')

[derm1ch1]

@lanewinfield you're right! - I made a small but crucial mistake while writing the code example - I fixed my example above.

My mistake was: You don't want to add the webcam.canvas object to the DOM, but the webcam.webcam object, which is basically a video tag.

If you, for some specific reason, want to use the canvas-object of the webcam stream as well - you can do so. But it's important to add the video-object in any case.

In other blogs I've read that hiding the video-object with css results in the original "freeze/ black screen - bug". So it would be an option for you or maybe others to place the canvas-object above the video-object. Pretty dirty, but the only working solution I found for webcam.canvas .

Have a nice day and let me know if it works for you now :-)