iphone issue

[googlecreativelab]

iphone issue

[derm1ch1]

[Solved] -- Working code example at the end --

Hello everyone,
after I experienced exactly the same bug with Safari in iOS and got stuck for days as how I get it running on an iPhone, here is the solution:

• Problem: Webcam Canvas freezes after one frame or stays black.

• Solution tested on an iPhone 11 Pro, iPhone 8, iPad Pro 2, macOS Safari, Chrome, Firefox, Opera

• Cause: The camera video stream generated in JavaScript is not added to the HTML DOM.
  Or rather: It is added too late. For users who experience the same problem in a different application, here is a short explanation of the approach - the camera stream for iOS Safari must be set up this way:
  (it will run on all other devices and operating systems as well, this is not a compromise, but only an extension of the current approach)

1. initialize video stream object in JS: let webcam = new tmImage.webcam(1800, 800, flip);
2. add video object to the DOM: document.getElementById('webcam-container').appendChild(webcam.webcam);
3. set video attributes "muted" and "playsinline" (otherwise the autoplay will be stopped by iOS)
4. start webcam video: webcam.play();

In the example snippet you can copy on the TM-Page:
First the video is started and then the video object is added to the DOM. Also the two attributes are missing. Therefore Safari automatically blocks the video stream and blocks the "autoplay" of the stream. Result: Video is frozen or stays black.

Working code/ adaptation of the code snippet:

Since I've been looking for a solution for days and hate to read long explanations, tinker the solution together myself and then get error messages again - here the adapted version of the TM code snippet that makes the camera stream work:

Replace/adapt the init function with this:

async function init() {
        const modelURL = URL + 'model.json';
        const metadataURL = URL + 'metadata.json';

        // load the model and metadata
        // Refer to tmImage.loadFromFiles() in the API to support files from a file picker
        // or files from your local hard drive
        model = await tmImage.load(modelURL, metadataURL);
        maxPredictions = model.getTotalClasses();

        // convenience function to setup a webcam
        const flip = true; // whether to flip the webcam
        webcam = new tmImage.Webcam(1800, 800, flip); // width, height, flip
        await webcam.setup({ facingMode: "environment" }); // use "user" to use front-cam on mobile phones

        // append elements to the DOM --> **before starting the webcam**
        // document.getElementById('webcam-container').appendChild(webcam.canvas); // just in case you want to use specifically the canvas
        document.getElementById('webcam-container').appendChild(webcam.webcam); // webcam object needs to be added in any case to make this work on iOS

        // grab video-object in any way you want and set the attributes --> **"muted" and "playsinline"**
        let wc = document.getElementsByTagName('video')[0];
        wc.setAttribute("playsinline", true); // written with "setAttribute" bc. iOS buggs otherwise :-)
        wc.muted = "true"
        wc.id = "webcamVideo";

        // only now start the webcam --> **after video-object added to DOM and attributes are set**
        webcam.play();
        window.requestAnimationFrame(loop); // update canvas by loop-function
    }

All right you beautiful people out there - I hope I could help you! This approach has been working for me personally beautifully since intensive testing! Let me know if you have any questions!
I wish you a great day - best greetings from Germany!

Oh: @HalfdanJ or @irealva - if the solution works for others as well - maybe you can change the code in the Github snippet and on the website to make it work out-of-the-box? Thanks! :-)

[irealva]

Hi @ashique12009 can you provide a bit more detail?

• What browser are you using on iPhone? Can you try the Chrome browser for iOS?
• Are you using the code snippet we provided? Can you include your code?
• Which model are you running (image, pose)?
• Can you open the model using the directly link to the model you get once you export it? (this link is the one at https://teachablemachine.withgoogle.com/models/YOUR_MODEL_ID)

[thedb]

[Solved] -- Working code example at the end --

Hello everyone,
after I experienced exactly the same bug with Safari in iOS and got stuck for days as how I get it running on an iPhone, here is the solution:

• Problem: Webcam Canvas freezes after one frame or stays black.
• Solution tested on an iPhone 11 Pro, iPhone 8, iPad Pro 2, macOS Safari, Chrome, Firefox, Opera
• Cause: The camera video stream generated in JavaScript is not added to the HTML DOM.
  Or rather: It is added too late. For users who experience the same problem in a different application, here is a short explanation of the approach - the camera stream for iOS Safari must be set up this way:
  (it will run on all other devices and operating systems as well, this is not a compromise, but only an extension of the current approach)

1. initialize video stream object in JS: let webcam = new tmImage.webcam(1800, 800, flip);
2. add video object to the DOM: document.getElementById('webcam-container').appendChild(webcam.webcam);
3. set video attributes "muted" and "playsinline" (otherwise the autoplay will be stopped by iOS)
4. start webcam video: webcam.play();

In the example snippet you can copy on the TM-Page:
First the video is started and then the video object is added to the DOM. Also the two attributes are missing. Therefore Safari automatically blocks the video stream and blocks the "autoplay" of the stream. Result: Video is frozen or stays black.

Working code/ adaptation of the code snippet:

Since I've been looking for a solution for days and hate to read long explanations, tinker the solution together myself and then get error messages again - here the adapted version of the TM code snippet that makes the camera stream work:

Replace/adapt the init function with this:

async function init() {
        const modelURL = URL + 'model.json';
        const metadataURL = URL + 'metadata.json';

        // load the model and metadata
        // Refer to tmImage.loadFromFiles() in the API to support files from a file picker
        // or files from your local hard drive
        model = await tmImage.load(modelURL, metadataURL);
        maxPredictions = model.getTotalClasses();

        // convenience function to setup a webcam
        const flip = true; // whether to flip the webcam
        webcam = new tmImage.Webcam(1800, 800, flip); // width, height, flip
        await webcam.setup({ facingMode: "environment" }); // use "user" to use front-cam on mobile phones

        // append elements to the DOM --> **before starting the webcam**
        // document.getElementById('webcam-container').appendChild(webcam.canvas); // just in case you want to use specifically the canvas
        document.getElementById('webcam-container').appendChild(webcam.webcam); // webcam object needs to be added in any case to make this work on iOS

        // grab video-object in any way you want and set the attributes --> **"muted" and "playsinline"**
        let wc = document.getElementsByTagName('video')[0];
        wc.setAttribute("playsinline", true); // written with "setAttribute" bc. iOS buggs otherwise :-)
        wc.muted = "true"
        wc.id = "webcamVideo";

        // only now start the webcam --> **after video-object added to DOM and attributes are set**
        webcam.play();
        window.requestAnimationFrame(loop); // update canvas by loop-function
    }

All right you beautiful people out there - I hope I could help you! This approach has been working for me personally beautifully since intensive testing! Let me know if you have any questions!
I wish you a great day - best greetings from Germany!

Oh: @HalfdanJ or @irealva - if the solution works for others as well - maybe you can change the code in the Github snippet and on the website to make it work out-of-the-box? Thanks! :-)

[Solved] -- Working code example at the end --

Hello everyone,
after I experienced exactly the same bug with Safari in iOS and got stuck for days as how I get it running on an iPhone, here is the solution:

• Problem: Webcam Canvas freezes after one frame or stays black.
• Solution tested on an iPhone 11 Pro, iPhone 8, iPad Pro 2, macOS Safari, Chrome, Firefox, Opera
• Cause: The camera video stream generated in JavaScript is not added to the HTML DOM.
  Or rather: It is added too late. For users who experience the same problem in a different application, here is a short explanation of the approach - the camera stream for iOS Safari must be set up this way:
  (it will run on all other devices and operating systems as well, this is not a compromise, but only an extension of the current approach)

1. initialize video stream object in JS: let webcam = new tmImage.webcam(1800, 800, flip);
2. add video object to the DOM: document.getElementById('webcam-container').appendChild(webcam.webcam);
3. set video attributes "muted" and "playsinline" (otherwise the autoplay will be stopped by iOS)
4. start webcam video: webcam.play();

In the example snippet you can copy on the TM-Page:
First the video is started and then the video object is added to the DOM. Also the two attributes are missing. Therefore Safari automatically blocks the video stream and blocks the "autoplay" of the stream. Result: Video is frozen or stays black.

Working code/ adaptation of the code snippet:

Since I've been looking for a solution for days and hate to read long explanations, tinker the solution together myself and then get error messages again - here the adapted version of the TM code snippet that makes the camera stream work:

Replace/adapt the init function with this:

async function init() {
        const modelURL = URL + 'model.json';
        const metadataURL = URL + 'metadata.json';

        // load the model and metadata
        // Refer to tmImage.loadFromFiles() in the API to support files from a file picker
        // or files from your local hard drive
        model = await tmImage.load(modelURL, metadataURL);
        maxPredictions = model.getTotalClasses();

        // convenience function to setup a webcam
        const flip = true; // whether to flip the webcam
        webcam = new tmImage.Webcam(1800, 800, flip); // width, height, flip
        await webcam.setup({ facingMode: "environment" }); // use "user" to use front-cam on mobile phones

        // append elements to the DOM --> **before starting the webcam**
        // document.getElementById('webcam-container').appendChild(webcam.canvas); // just in case you want to use specifically the canvas
        document.getElementById('webcam-container').appendChild(webcam.webcam); // webcam object needs to be added in any case to make this work on iOS

        // grab video-object in any way you want and set the attributes --> **"muted" and "playsinline"**
        let wc = document.getElementsByTagName('video')[0];
        wc.setAttribute("playsinline", true); // written with "setAttribute" bc. iOS buggs otherwise :-)
        wc.muted = "true"
        wc.id = "webcamVideo";

        // only now start the webcam --> **after video-object added to DOM and attributes are set**
        webcam.play();
        window.requestAnimationFrame(loop); // update canvas by loop-function
    }

All right you beautiful people out there - I hope I could help you! This approach has been working for me personally beautifully since intensive testing! Let me know if you have any questions!
I wish you a great day - best greetings from Germany!

Oh: @HalfdanJ or @irealva - if the solution works for others as well - maybe you can change the code in the Github snippet and on the website to make it work out-of-the-box? Thanks! :-)

@derm1ch1 Thx for your solution. Now，i can use this demo for my iphone safari. But i still cant run it with chrome.
The error message is

I dont know why chrome can not use WebGL2

My iphone version is 14.2

[derm1ch1]

Hey @thedb - that's odd. It works perfect for me in chrome.
Can you give me some more information about your setup/your browser version and you operating system?

I'm using chrome 87.0.4280.67 on macOS 10.15.6.

But in any case I don't think this has something to do with my suggested Safari-iPhone fix - but glad it helped! :-)

[thedb]

@derm1ch1
It's my fault, I didn't describe it clearly
This error is on the chrome of the iphone. And I know the limitation of iphone on webRTC. Thanks for your solution, I have already create pr too : )