"Open failed" on client.connect in iOS Telerik WebView [Cordova]

"Open failed" on client.connect in iOS Telerik WebView [Cordova]

I took the twilio-video.min.js and made the logger actually log to console (was an empty function):

2016-11-18 16:32:30.411 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | configuration parameters after validation: 2016-11-18 16:32:30.411 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · viaHost: "rvpffstgvelt.invalid" 2016-11-18 16:32:30.411 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · uri: sip:[e-mail] 2016-11-18 16:32:30.412 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · wsServers: [{"ws_uri":"wss://AC86a124ac4f3a1675b3bfe5385af4c820.endpoint.twilio.com","sip_uri":"<sip:AC86a124ac4f3a1675b3bfe5385af4c820.endpoint.twilio.com;transport=ws;lr>","weight":0,"status":0,"scheme":"WSS"}] 2016-11-18 16:32:30.412 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · password: NOT SHOWN 2016-11-18 16:32:30.413 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · registerExpires: 600 2016-11-18 16:32:30.415 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · register: false 2016-11-18 16:32:30.415 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · registrarServer: sip:ac86a124ac4f3a1675b3bfe5385af4c820.endpoint.twilio.com 2016-11-18 16:32:30.416 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · wsServerMaxReconnection: 3 2016-11-18 16:32:30.416 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · wsServerReconnectionTimeout: 4 2016-11-18 16:32:30.417 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · connectionRecoveryMinInterval: 2 2016-11-18 16:32:30.417 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · connectionRecoveryMaxInterval: 30 2016-11-18 16:32:30.417 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · keepAliveInterval: 30 2016-11-18 16:32:30.418 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · extraSupported: ["room-signaling","timer"] 2016-11-18 16:32:30.421 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · usePreloadedRoute: false 2016-11-18 16:32:30.422 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · userAgentString: "SIP.js/0.7.5" 2016-11-18 16:32:30.422 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · iceCheckingTimeout: 5000 2016-11-18 16:32:30.423 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · noAnswerTimeout: 60000 2016-11-18 16:32:30.423 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · stunServers: ["stun:stun.l.google.com:19302"] 2016-11-18 16:32:30.426 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · turnServers: [] 2016-11-18 16:32:30.426 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · traceSip: true 2016-11-18 16:32:30.427 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · hackViaTcp: false 2016-11-18 16:32:30.434 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · hackIpInContact: false 2016-11-18 16:32:30.434 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · hackWssInTransport: true 2016-11-18 16:32:30.435 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · hackAllowUnregisteredOptionTags: true 2016-11-18 16:32:30.435 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · contactTransport: "wss" 2016-11-18 16:32:30.435 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · forceRport: false 2016-11-18 16:32:30.435 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · autostart: false 2016-11-18 16:32:30.436 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · rel100: "none" 2016-11-18 16:32:30.438 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · replaces: "none" 2016-11-18 16:32:30.441 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · mediaHandlerFactory: function r(){function r(e){var r=i[e];if(r.length>1){var n="getDescription"===e;i[e]=t.Utils.promisify(i,e,n);}}var i=e.apply(this,arguments);return r("getDescription"),r("setDescription"),i;} 2016-11-18 16:32:30.442 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · authenticationFactory: undefined 2016-11-18 16:32:30.442 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · instanceId: "6c6ca3c0-f68d-4fbd-82b2-084ac248d775" 2016-11-18 16:32:30.442 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · sipjsId: "pnc3f" 2016-11-18 16:32:30.443 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · hostportParams: "AC86a124ac4f3a1675b3bfe5385af4c820.endpoint.twilio.com" 2016-11-18 16:32:30.443 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · authorizationUser: "0101123456_patient" 2016-11-18 16:32:30.446 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | · media: undefined 2016-11-18 16:32:30.451 Virtual Lab Testing[1242:751194] iosrtcPlugin#getUserMedia() 2016-11-18 16:32:30.451 Virtual Lab Testing[1242:751194] PluginGetUserMedia#call() 2016-11-18 16:32:30.451 Virtual Lab Testing[1242:751194] PluginGetUserMedia#call() | audio authorization: authorized 2016-11-18 16:32:30.452 Virtual Lab Testing[1242:751194] PluginGetUserMedia#call() | audio requested 2016-11-18 16:32:30.453 Virtual Lab Testing[1242:751194] PluginMediaStream#init() 2016-11-18 16:32:30.453 Virtual Lab Testing[1242:751194] PluginMediaStreamTrack#init() 2016-11-18 16:32:30.453 Virtual Lab Testing[1242:751194] PluginMediaStreamTrack#run() [kind:audio, id:B9FDE78F-E9BE-4C4C-B71E-EA1B696637CC] 2016-11-18 16:32:30.454 Virtual Lab Testing[1242:751194] PluginMediaStream#run() 2016-11-18 16:32:30.472 Virtual Lab Testing[1242:751194] iosrtcPlugin#MediaStreamTrack_setListener() 2016-11-18 16:32:30.474 Virtual Lab Testing[1242:751502] PluginMediaStreamTrack#setListener() [kind:audio, id:B9FDE78F-E9BE-4C4C-B71E-EA1B696637CC] 2016-11-18 16:32:30.479 Virtual Lab Testing[1242:751194] iosrtcPlugin#MediaStream_setListener() 2016-11-18 16:32:30.479 Virtual Lab Testing[1242:751335] PluginMediaStream#setListener() 2016-11-18 16:32:30.479 Virtual Lab Testing[1242:751194] Added a local microphone 2016-11-18 16:32:30.480 Virtual Lab Testing[1242:751194] iosrtcPlugin#getUserMedia() 2016-11-18 16:32:30.480 Virtual Lab Testing[1242:751194] PluginGetUserMedia#call() 2016-11-18 16:32:30.480 Virtual Lab Testing[1242:751194] PluginGetUserMedia#call() | video authorization: authorized 2016-11-18 16:32:30.481 Virtual Lab Testing[1242:751194] PluginGetUserMedia#call() | video requested (device not specified) 2016-11-18 16:32:30.481 Virtual Lab Testing[1242:751194] PluginGetUserMedia#call() | chosen video device: <AVCaptureFigVideoDevice: 0x128a11a90 [Front Camera][com.apple.avfoundation.avcapturedevice.built-in_video:1]> 2016-11-18 16:32:30.663 Virtual Lab Testing[1242:751194] PluginMediaStream#init() 2016-11-18 16:32:30.663 Virtual Lab Testing[1242:751194] PluginMediaStreamTrack#init() 2016-11-18 16:32:30.664 Virtual Lab Testing[1242:751194] PluginMediaStreamTrack#run() [kind:video, id:5F4BCD9B-8EFD-462E-9640-42C8A90FBE4B] 2016-11-18 16:32:30.664 Virtual Lab Testing[1242:751194] PluginMediaStream#run() 2016-11-18 16:32:30.664 Virtual Lab Testing[1242:751194] THREAD WARNING: ['iosrtcPlugin'] took '184.450928' ms. Plugin should use a background thread. 2016-11-18 16:32:30.666 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.ua | user requested startup... 2016-11-18 16:32:30.666 Virtual Lab Testing[1242:751194] Fri Nov 18 2016 16:32:30 GMT-0500 (EST) | sip.transport | connecting to WebSocket wss://AC86a124ac4f3a1675b3bfe5385af4c820.endpoint.twilio.com 2016-11-18 16:32:30.666 Virtual Lab Testing[1242:751194] ERROR: Failed to connect to the Room Error: Open failed

For anyone who isn't involved in our support ticket, we've been in contact with Twilio support (Jozsef Vass specifically) and were able to shim websockets on iOS (which requires that webviews are always Safari-based) to get past this error. We then had an issue with what we think is related to Web Workers on Safari not handling anything other than plain strings as Web Worker arguments very well.

At this point, we are moving toward authoring a Cordova plugin using the native Android and iOS Twilio SDKs and will be closing this ticket. Thank you all for the correspondence.

[markandrus]

Hi @pritchard,

Thanks for writing in. We haven't had an opportunity to start exploring frameworks like Cordova, but we are very interested in doing so come the New Year, so your initial explorations are valuable.

At this point, we are moving toward authoring a Cordova plugin using the native Android and iOS Twilio SDKs

Interesting—I look forward to seeing this develop.

Best,
Mark

@markandrus We did get native cordova Twilio Video plugins working for both Android and iOS. Did not end up using the twilio-video.js due to roadblocks.

I'm going to see if I have a little time after this project cools down to create a basic GitHub repo for the work we did. It's going to be a little messy, and our build process is project-specific, but hopefully others will find it valuable.

[drbeermann]

I've been able to hack through most of twilio-video.js to get things working on iOS with the iosrtc plugin. There are a bunch of kludges required, but iOS is now sending video and I'm receiving it with another peer. iOS can't display the remote participants media though. I think there are some conflicts with the cordova-plugin-ios.js functions that are defined still, I'm just hitting a limit of what I can modify to try to force this through.

It'd be great if you could share anything about the native plugins you've been able to develop. I had assumed that this would be a fairly quick process on Cordova, but I just can't get Twilio and the iosrtc plugin to cooperate.

Now that we've moved past the JavaScript approach, my above notes are the best record of what the issues were. Going native, we now have issues that both Android and iOS require us to manually modify build configuration settings every time we want to make changes to the application, due to the Twilio Video framework or Swift libraries requiring those additional steps because Cordova does not handle this scenario out of the box very well. So we build, the build fails, we modify settings, and then we can run the project... It's very cumbersome and not something we've had time to prepare for public review. (At the very least, it's painful just to compile the manual steps I have to perform into clearly written instructions.) Without the appropriate time and a developer resources to help me get past this point and write the Cordova hooks necessary to automate the process, the best I can do for now is correspond with you on where we are. Let me know if you want to schedule some time for us to have a deeper review together, hopefully to everyone's long-term benefit... I suppose I should consider recording a screen sharing session as an alternative to written instructions... hm...

…

On Mon, Jan 30, 2017 at 7:44 PM, Dale Beermann ***@***.***> wrote: I've been able to hack through most of twilio-video.js to get things working on iOS with the iosrtc plugin. There are a bunch of kludges required, but iOS is now sending video and I'm receiving it with another peer. iOS can't display the remote participants media though. I think there are some conflicts with the cordova-plugin-ios.js functions that are defined still, I'm just hitting a limit of what I can modify to try to force this through. It'd be great if you could share anything about the native plugins you've been able to develop. I had assumed that this would be a fairly quick process on Cordova, but I just can't get Twilio and the iosrtc plugin to cooperate. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#30 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABJ0Zb1ZeHo-ebYNuzFDXUTDFvcX6Xbyks5rXoPcgaJpZM4K26kl> .

[drbeermann]

Thanks for the response @pritchard. I've assumed that wrapping the native plugin would introduce a lot more complexity as well. I think that Twilio would be the defacto standard for implementation of webrtc on hybrid apps if this would actually work. I got pretty close, as referenced in the above issue, but I couldn't get past the last step and wasn't sure how deep I would have to keep going.

[CyborgMaster]

Just to throw my hat in the ring as well...

I'm also working on getting Twilio up and running on iOS inside Cordova. I haven't gotten very far yet and am running into the same issues that you guys listed above. I'll report on any progress I have here.

@drbeermann We hit a point where the issues were with Safari itself as well as how twilio-video or iosrtc were written in that they passed more than simple strings to web workers (we think, hard to debug).

ditto to @CyborgMaster To be clear and reiterate, we have succeeded in going the native plugin approach, and that is what we are using in Production... it's just cumbersome to then make changes and rebuild the app as a whole due to Cordova's inability to automate importing the additional build dependencies out of the box. We have to write a "hook" to do this and have not gotten around to it.

I would have liked to use the web-based approach as well, of course, so if anyone makes progress on that front we would certainly be happy to know! :)

[CyborgMaster]

@pritchard, is there any way we could get access to your plugin? If we could, maybe we could help get it the rest of the way to automated quality and help improve the situation for the community. I would love to get to the point where Twilio video is a simple include-and-go for Cordova apps.

[CyborgMaster]

Ok. I'm a little stumped. I'm not sure what it is, but I've poked and prodded a bit and for some reason the Twilio video js library and ios-rtc do not like each other. At this point I think our best bet is to go down the same path as @pritchard and use the native SDK's as well. I've never written a cordova plugin before, so if I could get access to your stuff @pritchard, no matter how raw, it would help a TON. I can try and figure out the build problems. And if I'm successful, I'll make sure that anything I discover will be shared.

@CyborgMaster There may be some delay on my end... Have been asking about the best way to share these efforts, but I'm finding the plugin hard to de-couple from our Production code, which is under an NDA on our end. Let me figure that out and see what I can come up with...

[CyborgMaster]

@pritchard, no worries. I'm 90% sure that this is the route our company is going to need to go down, so I'll be building it here soon. It would be great to leverage the work you guys have done that way we can both benefit from any advances, but I can start from scratch if I need to.

[drbeermann]

I'm swamped for the next week or so, but this will likely be the direction we go as well given the issues with iosrtc and Twilio. I'd be happy to work on a shared repo as we have time to look into this as well. It isn't ideal though. My worry is that there's a decent amount of complexity that is handled by the twilio js library. @pritchard even if you had some high level notes about any stumbling points you came across when building the plugin, it could be helpful.

@drbeermann @CyborgMaster I'll try to provide some high level notes for the native approach over the weekend.

[CyborgMaster]

Awesome. That would be super helpful. Thanks!

[jskrzypek]

Any update on this? I also think this would be the best approach. I started looking at this on my own, but I'm new to native dev in objective-C or swift and quickly got swamped without a clear picture of where to start.

I figure one place to start is with the iOS quick-starts that twilio provides.

@CyborgMaster @drbeermann If you guys are working on this I'd love to contribute as we definitely could use this solution.

[drbeermann]

Sorry for the radio silence, we've been swamped with a release. I'm actually going to be looking at this this week. We have an iOS dev resource that can help create an open source plugin that we'll make available. If anyone else has made some progress, it would be good to know before we get started.

[jskrzypek]

Fwiw I'm planning to sit with an iOS dev friend of mine this coming week to hack together a Cordova plugin based on the ios quickstart apps. I described the scope of the issue and he expects it to be pretty straightforward, but we'll see. I'll update when i know more.

…

On Sat, Mar 4, 2017, 20:44 Dale Beermann ***@***.***> wrote: Sorry for the radio silence, we've been swamped with a release. I'm actually going to be looking at this this week. We have an iOS dev resource that can help create an open source plugin that we'll make available. If anyone else has made some progress, it would be good to know before we get started. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#30 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABcXMbNOcYVEgf050P_DbUHmZ6RpkBVzks5ribEigaJpZM4K26kl> .

[Sachanski]

Hello everyone,

I have been tackling with the same issue for several days and managed to setup two clients (Android/iOS, Android/Browser and iOS/Browser) to communicate with each other with all four streams visible in each of the aforementioned cases. The changes I had to make in the twilio library were:

• after capturing the LocalMedia I pass the same object to Twilio Client's connect function and use it instead of calling getLocalMedia, which in my case was causing the local stream on the iOS device to freeze.
• changed the Track.prototype._attach function to call _MediaStream.create(this.mediaStream) instead of new _MediaStream() (this was suggested on some other place as well), since it was causing an error inside the iosrtc cordova plugin.
• inside the PeerConnectionV2 function I copy-pasted the stream events logic from the ChromeRTCPeerConnection function. Namely the code starting from if (!('ontrack' in webkitRTCPeerConnection.prototype)). I needed to change peerConnection.addEventListener('addstream', function ... to peerConnection.onaddstream = function ...
• Inside the shims I added a condition for the iosrtc plugin. In case it is true I export the functions from it (RTCIceCandidate, RTCPeerConnection and RTCSessionDescription)

I am reluctant to submit a pull request as the changes I made are specific to the iosrtc cordova plugin (although everything seems to be working on a browser as well, but is not thoroughly tested), however if anyone has any suggestions I can submit the diff.

[jskrzypek]

@Sachanski personally I think I might prefer to use the twilio-video native sdk for ios, but it may be instructive for me & other users if you can put your version up on github in a fork, even if you don't make a pr.

[CyborgMaster]

@Sachanski, I agree with @jskrzypek. I would love to see your patches. The native plugin is attractive to me, but I would definitely go with the JS version if we can get it to work. Not having to native code is a big win for us.

[Sachanski]

I have branched from the tag 1.0.0-beta4 (since this is currently what is fetched from the CDN) and committed the changes in this fork in branch twiliovideo-iosrtc. Please bear with me if there is something wrong, as this my first public "pull request".

[jskrzypek]

So for what it's worth my friend @igor-makarov and I started working on a plugin for cordova that follows the objc quick start

It's far from production-ready. We're going to see if we can switch over to using native views whose layout is handled by the js app instead of the storyboard, but we haven't had time to implement that yet.

I also plan to see if I can get the same kind of thing working with the native android sdk as well.

You can find it here for now: https://github.com/igor-makarov/cordova-plugin-twilio-video-ios

[drbeermann]

Just wanted to follow up here. We have a working iOS plugin as well. We had Android working on a build with Crosswalk using the javascript library but when I tried to get things running on my Google Pixel, I'm having a bunch of issues (non-crosswalk build). Just to clarify, do people have the javascript library working in Android? Did you have to do anything in particular to get it up and running? I'm wondering if I need to use the Android SDK for the cordova plugin as well.

@drbeermann I know I am incredibly late to respond to this. Unfortunately, the project I was contracted to work on failed to renew a budget right around the time we ran into this video plugin issue. Otherwise, it would have been our top priority at the time.

There were other major configuration issues with Cordova as well that we ran into on several occasions, and as a result, we have switched over to React Native for all future projects (until the next big thing comes around, of course).

Following up just to know if you and Pacifica have made any progress in this realm in general (video conferencing). Was reminded of this matter by a friend of mine suggested I download Pacifica, and I knew right away it was familiar.

[drbeermann]

@pritchard No worries. We actually have a fully working implementation right now that's logged about 12,000 minutes of use with just about zero bugs, so things seem to be working fairly well. There are a bunch of things we'd like to improve but it's getting the job done at the moment.

We are using the Javascript library with Android and the SDK with a custom Cordova plugin on iOS. I don't remember what the issue was that I had with my Pixel sadly, but I must have fixed it :)

[enraiser]

Hi @Sachanski I tried to use your JS file but it fails like this.
track.on is not a function. (In 'track.on(trackEvent, trackEventReemitters.get(trackEvent))', 'track.on' is undefined)