opentok/opentok-network-test

Sample app to test network connectivity and statistics (bps, packet-lost)

OpenTok Network Test

This repository contains sample code that shows how to diagnose if the client's call (publishing a stream to an OpenTok session) will be successful or not, given their network conditions. The network test can be implemented as a step the client runs before joining the session. Based on the test results, the app can decide if the client should be allowed to publish a stream to the session and whether that stream should publish video or use audio-only mode. The test is intended to be used in a session that connects two clients in a one-to-one call.

The network test is supported in:

• OpenTok Android SDK 2.12 and newer
• OpenTok iOS SDK 2.12 and newer

JavaScript clients should use the sample code found here: https://github.com/opentok/opentok-network-test-js.

How does it work

The sample apps each do the following:

1. Connect to an OpenTok session and publish a test stream to a test session.

   Note that the published test stream would be visible to all clients connected to the session. For this reason, you should use a separate test session (with a unique session ID) for the network test. Do not use the test session for your actual call. Use a separate OpenTok session (and session ID) to share audio-video streams between clients.

2. Subscribe to your own test stream for a test period.

   During the test period, the video quality will stabilize, based on the available network connection quality.

3. Collect the bitrate and packet loss statistics using the Network Stats API (see below).

4. Compare the network stats against thresholds (see below) to determine the outcome of the test.

Please see the sample code for details.

Network Stats API

This API lets you dynamically monitor the following statistics for a subscriber's stream:

• Audio and video bytes received

• Audio and video packets lost

• Audio and video packets received

This API is only available in sessions that use the OpenTok Media Router.

Thresholds and interpreting network statistics

You can use the network statistics to determine the ability to send and receive streams, and as a result have a quality experience during the OpenTok call.

Please keep in mind, every application's use case and every user's perception of the call quality is different. Therefore, you should adjust the default thresholds and timeframe in accordance with your use case and expectations. For example, the 720p, 30 fps video call requires a much better network connection than 320x480-pixel, 15 fps video. So, in that case, you need to set much higher threshold values in order to qualify a viable end user connection. Also, the longer you run the test, the more accurate the values you will receive will be. At the same time, you might want to switch between publishing video and audio-only, based on your specific use case.

The OpenTok Network Test is implemented as sample code to make it easier for developers to customize their application logic.

Below are examples of the thresholds for popular video resolution-frame rate combinations. The following tables interpret results (for audio-video sessions and audio-only sessions), with the following quality designations:

• Excellent - None or imperceptible impairments in media

• Acceptable - Some impairments in media, leading to some momentary disruptions

Audio-video streams

For the given qualities and resolutions, all the following conditions must met.

Quality	Video resolution @ fps	Video kbps	Packet loss
Excellent	1280x720 @ 30	> 1000	< 0.5%
Excellent	640x480 @ 30	> 600	< 0.5%
Excellent	352x288 @ 30	> 300	< 0.5%
Excellent	320x240 @ 30	> 300	< 0.5%
Acceptable	1280x720 @ 30	> 350	< 3%
Acceptable	640x480 @ 30	> 250	< 3%
Acceptable	352x288 @ 30	> 150	< 3%
Acceptable	320x240 @ 30	> 150	< 3%

Note that the default publish settings for video are 640x480 pixels @ 30 fps in the OpenTok iOS SDK. The default is 352x288 @ 30 fps in the OpenTok Android SDK.

You can calculate the video kbps and packet loss based on the video bytes received and video packets received statistics provided by the Network Statistics API. See the sample app for code.

The video resolutions listed are representative of common resolutions. You can determine support for other resolutions by interpolating the results with the closest resolutions listed.

Audio-only streams

For the given qualities, the following conditions must met.

Quality	Audio kbps	Packet loss
Excellent	> 30	< 0.5%
Acceptable	> 25	< 5%

Note that you can calculate the audio kbps and packet loss based on the audio bytes received and audio packets received statistics provided by the API. See the sample apps for code.

Sample code

This repo includes sample code showing how to build a network test using the OpenTok Android and iOS client SDKs. Sample code for the OpenTok JavaScript SDK can be found here: https://github.com/opentok/opentok-network-test-js. Each sample shows how to determine the the appropriate audio and video settings to use in publishing a stream to an OpenTok session. To do this, each sample app publishes a stream to a test session and then uses the Network Stats API to check the quality of that stream. Based on the quality, the app determines what the client can successfully publish:

• The client can publish an audio-video stream at the specified resolution.

• The client can publish an audio-only stream.

• The client is unable to publish.

Each sample subdirectory includes a README file that describes how the app uses the network stats API.

Frequently Asked Questions (FAQ)

• Why does the OpenTok Network Stats API values are different from my Speedtest.net results?

  Speedtest.net tests your network connection, while the Network Stats API shows how the WebRTC engine (and OpenTok) will perform on your connection.

• Why are the Network Stats API results inconsistent?

  The WebRTC requires some time to stabilize the quality of the call for the specific connection. If you will allow the network test to run longer, you should receive more consistent results. Also, please, make sure that you're using routed OpenTok session instead of a relayed on. For more information, see The OpenTok Media Router and media modes

• Why the output values are really low even though my user is streaming Netflix movies?

  WebRTC is conservative in choosing the allowed bandwidth. For example, if there is another high-bandwidth consumer on the network, WebRTC will try to set its own usage to the minimum.

• The network test shows the "Excellent" (or "Acceptable") result, but the video still gets pixilated during the call.

  You can increase the required thresholds to better qualify the end user connection. Please keep in mind, the network connection can change overtime, especially on mobile devices with changing network conditions.

• Why do I get compilation errors on iOS or Android.

  You need to be using a recent and supported version of the OpenTok iOS SDK or OpenTok Android SDK.

opentok-network-test's Issues

Update to use the latest OpenTok SDKs

Currently, the iOS SDK is at the 2.14 version, but this sample is still using the 2.12 version:
https://github.com/opentok/opentok-network-test/blob/master/iOS-Sample/Podfile#L12

Can't convert undefined to object

I used this script to derive network result for Client, but it gives error in console "can't convert undefined to object" at line no. 29 in app.js . Please rectify the problem

Getting very slow while attempting to publish a OTPublisher

Hello,
When my application is in background and receive a call from other at that time CallKit is showed up on screen. The code try to init a OTPublisher but init processing is very slow. It made CallKit became freeze, I can not accept or decline that call until iOS shut down CallKit automatically.
I found out that Tokbok Sample for iOS got exactly this issue.
Anyone got the answer for this situation?, I am in despair over this issue.

Minimum value for qualityTestDuration

Hello,
In sample, the qualityTestDuration =10. is it the minimum time? if not, how do we know the minimun qualityTestDuration to make sure network stable and check quality.

 [_networkTest runConnectivityTestWithApiKey:kApiKey
                                      sessionId:kSessionId
                                          token:kToken
                             executeQualityTest:YES
                            qualityTestDuration:10
                                       delegate:self];

dispatch_time_t delay = dispatch_time(DISPATCH_TIME_NOW,
                                              _qualityTestDuration * NSEC_PER_SEC);
        dispatch_after(delay,dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT,0),^{
            
            [self checkQualityAndDisconnectSession];
        });

Thansk!

Create React Native wrapper for ios/android network test

Feature request: It would be very convenient to create an implementation of this for ios/android in React Native. It would tie in nicely with the new opentok React Native repo. Is anyone planning on this?

Q: How to get all quality level of connections in session

We want to get the subscriber's and publisher's network quality level. But, I think this repository is not same that we want.

We are developing a group-video-call application. We want to show a connection quality indicator for each subscriber. Vonage provides the getStats function but we don't know how to calculate quality level with it.

Do you have any suggestions?

Thanks.

Why final result is considered the last stat received in the test ?

I am trying to understand how the network stats test works under the hood. I understand that the test stabilizes as the time goes up but is it the correct thing to do to present the final result as the last stat received ? Shouldn't we do an average of all stat values received ?

We implemented the test in both iOS and js.
In the javascript version we observed that that packet lost ratio could be > 0.0 in the test but the last stat received is == 0.0 so we consider that there is a good connection when this is not the case...Thats why we thought that we should compute the average of the stat values.

Audio only

I download this code, ran the JS sample locally and on our live server and I always get "Your bandwidth can support audio only".
I am on a very fast internet connection with the latest Chrome running. I also tried it on a home connection off the work network and it still says "Your bandwidth can support audio only".

Any ideas?
Many Thanks

Getting error in publisher(_:didFailWithError:) -> 1541 - Timed out while attempting to publish.

My App requirement is to test Audio and Video quality first, then start video call.
Network test is done properly.
And when I start actual call I'm getting above error. And unable to publish my feed and audio to subscriber.
I think issue might be in destroying audio, video session.

publisher(_:didFailWithError:)

1541 - Timed out while attempting to publish.

If I don't use network test class my app works fine.
I've tried in opentok version 2.19.1, 2.20.0

iOS Readme : checkQualityAndDisconnectSession confusion

I was a bit confused when reading the [OTNetworkTest checkQualityAndDisconnectSession] explanation where the end result are displayed :

BOOL canDoVideo = (video_bw < 50000 || video_pl_ratio > 0.03);
BOOL canDoAudio = (audio_bw < 25000 || audio_pl_ratio > 0.05);

canDoVideo looks a cannotDoVideo to me, it should be more something like

BOOL canDoVideo = (video_bw >= 50000 && video_pl_ratio <= 0.03);
BOOL canDoAudio = (audio_bw >= 25000 && audio_pl_ratio <= 0.05);

I would be even more explicit like :

BOOL canDoExcellent640x480 = (video_bw >= 600000 && video_pl_ratio <= 0.05);

Usage requires camera and microphone in web

If you do not have a camera AND a microphone, or do not allow either one of them, the JavaScript sample will not work.

A workaround is to modify the code to either:

1. Detect available devices using OT.getDevices() first, and use the publisher properties to only uses the devices you care about. Then change the getStats() related code to handle only the devices that are being used.
2. If you never care about a single kind of device (such as audio-only), you can remove the other from the implementation, and change the publisher intialization to pass false as the source for the one that you want to remove.

IE doesn't set the subscriber's volume

Even here the volume is specified IE 11 doesn't change the volume and the user can her herself while doing the network test. https://github.com/opentok/opentok-network-test/blob/master/js-sample/app.js#L100

I also tried doing subscriber.setAudioVolume(0); and it doesn't' work? Is this a Tokbox plugin issue? Is there any workaround from the Javascript side?

Just curious: What happens if this is used on session with media_mode 'relayed'?

I read in FAQ that it should be used on routed OpenTok session, but what will happen if it's relayed?

Will it just be inaccurate, or will it not work at all?

Cheers

how to save test result in session?

Network test is accessing microphone even though publishAudio is set to false (user Privacy issue)

I am using latest Opentok Android SDK v2.25.3, with device Google Pixel 6 Pro

Steps to reproduce:

1. Following the network test sample code provided, perform bandwidth test while setting publishAudio=false for the Publisher, as I am only interested in the video bandwidth

2. When the bandwidth test starts, the microphone access indicator led on the top right corner of the device lights up

3. Tapping on the microphone indicator, it says Mic access - being used by Lifestream (our app)

4. If I start a Whatsapp call before hand, then open our app and start bandwidth test, it takes over the microphone and the other side of the Whatsapp call could not hear me anymore.

PS: On a Huawei device that I have, when no.4 above happens, it shows this warning below, which is a huge concern for the users as they will thought we are recording them secretly while we are just running bandwidth test.

Discrepancy with Acceptable Packet Loss Threshold

There seems to be a discrepancy involving acceptable packet loss ratio between what's in the code (for both iOS and Android) and what's in the "acceptable threshold" table you have listed in the ReadMe of this project.

For example, you show in the table that an acceptable packet loss ratio for video is 0.3% or 0.003, yet you turn around in the code and claim 3%, NOT 0.3%: https://github.com/opentok/opentok-network-test/blob/master/iOS-Sample/Hello-World/OTNetworkTest.m#L244

Which is it? This is a considerable difference for us understanding if a call had good or bad video quality.

Furthermore, there are discrepancies in the Android and iOS versions when initializing packet loss ratio. I noticed for iOS you initialize with video_pl_ratio = -1 yet in Android it's private double mVideoPLRatio = 0.0. This matters because if iOS ever has an issue retrieving pack loss stats (it does at times), then our analytics will just show an array of [-1,-1,-1,-1] whereas Android will correctly show 0s. -1 Is difficult to use as a packet loss variable since it is a negative integer, so can you please explain why iOS is initialized that way versus the Android version? Should we use 0 instead of -1? What is recommended here?

Thanks!

VideoStats listener is not getting fired with Android SDK 2.13

We are using this with a Tokbox safari project. With the Safari project we upgraded the android SDK to 2.13. But ever since that upgrade the network test is failing. The implementation is pretty much as in the sample except for the fact that we are not listening for audio stats, only video stats.

What I found was that Audio Stats event fires and returns results. But the video stats event doesnt even gets fired.

Is this a known issue?

Error while building the project

node_modules/opentok-network-test-js/dist/NetworkTest/errors/types.d.ts (22,28): In ambient enum declarations member initializer must be constant expression.

mVideoBw is zero when publisher device is in low network

When the publisher device is moved to low network(2G), SubscriberVideoStats.videoBytesReceived is coming same for around 30 seconds. And during this time mVideoBw is calculated to 0.

private void checkSubscriberVideoStats(SubscriberKit.SubscriberVideoStats stats) {
    double videoTimestamp = stats.timeStamp / 1000;

    Log.d(TAG, "onVideoStats: stats considering same_timestamp:" + (s_prevVideoTimestamp==videoTimestamp)+" same_byte:"+(stats.videoBytesReceived == s_prevVideoBytes));
    //initialize values
    if (s_prevVideoTimestamp == 0) {
        s_prevVideoTimestamp = videoTimestamp;
        s_prevVideoBytes = stats.videoBytesReceived;
    }

    if (videoTimestamp - s_prevVideoTimestamp >= TIME_WINDOW) {
        //calculate video packets lost ratio
        if (s_prevVideoPacketsRcvd != 0) {
            long pl = stats.videoPacketsLost - s_prevVideoPacketsLost;
            long pr = stats.videoPacketsReceived - s_prevVideoPacketsRcvd;
            long pt = pl + pr;

            if (pt > 0) {
                s_videoPLRatio = (double) pl / (double) pt;
            }
        }
        s_prevVideoPacketsLost = stats.videoPacketsLost;
        s_prevVideoPacketsRcvd = stats.videoPacketsReceived;

        long mVideoBw = (long) ((8 * (stats.videoBytesReceived - s_prevVideoBytes)) / (videoTimestamp - s_prevVideoTimestamp));

        s_prevVideoTimestamp = videoTimestamp;
        s_prevVideoBytes = stats.videoBytesReceived;
        if (s_videoPLRatio > 0) {
            s_avgVideoPLRatio = (s_videoPLRatio + s_avgVideoPLRatio) / 2;
        } else {
            s_avgVideoPLRatio = s_videoPLRatio;
        }
        if (mVideoBw > 0) {
            s_avgVideoBw = (mVideoBw + s_avgVideoBw) / 2;
        } else {
            s_avgVideoBw = mVideoBw;
        }
        Log.d(TAG, "onVideoStats: stats considering avgVideoBw:" + s_avgVideoBw + " PLRatio:" + s_avgVideoPLRatio + " BytesReceived:" + stats.videoBytesReceived + " PacketsReceived:" + stats.videoPacketsReceived + " PacketsLost:" + stats.videoPacketsLost + " mVideoBw:" + mVideoBw);

    }

}

We're using v2.17.0

Test network while in actual call/video session

If i understand correctly, i can test video and audio quality and network stats before going into a real call/video session.

But i need to test these stats while i'm in actual call or video session. Is it possible to apply these checks in real enviroment (not pre-call, in an actual call) ?

iOS custom audio module used in Network test does not clean up

We are running Network test on iOS by first testing with test session id and later running an actual user session with actual session id for event. When running this use case, iOS App crashes. This crash does not happen if we use Android code. We looked at network test code for iOS and here is the issue:

• Network test overrides the microphone’s audio module with a custom audio module object
• After the test, the custom audio module object’s clean-up is attempted
• However, the clean-up seemingly fails, as when a real session tries to use the audio module later, it results in a crash
• We have tested that the crash is avoided when we bypass custom audio module settings in the sample. Here are snippets of code that we commented:

File: OTNetworkTest.m
Function Name: runConnectivityTestWithApiKey
Commented lines (66-72):
if(!_myAudioDevice)
{
_myAudioDevice = [[OTDefaultAudioDevice alloc] init];
}
[OTAudioDeviceManager setAudioDevice:_myAudioDevice];
[_myAudioDevice setAudioPlayoutMute:YES];

Function Name: dispatchResultsToDelegateWithResult
Commented lines (101-102)
[_myAudioDevice setAudioPlayoutMute:NO];
[OTAudioDeviceManager setAudioDevice:nil];
• In this case, the default audio device, picks up the audio, just as in case of a real session. But we get audio bandwidth difference as compared to when we use custom audio module

Video Packets Lost Always = 0 in VideoStatsListener

I'm developing an simple application one to one in Android:

• One subscriber and one publisher.
• Using Android 7.
• As the examples, I did and "re did" in many different forms.

In the event onVideoStats of the SubscriberKit.VideoStatsListener:

• can get stats.videoBytesReceived normally.
• the stats.videoPacketsLost always has the '0' value.
• not using SubscriberKit.AudioStatsListener.

Am I doing something wrong? Somebody has the same problem?

Video bitrate on Chrome is lower than on other browsers

On the same internet connection, the video bitrate on Chrome is much lower than on Firefox or Opera. I tested on Chrome Canary and other internet connections: same results. Any idea? Thanks!
Probably the same issue: #16

Chrome

Firefox

Opera

Low bandwidth check is wrong for web

I discovered that the bandwidth check for audio is wrongly implemented, because the amount of bits transferred depends on the volume level of the system.

We noticed that many of our clients were having bandwidth errors even when they had a good internet connection, and after testing we discovered that it was caused because they had a low volume level or the audio input was low during the test.

You should consider this on your test example.

Swift Support

I see that the codebase is out of date. It would be great if you guys have an updated example of this project with a swift version.