muxinc/stream.new

The repo for https://stream.new

stream.new

An open-source example application that allows users to record and upload videos using Mux

View Demo · Report a Bug · Request Features

Table of Contents

1. About The Project
   • Components
2. Getting Started
   • Step 1. Create an Account in Mux
   • Step 2. Set Up Environment Variables
   • Step 3. Deploy on Vercel
   • Step 4. (Optional) Slackbot Moderator
   • Step 5. (Optional) Add automatic content analysis to Slackbot Moderator (Google Vision API)
   • Step 6. (Optional) Add automatic content analysis to Slackbot Moderator Hive AI
3. Videos to Test in Development

About The Project

Components:

Mux:

• Direct uploads - this is an API for uploading video files from a client to create Mux Assets
• Webhook signature verification - webhook signature verification to make sure Mux webhooks are coming from a trusted source
• HLS.js - for doing HLS video playback of videos
• Mux Data - for tracking video quality metrics.

Slackbot moderator. This examples allows you to configure a SLACK_WEBHOOK_ASSET_READY. When a new Mux asset is ready, an Incoming Webhook for slack will be sent. This is an example of how you might integrate a Slack channel that can be used to moderate content. The Slack message contains the asset ID, playback ID and a storyboard of thumbnails from the video.

NextJS:

• SWR — dynamically changing the refreshInterval depending on if the client should be polling for updates or not
• /pages/api routes — a couple endpoints for making authenticated requests to the Mux API.
• Dynamic routes using getStaticPaths and fallback: true, as well as dynamic API routes.

This app was created with the NextJS with-mux-video example as a starting point.

Getting Started

Step 1. Create an account in Mux

All you need to set this up is a Mux account. You can sign up for free and pricing is pay-as-you-go. There are no upfront charges, you get billed monthly only for what you use.

Without entering a credit card on your Mux account all videos are in “test mode” which means they are watermarked and clipped to 10 seconds. If you enter a credit card all limitations are lifted and you get $20 of free credit. The free credit should be plenty for you to test out and play around with everything before you are charged.

Step 2. Set up environment variables

Copy the .env.local.example file in this directory to .env.local (which will be ignored by Git):

cp .env.local.example .env.local

Then, go to the settings page in your Mux dashboard, get a new API Access Token that allows for "Full Access" against Mux Video and set each variable in .env.local:

• MUX_TOKEN_ID should be the TOKEN ID of your new token
• MUX_TOKEN_SECRET should be TOKEN SECRET
• MUX_WEBHOOK_SIGNATURE_SECRET (optional) - the webhook signing secret if you set up webhooks (see below)
• SLACK_WEBHOOK_ASSET_READY (optional) - the slack webhook URL that will be used for the Slackbot moderator feature (see below)
• SLACK_MODERATOR_PASSWORD (optional) - this is the password when you want to take actions from the Slackbot moderator feature (see below)
• NEXT_PUBLIC_MUX_ENV_KEY (optional) - this is the mux environment key for Mux Data integration

Step 3. Deploy on Vercel

You can deploy this app to the cloud with Vercel (Documentation).

To deploy on Vercel, you need to set the environment variables using Vercel CLI (Documentation).

Install the Vercel CLI, log in to your account from the CLI, and run the following commands to add the environment variables. Replace the values with the corresponding strings in .env.local:

vercel secrets add stream_new_token_id <MUX_TOKEN_ID>
vercel secrets add stream_new_token_secret <MUX_TOKEN_SECRET>

Then push the project to GitHub/GitLab/Bitbucket and import to Vercel to deploy.

Step 4 (optional) Slackbot Moderator

This application uses a slackbot to send message to a slack channel every time a new asset is ready for playback. This requires a few steps for setup.

First, login to your Mux dashboard and in the left sidebar navigation find Settings > Webhooks. Create a new webhook and makes sure you are creating a webhook for the environment that matches the access token that you are using.

For local development you may want to use a tool like ngrok to receive webhooks on localhost. The route for the webhook handler is /api/webhooks/mux (defined in this NextJS app under ./pages/api/webhooks/mux).

Create a Slack 'Incoming Webhook'. Configure the channel you want to post to, the icon, etc.

When you're done with this, you should have a slack webhook URL that looks something like https://hooks.slack.com/services/....

Set the optional environment variables either directly in the vercel UI or by updating vercel.json and setting them as secrets for your organization. The optional environment variables are:

• MUX_WEBHOOK_SIGNATURE_SECRET - This is a security mechanism that checks the webhook signature header when the request hits your server so that your server can verify that the webhook came from Mux. Read more about webhook signature verification. Note that in ./pages/api/webhooks/mux the code will only verify the signature if you have set a signature secret variable, so this step is optional.
• SLACK_WEBHOOK_ASSET_READY - This is the https://hooks.slack.com/services/.... URL when you created the Slack Incoming Webhook.
• SLACK_MODERATOR_PASSWORD - This is the password that will be used to authorize deleting assets from the slack moderator (The button with the red text "DELETE (cannot be undone)")
• NEXT_PUBLIC_MUX_ENV_KEY - This is the env key to use with Mux Data. Note this is different than your API key and this environment key can be found on your environment page in the Mux dashboard
• TELEMETRY_ENDPOINT - This is an endpoint where instrumented telemetry data is sent. As of this update, the only place we collect/send telemetry data is around upload performance in order to test different configurations of UpChunk, but there may be more in the future.

After all of this is set up the flow will be:

1. Asset is uploaded
2. Mux sends a webhook to your server (NextJS API function)
3. (optional) Your server verifies the webhook signature
4. If the webhook matches video.asset.ready then your server will post a message to your slack channel that has the Mux Asset ID, the Mux Playback ID, and a thumbnail of the video.

Step 5 (optional) Add automatic content analysis to Slackbot Moderator (Google Vision API)

stream.new can automatically moderate content with the help of Google's Cloud vision API.

Follow these steps to help moderate uploaded content:

• GOOGLE_APPLICATION_CREDENTIALS - This is a base64 encoded JSON representation of your Google service account credentials. Follow instructions below.

1. First, you will need to set up a google developer account at cloud.google.com.
2. Create a project
3. Create a service account for your project and enable the "Cloud Vision API" for your project

Export a Google Service Account authentication file in JSON format. If you have a file that is like this:

service_account.json

{
  "type": "service_account",
  "project_id": "",
  "private_key_id": "",
  "private_key": "-----BEGIN PRIVATE KEY-----\",
  "client_email": "",
  "client_id": "",
  "auth_uri": "",
  "token_uri": "",
  "auth_provider_x509_cert_url": "",
  "client_x509_cert_url": ""
}

Get the base64 encoded string of this JSON file like so:

cat service-account.json | base64

^ This command will output one long string. This string is what you will use for the ENV var GOOGLE_APPLICATION_CREDENTIALS.

When the Slackbot Moderator message gets posted to slack, it will now include a "Moderation score (Google)" with 2 dimensions:

• "adult"
• "suggestive"
• "violent"

Each dimension will have a score from 1-5. You should interpret these scores in terms of likelihood that the video contains this type of content. This is based on Google Vision's Likelihood score

• 1: very unlikely
• 2: unlikely
• 3: possible
• 4: likely
• 5: very likely

Step 6 (optional) Add automatic content analysis to Slackbot Moderator (Hive AI)

stream.new can automatically moderate content with the help of Hive AI.

Follow these steps to help moderate uploaded content:

• HIVE_AI_KEY - This is a base64 encoded JSON representation of your Google service account credentials. Follow instructions below.

1. First, you will need to set up an account at thehive.ai.
2. Create a project
3. Get the API key for your prject

When the Slackbot Moderator message gets posted to slack, it will now include a section titled "Moderation score (hive)" with 2 dimensions:

• "adult"
• "suggestive"

Each dimension will have a score from 0-1 with a precision of 6 decimal places. These numbers come from the "Classification API" that Hive AI provides. Details here.

Hidden playback features via query params:

• time: will start the video at a specific timestamp in seconds, for example ?time=10 will start at 10 seconds like this
• color: a hex value without the # character will theme the Mux Player with the primaryColor. It's important to omit the # for example ?color=f97316 like this

Videos to test in development:

When developing, if you make any changes to the video player, make sure it works and looks good with videos of various dimensions:

Horizontal

• http://localhost:3000/v/Hi6we01h00uVvZc00GzvVXZW8C02Y8QC8OX7

Vertical

• http://localhost:3000/v/UNDUU7tU7vYt02CRMDTlZd1qKjvk41LN6yI5LbHgtxo8

Super vertical

• http://localhost:3000/v/seK501Bf00kyqSnGdMwQFi3lgqgdoS00qm5PAiV7Yjf2ew

Also be sure to check: Safari, Mobile Safari, Chrome, Firefox because they all behave a little differently.

stream.new's Issues

Storyboard support

https://docs.mux.com/docs/storyboard-guide

Error page says "video does not exist" when internet connectivity cuts during playback?

I was playing a stream.new video and my Internet cut out temporarily during playback. Instead of telling me that I had a connectivity issue, the error message instead said that the video does not exist (see image).

Subsequent testing indicated that when Internet is consistent during playback this does not appear.

Playback rate support

Add duration to slackbot moderator

Currently the slackbot moderator shows assetId, playbackId, and some actions.

It would be good to add another field duration. The duration field is convenient for a moderator to have when assessing content at first glance.

Stop recording button broken in Firefox

In Firefox 92, once I start screensharing in Firefox the "stop recording" button becomes non-functional.

Poor resolution while doing recording from iPhone

Tried to record a video from my iPhone 12 Pro, but the video resolution is degraded.

Controls are squished/overlapping on mobile

After uploading a vertical video on an iPhone 11 the volume and gear icons are overlapping and the progress circle and time are squished.

https://stream.new/v/jpZnXhZvSWhA7kEnz900HbAmmy1L2nJcJ8hIBJeBdbos

Setup Github actions to run 'yarn lint' and 'yarn test'

stream.new camera recording sometimes gets the wrong aspect ratio?

https://stream.new/v/1jbncFakyHiVvQm8BX372IDlGzQ8aqxWq51d56x9NEE

Firefox 78 on Windows using a StarTech USB3HDCAP capture device. Works fine with the OBS virtual camera (with the same capture device) routed into Firefox.

Super vertical videos are positioned incorrectly

https://stream.new/v/seK501Bf00kyqSnGdMwQFi3lgqgdoS00qm5PAiV7Yjf2ew

Chromecast support

camera/screenshare is stil active after stop recording

1. allow camera access
2. start recording
3. stop recording
4. [bug] camera is still active indicated by webcam light - it should not be

Same thing happens with screenshare

skipping recorder stop() b/c state is "inactive"

I think this log line indicates the issue

macOS Safari: fullscreen video isn't fullscreen

It seems these CSS properties are winning out in setting the size of the video element. Wasn't sure what the correct fixes were but in particular width: 1000px and max-height: 70vh are interfering.

Double controls on mobile

1. Open on mobile
2. Tap play - the player opens up into the native iOS full screen player.
3. Close full screen player
4. See double controls in the UI

paste URLs for upload

from the index page, if you paste a URL it should just create an asset from that URL

Add audio-only support + player

It's possible to upload an audio file, but it can't be played.

Error creating this asset: Please upload a valid video file (gifs are not supported)

🎯 Description:

• After recording my screen multiple times or a long video in the range of 3 to 5 minutes, I encountered this error when I tried to submit the recording.

🎞️ Steps:

1- Open

https://stream.new/

2- Record the entire screen multiple times videos between (3:5 min)

❌ Actual results:

• Error creating this asset: Please upload a valid video file (gifs are not supported)

✅ Expected results:

• Screen for copy video URL or waiting for the recorded video

Note:

• I tried refreshing it multiple times, but it did not work.

Example:

https://stream.new/assets/zt29WodXtqEpeagBpRwj1xmro01BTxtBgFecN3UcKJA4

Remove Curved Corners in Screenshare Recording

I would also recommend increasing the .webm quality. Is VP8 being used? VP9 might provide better quality. I wouldn't use VPX because of how slow it is. It might be as simple as increasing the bitrate, or perhaps allowing the end user to adjust that value prior to recording.

Great program overall! Thanks for developing and sharing.

OpenGraph Video preview missing / broken in Discord (and possibly other platforms?)

Sending a link to a video doesn't properly embed a player in Discord (I'd guess this also applies to some other platforms too)

Note that there isn't a play button and instead only shows the thumbnail of the video.

error when build

When run "npm run build", there is an error:
Nested Middleware is not allowed, found:
pages/_middleware
pages/v/[id]/_middleware

macOS Safari recording camera fails

• start camera recording in Safari
• it fails

[stream.new]
NotSupportedError: mimeType is not supported
[native code]
(anonymous function) — record-9215b8a9a8ad9eb9c43a.js:1:16393
l — commons.2f484fc1cee4e666eba9.js:1:31248
(anonymous function) — commons.2f484fc1cee4e666eba9.js:1:31033
r — 94d97e5b542eb1ef14fe677c87e30fb345c02fd6.0532e79688b18ec07277.js:1:19610
u — 94d97e5b542eb1ef14fe677c87e30fb345c02fd6.0532e79688b18ec07277.js:1:19814
(anonymous function) — 94d97e5b542eb1ef14fe677c87e30fb345c02fd6.0532e79688b18ec07277.js:1:19873
Promise
(anonymous function) — 94d97e5b542eb1ef14fe677c87e30fb345c02fd6.0532e79688b18ec07277.js:1:19764
p — record-9215b8a9a8ad9eb9c43a.js:1:9132

Upload / record buttons missing on iOS (chrome and safari)

Viewing https://stream.new from my iPhone, I'm unable to see the Record from camera, Record my screen buttons or upload a file from device. I've tried scrolling up and down, left and right, but no luck.

Device: iPhone 14 Pro
Browsers: Chrome and Safari

iOS 13.3 Safari issue?

Hi & thanks for this app - looking to use Mux for a web mobile prototype.

Have you been testing iOS? - After 'Upload a video', I don't get navigated to the upload screens.

I've tested iOS safari and Firefox for iOS

Thanks!

Starting recording after stopping breaks UI

After I hit "Stop Recording" I can hit Start again, and it doesn't work so well.

deleted assets: thumbnail flashes before showing error

Repro steps

• upload a video
• view the video
• delete the asset from Mux
• try to view the video
• see that the thumbnail flashes before showing an error message

Internally, this is what's happening:

• video player loads with poster= attribute
• even though the asset is deleted, previously rendered thumbnail images are cached by cdns for a while (days), so the request to load the poster image (thumbnail) succeeds
• the m3u8 playlist 404s, which triggers the correct error "This video does not exist"

Solution -- maybe there's something better but this is the best idea I have:

• before rendering the video player do a fetch GET to the m3u8, if it 404s then fallback into the error state

stream.new links do not reliabily unfurl in slack

Usually, they don't unfurl at all.

When using an alternate domain (https://stream-new.vercel.app) instead of (https://stream.new) it is more reliable. The alternate domain always seems to unfurl and sometimes it's the correct image, other times it's the image from the homepage.

My only conclusion is that slack is not properly recognizing the https://stream.new domain. This behavior is consistent across different slack workspaces.

This one shows no og:image for stream.new but when using stream-new.vercel.app it pulls in the cover image from the homepage.

Here's an example where the stream.new link doesn't unfurl at all, but stream-new.vercel.app unfurls correctly.

---

I'm talking with slack support about this, no resolution yet

3.. 2... 1... *now recording* countdown

Currently, clicking "start recording" will immediately start recording.

Particularly when recording screenshare, this is not great because the first few seconds of every recording is a view of the stream.new UI, usually immediately followed by switching to a different screen.

I think it would be good to have a "3... 2... 1..." countdown whenever recording starts (for both camera recording and screen recording)