jenslys/autovod

Automatically upload Twitch.tv / Kick.com streams in realtime to Youtube or a Rclone supported provider

AutoVOD

This script automates downloading and uploading Twitch.TV, Kick.com or YouTube Live streams to a selected upload provider.

Important

This does not download and upload the official Twitch/Kick/YouTube VOD after the stream is finished, but rather uses streamlink to record and upload the stream in realtime. So features like separating different audio track for the VOD are not supported. If that is something you are looking for, you should check out Twitch's manual export to YouTube feature.

Current available upload options:

• Youtube (Needs no transcoding, so no file is stored on the disc.)
  • Direct Upload
  • Re-stream
• Rclone - Should work with supported all providers
  • Direct upload (Needs transcoding, so the stream is temporally stored on the disc before uploading)
• Local
  • Local file (Downloads the stream locally to your machine)

Installation

Automatic Installation

Note Only supports APT or DNF. If you are using a different package manager, you will have to install the required packages manually.

sh -c "$(curl -fsSL https://raw.githubusercontent.com/jenslys/autovod/master/install.sh)"

Manual Installation

Required packages

PM2

apt-get install npm
npm install pm2 -g
pm2 startup

Streamlink

apt-get install python3-pip tar
pip3 install --upgrade streamlink

JQ

apt-get install jq

YoutubeUploader

If you want to upload to YouTube

Instructions

wget https://github.com/porjo/youtubeuploader/releases/download/23.03/youtubeuploader_23.03_Linux_x86_64.tar.gz
tar -xvf youtubeuploader_23.03_Linux_x86_64.tar.gz && rm youtubeuploader_23.03_Linux_x86_64.tar.gz
mv youtubeuploader /usr/local/bin/youtubeuploader

Rclone

If you want to upload to an any of the Rclone providers

Instructions

apt-get install rclone

FFMPEG

If you want to enable the re-encoding or re-streaming feature

Instructions

apt-get install ffmpeg

AutoVOD

git clone https://github.com/jenslys/autovod.git
cd autovod

Sample video

wget -c -O sample.mp4 https://download.samplelib.com/mp4/sample-5s.mp4

Setup

Youtube setup

Instructions

Set up your credentials to allow YouTubeUploader to upload videos to YouTube.

1. Create an account on Google Developers Console

2. Create a new project

3. Enable the YouTube Data API (APIs & Auth -> Library)

4. Go to the Consent Screen section, setup an external application, fill in your information and add the user/s that are going to be using the app (Channel/s you are uploading videos to). Enable the ".../auth/youtube.upload" scope. Then save.

5. Go to the Credentials section, click "Create credentials" and select "OAuth client ID", select Application Type 'Web Application'. Add a 'Authorised redirect URI' of http://localhost:8080/oauth2callback

6. Once created click the download (JSON) button in the list and save it as client_secrets.json

7. Getting token from YouTube:

   1. Due to recent changes to the Google TOS, if you are running this utility for the first time and want to run it on a Headless server, you have to first run youtubeuploader on your local machine (Somewhere with a web browser)

      youtubeuploader -filename sample.mp4

   2. and then simply copy/move request.token and client_secrets.json to the remote host. Make sure these are placed inside the autovod folder.

Note To be able to upload videos as either "Unlisted or Public" and upload multiple videos a day, you will have to request an API audit from YouTube. Without an audit your videos will be locked as private and you are limited to how many videos you can upload before you reach a quota.

Tips on passing the audit

I have applied for the audit twice (for two separate projects).

• First time, I was applying because I wanted to archive a particular streamer's streams to YouTube.
• Second time, I was applying because I needed a higher quota for the testing and development of AutoVOD.

Both times I was accepted fairly easily.

Since this tool isn't very complex, I typed almost the same thing on all fields, along the lines of:

"I am going to upload a certain twitch user VODS to YouTube and need a higher quote because the streamer streams multiple times a week for x amount of hours. The tool is internal, so the only person that is authenticating through it is me. This is using Youtube Data API to upload to videos."

I also linked/referenced this GitHub page (Don't know if that helped my case).

The field that wants you to upload a screen recording of the program; I just screen recorded myself doing the youtubeuplaoder --filename sample.mp4 command. Since that is how we get the token from youtube. You could also record the process starting AutoVOD.

Note It took around 20 days from submission to them accepting the audit.

I am leaving open the GitHub issue regarding this, in case people want to discuss or share their experience: #32

Rclone setup

Instructions

Refer to your provider on how to configure Rclone

https://rclone.org/#providers

Usage

Config file

We will create a dedicated config file for each steamer, in case are monitoring multiple streamers with different settings.

Create config file

Note: Case sensitive, make sure to type the capitalization for the username the same on all inputs and files.

cp default.config StreamerNameHere.config

Edit the config

Edit your newly created config

nano StreamerNameHere.config

Optional additional setup steps

Stream metadata

This currently only works if you are using Twitch.TV

If you want to add stream metadata to your video, you will need to deploy an api wrapper for the Twitch API. You can find the instructions on how to do that here. Once you have the wrapper deployed, you will need to add the url in the API_URL field in the config file and enable the API_CALLS field.

Disable ads

Fetching the OAuth token from

Follow the instructions here to get your OAuth token.

Then add the OAuth token: --twitch-api-header=Authorization=OAuth YOURCODEHERE to the STREAMLINK_OPTIONS field in the config file.

Other options

Other options can be found here

Start AutoVOD

pm2 start AutoVOD.sh --name <Streamer Name Here>
pm2 save

Check status

pm2 status

Check logs

pm2 logs

Using docker

This script can be used inside a docker container. To build a container, first execute all Setup-Steps, then build the image:

docker build --build-arg USERNAME=<Streamer Name Here> -t autovod .

You can now run this container

docker run -d autovod <Streamer Name Here>

Or you can run both commands in one line

./buildRunDocker.sh <Streamer Name Here>

FAQ

I am getting "[Error 32] Broken pipe"

There are multiple reasons this error can occur, check the following

YouTube

• That you have not reached your YouTube quota limit
• That your YouTube credential files have not expired
• You can check these by running youtubeuploader --filename sample.mp4 then checking the output.

Rclone

• You have configured rclone correctly
• You have inserted the correct variables inside the config.

Server resource exhaustion

• Uploading VODs require a lot of bandwidth, check if the upload fails because your provider is limiting or cutting of the upload.

My tokens keep getting revoked

• Visit the OAuth consent screen and click on the publish button to change from the testing status to the published status.

My video keeps getting marked as private

• To be able to upload videos as either "Unlisted or Public" and upload multiple videos a day, you will have to request an API audit from YouTube. Without an audit your videos will be locked as private and you are limited to how many videos you can upload before you reach a quota.

I cant upload videos longer then 15 minutes

• You will need to verify your phone number on youtube to upload videos longer then 15 min

One or more required files are missing

The following files are required for the script to work:

• nameOfStreamer.config
• request.token (Only if uploading to YouTube)
• client_secrets.json (Only if uploading to YouTube)

It should look something like this:

Credit

• Original script by arnicel
• YoutubeUploader by porjo
• Streamlink by streamlink
• Icon by xyaia

License

Licensed under the MIT License

autovod's Issues

[Feature request] automatic video scheduling for the "youtube" provider

Please Describe The Problem To Be Solved
The Twitch affiliate agreement includes a timed exclusivity period (Affiliate agreement §2.2), which requires VODs and livestreams to be available only on Twitch for at least 24hrs. This means even with a project like autovod, there's still some manual labor - mainly with scheduling the VODs.

(Optional): Suggest A Solution
youtubeuploader supports specifying when to schedule a video to (for that to work, the visibility has to be set to private). if autovod incorporated that in some way, it would make lives of streamers who maintain a VOD channel so much easier.

• Details of the technical implementation
  If autovod supported automatically taking system time say, at the beginning and add 24hrs (or user specified time) and supply youtubeuploader with that information - this could help automate that compliance.
  I can't find the specific CLI arg, but I know that if using the metaJSON argument, it's specified in the json as publishAt
  If that's only supported while using the JSON file based metadata, I assume autovod would need to spit out a temporary JSON file somewhere and point youtubeuploader to it.
• Tradeoffs made in design decisions
  Depending on the complexity of the code or any issues this might cause, removing the user specified offset feature and keeping it 24hrs only could be a thing that could be done as a last resort.
  Not sure how OS specific would be pulling the local system timezone to be able to supply youtubeuploader with the correct timestamp, but if it proves troublesome it could be beneficial to just make the timezone user specifiable (and assume the user specified the timezone in accordance to how the system keeps the time).
• Caveats and considerations for the future
  None at the moment.
  EDIT: I have remembered autovod starts piping Streamlink into youtubeuploader at stream start, which means it's probably passing the metadata to YouTube at the very beginning. I have reflected the implications of this by rephrasing a bit.

Username does not get defined properly

I've installed it properly, and I've verified that the script itself is working by hardcoding the username into AutoVOD.sh. When the username is hardcoded, the script runs like it's meant to be and uploads the VOD's to Youtube.

However when I run "pm2 start AutoVOD.sh --name usernamehere", and "pm2 save" after it, the script does not get the username.

Tried running it both without sudo and in sudo, same result.

Tried also adding "$@" into AutoVOD.sh and it does not show any parameters.

Logs;

out.log

0|twitch_s | Starting AutoVOD...      

0|twitch_s | Using Twitch user:       

0|twitch_s |       

0|twitch_s |         

0|twitch_s | Checking twitch.tv/ for a stream.       

0|twitch_s | Trying again in 1 minute     

error.log

0|twitch_s | error: No plugin can handle URL: twitch.tv/         

0|twitch_s | error: No plugin can handle URL: twitch.tv/       

0|twitch_s | error: No plugin can handle URL: twitch.tv/   

I managed to fix this in my end by launching it with "pm2 start AutoVOD.sh --name usernamehere -- usernamehereonceagain" and then replacing "STREAMER_NAME=$2" in AutoVOD.sh with "STREAMER_NAME=$1" but would be still cool to know what exactly is going wrong, and if it's issue in script's end to get it fixed.

Download VODs

There are many other tools out there to just download VODs but I'd love something I could spin up in Docker to just automatically watch a channel and download VODs.

403 Error on connection to Oauth

Hello, I have been trying to get this to work on a raspberry to use for my VODs. I have managed to rewrite some things in the scripts since the names in the youtubeuploader repo have changed (files now called youtubeuploader_22.02_Linux_.tar.gz)

My current issue is that I cant seem to get the Oauth to work. I have eveything setup so far but the actual connect to the API does seem to fail

Not uploading Videos to YouTube

Hi,

everytime i stream, pm2 error log shows me this error message:

error: Error when writing to output: [Errno 32] Broken pipe, exiting
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

What can i do?

[feat] - s3fs

https://github.com/s3fs-fuse/s3fs-fuse might resolve the issue of having to download the file to disc before uploading to the s3 bucket, since s3fs acts a normal filesystem

resources:
https://github.com/s3fs-fuse/s3fs-fuse
https://github.com/davidxjohnson/s3fs-howto
https://dev.to/aws-builders/how-to-mount-aws-s3-bucket-on-amazon-ec2-using-s3fs-1h8h

One or more required files are missing

When using starting autovod.sh nothing hapens and the message One or more requiered files are missing is in the logs, except I have all files set and configured

Steps to reproduce the behavior:

1. Install the script with automatic installation
2. Run pm2 start AutoVOD.sh --name StreamerNameHere
3. Run pm2 logs
4. See error

Expected behavior
Downloading and uploading of VODs

Screenshots

Desktop (please complete the following information):

• OS: Ubuntu
• Version 22.04

Additional context
Running as a VM in Proxmox
I managed to upload the sample file

[Feature] - Improve dockerfile

Overall refactor and improve the dockerfile
Maybe add:

• Option to mount volumes so you can store the VODs somewhere else besides inside the container

[Bug] autovod doesn't support Streamlink's twitch-api-header argument

Describe the bug
autovod doesn't work properly with twitch-api-header added to the Streamlink arguments in the streamer config

To Reproduce
Steps to reproduce the behavior:

1. Go to 'streamer-name.config'
2. Include --twitch-api-header "Authorization=OAuth <twitch-api-header-without-the-<>-brackets>" in the STREAMLINK_FLAGS section of the streamer config
3. Start autovod
4. The streamer goes live
5. Autovod doesn't work, logs reveal that Streamlink complains about invalid arguments.

Seems like autovod isn't passing something properly. Maybe it's not parsing the quotes required for that Streamlink arg (--twitch-api-header) properly.

Expected behavior
Autovod works as usual, and that argument is passed to Streamlink. Which results in ads not being passed through if the api header comes from a subscriber/broadcaster account.
Therefore no inset "Ads in progress" from twitch (or pause in the data stream when using --twitch-disable-ads) so long the api header hasn't expired yet.

Desktop (please complete the following information):

• OS: [Ubuntu Server 22, headless.]

Additional context
Autovod is being started by pm2.

[Chore] - Move docs to GitBook

[Errno 32] Broken pipe

Desktop (please complete the following information):

OS: Ubuntu
Version 22.04

Additional context
Running as a VM in Proxmox

It's me again, a stream is happening and

i'm getting broken pipes, I tried a youtube uploader and it worked

What could it be ?

Local upload provider

option to just save the VODS locally the server/machine

[feat] Chat replay

[chore] Add curl and git to requirement

add to readme

[feat] - Config file

It would be better if all the settings were stored in a separate config file instead of inside the script.

One approach could be using a separate bash file as the config:

config.sh

#!/bin/bash
set -a

hello="Hello World!"
now=$(date)
message="Current time is $now and message is $hello"

set +a

AutoVOD.sh

#!/bin/bash

# Loads the config
source config.sh

echo $message

[chore] Improve docs

• Improve docs for standard install and docker

Broken Pipe

This error is shown every minute as streamlink tries to get the stream info

[feat] - Support Kick.com

Please Describe The Problem To Be Solved
Support for multiple sources, most notably kick.com

(Optional): Suggest A Solution
Wait for Streamlink to make an official plugin, or use a third-part plugin: https://github.com/nonvegan/streamlink-plugin-kick

Other

• https://twitter.com/pauliekick/status/1682387647652589571?s=12&t=0KyHATbiQ93pvIBCXUUzJw

feat - Re-livestream

option to re-livestream to:

• Youtube
• Cloudflare stream

[Feature Request] - Stream title

Hi,

The script is working perfectly. I wondered how could change the title of the stream frome "STREAMER NAME + DATE" to the stream title on twitch. I see the variable : VIDEO_TITLE="$STREAMER_NAME - $TIME_DATE" but I don't know how to get the stream name.

[chore] Remove install script

• Remove install script
• add install steps for other package managers instead

[chore] Improve logs

• More logs for each event
• Current time to each echo
• colors?

[chore] Add STREAMLINK_QUALITY variable to change stream quality

[Bug] [Docker] - Youtube required files

• The dockerfile tries to copy the request.token and client_secrets.json regardless of if you are going to upload to YouTube or not

[chore] Version bump

Bump youtubeuploader

More GH templates

[question] - Downloading the Twitch VOD

1. its while im streaming, it auto streams to youtube?
2. or when im done stream, it presses the export to youtube button for me from twitch video manager?

im looking for solution to the 2nd option

[feat] - replace aws with rclone

replace the awscli tool with rclone

they offer support for multiple providers aswell:

• https://rclone.org/overview/
• https://rclone.org/commands/rclone_copy/
• https://forum.rclone.org/t/copy-single-file/1382/4

Problem with STREAMER_TITLE and STREAMER_GAME variables

Describe the bug
I have the api that works and I receive the titles well, however on the youtube video there is an initial_title mark

Screenshots

Desktop (please complete the following information):

• OS: DEBIAN 12

Stream variables not working

Describe the bug
When trying to use stream variables it just returns "initial_title" instead of the actual title.

To Reproduce
Steps to reproduce the behavior:

1. Turn API calls on
2. Use the stream title variable
3. See that it does not actually pull the title

Expected behavior
The stream title should be pulled.

Desktop (please complete the following information):

• OS: Ubuntu
• Version 2.1.0

Replace docker OS image to Alpine

Would it be possible to change the docker image from Ubuntu to Alpine?
https://jfrog.com/knowledge-base/why-use-ubuntu-as-a-docker-base-image-when-alpine-exists/#:~:text=Larger%20image%20size%3A%20Alpine%20base,(in%20most%20cases)%20run.

Show streamlink log output

Right now it's all sent to dev/null, but it would be very helpful to see the log output from streamlink

Improve install script

1. Detect the systems package manger
2. Check if the package is installed before trying to install
3. Add option for youtube and s3

[feat] - Support more providers

[question] - Tips on passing Youtube Form

would it be possible to get some more details about how to fill in the form?
my initial request was denied because i picked the wrong form (periodic) instead of quota extension request

How much additional quota am i requesting?

Justification for additional quota

What potential workarounds would you use to compensate for less quota? (ex. decreased feature set, estimations, smaller sampling)

[feat] optional transcoding option for S3

Transcode to AV1

chore - refactor fetching extra stream metadata

[chore] Add note regarding Affiliate agreement

https://www.twitch.tv/p/en/legal/affiliate-agreement/
section 2.2

[DOCKER] Volumes

Hello, Little question since we don't use the -v, where are the vods stored?

THANKS

How to add multiple channels?

Hi,

Apologies if this question seems silly but this is my first time setting up something like this so I am a bit ignorant.

I was wondering if it's possible to add multiple channels with AutoVOD or is it just single channel based? For instance there's a broadcaster that has a main channel and an alt. Would I be able to add both of their channels or is one instance only allowed?

Thanks

Streamlink usage error with weird behavior

When running, the script pushs out an error with an incressing amount of of "best"

Steps to reproduce the behavior:

1. Install the script with automatic installation
2. Run pm2 start AutoVOD.sh --name StreamerNameHere
3. Run pm2 logs
4. Wait more than 1 minute
5. See error

Expected behavior
Not outputting errors

Screenshots

Desktop (please complete the following information):

• OS: Ubuntu
• Version 22.04

Additional context
Running as a VM in Proxmox

[Feature] - Docker container

I am not familiar with docker, so if anyone wants to help out and create a docker container for this script that would be awesome 🚀

Removal of -headlessAuth in instructions

Hi!

First off, I wanted to say thanks for creating such an amazing vod automator. I've been using it for a couple of weeks and it's been a blessing.

I wanted to make this issue to update the instructions on getting this setup correctly. The author of youtubeuploader has removed the -headlessAuth parameter as of 22.03: porjo/youtubeuploader#116

Per youtubeuploader instructions on headless servers:

To run the utility on a headless-server, generate the token file locally first, then simply copy the token file along with youtubeuploader and client_secrets.json to the remote host.

[bug] shellcheck action not working

title

[chore] FAQ Section

"BrokenPipeError: [Errno 32] Broken pipe"

• e

"my tokens keep getting revoked"

• Visit the OAuth consent screen and click on the publish button to change from the testing status to the published status.

"can I add more users?"

• yes, just create a new config for that user and start autovod with the new name

"my video keeps getting marked as private"

• brrrr

[feat] - Upload to S3

UPLOAD_SERVICE="s3"
STREAMER_NAME="nameHere"
BUCKET_NAME="bucketName"
OBJECT_KEY="vods/stream.mkv"
ENDPOINT_URL="https://123123.r2.cloudflarestorage.com" # https://s3.auto.amazonaws.com

if [ "$UPLOAD_SERVICE" = "youtube" ]; then
  # Use youtubeuploader to upload the stream
elif [ "$UPLOAD_SERVICE" = "s3" ]; then
  # Upload to S3
else
  echo "Invalid upload service specified: $UPLOAD_SERVICE" >&2
  exit 1
fi

Would it be beneficial to re-encoding using AV1?

• file sizes might be smaller, but It may be resource intensive to re-encode.

Resources:
streamlink/streamlink#1478

[Feature] - Failure to upload file/tempfile

Handling on what happens when a upload fails to rclone.

• retry upload?
• save file to disk xqc_08-04-2023_failed_upload ?

Autovod doesnt work

Hi,

i installed autovod and configured everything. The sample.mp4 also uploaded correctly to youtube. But now it doesnt upload old vods (or new, if i go live) to youtube.

pm2 status:

What can i do?

Add docker environment

Note: I am working on this already. Just wanted to note it down here.

A way to run this in Docker would be ideal as manually installing all this stuff is annoying.

[Bug] - Docker target channel issue

Docker container keeps setting twitch.tv/dockeruser as the target channel

18:29:54| Selected streamer: dockeruser
18:29:54| Found config file
18:29:54| Starting AutoVOD
18:29:54| Loading dockeruser.config
18:29:54| Checking twitch.tv/dockeruser for a stream
18:29:54| All required files found
error: No playable streams found on this URL: twitch.tv/dockeruser
18:29:58| Trying again in 1 minute

cp default.config targetchannel.config
sudo docker build --build-arg TWITCH_USER=targetchannel -t autovod .
sudo docker run -d autovod

despite the channel being live, it's not picking it up.
am I doing something wrong?

autovod v2.0.0