livepeer / catalyst Goto Github PK
View Code? Open in Web Editor NEWLivepeer's Decentralized Media Server
License: MIT License
Livepeer's Decentralized Media Server
License: MIT License
https://discord.com/channels/423160867534929930/992554008776540180/1006636908836827267
having issues with 4K streams. At some point I thought they just didn't work, but now only transcoding is broken. only source renditions is available
likely transcoding is taking too long and then being ignored as if it timed out? i know it's normally close to/worse than real-time for 4K videos
live for the next ~2h:
https://lvpr.tv/?url=https://sao-canary-catalyst-0.livepeer.fun/hls/video+196a6qz7pg06nl5h/index.m3u8?mute=false
Streaming command:
ffmpeg -re -i ./vintage/vintage-cercle.mp4 -c:v copy -c:a copy -strict -2 -f flv rtmp://sao-canary-catalyst-0.livepeer.
fun/live/196a-em8j-4822-1hun
File is 4K with H264 and AAC as required. Can send it over to someone but I don't think there's anything special with it. Works fine on livepeer.com.
Initial targets: Linux AMD64, Linux ARM64, macOS AMD64, macOS ARM64
@leszko Please add a comment about the stream redirection logic
@emranemran Please add a comment about DTSC(S) stream replication
@hjpotter92 Please add a comment explaining the the e2e test flow in this repo
https://www.notion.so/livepeer/Multi-Node-Stream-Replication-Docs-43f2d8982da042f8894ad452f99cf9fc
A lot of our testing is going to be Docker integration testing. That makes it important to have fast rebuilds... and currently, we have an ADD . .
command that means everything gets redownloaded on every change.
Consolidating all of the downloader stuff in cmd/downloader
should fix that; it probably won't share code with other things like catalyst-node
anyway.
It was evident from #54 that we need a third container that launches all tester binaries instead of relying on the CI host to have certain dependencies fulfilled. A workaround was used for the linked PR to copy over the relevant binaries required for load-testing into the CI host. These binaries should instead live in their own containers.
Minimally we'll want to auto-populate the retry-join
flag so we can get rid of the two separate config files in the config
directory. It can easily be set with the CATALYST_RETRY_JOIN
environment variable once #51 ships.
Some other cases that we'd want once we can generate configs:
There is some race condition, which makes the execution of the livepeer/catalyst
container flaky. In result, sometimes, when catalyst is started, there are no logs at all and the only running process is MistController
.
for i in $(seq 0 100); do CID=$(docker run -d --rm livepeer/catalyst); sleep 5; LOGS=$(docker logs $CID); [[ -z "$LOGS" ]] || docker kill $CID;done
This command will start 100 container one by one and kill all expect the ones that produce no logs. Then check the running containers and if there are none, then retry the command above. In my case, there was always at least one running container left.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
79e3ec38e0c6 livepeer/catalyst "/usr/bin/MistContro…" 8 minutes ago Up 8 minutes 1935/tcp, 4242/tcp, 8080/tcp, 8889/udp sleepy_poincare
You should find that the only running process is MistController
.
$ docker exec -it 79e3ec38e0c6 sh
# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 16080 4340 ? Ssl 13:47 0:00 /usr/bin/MistController -c /etc/livepeer/catalyst.json
root 14 0.1 0.0 2060 496 pts/0 Ss 13:57 0:00 sh
root 20 0.0 0.0 5480 2384 pts/0 R+ 13:57 0:00 ps aux
project-platform-arch.tar.gz
format, untars to bin
In #86, we added stream interruption test i.e. verify playback still recovers after ingest stream is killed and restarted. However, we set it to forcefully pass even with failures - this was temporary until we root-caused the issue. With #65 now resolved, this stream interruption test can be correctly set to fail if playback is not able to recover correctly during an ingest stream interruption.
Initial Setup:
docker-compose
stuff~/.config/livepeer/catalyst.json
mist-api-connector
authWebhookUrl
from broadcasterhardcoded_broadcasters
field set to [{"address":"http://localhost:8935"}]
video+foo
field using ffmpeg and OBS (gstreamer too!)http://localhost:8080/hls/video+foo/index.m3u8
http://localhost:8090/hls/foo/index.m3u8
Build-to-run workflow:
make mistserver
for Mistmake livepeer
, make livepeer-catalyst-node
killall livepeer
and let MistController bring it back upCurrently stream in Livepeer Studio can have two prefixes, corresponding to two Mist stream configurations: video
for normal streams, and videorec
for streams with recording enabled.
So our playback URLs look like this:
https://catalyst-node.example.com/hls/video+PLAYBACK_ID/index.m3u8
https://catalyst-node.example.com/hls/videorec+PLAYBACK_ID/index.m3u8
But in the dashboard, our playback URLs don't have any prefixes, and look like this:
https://catalyst-node.example.com/hls/PLAYBACK_ID/index.m3u8
The legacy system uses etcd routes and middleware in Traefik to handle this sort of thing, but we're getting rid of that. Other Mist configurations use the defaultStream
parameter to automatically rewrite PLAYBACK_ID
to e.g. stream+PLAYBACK_ID
, but that won't work for us because we have two stream configurations, video
and videorec
.
I'm leaning toward handling this on the catalyst-node side, so it can issue a redirect from https://global-playback-url.example.com/hls/PLAYBACK_ID/index.m3u8
to https://catalyst-node.example.com/hls/PLAYBACK_ID/index.m3u8
.
New command line option: -redirectPrefixes=video,videorec
. Then when we get an incoming playback request for PLAYBACK_ID
, we'll query the Mist load balancer concurrently for video+PLAYBACK_ID
and videorec+PLAYBACK_ID
. (If an incoming request comes in for video+playbackId
we can just forward it as-is.) Then we can issue a redirect to whichever one has a stream, if any.
Following the pattern set by go-livepeer:
livepeer-darwin-amd64.tar.gz
livepeer-linux-amd64.tar.gz
livepeer-windows-amd64.zip
Starting with the Docker images
There are lots right now. Let's move to like...
Discovered while writing #57: we're feeding IP addresses rather than hostnames into the Mist load balancer:
▶ curl 'http://localhost:8042/video+PLAYBACK_ID'
185.102.219.180
It would be good to use hostnames instead so that we can redirect people to valid HTTPS URLs and do DTSCS properly.
Commit is here: https://github.com/DDVTECH/DMS/pull/105
So we need that to work and then we need to teach MistController/MistUtilLoad to utilize it when connecting across to different regions.
Check out this build. It's for commit cc9d188, right? Here's manifest.yaml for that commit, showing MistServer version DDVTECH/mistserver@2c33311.
However, the build itself downloads DDVTECH/mistserver@52e1cb9. This shows up in the logs when it does verification:
#24 120.5 I0815 18:17:07.185310 474 downloader.go:67] Doing SHA checksum verification for service=mistserver
#24 120.7 I0815 18:17:07.375632 474 shasum.go:13] Verifying shasum with file=52e1cb9c35555ca67ab215d3f80919b344fb0c32_checksums.txt
I'm guessing the offending error is this, occurring right at the git clone
step?
fatal: Not a valid object name cc9d1881159057f782222e725d03a8f3dd047634^{commit}
Blocked by https://github.com/livepeer/livepeer-infra/issues/917
Canary branch works like #55, except:
canary
like we have with staging
, because we expect upstream projects to push new semver releases when they're ready to goParticularly trying to detect failures shown in #65
mistserver builds in catalyst is broken and reports the following when running make mistserver
:
Builing release "Generic_64" for version "mist-v3.0-130-g3f81819" @ debug level 4
Building with SRT
Building without RIST support
SRT-LIB-IS-XXX="/home/parallels/catalyst/build/compiled/lib/libsrt.a"
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
Not cross compiling - building sourcery
-- Configuring done
-- Generating done
-- Build files have been written to: /home/parallels/mistserver
+ make -j2
make[1]: Entering directory '/home/parallels/catalyst/build/mistserver'
make[1]: *** No targets specified and no makefile found. Stop.
make[1]: Leaving directory '/home/parallels/catalyst/build/mistserver'
make: *** [Makefile:60: mistserver] Error 2
We need have a suite of integration tests and run them as part of the CI pipeline.
This GH Issue includes:
Test scenario:
At first glance, it seems we could use test-containers and start a few DMS instances locally. We already used test-containers library inside go-livepeer, for the purpose of E2E Testing.
Blocked by DDVTECH/mistserver#61; blocking auto-deploys of Mist
After make-dev
in latest catalyst repo:
Error: unknown shorthand flag: 'j' in -j
Usage:
livepeer-transcode [command]
Available Commands:
help Help about any command
list-presets Lists transcoding presets
transcode Transcodes video file using Livepeer API
Flags:
-a, --api-host string Livepeer API host (default "livepeer.com")
-k, --api-key string Livepeer API key
-h, --help help for livepeer-transcode
-v, --version version for livepeer-transcode
Use "livepeer-transcode [command] --help" for more information about a command.
make release
or whatever), a new CI action gets kicked offmanifest.yaml
staging
. This branch is always force-pushed, so there's just one commit to merge when we want to cut a release.staging
to main
.Probably this'll mean just allowing Mist to pass through stuff that's in a format it's not expecting?
We need to handle incoming requests for e.g. https://playback.livepeer.studio/hls/PLAYBACK_ID/index.m3u8
and 302 redirect them to the address of a Catalyst server that has the stream in question. Previously, we proxied all of these requests behind the scenes, but that led to some suboptimal performance characteristics, especially around segment download latencies.
The workflow will be basically:
GET https://playback.livepeer.studio/hls/PLAYBACK_ID/index.m3u8
▶ curl 'http://localhost:8042/video+PLAYBACK_ID'
185.102.219.180
https://185.102.219.180/hls/PLAYBACK_ID/index.m3u8
There will be some nuance in getting this right, I think... for example, we'll want to check for the X-Forwarded-Proto: https
header and return an HTTPS url if present, that sort of thing.
@iameli TODO: Document why we're doing the flow this way
There's a start here but we need to account for the DTSCS routing (should be another IngressRouteTCP
using SNI routing to route incoming requests to the right place) and the externally-routable HTTPS requests.
We also want to move to separate Kubernetes namespaces for everything, so that should be integrated alongside livepeer/livepeer-infra#875
From time to time, the stream is not load balanced correctly between Mist nodes. I attach the logs of both catalyst nodes.
logs-node-1.txt
logs-node-2.txt
Check out the branch reproduce-load-balancing-issue
. It adds additional logs and increases the timeout of the E2E Test
for i in $(seq 0 30); do go test -timeout 30m -v ./test/e2e/*; done
You should see a message that the test is stuck.
#####
STUCK in requireReplicatedStream
Now, you can check the logs of the containers and investigate the issue.
docker ps
Minimally we need /cmaf/
alongside /hls/
, but ideally we'd have a few more... here's a full list taken from the Mist embed
page:
http://localhost:8080/stream%2bfoo.sdp
ws://localhost:8080/stream%2bfoo.mp4
http://localhost:8080/hls/stream%2bfoo/index.m3u8
http://localhost:8080/stream%2bfoo.mp4
http://localhost:8080/stream%2bfoo.webm
http://localhost:8081/hls/stream%2bfoo/index.m3u8
http://localhost:8080/cmaf/stream%2bfoo/Manifest
http://localhost:8080/cmaf/stream%2bfoo/index.m3u8
http://localhost:8080/cmaf/stream%2bfoo/index.mpd
rtmp://localhost/play/stream%2bfoo
http://localhost:8080/dynamic/stream%2bfoo/manifest.f4m
http://localhost:8080/stream%2bfoo.flv
rtsp://localhost:5554/stream%2bfoo
http://localhost:8080/stream%2bfoo.ts
http://localhost:8080/stream%2bfoo.aac
Okay, we don't want quite all of those...
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.