dreamvo/gilfoyle

Distributed video encoding, hosting and streaming

⚠️ Development on this project has been stopped due to lack of activity. Project should be considered as unstable and outdated.

Distributed video encoding, hosting and delivery. By Dreamvo

Gilfoyle is a web application from the Dreamvo project that runs a self-hosted media streaming server. Allowing you to easily implement media streaming in any application at any scale.

It's written in Golang, mostly designed for Kubernetes and runs as a single Linux binary with PostgreSQL and RabbitMQ.

Table of content

• Features
• Current status
• Design
• Roadmap
• Documentation
• Contributing
• Discussion

Features

• Deploy a RESTful API and HLS server to manage and stream audio & video
• Handle media compression and encoding with FFmpeg
• Customize media renditions
• Monitoring: Prometheus exported metrics, embedded Web UI
• Media thumbnail & video preview generation
• Enjoy highly scalable & cloud-native architecture

What's next ?

• Authentication and delegated upload
• Caption support
• Player customization
• Multi stream support (e.g: one audio stream per language)
• Live streaming
• IPFS support
• Encryption support

Current status

It's a Work In Progress. As this project is very recent, it's under heavy development and not suitable for production yet. Please consider v0 as unstable. Want to contribute ? Check the backlog.

Design

See this document for a high level design and goals.

Roadmap

Phase 1 (v0.1)

This first phase aim to build a first working version of this software with a minimum of test coverage and bug fixes. The first working version is v0.1.0 whose roadmap is available here.

Phase 2 (v0.x - current)

This second phase is about getting more contributors and feedbacks to fix more bugs and improve testing. Feedbacks and tests should make us able to validate design and create a roadmap for the first stable release (v1.0.0).

• Roadmap for v0.2.0

Phase 3 (v1)

This third phase should allow us to make the program more stable and welcoming to new users. Stability will make us able to publish donation goals in order to continue improve the OSS project and build a SaaS product.

Documentation

• For contributors: see godoc, high-level design documentation
• For users: see user guide and API documentation
• The GPL v3 license

Contributing

This project is in an early stage. We appreciate feedbacks and discussions about the design and features.

Discussion

• Discuss on GitHub
• Report a bug
• Follow us on Twitter
• Contact us at [email protected]

gilfoyle's Issues

Use go:embed instead of statik for static assets

Summary

Go 1.16 has an official support for static assets. Unofficial implementation should now be deprecated.

References

• https://pkg.go.dev/embed

Fix OpenAPI spec

spec is invalid

• Add OperationId tag (// @ID get-string-by-int)

Roadmap for v0.1.0

List of features/changes we want to finish before releasing an official v0.1.0 version:

• Web framework #14
  • OpenAPI specs
  • ORM & schema implementation
  • Validation
  • JSON logging
• Storage API #32 - done
  • Filesystem
  • S3 storage
  • Google Cloud Storage
• Upload tunnel (video only) - done
  • REST routes #36
  • Async transcoding #85
• Streaming tunnel #85 done
  • REST routes
  • HLS playlists
• Worker implementation #62 - done
• Dashboard base implementation #46 - done
• Dashboard UI framework implementation #73 - done

Please note that this list is changing.

Configuration watcher

Add hot reloading on configuration file, when provided. This can be useful when configuration is provided by a k8s configmap.

Implement Player generation & oEmbed endpoints

This feature should allow a user to easily implement a HTML5 production-ready video player into his own web page.

Example use case

1. User makes an HTTP request to POST /players player options included, to create a new player
2. User retrieve the newly created player if needed by making a request to GET /players/:player_uuid
3. User can now make a request to GET /players/:player_uuid/oembed?url=media_url&format=json to embed the player in its own page

Implementation

Routes

• POST /players - Create a player
• PUT /players - Edit a player
• DELETE /players - Delete a player
• GET /players - Retrieve all players
• GET /players/:uuid - Get one player
• GET /players/:uuid/oembed - Get oEmbed object
• GET /players/:uuid/embed/:media_uuid - Get HTML5 player to embed

Player generation

• Solution 1 : store static assets in the VCS, then embed them in the binary then serve the content as a single HTML response. Assets include : videojs-css, videojs-js. VideoJS options are altered according to the player's options.

Example oEmbed response

• GET /players/fa3b3ec9-2388-4196-96cc-d7ca1a86f3da/oembed?media=fa3b3ec9-2388-4196-96cc-d7ca1a86f3da&format=json

{
  "title": "cyberpunk 2077 gameplay",
  "author_name": "",
  "author_url": "",
  "type": "video",
  "height": 640,
  "width": 360,
  "version": "1.0",
  "provider_name": "Gilfoyle",
  "provider_url": "https://demo.gilfoyle.dreamvo.com/",
  "thumbnail_height": 360,
  "thumbnail_width": 480,
  "thumbnail_url": "https://cdn.demo.gilfoyle.dreamvo.com/fa3b3ec9-2388-4196-96cc-d7ca1a86f3da/previews/fa3b3ec9-2388-4196-96cc-d7ca1a86f3da.jpg",
  "html": "<iframe width=\"200\" height=\"113\" src=\"https://demo.gilfoyle.dreamvo.com/players/fa3b3ec9-2388-4196-96cc-d7ca1a86f3da/embed/fa3b3ec9-2388-4196-96cc-d7ca1a86f3da?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe>"
}

We'll only support JSON format for now. XML will come later.

Player options

• loop: bool
• controls: bool
• autoplay: bool
• muted: bool
• preload: string
• playbackRates: []float32

References

• https://oembed.com/

Use goreleaser to push Docker image

Summary

Goreleaser have native support for pushing Docker images. We should take advantage of it.

References

• https://goreleaser.com/customization/docker_manifest/

Examples

dockers:
- image_templates:
  - "dreamvo/gilfoyle:{{ .Version }}-amd64"
  use_buildx: true
  dockerfile: Dockerfile
  build_flag_templates:
  - "--platform=linux/amd64"
- image_templates:
  - "dreamvo/gilfoyle:{{ .Version }}-arm64"
  use_buildx: true
  goarch: arm64
  dockerfile: Dockerfile
  build_flag_templates:
  - "--platform=linux/arm64"
docker_manifests:
- name_template: dreamvo/gilfoyle:{{ .Version }}
  image_templates:
  - dreamvo/gilfoyle:{{ .Version }}-amd64
  - dreamvo/gilfoyle:{{ .Version }}-arm64

Creatre control plane application

Goal: create a control plane application to control and monitor a Gilfoyle instance from the command line, through the REST API.

The application would use github.com/dreamvo/gilfoyle/sdk package.

Command examples

# Add a new server
$ gilfoylectl add 127.0.0.1:8080 --api v1 --name default

$ gilfoylectl get jobs --server default --status INPROGRESS
ID       STATUS                 QUEUE                  CLASS
1         Processing            TranscodeVideo     1080@60

$ gilfoylectl get info --all-servers
HOST                 NAME        VERSION
127.0.0.1:8080    default     0.1betaRC1
127.0.0.1:8081    server2     0.1betaRC1

Add custom Prometheus metrics

Goal: create custom Prometheus metrics to monitor an instance.

• Number of medias (counter)
• Number of medias with status Errored (counter)
• Number of medias with status Processing (counter)
• Number of medias with status Ready (counter)
• Total gigabytes of storage used (counter)

References

• https://github.com/siimon/prom-client
• https://github.com/apertureless/vue-chartjs

Implement dashboard

Add a command that launches a web server which serves a GUI for interacting with a Gilfoyle instance. It can be done with Vue and a UI framework. The implementation method still needs to be specified. We can either inject assets directly into the binary, or put the code in another repo, which will be then downloaded by the CLI.

Important: requests to instance must be server-sided in order to prevent user from exposing the API endpoint, if he don't want to. So the dashboard would have its own API for client-side requests. User can put the dashboard behind a bastion or basic auth.

$ gilfoyle dashboard -p 5000 --endpoint 127.0.0.1:3000

Features

• Manage medias, get info about media files
• Manage media attachments
• Upload a new media
• Watch a media
• Get info and metrics about the instance (version, storage usage, ...)

Delete media files and storage folder on media delete

Route "DELETE /medias/:id" should delete storage related to the media. Also, media deletion should be forbidden if the media has state "Processing".

Create HLS streaming route

After #5 and #48

Goal: create a route to receive a blob stream of the requested video file.

Routes

• GET /medias/:id/stream/playlist
• GET /medias/:id/stream/playlist/:playlist/:filename

Example

# Master playlist
curl -X GET localhost:3000/medias/:uid/playlist

# Rendition playlist
curl -X GET localhost:3000/medias/:uid/playlist/:playlist/:file

# Examples
curl -X GET localhost:3000/medias/e6d8ea03-3a1c-41b4-9c50-e200d967b5a9/playlist -o playlist.m3u8

curl -X GET localhost:3000/medias/e6d8ea03-3a1c-41b4-9c50-e200d967b5a9/playlist/medium/index.m3u8
curl -X GET localhost:3000/medias/e6d8ea03-3a1c-41b4-9c50-e200d967b5a9/playlist/medium/source0.ts

References

• https://qvault.io/2019/12/03/golang-video-stream-server/
• https://github.com/gwuhaolin/livego

Implement video player

References

• https://videojs.com/city
• https://github.com/video-dev/hls.js

Pagination & ordering

GET /v1/videos?limit=100&offset=10
GET /v1/videos?orderBy=created_at&order=desc
GET /v1/videos?status=ready

limit cannot be lower than 1, higher than 100 ; default value is 50

Command version

A command that prints the program's version.

$ gilfoyle version
Gilyfoyle version v0.0.1 commit b02d446

SDK implementation

Create a public interface in gilfoyle package to interact programmatically with a Gilfoyle instance. The implementation can be generated from the swagger specs.

Example

type SDK interface {
    GetVideos(...gilfoyle.ListOpts) []*gilfoyle.Video
    GetVideo(string) *gilfoyle.Video
    CreateVideo(*gilfoyle.Video) *gilfoyle.Video
}

// usage
g := gilfoyle.NewClient(&gilfoyle.ClientOpts{})

v, err := g.CreateVideo(&gilfoyle.Video{})

Configuration handling

Handle configuration options with config file or env vars.

• Configuration must follow config.Config structure
• Configuration file format can be yaml or json

Examples

$ gilfoyle serve --config ./gilfoyle.yml
$ gilfoyle serve --config ./gilfoyle.json

Original policy

Create a config option to set the Original Policy. Original Policy defines how the original file must be handled after processing succeeded.

Values :
Delete : delete the file
Retain : keep the file in the storage

Docker image deployment

Continuously deploy image to Docker hub : dreamvo/gilfoyle:latest

Command generate-keys

A command that generate self-signed certificate and an event signing key for production SSL. Certificate will be used to serve on port 443.

$ gilfoyle generate-keys --tls-cert server.crt --tls-key server.key
$ gilfoyle serve -p 443 --tls-cert server.crt --tls-key server.key

Once created, user needs to set cert and key paths in config. This configuration is optional and the application always fallback to port 80 if there's no TLS configuration.

tls:
  cert: ./server.crt
  key: ./server.key

// gin example
r.RunTLS(":8080", "./server.pem", "./server.key")

References

• https://github.com/matrix-org/dendrite/blob/master/cmd/generate-keys/main.go

Remove IPFS references

Remove references to IPFS to abandon support of it. We don't want to support it anymore because it's not compatible with the current architecture.

User guide

Goal: create a welcoming documentation website, hosted on GH pages, to make new users want to try the product, or use it in production. Gilfoyle is already easy to setup and made for cloud environments. So the docs should be able to show that.

Choose a docs framework:

• Hugo
• Mkdocs
• Docusaurus
• Docsify
• VuePress

Content :

• Home page
• Blog
• GitHub
• Documentation
  • About
  • Getting started
    • Installation
    • Configuration
    • Deployment
  • Uses cases
    • HLS video streaming
    • Audio streaming
  • Integrations
    • Kubernetes
    • Application SDKs
  • Concepts
    • Lifecycle of medias
    • Processing strategies
    • Worker pool
  • References
    • OpenAPI specs
    • Configuration

Upload media file

Goal: Implement some REST routes to upload audio or video file, handle the transcoding process asynchronously then save the files.

Routes

• POST /medias/:id/upload Upload source media file

Accepted extensions: mp4, avi, mkv, webm

Upload tunnel

• Start: Upload the file and store it in a temporary directory with an unique name (e.g: /tmp/<media_file_uuid>), and check its file size
• Perform an analysis
  • Check the file with ffprobe
  • Gather metadata from file (size, duration in sec...)
• Add source to storage
• Save a MediaFile object for "source" preset
• Set media status to Processing
• Start background job
  • Convert source file to webm then add it to storage
  • Save a MediaFile object
  • Continue processing for other formats and generate attachments...
• End: Set media status to Ready

Note: In the first phase of this task, we'll only process the source quality. Then add other background jobs to processing additional formats.

Media entity

• uuid (uuid)
• status (enum)
• name (string)
• kind (enum)
• created_at (date)
• updated_at (date)

MediaFile entity

• uuid (uuid)
• media (1:1 - Media)
• type (enum: video|audio)
• format (enum: m3u8)
• preset (enum: source|audio|144@30|240@30|360@30|480@30|720@30|1080@30|1080@60)
• duration_seconds (float64)
• width (int)
• height (int)
• frame_rate (int)
• created_at (date)
• updated_at (date)

Directory structure

root/
    <media_uuid>/
        source # uploaded file, used as input for transcoding
        playlist.m3u8
        audio/
            index.m3u8
            audio_001.ts
            audio_002.ts
        source/
            index.m3u8
            source_001.ts
            source_002.ts
        144p/
            index.m3u8
            144p_001.ts
            144p_002.ts
    attachments/
        - thumbnail.webp # generated video thumbnail
        - preview.webm # generated video preview
        - subtitles_fr_FR.vtt

References

• https://docs.peer5.com/guides/production-ready-hls-vod/#multiple-renditions
• https://github.com/video-dev/hls.js
• https://developer.mozilla.org/en-US/docs/Web/Guide/Audio_and_video_delivery/Live_streaming_web_audio_and_video
• https://pkg.go.dev/gopkg.in/vansante/go-ffprobe.v2
• https://www.sample-videos.com

User documentation

Goal: create a welcoming user documentation to make new users want to try the product. Gilfoyle is already easy to setup and made for cloud environments. So the docs should be able to show that.

Choose a docs framework:

• Hugo
• Mkdocs
• Docusaurus
• Docsify

Improve validation

• Return error on extra params
• Create a validation error response schema

References

https://github.com/go-playground/validator/blob/master/_examples/simple/main.go

Improve test suites for storage drivers

Goal: improve test suites for drivers of the storage API. Storage drivers are : S3 (S3-compatible APIs), GCS (google cloud storage), FS (filesystem).

Integration tests already exist for all drivers, but S3 uses the minio-sdk. S3 tests currently uses the public instance of minio, which is time and bandwidth consuming. GCS's tests exist but were skipped because it requires an access to an actual GCS bucket. We need a solution to avoid using the network but with a good level of code coverage.

Here's few solutions I came across:

Solution 1: Mock HTTP calls of the storage client. Libraries such as gock can help to mock API calls.

Solution 2: Refactor packages to use an interface which will be mocked in the tests.

Two pass encoding

Two pass encoding allow user to have more control on the output filesize.

Use case

Let's say you want to upload a video of 10min, that is 400MiB for some reasons. That'd mean 1 min of video is 40MiB. We want each video to not be upon 20MiB per minute.

We need to use two-pass encoding to achieve this :

• 10 minutes = 600 seconds
• 20 MiB = 20 000 KiB

(20k / 600) * 60 = 2000 MiB

2M - 192k = 1 808 000 kBit/s is our target video bitrate.

That'd produce the following settings :

encoding:
    strategy: crf
    minute_filesize: 40000
    renditions:
     - name: <defined by user>
        width: <defined by user>
        height: <defined by user>
        video_bitrate: ( (minute_filesize [kB] / duration [sec]) * 60 )
        audio_bitrate: 192000
        framerate: <from original or defined by user>
        audio_rate: 48000
        video_codec: libx265
        audio_codec: aac
        target_bandwidth: (video_bitrate + audio_bitrate)
        preset: medium

Which would produce the following ffmpge options :

$ ffmpeg -i original -y -vf scale=w=1280:h=720:force_original_aspect_ratio=decrease -c:v libx265 -b:v 778k -c:a aac -b:a 192k -sc_threshold 0 -g 48 -keyint_min 48 -hls_time 4 -hls_playlist_type vod -hls_segment_filename output/360p_%03d.ts output/360p.m3u8

References

• https://trac.ffmpeg.org/wiki/Encode/H.264
• https://trac.ffmpeg.org/wiki/Limiting%20the%20output%20bitrate

Low resolutions should scale up to the lowest configured resolution

When uploading a video that has resolution of 144x144 when the lowest resolution in the config is 842x480, the video is ignored and any media file is created.

Storage API

Create a common interface to interact with various storage backends.

Example

storage:
  # choices: fs, s3, gcs, ipfs...
  kind: fs
  cache_path: string # default to ./tmp, env: CACHE_PATH
  fs:
    data_path: string # default to ./data, env: FS_DATA_PATH
  s3:
    host: string
    access_key_id: string
    secret_access_key: string
    region: string
    bucket: string
    useSSL: bool
  gcs:
    credentials_file: string # env: GCS_CREDENTIALS_FILE
  ipfs:
    gateway: string # default to gateway.ipfs.io, env: IPFS_GATEWAY

func main() {
  s, err := storage.New(storage.Filesystem)

  s.Save(ctx, strings.NewReader("(╯°□°）╯︵ ┻━┻"), "test")
}

References

• https://github.com/ulule/gostorages

Webhooks

Allow user to create webhooks for various events related to Medias.
Example include: OnUpload, OnMediaCreate, OnMediaReady.

This can be useful to let user interact with the lifecycle of a media.

Attachment generation: preview image

Goal: create a queue that generates default attachments for video medias.

Example

# Preview image
ffmpeg -i input.mp4 -ss 00:00:03 -frames:v 1 -s 1280x720 -y preview_image.webp

# Preview animation
ffmpeg -i input.mp4 -lossless 0 -ss 00:00:00 -t 00:00:03 -s 320x180 -qscale 60 -y preview_animation.webp

Anonymous tracking feature

Goal

Implement a tracking feature to count views and watchtime for a unique user identifier. This feature should consider user privacy by not using personal information such as IP address. But still can use a SHA-256 hash for example. Its value can be anything matching alphanumeric characters.

Example

• X-Forwarded-For-User: <unique_hash>
• X-Forwarded-For-User: <user_uuid>

Support for unexpected aspect ratio

When uploading a video that has a non 16:9 frame size, one of the encoding jobs fails because of the aspect ratio.

Solution 1

• Case A: we have a configured frame size of 842x480px. When uploading a video with frame size of 513x1003px, we can scale down to the closest width according to the settings and keep the original aspect ratio. The result is : 245x480px ((513/1003)*480).

• Case B: we have a rendition with frame size of 1920x1080px. When uploading a video with frame size of 1920x800px, we can keep the original ratio since it's the same width. The result is : 1920x800px.

• Case C: we have a rendition with frame size of 1280x720px (16:9). When uploading a video with frame size of 1280x1024px (4:3), for each rendition whose width is <= 1280, recalculate the resolution : ((1280/1024)*720) = 900x720px

Implement ffmpeg & ffprobe wrapper packages

References

• https://github.com/dreamvo/go-ffprobe
• https://github.com/dreamvo/transcoder
• https://github.com/giorgisio/goav

Metrics resource

Implement a /metrics route that exposes metrics about the instance. This can be useful to monitor instance quotas.

• https://prometheus.io/docs/guides/go-application/

Result filtering & sorting (medias)

A feature to only get desired results based on query filters.

Filters

• _eq Equal
• _in In array
• _before Before date
• _after After date
• sort Sort results by field
• q search query

List of params

• q
• sort
• status_eq
• status_in
• created_at_after
• created_at_before
• updated_at_after
• updated_at_before

Example

GET /medias?sort=title:DESC
GET /medias?status_eq=Processing
GET /medias?status_in=Processing,Ready
GET /medias?created_at_after=2021-01-23
GET /medias?q=my%20title

Implement JSON http logging

Create a logging / log package to initialize a new logger object and replace the default Gin logger.

The logger :

• can be disabled (e.g: by a CLI flag)
• always logs to JSON

Interface

type Logger interface {
  Enable(bool) Logger
  IsEnabled() bool
  Info(string, map[string]interface{}) Logger
  Warning(string, map[string]interface{}) Logger
  Error(error, map[string]interface{}) Logger
}

func NewLogger() Logger {
  // ...
}

Stdout example

{"level": "info", "message": "server running", "data": {"key": "value"}}

Configuration

# Config file
logging:
  enable: true
  json: true

# Env vars
LOGGING_ENABLE=true LOGGING_JSON=true gilfoyle serve

References

• https://github.com/uber-go/zap
• https://github.com/thoas/picfit/tree/master/logger

Asynchronous transcoding

Goal: Implement the transcoding process asynchronously for each new media files.

Upload tunnel

• Start: Upload the file and store it in a temporary directory with an unique name (e.g: /tmp/<media_file_uuid>), and check its file size
• Perform an analysis
  • Check the file with ffprobe
  • Gather metadata from file (size, duration in sec...)
• Add source to storage
• Save a MediaFile object for "source" preset
• Set media status to Processing
• Start background job in the TranscodingQueue
  • Convert source file to HLS format then add it to storage
  • Save a MediaFile object
  • Continue processing for other formats and generate attachments...
• End: Set media status to Ready

Conversion

formats: HLS (m3u8)
resolutions: 144, 240, 360, 480, 720, 1080 (ignore resolutions higher than source!)
frame rates: default frame rate

Media streaming

Medias can then be streamed using the HLS format.

Example FFmpeg commands

# Encode source to HLS
ffmpeg -hide_banner -y -i input.mp4 -c:a aac -ar 48000 -c:v h264 -profile:v main -hls_time 4 -hls_playlist_type vod -hls_segment_filename source/source_%03d.ts source/index.m3u8

# Convert to HLS
ffmpeg -hide_banner -y -i original.mkv \
  -vf scale=w=640:h=360:force_original_aspect_ratio=decrease -c:a aac -ar 48000 -c:v h264 -profile:v main -crf 20 -sc_threshold 0 -g 48 -keyint_min 48 -hls_time 4 -hls_playlist_type vod  -b:v 800k -maxrate 856k -bufsize 1200k -b:a 96k -hls_segment_filename beach/360p_%03d.ts beach/360p.m3u8 \
  -vf scale=w=842:h=480:force_original_aspect_ratio=decrease -c:a aac -ar 48000 -c:v h264 -profile:v main -crf 20 -sc_threshold 0 -g 48 -keyint_min 48 -hls_time 4 -hls_playlist_type vod -b:v 1400k -maxrate 1498k -bufsize 2100k -b:a 128k -hls_segment_filename beach/480p_%03d.ts beach/480p.m3u8 \
  -vf scale=w=1280:h=720:force_original_aspect_ratio=decrease -c:a aac -ar 48000 -c:v h264 -profile:v main -crf 20 -sc_threshold 0 -g 48 -keyint_min 48 -hls_time 4 -hls_playlist_type vod -b:v 2800k -maxrate 2996k -bufsize 4200k -b:a 128k -hls_segment_filename beach/720p_%03d.ts beach/720p.m3u8 \
  -vf scale=w=1920:h=1080:force_original_aspect_ratio=decrease -c:a aac -ar 48000 -c:v h264 -profile:v main -crf 20 -sc_threshold 0 -g 48 -keyint_min 48 -hls_time 4 -hls_playlist_type vod -b:v 5000k -maxrate 5350k -bufsize 7500k -b:a 192k -hls_segment_filename beach/1080p_%03d.ts beach/1080p.m3u8

# Create Master playlist with multiple formats
# see https://hlsbook.net/creating-a-master-playlist-with-ffmpeg/
ffmpeg -y -i original.mp4 \
  -preset slow -g 48 -sc_threshold 0 \
  -map 0:0 -map 0:1 -map 0:0 -map 0:1 \
  -s:v:0 640x360 -c:v:0 libx264 -b:v:0 365k \
  -s:v:1 842x480 -c:v:0 libx264 -b:v:0 1400k \
  -s:v:2 1280x720 -c:v:1 h264 -b:v:1 2800k  \
  -s:v:3 1920x1080 -c:v:1 h264 -b:v:1 5000k  \
  -c:a copy \
  -var_stream_map "v:0,a:0 v:1,a:1 v:2,a:2 v:3,a:3" \
  -master_pl_name master.m3u8 \
  -f hls -hls_time 6 -hls_list_size 0 \
  -hls_segment_filename "v%v/fileSequence%d.ts" \
  v%v/prog_index.m3u8

References

• https://docs.peer5.com/guides/production-ready-hls-vod/#multiple-renditions
• https://github.com/video-dev/hls.js
• https://developer.mozilla.org/en-US/docs/Web/Guide/Audio_and_video_delivery/Live_streaming_web_audio_and_video
• https://pkg.go.dev/gopkg.in/vansante/go-ffprobe.v2
• https://www.sample-videos.com

connection refused on ./gilfoyle serve

Description
I get this error when i try to start with "./gilfoyle serve -p 8080 --config ./gilfoyle.yml" or "./gilfoyle serve":

{"level":"fatal","time":"2021-12-31T13:09:57.985Z","caller":"gilfoyle/serve.go:65","message":"failed creating schema resources","error":"dial tcp [::1]:5432: connect: connection refused","stacktrace":"github.com/dreamvo/gilfoyle/cmd/gilfoyle.glob..func3\n\t/home/runner/work/gilfoyle/gilfoyle/cmd/gilfoyle/serve.go:65\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:854\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:958\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:895\ngithub.com/dreamvo/gilfoyle/cmd/gilfoyle.Execute\n\t/home/runner/work/gilfoyle/gilfoyle/cmd/gilfoyle/root.go:45\nmain.main\n\t/home/runner/work/gilfoyle/gilfoyle/cmd/main.go:6\nruntime.main\n\t/opt/hostedtoolcache/go/1.15.6/x64/src/runtime/proc.go:204"}

Environment

• OS: Debian 9.8
• Gilfoyle version: 0.1.0 commit 23c0da8
• Go version: 1.13
• FFmpeg version: 3.2.14-1

Docker installation does not work

Hey @sundowndev , I've encountered this issue too,

Here is what I've done so far

I have these docker containers up and running

And this is the docker compose config file i've used to create the containers

This is the Docker postgres container output

And i've confirmed that the database is up and running and I can connect through PgAdmin just fine

And this is the config for gilfoyle docker image I have

I've set up the compose variable based on the config on the gilfoyl image.
When i try to start the app I've get this error

{"level":"info","time":"2022-07-13T19:33:38.242Z","caller":"gilfoyle/serve.go:39","message":"Initializing API server"}
{"level":"info","time":"2022-07-13T19:33:38.242Z","caller":"gilfoyle/serve.go:40","message":"Environment","debug":false}
{"level":"fatal","time":"2022-07-13T19:33:38.243Z","caller":"gilfoyle/serve.go:65","message":"failed creating schema resources","error":"dial tcp 127.0.0.1:5432: connect: connection refused","stacktrace":"github.com/dreamvo/gilfoyle/cmd/gilfoyle.glob..func3\n\t/app/cmd/gilfoyle/serve.go:65\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:854\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:958\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:895\ngithub.com/dreamvo/gilfoyle/cmd/gilfoyle.Execute\n\t/app/cmd/gilfoyle/root.go:45\nmain.main\n\t/app/cmd/main.go:6\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204"}

Originally posted by @SarkarKurdish in #293 (comment)

Job scheduling

Choose and implement a Go worker library to perform asynchronous actions using Redis, NSQ or RabbitMQ.

Libs

• https://github.com/hibiken/asynq
• https://github.com/benmanns/goworker
• https://github.com/jrallison/go-workers
• https://github.com/mattermost/mattermost-server/tree/master/jobs

References

• https://www.rabbitmq.com/consumers.html#single-active-consumer
• https://github.com/streadway/amqp

API access mode

A feature to run the instance in read only mode. That means any resource can be edited through the REST API. This can be useful when all of your instance is public but only your application is allowed to manage resources of the instance. Concerned routes will now have a middleware deciding whether to allow the request or not. If ReadOnly mode & auth are enabled and the request does not contain a "Authorization" header, disallow the request.

Config example

settings:
    access_mode: ReadOnly # values: ReadOnly, ReadWrite, Restricted (default: ReadWrite)
    auth:
        enabled: true
        key: <bearer_token>

Media attachments

Goal: Implement a feature allowing users to add additional attachment files to medias (subtitles, thumbnails...).

After #36

---

Attachments would allow user to upload additional content (e.g: preview images, subtitles...) for a particular media.

Routes

• GET/POST/DELETE/PATCH /medias/:id/attachments/:key Manage media's attachments

MediaAttachment entity

• uuid (uuid)
• media (1:1 - Media)
• key (string)
• filename (string)
• content_type (string)

Directory structure

root/
    <media_uuid>/
        - playlist.m3u8
        - source.m3u8
        attachments/
          - thumbnail.webp # generated video thumbnail
          - preview.webm # generated video preview

ent: validator failed for field \"resolution_width\": value out of range

When setting a width of 3840 for a rendition, the job fails because value is out of range. We can set the limit to 15360×8640 (16K UHD).

"directory is not empty" error when attempt to delete a ready media

Deleting media files fails when attempt to delete the entire media object. Also we should disable the delete button on dashboard when media has state Processing.

Sentry optional integration

To report & monitor errors

Add metadata to healthcheck route response

• Debug value
• Storage driver
• MaxFileSize value
• ExposeSwaggerUI value
• Database dialect

Roadmap for v0.2.0

List of features/changes we want to finish before releasing an official v0.2.0 version:

• Improve Media deletion #98
• New encoding tunnel #139
• Original file policy #99
• Fix aspect ratio bug #134
• Fix low resolutions issue #104
• Fix validation issue with high resolutions #137
• Fix issues with storage when deleting a media #138
• Improve dashboard
  • UI/UX research #102
  • Media streaming #86
  • Media management (add, edit, delete)
• Setup user documentation website #130

Please note that this list is changing.

Make video encoding configurable

After #5, create a configuration for video conversion.

Example

Here's an example from owncast :

hls:
  render_audio: true
  video_renditions:
    - name: low
      videoBitrate: 800
      scaledWidth: 640
      fps: 30
      codec: libx64
    - name: medium
      videoBitrate: 1200
      fps: 30
      codec: h264
    - name: high
      videoBitrate: 1600
      fps: 30
      codec: libx64

Create master playlist after each rendition

When a rendition succeed, the media is not set as ready once all renditions succeeded. This prevent user of consuming already available rendition of the media.

Also, temporary files should be deleted after processing.

Setup security scan in CI

• https://github.com/securego/gosec