reel2bits/reel2bits

Self-hosted Soundtracks and Podcasts sharing, with ActivityPub federation.

reel2bits

Demo instance — Matrix room: #reel2bits:otter.sh - Installation & Documentation

Reel2bits is a soundcloud-like self-hosted opensource web application. It allows you to upload tracks, transcode them if needed and publish podcasts or albums.

ActivityPub federation is still a work in progress, everything else works.

Installation

The official documentation is available here: https://docs-develop.reel2bits.org/admin/index.html

Development notes

The Developper documentation is available here: https://docs-develop.reel2bits.org/contributing.html

Translators documentation

Refers to https://docs-develop.reel2bits.org/translators.html

Docker

See documentation for contributing or installing with docker

Default config

• App Name: My reel2bits instance
• App description: This is a reel2bits instance
• App notice: none

Licensing

• AGPL v3

Others projects inspired from

• https://github.com/tsileo/microblog.pub from Little-Boxes ActivityPub backend
• https://funkwhale.audio

reel2bits's Issues

Search

Using postgresql FTS (there is a sqlalchemy lib), should search through Sound and Album, when both .private == False.

The actual search endpoint is /api/reel2bits/search?q=.

We should do FTS on the Activities to return them directly.

Albums can use the Album directly, will uses Activity when they will federate.

Users we can search on Actors (meta_deleted=False)

Correctly handle remote users in frontend

• /:user == local user, /users/:id == remote user
  • handle the route correctly when generating links in UserCard and UserCardList
  • handle the route properly in
    • user profile
    • user followings and followers

Computed user() this.$store.getters.findUser(this.userId) needs to be somehow updated because right now it returns null since it's not an id.
Maybe use username then ID or ID then username.. whatev works best.
Worst cast : ditch the user() if useless, and uses track.account or album.account maybe

Update of js

manual process, bump a few js libs : jquery, bootstrap, wavesurfer, datatables

Tracks improvements

• Genre (https://musicbrainz.org/genres)
• Tags + autocompletion
• Cover picture (+ delete file handling)
• Store original file size in db
• Store transcoded file size in db
• Display both file sizes in the UI (download links and metadatas)

Front translation selection in user profile is overriden by browser language

as said in title.

might need some fixes in app.vue:created(), maybe check if this.currentUser, then dispatch the interfaceLanguage with this.currentUser.reel2bits.lang from it, else use this.autodetect....

Frontend refactoring

Doing a massive front refactoring, including #63 would most likely require switching to a SPA like Vue.js.

However I have never touched ever Vue.js or anything like that, so...

Sentry in frontend

We needs to have Sentry integrated in the frontend.

https://docs.sentry.io/platforms/javascript/vue/ (with logErrors: true).

Backend:

• Add sentry_dsn to NodeInfo

Frontend:

• In after store setup, read sentry_dsn metadata and add it to instance store
• Plug Sentry.init in main.js, probably right after afterStoreSetup so nodeinfo has populated the instance store.

[question] Funkwhale compatibility?

Is this project compatible with Funkwhale (since they use the same protocols)?

Refactore mail sending usage

Various backend parts currently do import mail from app, we can use instead:

    mail = current_app.extensions.get("mail")
    mail.send(msg)

This will resolve potentially recursive imports, and it better to use.

The app.py then needs to be used like:

mail = Mail(app)

instead of mail defined outside of create_app() and mail.init...(app).

OAuth: Better template for /oauth/authorize

We needs to have a not-too-ugly very basic template returned on GET /oauth/authorize to ask for user/password and redirect or show the token thing.

File: api/controller/templates/oauth/authorize.jinja2

Seconds to human is wrong

displays "54" instead of "1:54" for 6869.6333106576 duration.

AP_ENABLED=False should only disable the endpoints and federating the object

... not creating it, because timelines rendering needs the activities to build them.

implemented in post_to_outbox etc. should work. need some testing just in case something was missed.

bring back avatar + uploaded a track/album xx days ago to timelines

as said in title

Investigate AUTHLIB_INSECURE_TRANSPORT=1 needed

Somehow Authlib only works with AUTHLIB_INSECURE_TRANSPORT=1 even in production.

Is it a bug on our side ? Authlib side ? HTTP Headers not recognized ?

Atom feeds

for Albums and User, only public content

/users/xxx/feed.atom
/users/xxx/album/xxx/feed.atom

served by backend

Extract player into component

We should extract the player into its own component

name: ContentAudioPlayer
will contains: wavesurfer div, cover, title, actions and timers
props: track (object)

notes: logo rotation should works

bonus: unique in store WebAudio backend

backend improvements

• Better config management with loading from ENV
• Move backend to api/

createuser slug not populated

Describe the bug
When using reel2bits fresh install, upload a file give an error:

Internal Server Error

The server encountered an unexpected internal server error

(generated by waitress)

• reel2bits-web logs
• reel2bits-worker logs

To Reproduce
Steps to reproduce the behavior:

1. Install reel2bits on Yunohost
2. Register
3. Click on top left menu UPLOAD
4. Click on Choose a file and choose an audio file
5. Click on form button UPLOAD

Expected behavior
Not an error

Screenshots
With DEBUG = True

With DEBUG = False

Desktop :

• OS: windows 10
• Browser : chrome 73

Additional context

1. When manually putting a value to slug field for the user in the postgresql user table, problem is solved
2. slug field is empty when creating user from the website or flask createuser

ActivityPub support tracking issue and audit

ActivityPub tracking:

Custom AP Context is in api/activitypub/vars.py and used for Audio create and update.

Frontend

• Handle correctly remote users (IDs) vs local users (username) (#155)

Various to implement

• /api/v1/instance
• /.well-known/host-meta
• Deactivation of broadcasting the federation (#142)

Mandatory Actor changes

• Every Actor should have an User, with local=False
  Best way might be to change models.py::create_remote_actor() to create the User.local=true at the same time.
  Returning an actor, user, and editing functions using create_remote_actor() to handle thoses two and save them.

Timelines

• Home "Your feed"

Sound (re)publishing

• Create
• Update (not handled in Pleroma or Mastodon)
• Handle fetching the original activity if getting an Update for an unknown track
• Delete
• Boosts
• Likes
• Important for incoming track handling: Federates track (#283):
  • Genre (string)
  • Tags (array of string)
  • licence (dict{licid: integer, name: string})
  • artwork (string, url)

Sound incoming (#283)

• Create
• Update
• Delete
• Various listing
• Boosts
• Likes

To check:

• Actor Delete properly sent (to=)
• Sound Delete properly sent too (to=)

Things:

• We probably should compute and store the count of followings/followers/tracks for remote users instead of having "0" displayed
• Check if user Updates are properly handled

Missing in frontend:

• Comments tab on track
• Likes tab on track
• Boosts tab on track

Reports

• handle the same AP implementation as mastodon
• implement AdminFE pleroma backend endpoints
• leverage the AdminFE reports support #127
• add reporting modal in UI for end-users (logged-in)
• Note: this would needs to add an Instance Actor, which would be used for the report origin user, so the reports are not tied to the real user for privacy reasons.

Static instance announcement

Show it right on top of timelines tabs, only in home.

Backend:

• Add column 'announcement' in Config table
• Export this announcement through nodeinfo 'metadatas.announcement'

Frontend:

• Display it in an alert, dismissable, in home

Sentry update

IIRC raven is deprecated, should use new method here https://sentry.io/for/flask/

Move track_size_limit to config

Uses config setting for track_size_limit instead of Reel2bitsDefaults.track_size_limit

Albums improvements

• tags / categories (like #149)
• cover upload

`flask users list` shows wrong confirmable/active flag

As said, if user isn't confirmed it says it is activated, but it should more reflect the confirmed state or no.

Maybe add another column for confirmed (date).

Admin frontend

Right now only the basics of user management provided by pleroma Admin-FE have been implemented.
Forking admin-fe could be a good choice (we get the reports UI as a bonus)

Should also have celery things:

• admin interface
  • jobs failed
  • retry

Nodeinfo 2.1

Nodeinfo 2.1 should be added to use the URL field (and be able to set it to a custom repository)

Should also reflect in the frontend

Better user deletion

• Proper User deletion
  • should delete the User
  • all Album, Sound, SoundInfo, UserLogging, Oauth2*, Logging and Follower should be deleted
  • Activity are kept, but meta_deleted=True
  • the Actor should be kept, but introducing a new flag : deactivated
  • if Actor.deactivated = returns 404, whatever
  • needs to be used for new user registration (no username reuse)

New user gets undefined quota on upload page

Better email templates

Templates to update (add a footer with instance name, url, infos etc.):

• templates/email/*
• templates/security/email/*

Dependabot can't evaluate your Python dependency files

Dependabot can't evaluate your Python dependency files.

As a result, Dependabot couldn't check whether any of your dependencies are out-of-date.

The error Dependabot encountered was:

Illformed requirement ["==0.34."]

You can mention @dependabot in the comments below to contact the Dependabot team.

Catch FSConfirmable errors in registration

If FSConfirmable explodes it should be try/except and return a proper json error to be catched on frontend.
Should be fixed in registration endpoint (/api/accounts something something)

Add mastodon-api-compatible rendering of timelines

This will make whem mastodon-api compatible too.

Add paginated=true in frontend (in linkGen of timeline.vue I think).

if request.args has 'paginated=true' then it's the paginated method, else we need to handle since_id and count, and it's 'mastodon-method'

Timelines sidebars

Timelines should have dedicated sidebar with popular tags and genres.

Backend:

• Endpoint for populars tags
• Endpoint for popular genres

Frontend:

• Timeline to mockup (they should have the actual Footer element !)

Add a CLI command to test emails

As said, something like flask reel2bits test-email.

Click ask for an email, subject and body will be hardcoded.

Should be added to docs:

• commands help
• "mandatory" installation step

User quota

User quota limit (disk space)
Admin config possible.
track_size_limit to add to nodeinfo custom metadatas

Re tweak pixels-per-second

100 instead of 10 ?

OpenGraph / Card thing

We need to implement that thing

Refresh user object in frontend

After uploading or deleting tracks, the user object isn't refreshed, the quota won't change or track count, something needs to be done on that side.

Two things can be done:

• Anything changing the content of a Track or User in the backend should returns a Track and/or User, that way it is going to be updated in the frontend (apiService might needs some edit)

• Using the notifications timelime to push new User on it that would be updated, we don't have that timeline implemented in the backend however.

The latter needs to have a notification read implemented to avoid the timeline returning all objects every time.

Documentation review and improvements

Issue to track improvements and things to do on the documentation (https://docs-develop.reel2bits.org/)

Tagging @smonff and @papey when you have time.

Self TODO for what needs to be documented (not exhaustive):

• Targets from flask db-datas in Administration part

Continuous player on listings

Like on SC, being able to play directly from a profile list by example.

Some things that might needs to be done:

• each player initiate a wavesurfer into a list or dict in the player state.
• a queue needs to be implemented in the player state.
• if a timeline loads, the queue gets feeds with the first page
• playing one track will switch to the next in queue, marking the previous one as played (the queue might needs to be a dict then or list of dict)
• when track is track position == length(queue)-1 : load the next timeline page in background and feed the queue --> this means not having a displayed wavesurfer in the page, except in the bottom player
• the bottom player uses http://wavesurfer-js.org/example/minimap/index.html
• bottom player has volume (if possible), previous, next, the minimap, the artwork, and a truncated clickable title
• the bottom player should stay on top (how we can do that ?)

Note that we might get bitten by the wavesurfer 1462 bug

Get the docker images back to life

Two images:

• web

• workers

• configurable storage volume

• settings possible using ENV

• docker-compose.yml for dev environment

• documentation update

Atom feeds improvements

Feedgen does handle a podcast extension (https://feedgen.kiesow.be/#extensions)

>>> fg.load_extension('podcast')
...
>>> fg.podcast.itunes_category('Technology', 'Podcasting')

We should add an input field to Albums, with auto completion maybe, comma separated list of categories to be exposed to the Atom podcast feed

Embedded player/iframe

We would ideally need an embedded player/iframe.

Rendering code placeholder is in api/controllers/embed.py:iframe().

Should use the less possible CSS files or libraries, a bundled CSS should be sufficient, and a static wavesurfer js file (to put in api/static/js/.

Rendering is done by the server using a jinja2 template.

Authlib crimes clean

We have a few authlib horrors I would like to get rid of:

• oauth2_client never get a proper user_id set..

User search is wrong and broken

We should rework the whole search to include users (remote or local) and songs, maybe albums.

Currently it's only user search, and SQL JOIN following part is broken.

New search needs to account user being not logged (no following status !)
And search should returns search[accounts] plus search[songs]

Something like

search[songs] == songs matching search, FTS
search[accounts] ==
  if s https://
     match actor url in db ?
  else if s user@instance
     match actor username and instance in db ?
  else
     match actor username in db using LIKE ?
  or, no results for the three cases
     do a webfinger, it will return only one user

Transcoding improvements

If the file can't be found or some errors happens the file might not get transcoded/processed and cannot be again.

• properly reflect a not available song in frontend and API
• add a way to clear the song and allows processing and transcoding again

Debian provides python >= 3.6, now

You can't say "bye bye Debian stable" anymore.

https://packages.debian.org/fr/buster/python3

CI: skip building docker for PRs

I should look to make drone ignore building the docker images when pushing a PR to reduce build time

License change

Change license from MIT to AGPL.

The new front is going to use snippets from elsewhere and since pretty much everything else is already AGPL there is little to no choice but to change licence.

Handle frontend errors correctly

Fails in register:

Should be checked/implemented/fixed in register and login, perhaps elsewhere.