ssssam/calliope

Playlist toolkit - MOVED, now at https://gitlab.com/samthursfield/calliope/

calliope's Issues

Test suite broken

Commit 5e70f85 broke the test suite completely as we can no longer import the calliope module in the tests.

I have no idea how to fix this without regressing the changes from that patch. Perhaps, rather than templating the init.py file, we should just dump the build configuration as a .json file and read the file every time init.py is imported??

Rhythmbox plugin to show how 'mainstream' each artist is

We could generate a 'mainstream' percentage based on how many listeners an artist has on Spotify, Last.fm. There may be other data sources too. This could be shown in Rhythmbox to allow filtering music by how 'mainstream' it is.

Add `cpe tracker expand` command

This would expand the 'albums' field to contain actual tracks. Currently annotate does this, but that is a bit weird -- see #11 .

We could also add a --with-tags or --without-tags option to include, for example, only studio albums by passing --without-tags live. This depends on #10 to be useful.

Fully document playlist format

We should have human-readable documentation of the playlist format.

We should also have a way of automatically validating a playlist.

So far the format is as follows:

• 'track' entries are 'normal'
• 'album' or 'artist' entries can also exist, as a useful shorthand
• standard keys don't contain a '.'; tool-specific keys do

Some points to consider:

• using '.' in key names doesn't play well with jq
• at one point I thought it was a good idea to have multiple tracks inside a single toplevel item. cpe tracker local-albums still does this (#17 should fix that). Since we switched to the json-seq format, it's become obvious that nesting multiple tracks inside of one toplevel json object will negate the benefits of that format.

Allow selectively enabling components at build time, instead of selectively disabling

Calliope's build system allows you to disable components you don't want. However, every time new component is added, dependent projects might need to extend their list of opt-outs. It would be easier for them if the default was opt-in.

The Meson 'feature' option type might help here -- users could use -Dauto_features=disabled to disable everything.

Link to individual commands in documentation

Currently inside use-cases/ we say things like "run cpe diff" but we don't link to the documentation. I think that sphinx-click doesn't provide the necessary 'target' entries in the doctree to allow that. This probably needs to be fixed upstream in sphinx-click.

Desktop playlist manager app

I'd like a simple GNOME application that lets me easily see and modify playlists.

Features:

• manage remote playlists (in spotify, youtube etc.) as well as local ones

Minimum feature set:

• viewing and editing playlists

Not needed for first version:

• playback
• recommendations when adding songs

Automate test suite on every push

I make too many mistakes to leave this sort of thing to chance.

We need a Docker image that contains all of Calliope's dependencies. Then, a circleCI job should be easy enough to set up.

Add a Python API

The primary API to Calliope is the commandline interface. It can be called from any programming language in the world and can also be used directly from Bash. Great.

As an optimization we could expose the underlying Python API as well. This is how I imagine it working:

• each module inside calliope/ contains an api.py file, which defines the public API.
• the API is documented using docstrings in the api.py files
• sphinx-autodoc is used to generate reference documentation, and we use the existing CALLIOPE_DOCS_BUILD hook so that we don't require the full set of dependencies when building the docs
• the cli.py file becomes a wrapper around the api.py files.

The only big downside here is that we'll have mostly duplicate documentation in cli.py and api.py. We may be able to use some clever string substitution to minimize duplication, or we may just have to live with it.

More efficient cache

The current method of caching data returned by the API is very simple and stupid: we rewrite a .json file on each cache update.

Reads are fairly cheap in this model but writes are expensive, it's starting to burn my CPU and my SSD.

One fix would be to throttle writes to e.g. once every 30 seconds. We would need to ensure the write is completed on process shutdown, though. An explicit cache.close() method would help, but it would be easy to forget to call this & not notice.

Alternately, a more sophisticated key-value data store could be used. First let's look at the built-in UNIX databases ndbm / gdbm, these are likely to be available on all systems.

If that fails we could investigate other tools such as Redis. These will be an extra dependency though.

`cpe tracker show` outputs bad results

It currently outputs invalid YAML, and seems to list all albums totally incorrectly.

Rhythmbox plugin to show country associated with each artist

The cpe musicbrainz tool can provide country information. It would be interesting to show a 'country' column in Rhythmbox and allow sorting by country.

Automated tests for each use case

Now that each use case is documented in docs/, we can add a corresponding test case for each one to ensure that it still works.

Website to explore and suggest music by record label

Render playlists and collections to HTML

We need to have a pretty view on the data, and HTML is about the most convenient and platform-independent way to achieve that.

Import images from last.fm

We need a cpe lastfm command that will annotate playlist with metadata using the Last.fm API.

The best API wrapper to use seems to be: https://github.com/pylast/pylast

`cpe tracker annotate` shouldn't expand 'albums' lists by default

This is convenient but inconsistent behaviour. It could perhaps be enabled by a flag.

sync: Take into account numbered directories already present

If I cpe sync --number-dirs to a location that already has some numbered directories, it should be possible to start numbering the new directories higher than the existing ones rather than overlapping.

E.g. if I have in my target dir:

001_album1
002_album2

if I now sync a playlist containing 'album3', it should go to 003_album3 not 001_album3

Integrate pylint with Meson

I want to have pylint running at build time. Ideally the warnings could be treated as errors by the CI system too.

This will require some selective disabling of pylint warnings, and some fixing of mistakes.

Rhythmbox plugin to recommend music from user's collection

I'd like a plugin that gives a 'similar artists' and 'similar albums' results based on the track currently playing.

This would draw on external data such as:

• last.fm tags from cpe lastfm
• last.fm 'related artists' info

Spotify also provides 'related artists' but we must authenticate with the Spotify API for this info, which Rhythmbox so far doesn't do.

This could possibly be implemented in the Rhythmbox last.fm plugin, although that code is in C and is part of Rhythmbox core.

Use case: find music that you listen to regularly but you don't have in your local music collection

Ways of discovering 'music that you listen to regularly':

• lastfm listening history
• lastfm library
• spotify library
• spotify playlists

The most complete data should be lastfm listening history.

Add initial `cpe suggest` command

First cut of this will be to suggest tracks from a collection given a set of existing 'model' playlists.

Online documentation

cpe tracker: List music in arbitrary directories

Currently cpe tracker only finds media that the session-wide Tracker daemon is configured to find.

I prefer to control everything with commandline arguments to the process itself. I started a helper library named https://gitlab.gnome.org/sthursfield/tracker-app-domain to try and make this possible by wrapping the existing Tracker API.

Add `cpe merge` to combine metadata from different tools

My current requirement is: I need the 'tracker.url' field to be promoted to the 'url' field in the never-listened use case, so that I can create an .m3u playlist from the result.

This tool is crucial for Calliope to be useful, we need to be able to combine metadata from multiple sources to produce the 'best' output. For example if Tracker and Musicbrainz have different options about the 'artist' for a track, the Musicbrainz version should be preferred.

Allow loading playlists in multiple formats

I made a mistake when I changed the input format to be JSON only.

The problem is that tools now can't load normal JSON files, because they expect each line to contain a valid JSON object. So a file that starts [\n will trigger an error like this:

*** json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 2)

Very bad.

I want to support both whole files and jq-style streams, so it seems we need to do some kind of autodetection. This could be as simple as trying 2 loaders in sequence by default. We would need a flag to avoid the autodetection.

This could also provide a way to reintroduce YAML support, as the performance issues with large YAML files will go away if we have a way to tell the parser to treat it as JSON.

It could even provide a way to remove the cpe import command; but I feel like this is inviting too much complexity in the core. The core can deal with multiple serialization formats but not different data formats.

`cpe tracker scan` doesn't seem to pick up every album

Some albums do not appear in the collection even when cpe tracker scan is explicitly pointed to the directory containing them. No error is reported either.

Examples from my machine:

• Squarepusher - Hello Everything

Tracks with multiple artists

A lot of Calliope expects tracks to look like this:

artist: Queen
track: Under Pressure

However, some tracks have more than one artist. We throw away this info where it is available. A better approach would be something like this:

artists:
  - Queen
  - David Bowie
track: Under Pressure

See if defaulting to JSON-only input provides a performance boost

Parsing a large YAML collection can take 10 or 20 seconds on a fast machine.

If we default to JSON output (or JSON-LD :-) we may see a significant performance boost. It's still a requirement that users can write playlists as YAML, so we will either need to add a --yaml option to enable YAML parsing or suggest use of a tool such as yq.

Concurrency issues with GdbmCache

Sometimes cache operations fail with 'Resource temporarily unavailable' due to the locking in GDBM. We should catch these and retry a few times before failing.

Allow tagging albums, tracks and artists

Tags are generally great for everything; one specific use case is tagging live albums with 'live' so they can be separated from studio albums.

We could add cpe tracker tag command, or add tags to an existing collection file using 'tags' key and pipe the results to a cpe tracker store command.

cpe tracker show should also show whatever tags are stored.

cpe tracker show: Allow showing flat list of artists, tracks or albums

Currently cpe tracker show outputs a structured list grouped by artist, then album, then track. This is a good default but it doesn't serve all purposes.

I'd like to pick a random set of artists, or albums or tracks, for that I need a flat list that I can pipe to sort -r.

We need cpe tracker show-artists, cpe tracker show-tracks and cpe tracker show-albums; the current cpe tracker show should become show-albums.

Automatically fix tags

Some songs in a local music collection often lack tags.

We can identify these with cpe tracker somehow, find the correct tag information using cpe musicbrainz, and then write the tags to the file using a new component. (Possibly extending cpe tracker to support writeback).

Use case: explore geographical distribution of artists you listen to

First cut would be:

• list artists in collection with cpe tracker
• use cpe musicbrainz to annotate collection with country codes
• output data suitable for visualizing in some web-based tool

Ideas for web visualization:

• https://bl.ocks.org/mbostock/2206590 (d3.js zoomable map)
• http://bl.ocks.org/lokesh005/7640d9b562bf59b561d6 (d3.js lat/long points on map)
• http://leafletjs.com/examples/choropleth/ (leaflet.js heatmap of US states)
• http://www.simile-widgets.org/exhibit/ (interactive map widget)

Further extensions:

• more ways to import data: last.fm history, Spotify play history, Bandcamp follows, ...
• publish an example map
• document how to create your own map

Annotate artists with tags from last.fm

We need a cpe lastfm annotate-tags command, which would add the lastfm top tags to each item in a playlist.

This could be used to find everything in your collection that is considered 'ambient', or whatever.

Playlist viewer desktop app

I would like a desktop app that can show me the items in a playlist. I will make a simple proof of concept using GTK+ and see how it goes.

`cpe tracker` should warn if no content is indexed yet

It's not obvious that the cpe tracker scan command needs to run before any of the commands will do anything. So we should warn the user in a situation where cpe tracker scan needs to be run.

Include reference documentation in online documentation

We should document each cpe subcommand as part of the online documentation, which is currently hosted at https://calliope-music.readthedocs.io/

The information can be generated from the code using sphinx-click. This works fine locally but it's much harder to get this to build in readthedocs.org.

Options are:

• hack something such that the sphinx build can do from calliope import cli right from the source tree. This should be doable if we separate all click-related code into a single file; we can then add that file to sys.path manually in docs/conf.py, and set a _CALLIOPE_CLI_STUB flag to avoid that module importing any others. It will require a bit of churn to set up a consistent internal API between the .cli module and the various others, but that may have other benefits too.
• write a requirements.txt and a setup.py file that installs enough of Calliope that the docs build succeeds.

I prefer the 1st option.

Deduplicate two different encodings of the same album

If there is an .mp3 and .flac version of the same album for whatever reason then it is treated as two different albums. It would be better if one 'shadowed' the other and the highest quality encoding was used.

It would be nice to handle this in Tracker itself rather than in Calliope, but that would be tricky to implement as it would have to implement its own heuristics.

A simpler option would be to have cpe musicbrainz annotate figure out the musicbrainz ID for each album. We could then easily spot duplicates.