nielsdejong/neo4j-playlist-builder

A tool to dynamically generate Spotify playlists using Neo4j graph data science.

neo4j-playlist-builder

A tool to generate Spotify playlists using Neo4j graph data science.

Taking a single (large, public) playlist as an input, the tool uses node similarity algorithms and kmeans clustering to group songs based on genre, energy and mood.

The script is set up to be a complete end-to-end tool for organizing your music:

• It loads your data directly into Neo4j using the Spotify API...
• ...runs graph algorithms to build playlists...
• ...uses keywords to name playlists...
• ...and eventually creates new playlists in Spotify.

Quick setup

This project uses a single Python script: neo4j_spotify_playlist_builder.py. In here, you specify the parameters needed to connect to your spotify account using the Spotify API:

user_id = "[ADD YOUR SPOTIFY USER ID HERE]"
client = "[ADD YOUR SPOTIFY CLIENT ID HERE]"
secret = "[ADD YOUR SPOTIFY CLIENT SECRET HERE]"
playlist_uri = "[ADD YOUR PUBLIC PLAYLIST TO SORT HERE]"

Steps:

1. Your user_id can be found by using the web version of spotify and going to your profile overview. A spotify user id will look something like this: 111306XXXXX
2. You need the Spotify developer dashboard to obtain a client id/secret. You can access it here: https://developer.spotify.com/dashboard/login. Next, create an app to obtain a client_id and a client_secret.
3. Your public playlist_uri can be found using the spotify application. Right click a playlist, select 'Share' and click 'Copy Spotify URI'. Your URI will have the following format: spotify:playlist:XXXXXXXXXXXXXXXXXX
4. Ensure that your Spotify developer app has the right redirect_url configured. For this tool to work, go to the Spotify developer dashboard and open the app you created. Click 'edit settings', and add the following url to "Redirect URIs": http://localhost:8888/callback.
5. Set up your Neo4j connection in neo4j_spotify_playlist_builder.py:

   neo4j_url = "bolt://localhost:7687"
   neo4j_username = "neo4j"
   neo4j_password = "neo" 
   

   Keep in mind this application clears your database, so best use a fresh DB. Ensure that your Neo4j installation has the Neo4j Graph Data Science plugin installed!
6. Install python dependencies in requirements.txt.
7. Run neo4j_spotify_playlist_builder.py and watch the magic happen!

Notes

• An input playlist must be made public for the tool to work. Output playlists will be private by default.
• The tool can also be run without creating new playlists in Spotify by disabling the write_to_spotify parameter.

neo4j-playlist-builder's Issues

Error

Getting the following error:

(venv) KTEL-MBP-1312:neo4j-playlist-builder djk-tel$ python3 neo4j_spotify_playlist_builder.py
START load graph using spotify api...
b4 create neo4j session...
dropping and creating constraints...
get_tracks...
get_track_audio_featurescks...
Traceback (most recent call last):
File "/Volumes/Panko/zz Programming Transfers/Machine Learning/Neo 4J/neo4j-playlist-builder/neo4j_spotify_playlist_builder.py", line 340, in
load_graph_using_spotify_api()
File "/Volumes/Panko/zz Programming Transfers/Machine Learning/Neo 4J/neo4j-playlist-builder/neo4j_spotify_playlist_builder.py", line 42, in load_graph_using_spotify_api
neo4j.run("UNWIND $tracks as track CREATE (t:Track{id: track.id}) SET t = track", parameters={'tracks': list(tracks.values())})
File "/Volumes/Panko/zz Programming Transfers/Machine Learning/Neo 4J/neo4j-playlist-builder/venv/lib/python3.9/site-packages/neo4j/work/simple.py", line 217, in run
self._autoResult._run(query, parameters, self._config.database, self._config.default_access_mode, self._bookmarks, **kwparameters)
File "/Volumes/Panko/zz Programming Transfers/Machine Learning/Neo 4J/neo4j-playlist-builder/venv/lib/python3.9/site-packages/neo4j/work/result.py", line 101, in _run
self._attach()
File "/Volumes/Panko/zz Programming Transfers/Machine Learning/Neo 4J/neo4j-playlist-builder/venv/lib/python3.9/site-packages/neo4j/work/result.py", line 202, in _attach
self._connection.fetch_message()
File "/Volumes/Panko/zz Programming Transfers/Machine Learning/Neo 4J/neo4j-playlist-builder/venv/lib/python3.9/site-packages/neo4j/io/_bolt4x0.py", line 330, in fetch_message
response.on_failure(summary_metadata or {})
File "/Volumes/Panko/zz Programming Transfers/Machine Learning/Neo 4J/neo4j-playlist-builder/venv/lib/python3.9/site-packages/neo4j/io/_bolt4x0.py", line 529, in on_failure
raise Neo4jError.hydrate(**metadata)
neo4j.exceptions.CypherTypeError: Property values can only be of primitive types or arrays thereof. Encountered: Map{spotify -> String("https://open.spotify.com/track/1MUgIUkMkEzybR9HZnyVmj")}.

any ideas?

Fetching Tracks Total Count is Off

My Spotify Playlist contains 226 tracks but the program only fetches 200:

I inserted the following logging in the getTracks()

def get_tracks():
	playlist = spotify.playlist(playlist_uri)
	results = playlist['tracks']
	print("Playlist Name:", playlist['name'])
	print("  total tracks", playlist['tracks']['total'])
	
	items = {}
	while results['next'] or results['previous'] is None:
		print("Current Results Items Total:", len(results["items"]))
		print("Current Results NEXT is:", results['next'])
		print("Current Results PREVIOUS is:", results['previous'])
		for track in results["items"]:
			if track['track']['id']:
				track['track']['artists'] = [artist if type(artist) == str else artist['id'] for artist in
											 track['track']['artists']]
				track['track']['album'] = track['track']['album'] if type(track['track']['album']) == str else \
					track['track']['album']['id']
				items[track['track']['id']] = track['track']
			for field in track['track']:
				if track is not None and type(track['track'][field]) == dict:
					track['track'][field] = None
		if not results['next']:
			break
		results = spotify.next(results)
		print("Next Results NEXT is:", results['next'])
		print("Next Results PREVIOUS is:", results['previous'])
	return items

When run the output is:

creating tracks...
Playlist Name: Spotify G-Funk All
  total tracks 226
Current Results Items Total: 100
Current Results NEXT is: https://api.spotify.com/v1/playlists/5U2511keju1CfHJU90cGXS/tracks?offset=100&limit=100&additional_types=track
Current Results PREVIOUS is: None
Next Results NEXT is: https://api.spotify.com/v1/playlists/5U2511keju1CfHJU90cGXS/tracks?offset=200&limit=100&additional_types=track
Next Results PREVIOUS is: https://api.spotify.com/v1/playlists/5U2511keju1CfHJU90cGXS/tracks?offset=0&limit=100&additional_types=track
Current Results Items Total: 100
Current Results NEXT is: https://api.spotify.com/v1/playlists/5U2511keju1CfHJU90cGXS/tracks?offset=200&limit=100&additional_types=track
Current Results PREVIOUS is: https://api.spotify.com/v1/playlists/5U2511keju1CfHJU90cGXS/tracks?offset=0&limit=100&additional_types=track
Next Results NEXT is: None
Next Results PREVIOUS is: https://api.spotify.com/v1/playlists/5U2511keju1CfHJU90cGXS/tracks?offset=100&limit=100&additional_types=track
tracks count is: 200

the
while results['next'] or results['previous'] is None:
fails because both the NEXT and PREVIOUS values have URLs
I'm not familiar with Python's Looping and Clauses. Especially where / the loop takes into the value
of checking continuation..

Missing GDS????

Was able to final get a bit further.
Realized I had to set up AuroaDB and enter my info there.
Getting further but now it fails trying to excute first call in function:
Am I missing the GDS library?

def cluster_genres_with_gds(neo4j):
    result = neo4j.run("""
    CALL gds.graph.exists($name) YIELD exists WHERE exists CALL gds.graph.drop($name) YIELD graphName
    RETURN graphName + " was dropped." as message
    """, name='genre-has-artist')

Full Output from Start

KTEL-MBP-1312:neo4j-playlist-builder djk-tel$ python3 neo4j_spotify_playlist_builder.py
dropping and creating constraints...
creating tracks...
creating albums...
creating artists..
creating genres..
Linking tracks to albums, genres, and artists...
Clustering genres using the GDS library and creating super-genres...
Traceback (most recent call last):
  File "/Volumes/Panko/zz Programming Transfers/Machine Learning/Neo 4J/neo4j-playlist-builder/neo4j_spotify_playlist_builder.py", line 338, in <module>
    load_graph_using_spotify_api()
  File "/Volumes/Panko/zz Programming Transfers/Machine Learning/Neo 4J/neo4j-playlist-builder/neo4j_spotify_playlist_builder.py", line 63, in load_graph_using_spotify_api
    cluster_genres_with_gds(neo4j)
  File "/Volumes/Panko/zz Programming Transfers/Machine Learning/Neo 4J/neo4j-playlist-builder/neo4j_spotify_playlist_builder.py", line 114, in cluster_genres_with_gds
    result = neo4j.run("""
  File "/Users/djk-tel/Library/Python/3.9/lib/python/site-packages/neo4j/work/simple.py", line 217, in run
    self._autoResult._run(query, parameters, self._config.database, self._config.default_access_mode, self._bookmarks, **kwparameters)
  File "/Users/djk-tel/Library/Python/3.9/lib/python/site-packages/neo4j/work/result.py", line 101, in _run
    self._attach()
  File "/Users/djk-tel/Library/Python/3.9/lib/python/site-packages/neo4j/work/result.py", line 202, in _attach
    self._connection.fetch_message()
  File "/Users/djk-tel/Library/Python/3.9/lib/python/site-packages/neo4j/io/_bolt4x0.py", line 330, in fetch_message
    response.on_failure(summary_metadata or {})
  File "/Users/djk-tel/Library/Python/3.9/lib/python/site-packages/neo4j/io/_bolt4x0.py", line 529, in on_failure
    raise Neo4jError.hydrate(**metadata)
neo4j.exceptions.ClientError: There is no procedure with the name `gds.graph.exists` registered for this database instance. Please ensure you've spelled the procedure name correctly and that the procedure is properly deployed.

Playlist Naming Issue - Playlist Parameters Questions

I was having this before when I had it working.
Was getting this error: See Output. (Trimmed from Naming Playlists)

Naming playlists...

Traceback (most recent call last):
  File "/Volumes/Panko/zz Programming Transfers/Machine Learning/Neo 4J/neo4j-playlist-builder/neo4j_spotify_playlist_builder.py", line 338, in <module>
    load_graph_using_spotify_api()
  File "/Volumes/Panko/zz Programming Transfers/Machine Learning/Neo 4J/neo4j-playlist-builder/neo4j_spotify_playlist_builder.py", line 69, in load_graph_using_spotify_api
    name_playlists_based_on_keywords(neo4j)
  File "/Volumes/Panko/zz Programming Transfers/Machine Learning/Neo 4J/neo4j-playlist-builder/neo4j_spotify_playlist_builder.py", line 93, in name_playlists_based_on_keywords
    neo4j.run("""    
  File "/Users/djk-tel/Library/Python/3.9/lib/python/site-packages/neo4j/work/simple.py", line 217, in run
    self._autoResult._run(query, parameters, self._config.database, self._config.default_access_mode, self._bookmarks, **kwparameters)
  File "/Users/djk-tel/Library/Python/3.9/lib/python/site-packages/neo4j/work/result.py", line 101, in _run
    self._attach()
  File "/Users/djk-tel/Library/Python/3.9/lib/python/site-packages/neo4j/work/result.py", line 202, in _attach
    self._connection.fetch_message()
  File "/Users/djk-tel/Library/Python/3.9/lib/python/site-packages/neo4j/io/_bolt4x0.py", line 330, in fetch_message
    response.on_failure(summary_metadata or {})
  File "/Users/djk-tel/Library/Python/3.9/lib/python/site-packages/neo4j/io/_bolt4x0.py", line 529, in on_failure
    raise Neo4jError.hydrate(**metadata)
neo4j.exceptions.ConstraintError: Node(516) already exists with label `Playlist` and property `name` = '[NPB G-Funk] Southern Hyphy Hardcore - active, lively'

I had originally solved this by adding the names that would pop up in the already exists error:
IE for this one I would add "hyphy", "hardcore" to the list and try again.
Currently my list is pretty long like this:
filtered_keywords = '"and", "rap", "hip", "hop", "new", "coast", "gangster", "west", "funk", "g", "pop"'

I still noticed that after got over the exists error that many playlists were being named very similar just swapped names:

[NPB G-Funk] Hardcore Hyphy Southern - active, cheerful
[NPB G-Funk] Southern Hyphy Hardcore - active, lively
[NPB G-Funk] Hardcore Trap Hyphy - energetic, lively

Here Hardcore is used 3 times, Hyphy x 3, Southern x 2.
Is there anyway that could store a array that has names already used (even its index uses in playlist name)
and not use them for the naming?

---

PLAYLIST LIMITS QUESTIONS

could you explain a bit more the

Cut off for playlists to be grouped as 'misc'

min_playlist_size

min size for playlists to be chopped up in smaller ones.

playlist_split_limit

Number of keywords to use in dynamic playlist names.

playlist_keywords_count = 3

---

Does this only affect the name of the playlist or does it also affect the criteria for the playlists?
playlist_keywords_count = 3

Client Error there already exists (Constraint)

I've tried a few things to reset / clear the dB.
Currently it's still showing:
Labels - 1
Properties - 43

here's my current output

KTEL-MBP-1312:neo4j-playlist-builder djk-tel$ python3 neo4j_spotify_playlist_builder.py
dropping and creating constraints...
Traceback (most recent call last):
  File "/Volumes/Panko/zz Programming Transfers/Machine Learning/Neo 4J/neo4j-playlist-builder/neo4j_spotify_playlist_builder.py", line 338, in <module>
    load_graph_using_spotify_api()
  File "/Volumes/Panko/zz Programming Transfers/Machine Learning/Neo 4J/neo4j-playlist-builder/neo4j_spotify_playlist_builder.py", line 34, in load_graph_using_spotify_api
    recreate_contraints(neo4j)
  File "/Volumes/Panko/zz Programming Transfers/Machine Learning/Neo 4J/neo4j-playlist-builder/neo4j_spotify_playlist_builder.py", line 82, in recreate_contraints
    neo4j.run("CREATE CONSTRAINT ON (g:Genre) ASSERT g.name IS UNIQUE")
  File "/Users/djk-tel/Library/Python/3.9/lib/python/site-packages/neo4j/work/simple.py", line 217, in run
    self._autoResult._run(query, parameters, self._config.database, self._config.default_access_mode, self._bookmarks, **kwparameters)
  File "/Users/djk-tel/Library/Python/3.9/lib/python/site-packages/neo4j/work/result.py", line 101, in _run
    self._attach()
  File "/Users/djk-tel/Library/Python/3.9/lib/python/site-packages/neo4j/work/result.py", line 202, in _attach
    self._connection.fetch_message()
  File "/Users/djk-tel/Library/Python/3.9/lib/python/site-packages/neo4j/io/_bolt4x0.py", line 330, in fetch_message
    response.on_failure(summary_metadata or {})
  File "/Users/djk-tel/Library/Python/3.9/lib/python/site-packages/neo4j/io/_bolt4x0.py", line 529, in on_failure
    raise Neo4jError.hydrate(**metadata)
neo4j.exceptions.ClientError: There already exists an index called 'constraint_1a952f13'.