rixx/goodreads-to-sqlite

Export your (or other people's) Goodreads data to SQLite

goodreads-to-sqlite

Save data from Goodreads to a SQLite database. Can save all your public shelves and reviews, and also the public reviews and shelves of other people.

How to install

$ pip install goodreads-to-sqlite

Add the -U flag to update. Change notes can be found in the CHANGELOG file, next to this README.

Authentication

Create a Goodreads developer token: https://www.goodreads.com/api/keys

Run this command and paste in your token and your profile URL:

$ goodreads-to-sqlite auth

This will create a file called auth.json in your current directory containing the required value. To save the file at a different path or filename, use the --auth=myauth.json option.

Retrieving books

The books command retrieves all of the books and reviews/ratings belonging to you:

$ goodreads-to-sqlite books goodreads.db

Note that your Goodreads profile must be public in order for this to work - if it is not already, you can enable this by visiting https://www.goodreads.com/user/edit?ref=nav_profile_settings and selecting "anyone (including search engines)" within the "Settings" tab.

You can also specify the user to target, to fetch books on public shelves of other users. Please provide either the user ID (the numerical part of a user's profile URL), or the name of their vanity URL.

$ goodreads-to-sqlite books goodreads.db rixx

Sometime in 2018 or 2017, Goodreads started leaving out some "read_at" timestamps in their API. If you want to include these datapoints regardless, you can add the --scrape parameter, and the dates will be scraped from the website. This will take a bit longer, by maybe a minute depending on the size of your library.

$ goodreads-to-sqlite books goodreads.db --scrape

The auth.json file is used by default for authentication. You can point to a different location of auth.json using -a:

$ goodreads-to-sqlite books goodreads.db rixx -a /path/to/auth.json

Limitations

• The order of books in shelves is not exposed in the API, so we cannot determine the order of the to-read list.
• Goodreads also offers a CSV export, which is currently not supported as an input format.
• Since the Goodreads API is a bit slow, and we are restricted to one request per second, for larger libraries the import can take a couple of minutes.
• The script currently re-syncs the entire library instead of just looking at newly changed data, to make sure we don't lose information after aborted syncs.

Thanks

This package is heavily inspired by github-to-sqlite by Simon Willison.

The terminal recording above was made with ASCIInema.

goodreads-to-sqlite's Issues

TypeError: fetch_books() got an unexpected keyword argument 'commit'

Traceback (most recent call last):
  File "/Users/simonw/Library/Python/3.7/bin/goodreads-to-sqlite", line 10, in <module>
    sys.exit(cli())
  File "/usrTypeError: fetch_books() got an unexpected keyword argument 'commit'/local/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/simonw/Library/Python/3.7/lib/python/site-packages/goodreads_to_sqlite/cli.py", line 109, in books
    utils.fetch_books(db, user_id, token, commit=True)

/tmp $ 

Dead after goodreads not issuing new dev Keys?

"As of December 8th 2020, Goodreads is no longer issuing new developer keys for our public developer API and plans to retire these tools."

https://joealcorn.co.uk/blog/2020/goodreads-retiring-API

Do You have any plans how to change this to still export it all to sql, after api deprecation?
I regret that only now I found this application :C

Suggestion: add the dogsheep tag to the repo

Then it will show up on https://github.com/topics/dogsheep

Track percentage done status, maybe via the RSS feed?

I'd like to have my "percentage done" information stored in my database for each book that I'm reading.

That information doesn't appear to be available in the API... but apparently it IS available in the RSS feed!

https://www.goodreads.com/user/updates_rss/103923094

So one option could be to subscribe to that. That's what this JS library appears to do: https://github.com/kylegoetz/goodreads-rss

Upgrade to sqlite-utils 2.x

I'm upgrading the other Dogsheep tools at the moment. I want to run them all in the same virtual environment, so I'd like to bump this to sqlite-utils 2.x as well.