arzkar/ao3-cli

A CLI to download from archiveofourown.org using their built-in download option.

Hi there, I'm Arbaaz Laskar

I'm a software developer creating fun open source projects during my free time!
I have experience with Python, JavaScript, TypeScript, Go, C#, C++ & Rust.

If you want to support me you can consider:

Projects Summary

• Apps
  • todo-app: A todo app built using NextJS, NodeJS, MySQL
• Extensions
  • VSNotes: A tool to organize notes by workspaces
• Libraries
  • fetch-ao3: A library to fetch data from ArchiveOfOurOwn.org
• APIs
  • Fanfiction-Finder-API: An API to scrape both ffnet and ao3 fanfiction metadata
  • ao3-api-js: An API to scrape archiveofourown.org fanfiction works metadata
  • Product-Logistics-API: A basic Product Logistics API which can be used to track the transactions of different products and its delivery to different cities
• CLIs
  • git-utils: A CLI for performing various bulk operations on git repositories
  • adb-sync: A CLI to sync between android & local system using adb
  • pyenv-win-venv: A CLI to manage virtual envs with pyenv-win
  • calibre-ebook-convert-helper: A helper CLI for calibre's ebook-convert CLI which is used to convert all files in a directory into another format
  • Contributions to fichub.net project:
    • fichub-cli: A CLI for the fichub.net
    • fichub-cli-metadata: A metadata plugin for the fichub-cli to fetching Metadata from the Fichub API
    • hermes: web fiction (fanfic and web serial) browser, reader, library, etc
  • ao3-cli: A CLI to download from archiveofourown.org using their built-in download option
• Hermes-GUI: A GUI based on my Quote Finder bot using PyQt
• Bot-Devel: A bunch of Discord bots written in Python
  • Fanfiction-Finder: A fanfiction metadata finder bot for all the fanfiction sites supported by Fichub.net
  • Quote-Finder: A discord bot which searches a text file using RegEx to find a quote from a book
  • The-Discord-Archivist: A discord bot to archive a channel and convert it into an epub file. It also includes image files such as png & jpeg.

My Stack

Languages

Frameworks

Python

JavaScript

Databases

Utilities

Github Metrics

ao3-cli's Issues

Add a rety-after to cool down once an user hits AO3's rate limit

Possibly. Can you share the fic url in which you are getting this error. I will try to find a fix.

This isn't a situation where a specific URL triggers it at all. It gets triggered after downloading anywhere from 30-50 fics in a row, after which every URL for a good long while will give "Fanfiction not found". Eventually it'll escape that and get back to working (whenever my IP gets out of AO3 jail) but until then, it doesn't matter which URL I try; they'll all fail. (If I attempt to browse to one in my browser at that moment, it'll give me the "Retry later" message, too.)

Another AO3-related script had this about it:
"Try to keep your ao3 browsing to a minimum while the script is running. It won't break anything, but it may cause you to hit ao3's limit on how many hits to the site you are allowed within a certain time frame. This limit is per user, or per IP if you are not logged in. If this happens, the script will pause for 5 minutes to let the limit reset, and you may see a "Retry later" message when you try to open an ao3 page during that time. Don't be alarmed by this, just wait it out."

You can find more information on this error in the issue for this script here: radiolarian/AO3Scraper#24
They note it's a 429 error. You'll notice the previous script (whence I copied the info from) said it was designed to pause for five minutes to wait out the error - is that something that could be included in your script?

This script works great for one or two files here and there, but for a large list of them (I'm attempting to grab 20,000+ fics for a friend, the ids of which I grabbed using that AO3Scraper above), it's going to run into AO3's limits on usage over and over.

Originally posted by @Kyther in #2 (comment)

"Fanfiction not found" when restricted to registered users

AO3 allows authors to restrict a story's visibility to "registered users," meaning users who are currently logged in. Such stories will not appear in search results for a user browsing the site anonymously, and if such a user attempts to view the story using the direct URL, he or she receives a login page saying "This work is only available to registered users of the Archive ..."

When I try to use ao3_cli to download such a story (such as this one), it fails with the error "Fanfiction not found". It would be nice to be able to provide the tool with account credentials to use to download such "restricted" stories.

Error, No connection adapters were found

Hi, I apologize ahead of time because I don't really understand anything about Python or script/coding. I changed my directory location to the desired folder. Put my .txt file within that folder. And ran it two different ways, receiving the same error.

This is the first way I ran it:
C:\Users\[REDACTED]\Documents\STFicsDL>ao3_cli -i urls.txt -f PDF

This is the second:
C:\Users\[REDACTED]\Documents\STFicsDL>ao3_cli -i urls.txt -o C:\Users\[REDACTED]\Documents\STFicsDL -f PDF

Here is the error:

Processing https://archiveofourown.org/works/33413554
  0%|                                                                                        | 0/19633, ?file/s, ETA: ?
Traceback (most recent call last):
  File "c:\python 3.9.1\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\python 3.9.1\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Python 3.9.1\Scripts\ao3_cli.exe\__main__.py", line 7, in <module>
  File "c:\python 3.9.1\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\python 3.9.1\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\python 3.9.1\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\python 3.9.1\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\python 3.9.1\lib\site-packages\ao3_cli\cli.py", line 44, in run_cli
    fic.get_fic_with_infile(infile)
  File "c:\python 3.9.1\lib\site-packages\ao3_cli\utils\fetch_data.py", line 67, in get_fic_with_infile
    fic.get_fic_metadata(self.format_type)
  File "c:\python 3.9.1\lib\site-packages\ao3_cli\utils\ao3.py", line 23, in get_fic_metadata
    response = self.session.get(self.BaseUrl, params=params)
  File "c:\python 3.9.1\lib\site-packages\requests\sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "c:\python 3.9.1\lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "c:\python 3.9.1\lib\site-packages\requests\sessions.py", line 649, in send
    adapter = self.get_adapter(url=request.url)
  File "c:\python 3.9.1\lib\site-packages\requests\sessions.py", line 742, in get_adapter
    raise InvalidSchema("No connection adapters were found for {!r}".format(url))
requests.exceptions.InvalidSchema: No connection adapters were found for 'https://archiveofourown.org/works/33413554'

Problems with fics with / in title

I used this on a file with a list of URLs; it threw a "FileNotFoundError [Errno 2] No such file or directory" whenever it reached one which had a / in the fic name, because it kept the / in the name which meant adding a subdirectory to the file path (I'm running Linux). Running it directly on the URL itself didn't help.

It probably should be coded to ignore all forward slashes in the file names when saving…

Also, I observed that it would hit a point where it would not get a proper response from AO3, and would merrily skip fic after fic after fic with "Fanfiction not found". When I aborted, removed all the successful links from the list, and restarted it, it usually would successfully grab the same fics it had just said weren't there. It would probably be helpful if there were something built in to handle the vagaries of AO3 responses, so that one did not have to monitor the downloading so closely to avoid missing them.