dragoonaethis/itch-dl

Download all games from a public Itch.io Game Jam

itch-dl's Issues

NotImplementedError: Unknown infobox block name 'Category'

Hi!

INFO:root:Downloading https://andreipasynkov.itch.io/upandown
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/tmp/itch-dl/itch_dl/__main__.py", line 3, in <module>
    run()
  File "/tmp/itch-dl/itch_dl/cli.py", line 68, in run
    return drive_downloads(jobs, download_to, args.mirror_web, args.api_key, keys, parallel=args.parallel)
  File "/tmp/itch-dl/itch_dl/downloader.py", line 350, in drive_downloads
    results = [downloader.download(job) for job in tqdm(jobs, **tqdm_args)]
  File "/tmp/itch-dl/itch_dl/downloader.py", line 350, in <listcomp>
    results = [downloader.download(job) for job in tqdm(jobs, **tqdm_args)]
  File "/tmp/itch-dl/itch_dl/downloader.py", line 240, in download
    metadata = self.extract_metadata(game_id, url, site)
  File "/tmp/itch-dl/itch_dl/downloader.py", line 149, in extract_metadata
    infobox = parse_infobox(infobox_div)
  File "/tmp/itch-dl/itch_dl/infobox.py", line 116, in parse_infobox
    parsed_block = parse_tr(name, content_td)
  File "/tmp/itch-dl/itch_dl/infobox.py", line 100, in parse_tr
    raise NotImplementedError(f"Unknown infobox block name '{name}' - please file a new itch-dl issue.")
NotImplementedError: Unknown infobox block name 'Category' - please file a new itch-dl issue.

...so I did! Got this from https://itch.io/jam/game-boy-showdown.

Something about extracting time breaks the script

Lately downloading from Itch stopped working for me, I tested 3 different pages.

itch-dl --verbose --mirror-web https://beth-and-angel-make-games.itch.io/a-winter-tale

DEBUG:root:Found config file: /home/xxx/.config/itch-dl/config.json
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.itch.io:443
DEBUG:urllib3.connectionpool:https://api.itch.io:443 "GET /profile HTTP/1.1" 200 None
INFO:root:Found 1 URL(s).
INFO:root:Fetching all download keys...
INFO:root:Downloading page 1 (found 0 keys total)
DEBUG:urllib3.connectionpool:https://api.itch.io:443 "GET /profile/owned-keys HTTP/1.1" 200 None
INFO:root:Downloading page 2 (found 50 keys total)
DEBUG:urllib3.connectionpool:https://api.itch.io:443 "GET /profile/owned-keys HTTP/1.1" 200 None
INFO:root:Downloading page 3 (found 100 keys total)

…

INFO:root:Fetched 500 download keys.
Games:   0%|                                                                                                                                                            | 0/1 [00:00<?, ?game/s]INFO:root:Downloading https://beth-and-angel-make-games.itch.io/a-winter-tale
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): beth-and-angel-make-games.itch.io:443
DEBUG:urllib3.connectionpool:https://beth-and-angel-make-games.itch.io:443 "GET /a-winter-tale HTTP/1.1" 200 None
Traceback (most recent call last):
  File "/run/media/xxx/eeea9c45-51a7-4354-94d3-0e538cb28013/itch_downloads/envname/bin/itch-dl", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/run/media/xxx/eeea9c45-51a7-4354-94d3-0e538cb28013/itch_downloads/envname/lib/python3.11/site-packages/itch_dl/cli.py", line 81, in run
    return drive_downloads(jobs, download_to, args.mirror_web, settings, keys, parallel=args.parallel)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/run/media/xxx/eeea9c45-51a7-4354-94d3-0e538cb28013/itch_downloads/envname/lib/python3.11/site-packages/itch_dl/downloader.py", line 353, in drive_downloads
    results = [downloader.download(job) for job in tqdm(jobs, **tqdm_args)]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/run/media/xxx/eeea9c45-51a7-4354-94d3-0e538cb28013/itch_downloads/envname/lib/python3.11/site-packages/itch_dl/downloader.py", line 353, in <listcomp>
    results = [downloader.download(job) for job in tqdm(jobs, **tqdm_args)]
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/run/media/xxx/eeea9c45-51a7-4354-94d3-0e538cb28013/itch_downloads/envname/lib/python3.11/site-packages/itch_dl/downloader.py", line 240, in download
    metadata = self.extract_metadata(game_id, url, site)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/run/media/xxx/eeea9c45-51a7-4354-94d3-0e538cb28013/itch_downloads/envname/lib/python3.11/site-packages/itch_dl/downloader.py", line 150, in extract_metadata
    infobox = parse_infobox(infobox_div)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/run/media/xxx/eeea9c45-51a7-4354-94d3-0e538cb28013/itch_downloads/envname/lib/python3.11/site-packages/itch_dl/infobox.py", line 121, in parse_infobox
    parsed_block = parse_tr(name, content_td)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/run/media/xxx/eeea9c45-51a7-4354-94d3-0e538cb28013/itch_downloads/envname/lib/python3.11/site-packages/itch_dl/infobox.py", line 58, in parse_tr
    return "published_at", parse_date_block(content)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/run/media/xxx/eeea9c45-51a7-4354-94d3-0e538cb28013/itch_downloads/envname/lib/python3.11/site-packages/itch_dl/infobox.py", line 38, in parse_date_block
    time = datetime.strptime(time_str.strip(), "%H:%M")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/_strptime.py", line 352, in _strptime
    raise ValueError("unconverted data remains: %s" %
ValueError: unconverted data remains:  UTC
Games:   0%|                                                                                                                                                            | 0/1 [00:00<?, ?game/s]

Site Archive Inconsistencies

I'm backing up games on Itch and I've noticed multiple inconsistencies with the archived pages generated by itch-dl.

• Cyrillic (All unicode?) is garbled.
• 'Updated' and 'Published' dates are missing under 'More information'
• Some images like the cover and screenshots seem to be archived but not linked to the local mirror, and other images are missing like all store banners and user avatars.
• Developer log posts are not saved, and if I understand correctly the devlogs can also include past versions of software that itch-dl does not download. That would be super slick if itch-dl could back up all the past versions as well with an argument like --devlog.
• Useful information in community posts is also not archived.

I tried with and without using --mirror-web but there was not much of a difference. Screenshots are saved when specified but I did not note any additional benefit.

Download "My Games" for the currently logged-in user

There's a /my-games endpoint available in the API - it'd be useful to be able to archive all games in the user's library.

Support downloading non-latest versions

Originally part of #8 @JoshuaFern - Itch supports storing multiple game versions (and exposes that partially through the API), but the downloader currently grabs just the latest one. There's no way to specify which versions to download, or to list existing versions. Would be neat to make this possible.

Support downloading bundles "NotImplementedError: itch-dl cannot download bundles yet."

This is very useful! Please add support for downloading entire bundles if you can. 😃

UnicodeEncodeError: 'charmap' codec can't encode character '\x80' in position 23385: character maps to <undefined>

Running itch-dl on Python 3.11.1 installed via pip on WIndows 11 powershell.

...
INFO:root:Downloading page 49 (found 2400 keys total)
INFO:root:Downloading page 50 (found 2450 keys total)
INFO:root:Downloading page 51 (found 2500 keys total)
INFO:root:Downloading page 52 (found 2550 keys total)
INFO:root:Downloading page 53 (found 2600 keys total)
INFO:root:Fetched 2648 download keys.
Games:   0%|                                                                               | 0/49557 [00:00<?, ?game/s]INFO:root:Downloading https://dkelroqur.itch.io/black-jumper
https://img.itch.zone/aW1hZ2UvMzMwNzcvMTQxODU4LnBuZw==/original/oKU282.png: 100%|██| 19.8k/19.8k [00:00<00:00, 142kB/s]
Traceback (most recent call last):NzcvMTQxODU4LnBuZw==/original/oKU282.png: 100%|██| 19.8k/19.8k [00:00<00:00, 142kB/s]
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Proxy_ydkavi9\AppData\Local\Programs\Python\Python311\Scripts\itch-dl.exe\__main__.py", line 7, in <module>
  File "C:\Users\Proxy_ydkavi9\AppData\Local\Programs\Python\Python311\Lib\site-packages\itch_dl\cli.py", line 81, in run
    return drive_downloads(jobs, download_to, args.mirror_web, settings, keys, parallel=args.parallel)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Proxy_ydkavi9\AppData\Local\Programs\Python\Python311\Lib\site-packages\itch_dl\downloader.py", line 353, in drive_downloads
    results = [downloader.download(job) for job in tqdm(jobs, **tqdm_args)]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Proxy_ydkavi9\AppData\Local\Programs\Python\Python311\Lib\site-packages\itch_dl\downloader.py", line 353, in <listcomp>
    results = [downloader.download(job) for job in tqdm(jobs, **tqdm_args)]
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Proxy_ydkavi9\AppData\Local\Programs\Python\Python311\Lib\site-packages\itch_dl\downloader.py", line 324, in download
    f.write(site.prettify())
  File "C:\Users\Proxy_ydkavi9\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\x80' in position 23385: character maps to <undefined>
Games:   0%|                                                                               | 0/49557 [00:01<?, ?game/s]

As a workaround I've been running the script under Ubuntu WSL2 where it seems to be working fine.

NotImplementedError: Unknown infobox block name 'Code license'

Is there a quick way to skip this error?

Allow toggling Download Key caching

It's possible to query the API for any available Download Keys directly for a Game ID, instead of downloading them all before the fact. This would be faster for a small amount of titles to download, especially if the user has a ton of purchased/claimed games.