Giter Site home page Giter Site logo

kalanyr / gogrepoc Goto Github PK

View Code? Open in Web Editor NEW

This project forked from eddie3/gogrepo

206.0 31.0 28.0 247 KB

Python-based tool for downloading all your GOG.com game and bonus collections to your local computer for full offline enjoyment.

Python 100.00%

gogrepoc's Introduction

gogrepo gamma

Python-based tool for downloading your GOG.com game collections and extras to your local computer for full offline enjoyment.

It is a clean standalone python script that can be run from anywhere. It requires a typical Python 2.7 or Python 3 installation and html5lib, requests and pyOpenSSL.

By default, game folders are saved in the same location that the script is run in. You can also specify another directory. Run gogrepo.py -h to see help or read more below. Each game has its own directories with all game/bonus files saved within.

License: GPLv3+

Features

  • Ability to choose which games to download based on combinations of OS (windows, linux, mac) and language (en, fr, de, etc...)
  • Saves a !info.txt in each game folder with information about each game/extra item.
  • Creates a !serial.txt if the game has a special serial/cdkey (I know, not 100% DRM-free, is it?). Sometimes coupon codes are hidden here!
  • Verify your downloaded collection with full MD5, zip integrity, and expected file size checking.
  • Auto retrying of failed fetch/downloads. Sometime GOG servers report temporary errors.
  • Download resume support for interrupted downloads where possible.
  • Ability to import your already existing local collection.
  • Easy to throw into a daily cronjob to get all the latest updates and newly added content!
  • Clear logging prints showing update/download progress and HTTP errors. Log files are created by default but can be disabled.

Quick Start -- Typical Use Case

  • Login to GOG and save your login cookie for later commands. Your login/pass can be specified or be prompted. You generally only need to do this once to create a valid gog-cookies.dat

    gogrepoc.py login

  • Fetch all new and updated game and bonus information from GOG for items that you own and save into a local manifest file. Run this whenever you want to discover newly added games or game updates.

    gogrepoc.py update

  • Download the games and bonus files for the OS and languages you want for all items known from the saved manifest file.

    gogrepoc.py download

  • Verify and report integrity of all downloaded files. Does MD5, zip integrity, and expected filesize verification. This makes sure your game files can actually be read back and are healthy.

    gogrepoc.py verify

Advanced Usage -- Common Tasks

  • Add new games from your library to the manifest.

    gogrepoc.py update -os windows -lang en de -skipknown

  • Update games with the updated tag in your libary.

    gogrepoc.py update -os windows -lang en de -updateonly

  • Update one or more specified games in your manifest.

    gogrepoc.py update -ids trine_2_complete_story

  • Download one or more specified games game in your manifest.

    gogrepoc.py download -ids trine_2_complete_story

Commands

gogrepoc.py login Authenticate with GOG and save the cookie locally in gog-cookies.dat file. This is needed to do update or download command. Run this once first before doing update and download.

login [username] [password]
username    GOG username/email
password    GOG password

--

gogrepoc.py update Fetch game data and information from GOG.com for the specified operating systems and languages. This collects file game titles, download links, serial numbers, MD5/filesize data and saves the data locally in a manifest file. Manifest is saved in a gog-manifest.dat file

update [-os [OS [OS ...]]] [-lang [LANG [LANG ...]]] [-skipknown | -updateonly | -id <title>]
-os [OS [OS ...]]    	operating system(s) (ex. windows linux mac)
-skipos				 	skip operating system(s)
						Can't be used with -os
-lang [LANG [LANG ...]] game language(s) (ex. en fr de)
-skiplang				skip game language(s)
						Can't be used with -lang
-standard				update new and updated games only (default unless -ids used)
						Can't be used with -skipknown, -updateonly, -full
-skipknown            	only update new games in your library
-updateonly           	only update games with the updated tag in your library
-full				  	update all games on your account (default if -ids used)
-ids <title>		  	id(s)/titles(s) of (a) specific game(s) to update
-id <title>           	specify the game to update by 'title' from the manifest. Deprecated by -ids
						<title> can be found in the !info.txt of the game directory
-skipids <title>	  	id(s)/titles(s) of (a) specific game(s) not to update
-resumemode			  	how to handle resuming if necessary: noresume, resume, or onlyresume. Default: resume
-strictverify 		  	clear previously verified unless md5 match
-skiphidden			  	skip games marked as hidden
-installers			  	GOG Installer type to use: galaxy, standalone or both. Default: standalone
-wait <WAIT>			wait this long in hours before starting

--

gogrepoc.py download Use the saved manifest file from an update command, and download all known game items and bonus files.

download [-dryrun] [-skipextras] [-skipextras] [-skipgames] [-wait WAIT] [-id <title>] [savedir]
savedir      	   directory to save downloads to
-dryrun      	   display, but skip downloading of any files
-skipextras  	   skip downloading of any GOG extra files
-skipgames   	   skip downloading of any GOG game files. Deprecated by -skipgalaxy, -skipstandalone and -skipshared
-skipgalaxy  	   skip downloading Galaxy installers
-skipstandalone    skip downloading standlone installers
-skipshared 	   skip downloading installers shared between Galaxy and standalone
-skipfiles <file>  file name (or glob patterns) to NOT download
-wait <WAIT>   	   wait this long in hours before starting
-ids <title>	   id(s) or title(s) of the game in the manifest to download
-id <title>  	   specify the game to download by 'title' from the manifest. Deprecated by -ids
				   <title> can be found in the !info.txt of the game directory
-skipids <title>   id(s) or title(s) of the game(s) in the manifest to NOT download, default=[])
-os [OS [OS ...]]  download game files only for operating system(s) (ex. windows linux mac)
-skipos			   skip downloading game files for operating system(s)
				   Can't be used with -os
-lang [LANG [LANG ...]] download game files only for language(s) (ex. en fr de)
-skiplang		   skip downloading game files for language(s)
				   Can't be used with -lang

--

gogrepoc.py verify Check all your game files against the save manifest data, and verify MD5, zip integrity, and expected file size. Any missing or corrupt files will be reported.

verify [-skipmd5] [-skipsize] [-skipzip] [-delete] [gamedir]
gamedir     directory containing games to verify
-forceverify (also verify files that are unchanged (by gogrepo) since they were last successfully verified)
-skipmd5    	   do not perform MD5 check
-skipsize   	   do not perform size check
-skipzip    	   do not perform zip integrity check
-skipextras  	   skip verification of any GOG extra files
-skipgames   	   skip verification of any GOG game files. Deprecated by -skipgalaxy, -skipstandalone and -skipshared
-skipgalaxy 	   skip verification of any GOG Galaxy installer files
-skipstandalone    skip verification of any GOG standalone installer files
-skipshared 	   skip verification of any installers included in both the GOG Galalaxy and Standalone sets
-skipfiles <file>  file name (or glob patterns) to NOT verify
-ids <title>	   id(s) or title(s) of the game in the manifest to verify
-id <title>  	   specify the game to verify by 'title' from the manifest. Deprecated by -ids
				   <title> can be found in the !info.txt of the game directory
-skipids <title>   id(s) or title(s) of the game(s) in the manifest to NOT verify, default=[])
-os [OS [OS ...]]  verify game files only for operating system(s) (ex. windows linux mac)
-skipos			   skip verification of game files for operating system(s)
				   Can't be used with -os
-lang [LANG [LANG ...]] verify game files only for language(s) (ex. en fr de)
-skiplang		   skip downloading game files for language(s)
				   Can't be used with -lang
-delete    		   delete any files which fail integrity test
-clean 			   clean any files which fail integrity test

--

gogrepoc.py import Search an already existing GOG collection for game item/files, and import them to your new GOG folder with clean game directory names and file names as GOG has them named on their servers.

import [src_dir] [dest_dir]
src_dir     	   source directory to import games from
dest_dir    	   directory to copy and name imported files to
-skipgalaxy  	   skip importing Galaxy installers
-skipstandalone    skip importing standlone installers
-skipshared 	   skip importing installers shared between Galaxy and standalone
-ids <title>	   id(s) or title(s) of the game in the manifest to import
-skipids <title>   id(s) or title(s) of the game(s) in the manifest to NOT import, default=[])
-os [OS [OS ...]]  import game files only for operating system(s) (ex. windows linux mac)
-skipos			   skip importing game files for operating system(s)
				   Can't be used with -os
-lang [LANG [LANG ...]] import game files only for language(s) (ex. en fr de)
-skiplang		   skip importing game files for language(s)
				   Can't be used with -lang

--

gogrepoc.py backup Make copies of all known files in manifest file from a source directory to a backup destination directory. Useful for cleaning out older files from your GOG collection.

backup [src_dir] [dest_dir]
src_dir     	   source directory containing gog items
dest_dir    	   destination directory to backup files to
-skipextras  	   skip backup of any GOG extra files
-skipgames   	   skip backup of any GOG game files. Deprecated by -skipgalaxy, -skipstandalone and -skipshared
-skipgalaxy  	   skip backup of Galaxy installers
-skipstandalone    skip backup of standlone installers
-skipshared 	   skip backup of installers shared between Galaxy and standalone
-ids <title>	   id(s) or title(s) of the game in the manifest to backup
-skipids <title>   id(s) or title(s) of the game(s) in the manifest to NOT backup, default=[])
-os [OS [OS ...]]  backup game files only for operating system(s) (ex. windows linux mac)
-skipos			   skip backup of game files for operating system(s)
				   Can't be used with -os
-lang [LANG [LANG ...]] backup game files only for language(s) (ex. en fr de)
-skiplang		   skip backup of game files for language(s)
				   Can't be used with -lang

--

gogrepoc.py clean Clean your games directory of files not known by manifest. Moves files to the !orphaned folder.

clean [-dryrun] [cleandir]
cleandir    root directory containing gog games to be cleaned
-dryrun     do not move files, only display what would be cleaned

--

gogrepoc.py trash Permanently remove orphaned files in your game directory.

trash [-dryrun] [-installersonly] [gamedir]
gamedir    		root directory containing gog games
-dryrun     	do not move files, only display what would be trashed
-installersonly only delete file types used as installers

--

Other arguments: -h, --help show help message and exit. Used in all commands. -nolog don't write to log file gogrepo.log. Used in all commands. -v, --version show version number and exit. Used in all commands.

Requirements

Optional

  • html2text 2015.6.21 or later (https://pypi.python.org/pypi/html2text) (optional, used for prettying up gog game changelog html) *nix:
  • dbus-python and required dependencies (*nix, optional, used to prevent suspend/sleep interrupts on *nix, where supported) (this will likely move to pydbus as it matures) Mac:
  • caffeinate support (optional, required to prevent suspend/sleep interrupts)

TODO

  • add ability to update and download specific games or new-items only
  • ~~add 'clean' command to orphan/remove old or unexpected files to keep your collection clean with only the latest files
  • support resuming manifest updating
  • add support for incremental manifest updating (ie. only fetch newly added games) rather than fetching entire collection information
  • ability to customize/remap default game directory name
  • add GOG movie support
  • ... feel free to contact me with ideas or feature requests!

gogrepoc's People

Contributors

disi avatar eddie3 avatar h143570 avatar hamsterexastris avatar idamkini avatar jpenney avatar kalanyr avatar resonantworks avatar sarkanyi avatar sta2s avatar stevensmedia avatar trilarion avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gogrepoc's Issues

xml parsing error

Tried to update today my offline games… to meet with following errors

17:52:04 | xml parsing error occurred trying to get md5 data for patch_age_of_wonders_iii_1.802_(24203)_to_1.802_fix_(28713).exe
17:52:05 | xml parsing error occurred trying to get md5 data for patch_age_of_wonders_iii_1.801_b26272_(20409)_to_1.802_(24203).exe
17:52:06 | xml parsing error occurred trying to get md5 data for patch_age_of_wonders_iii_1.801_(19572)_to_1.801_b26272_(20409).exe
17:52:08 | (  6 / 263) fetching game details for akalabeth_world_of_doom...
17:52:10 | xml parsing error occurred trying to get md5 data for setup_akalabeth_2.0.0.1.exe
17:52:11 | xml parsing error occurred trying to get md5 data for akalabeth__world_of_doom_enUS_1_0_30902.pkg
17:52:12 | xml parsing error occurred trying to get md5 data for gog_akalabeth_world_of_doom_2.0.0.3.sh
17:52:13 | (  7 / 263) fetching game details for alders_blood_prologue...
17:52:28 | xml parsing error occurred trying to get md5 data for setup_alders_blood_prologue_1.0.20a_(64bit)_(37554).exe
17:52:28 | (  8 / 263) fetching game details for aliens_versus_predator_classic_2000...
17:52:57 | xml parsing error occurred trying to get md5 data for setup_avp_classic_20160330_(2149).exe
17:52:58 | (  9 / 263) fetching game details for amnesia_a_machine_for_pigs...
17:53:01 | xml parsing error occurred trying to get md5 data for setup_amnesia_-_a_machine_for_pigs_1.0_(20583).exe

Did gog change something on their end or is it temporary?

Wishlist: Downloading ONE language by preference

So, would it be possible to have a parameter like "langpref de en" that says: Take the German version and if you can't find one the english version. This would avoid downloading two versions of a game. Or am I missing something in the docs?

Thanks for this excellent script!

Games that corrupt the manifest DAT file

Version String: Python 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)] on win32

Hi Kalanyr,

I thought (in a similar vein to #60 and #61) I'd pass along a number of games that seem to corrupt the manifest when running py -m gogrepoc update (without a gog-manifest.dat) and the py -m download—related explicitly to null characters being present in the data file (which can be manually resolved by looking for the special regex character \0 and removing the entire game entry (e.g. from {'bg_url: to its corresponding right bracket }. The affected games are listed below the ######################### .

I have also noticed that many of the affected sections have excessively long changelog entries if that helps with diagnostics.

An excerpt of the series of outputs from running py -m gogrepoc download can be found here: https://pastebin.com/dwgKU9Jp

NOTE: In the pastebin ">>> preceded by an empty line is my redacted prompt

#########################

Cossacks Anthology: https://www.gog.com/en/game/cossacks_anthology
Cyberpunk 2077: https://www.gog.com/en/game/cyberpunk_2077
Gabriel Knight: Sins of the Fathers: https://www.gog.com/en/game/gabriel_knight_sins_of_the_fathers_20th_anniversary_edition
Geneforge 1: https://www.gog.com/en/game/geneforge_15
Planetscape - Torment: https://www.gog.com/en/game/planescape_torment_enhanced_edition
SpaceChem: https://www.gog.com/en/game/spacechem
Spelunky: https://www.gog.com/en/game/spelunky
Neverwinter Nights Diamond Edition: https://www.gog.com/en/game/neverwinter_nights_enhanced_edition_pack
Risen: https://www.gog.com/en/game/risen
Rise of the Triad: https://www.gog.com/en/game/rise_of_the_triad__dark_war and https://www.gog.com/en/game/rise_of_the_triad
Stellaris: https://www.gog.com/en/game/stellaris
Stronghold Crusader/HD: https://www.gog.com/en/game/stronghold_crusader
The Witcher: https://www.gog.com/en/game/the_witcher
The Witcher 2: Enhanced Edition (there seemed to be two entries for this game although only one was affected): https://www.gog.com/en/game/the_witcher_2
SteamWorld Dig: https://www.gog.com/en/game/steamworld_dig
Thea 2 The Shattering: https://www.gog.com/en/game/thea_2_the_shattering
Yooka Laylee: Impossible Lair: https://www.gog.com/en/game/yookalaylee_and_the_impossible_lair
Alan Wake: https://www.gog.com/en/game/alan_wake
Alan Wake: American Nightmare: https://www.gog.com/en/game/alan_wakes_american_nightmare

Clean handles both pre/post Download maintenance tasks

Clean currently handles renames (which needs to be down before downloads) and orphaning files (which should be done afterwards in case the file is similiar to it's successor) need to either make these separate commands (maintain and clean?) or have flags.

Problem logging in

I am trying to log in with my user and password. Always fails with cannot continue, gog is asking for a reCAPTCHA :( try again in a few minutes. message. I exported the cookies from firefox with export cookies extension into cookies.txt into the same folder and got same message. Tried also downloading from chrome with get cookies.txt extension and same message. What else can I try ?

How to manually create cookie file?

Since gogrepoc always returns:
python3 gogrepoc.py login login password 10:04:16 | attempting gog login as '[email protected]' ... 10:04:21 | cannot continue, gog is asking for a reCAPTCHA :( try again in a few minutes. 10:04:21 | exiting...

I'd like to create cookie file manually - how to do so? What format it need to have and what file name use?

[QoL] Use .format() instead of %s and %d where possible

.format() is kinda common way to handle string formatting in python nowadays. Its way easier to read than many %s'es and %d'es and a bit more flexible.
For example, this:
info("not moving uncompleted download '%s', success: %s remaining bytes: %d / %d " % (downloading_path,str(succeed),sizes[path],sz))

Can be improved into something like this:
info("not moving uncompleted download {}, success: {} remaining bytes: {} / {} ".format(downloading_path,str(succeed),sizes[path],sz))

Feature request: file blacklisting

Main reason for me - no language attribute for extras, no good way to skip downloading extras for all languages.
Proposed solution:

  1. Add command to add file(s) to blacklist, added files not count as missing on running gogrepoc.py verify etc. Count as orphaned, maybe.
  2. gogrepoc.py download -skipfiles skips blacklisted files if no filenames specified directly, or add another option, or make blacklisted files skipped by default.

Orphaning dir fails because path already exists

I'm using gogrepoc for a few years now, and I am very thankful that it exists, because it enables me to download and manage my entire collection which has become quite big nowadays.
However, recently I have repeatedly stumbled upon this annoying bug, which appears to occur whenever a game on GOG is renamed (or removed) AND this exact game has already had orphaned file entries in the past.
Thus when moving the whole game directory into the orphaned folder, it fails, because a folder named like this is already there.

Subsequently the script completely stops orphaning the remaining files, because there is no proper error handling.

This error occurred with Master of Magic and some other game, and now with Wolfenstein 3D.

I always have to manually remove/rename the problematic folders, because orphaning always fails when the folder is present.

Here is the log output:

08:25:59 | orphaning dir 'wolfenstein_3d'
08:25:59 | fatal...
Traceback (most recent call last):
File "gogrepoc.py", line 2891, in
main(process_argv(sys.argv))
File "gogrepoc.py", line 2711, in main
cmd_clean(args.cleandir, args.dryrun)
File "gogrepoc.py", line 2523, in cmd_clean
shutil.move(cur_fulldir, orphan_root_dir)
File "/usr/lib/python3.8/shutil.py", line 789, in move
raise Error("Destination path '%s' already exists" % real_dst)
shutil.Error: Destination path './gog/!orphaned/wolfenstein_3d' already exists

Use JSON to store the manifest

This allows for other projects to read the manifest and closes the security hole that is the eval() being called on a data file. JSON is not YAML, but it also does not need to look bad.

import json
import gogrepo
manifest = gogrepo.load_manifest()
print(json.dumps(manifest[3], indent=1, sort_keys=True)

You could perhaps replace all the manifest serialization and deserialization code with less than 10 lines of code.

Download Rate Limiting & Concurrent Downloads

I was wondering if it would be possible to add arguments to control the following behaviour:

  • Download rate limit
    Something like -ratelimit x which would ensure the download rate of any individual worker does not exceed x MB/s.

  • Concurrent downloads (workers)
    Something like -workers x which would ensure x workers are spawned to do the downloads in parallel.

These two arguments would allow me to use this script like a scheduled task or service that does not consume all of my bandwidth in the background. Downloading one file at a time also helps with resuming if the script is terminated since there will be only one partial file that needs to be re-downloaded instead of 4 partial files from cancelling.

Pause and resume downloads?

Hi there,

Will this let you pause and resume downloads? In some cases I can't finish downloading in one session and I would like to pause and resume.

Cheers

[bug] "GAME_STORAGE_DIR" variable is unused

Tried to implement os-specific directories, but it all came down to one issue - while cookies, manifest, !downloading and !orphaned are easy to move, GAME_STORAGE_DIR variable (which looks like something thats supposed to represent directory where game files will be moved at the end), despite being specified at the very beginning of script - in reality is never used anywhere.

Suggestion: well, just make it usefull (say, at the end of cmd_download() function)

Possibility to add an option to download only games and ignore patches?

Hello,

While reviewing the code/options available I may have answered myself my question. But I was wondering if there was a possibility to skip patches of games and just download the new full release of it where there is an update or during the initial download?

I assume that can be done with simply using -skipfiles, and that it is probably the intended way. So sorry if I asked the question for nothing!

Thanks in advance for your time! And for this great project :)

Required libraries are not mentioned in the README

The current README looks unmodified compared to the source of the fork (eddie3/gogrepo) and makes no mention of the requirement for the requests or OpenSSL libraries. While requests can be installed with the obvious (pip install requests), it it not so obvious how to install OpenSSL (pip install pyopenssl), so a mention of this in the README would be most welcome.

[Feature] Configuration file

It could be nice to have some configuration file (say, in ini format) that can be used to override some default variables automatically (e.g instead of using command-line options each time). Example options that may be tweaked this way:

  • Directory used to store games.
  • Default OS, for which game will be downloaded (win/mac/lin)
  • Should game's goodies be downloaded or not

The logic is following:

  • If config.ini doesnt exist, script makes one with default values
  • If config.ini exists, but not every necessary variable present there - for missing variables, script falls back to defaults
  • If config.ini contains incorrect variables - script throws error and sys.exit(1)
  • If config.ini exists and is correct - variables stored inside are used instead of script's defaults
  • If script has been launched with additional flags that affect the same things as config.ini - options described in flags will be used instead of options in config.ini

Look into adding a GUI (via Gooey?)

(Possibly have a seperate py file that serves as the GUI frontend (using overridng/inheritance ), if can't somehow combine), probably Python 3 only ?

Cyberpunk oddities

Any ideas why Cyberpunk refuses to get to the latest versions, even when you manually force full reload of the manifest of its ID? Is this GOG actively blocking full downloads for every point release? Website has 2.12 for some days now, gogrepoc can only see 2.11.

Cheers, Boris

import fails

22:01:37 | loading local manifest...
22:01:38 | collecting md5 data out of the manifest
22:01:38 | fatal...
Traceback (most recent call last):
File "/mnt/usb/gog/games - gog/downloader/gogrepoc.py", line 2777, in
main(process_argv(sys.argv))
File "/mnt/usb/gog/games - gog/downloader/gogrepoc.py", line 2558, in main
cmd_import(args.src_dir, args.dest_dir,args.os,args.lang,args.skipextras,args.skipids,args.ids,args.skipgalaxy,args.skipstandalone,args.skipshared)
File "/mnt/usb/gog/games - gog/downloader/gogrepoc.py", line 1452, in cmd_import
if game_item.md5 is not None:
UnboundLocalError: local variable 'game_item' referenced before assignment

GOGRepoC cannot find the serial key(s) for the game "X4: Foundations".

Hi there,

GOGRepoC cannot find the serial key(s) for the game "X4: Foundations".

In my library, under "X4: Foundations", in the serial keys section, I have two keys:
X4: Foundations Collector's Edition Key
X4: Foundations Key

I purchased "X4: Foundations Collector's Edition" package, which is now delisted.
None of the current editions/packages "X4: Foundations", "X4: Community of Planets Edition", "X4: Community of Planets Collector's Edition" are marked as owned, but I doubt it has any relevance, as GOGRepoC only looks at the library, not the store (AFAIK).

Lose Python 2 compatibility

Personally, I would consider dropping support for Python 2 (people who need it can continue to use the current version of gogrepo) to get a much cleaner, simpler and more modern codebase.

Then the option of using Async IO could be put on the table. There would be no need to adjust the number of concurrent workers and the system would scale better with the bandwidth that is available. But that is just my opinion.

Won't download games, only bonuses

I tested with the last version (master). It does not download the games but only the bonuses.

I am using linux (Manjaro 21.2.0) and python 3.9

I used these commands (without error):

gogrepoc.py login
gogrepoc.py update
gogrepoc.py download
gogrepoc.py verify

Games:

The Terraria game is missing for example, there are only bonuses:

python gogrepoc.py download -ids terraria                                                                                                                                                                                                                                 127 ✘ 
07:39:14 | loading local manifest...
07:39:14 | downloading games with id(s): {terraria}
07:39:14 | scanning manifest for renames...
07:39:14 | Cleaning up ./!downloading
07:39:14 | {terraria}
07:39:15 |      pass       terraria_soundtrack_vol_1_mp3.zip
07:39:15 |      pass       terraria_soundtrack_vol_2_mp3.zip
07:39:15 | nothing to download
07:39:15 | --
07:39:15 | total time: 0:00:00.398467
07:39:15 | exiting...

The script has nothing to download with metro 2033 redux:

python gogrepoc.py download -ids metro_2033_redux                                                                                                                                                                                                                             ✔ 
07:40:36 | loading local manifest...
07:40:36 | downloading games with id(s): {metro_2033_redux}
07:40:36 | scanning manifest for renames...
07:40:36 | Cleaning up ./!downloading
07:40:36 | {metro_2033_redux}
07:40:36 | nothing to download
07:40:36 | --
07:40:36 | total time: 0:00:00.046324
07:40:36 | exiting...

Maybe I misunderstood something ?

Is the resume manifest really necessary?

If we store a JSON object with the list of all games at the top of the manifest, and then add one JSON object entry per game, we can always know where we left off and proceed. Say, "our collection is 54 games, I could read only 43, let me work on the missing ones".

If we want to start over, we could just delete the manifest file! I think it is a pretty clean solution.

Clean function tries to orphan Windows hidden files on external hard drive.

I had my GOG games stored on the root of an external hard drive I have and the clean function would always error out because it didn't have permission to delete $RecycleBin and/or System Volume Information folders (which it shouldn't do anyways). Had to move all the games into a subfolder for it to function correctly.

Linux installers not found?

The game Master of Orion 2 has an installer for Linux (since the end of 2015, according to the GOG changelog). However, gogrepoc does not see it? Is this an issue on my side only?
This is what I tried:

python gogrepo.py update -ids 1207661633 -installers standalone 
python gogrepo.py download -ids 1207661633 -os linux -skipextras -- gog

I found no mention of a Linux installer on the manifest file and found no Linux installer found on the game directory. Am I doing something wrong?

Latest commits broke the script

I'm having errors while trying to update the manifest file. Every game gives an error similar to the following:

22:19:55 | (  1 / 901) fetching game details for 140_game...
22:20:04 |   -> title has changed "140_game" -> "140"
22:20:04 | The handled exception was:
22:20:04 | error
Traceback (most recent call last):
  File "/run/media/2844327D5A8D495D/gog/../gogrepoc/gogrepoc.py", line 313, in __getattr__
    return self[key]
           ~~~~^^^^^
KeyError: 'folder_name'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/run/media/2844327D5A8D495D/gog/../gogrepoc/gogrepoc.py", line 1715, in cmd_update
    handle_game_updates(gamesdb[item_idx], item,strict)
  File "/run/media/2844327D5A8D495D/gog/../gogrepoc/gogrepoc.py", line 657, in handle_game_updates
    if olditem.folder_name != newitem.folder_name:
                              ^^^^^^^^^^^^^^^^^^^
  File "/run/media/2844327D5A8D495D/gog/../gogrepoc/gogrepoc.py", line 315, in __getattr__
    raise AttributeError(key)
AttributeError: folder_name
22:20:04 | End exception report.

Empty downloads property

Hi.

Haven't seen this in the issues, and I am not willing to read 125 pages on the GOG forum, so please excuse me if this is ovious.
Just tried from a completely new folder and clean slate, still doing this.

python3 ../gogrepoc.py login
python3 ../gogrepoc.py update -full
python3 ../gogrepoc.py download -dryrun

It reports nothing but the extras to download (I have 147 games, they cannot just be 48G :D ). The manifest seems to be missing the URL to download the game itself:
'downloads': [],

Every game has this issue.

Is this something I messed up on the CLI or python dependencies ?

EDIT: Should point out I'm using the dev branch.
gogrepoc.py (version 0.4.0-a)

Revise the imporing of files - less hashing

When importing files, we don't need to compute the MD5 on all files. If the size of the file does not match the size of a file on your manifest, we can be pretty sure the MD5 won't match either.

So we should filter for file size first and for MD5 second. You'll see big improvements on importing speed.

Download everything?

Dumb question...Don't know if I should ask here or on gog, so doing both...

Is there a simple way to download EVERYTHING. Every language, every installer, every os, everything?

I keep trying different cmd options and all I get is the bonus files. I'm experimenting with the bash script now. But it looks like even that is somewhat limiting.

Any help?

savedir at the end of the command string fails to use path when following -os -lang or -skipfiles

All three options seem to treat the last argument, savedir as an attempt to give them an option. With -os it will complain that the specified os is not one of the valid os types. With -lang it will complain that specified language is not one of the valid languages, and finally with -skipfiles it will run but it will act on your current directory and not the supplied path, implying that the path has been entered as string to skip instead of accepted as savedir.

I don't see an option like -savedir, so I don't see a way to break the option out of itself. I've adjusted to just run the script in the folder I want the work done in order to work around this issue.

[Feature] OS-specific directories

Suggestion: instead of storing everything in the same directory as script, make use of OS-specific directories that serve for that purpose. For example, on linux it would be something like that:

  • "$HOME/.config/gogrepo/" for cookies and manifest
  • "$HOME/.local/share/gogrepo/" for downloaded games
  • "$HOME/.cache/gogrepo/" for downloading_dir, resume_manifest, netscape_cookies_tmp

Dirty example of how that may be achieved (covering just linux, but it can be adjusted - I just dont know the correct paths on windows and mac):
In imports:
from os.path import expanduser
In # filepath constants:
https://gist.github.com/moonburnt/275c67519d055f01afebf9a72fcc9d5f
In main:
https://gist.github.com/moonburnt/9e3b54d4290b046e83e9755b8f75cc2c

I've tested it and it works for everything but finished downloads - more about this bug there: #31

Revise the imporing of files - more trusting

If the file size and the file name are the same, and no MD5 information is provided by GOG, then perhaps I could trust that the file is the same. I would like to see this option added to make it easy to move/change my repository around.

"no md5 data found for" Error for (nearly) all .sh files

I'm not sure if GOG broke something recently or what, but for the past few months I've been getting "request failed: 404 Client Error: Not Found for url: <...> .sh.xml" and "no md5 data found for" errors for pretty much all Linux (.sh) files without exception. Windows and Mac files seem to be fine.

Please add a "shebang" into script

Calling script in linux shell as simple executable leads to some unpredictable results, since shebang is missing

#!/usr/bin/env python3 for example.

./gogrepoc.py fails

python3 ./gogrepoc.py works

Review code organization

The current code is quite long (almost 2400 lines) and not overly welcoming to newcomers in its structure or lack of comments. (I recall a function that returns nothing but alter their inputs, like filter_ functions or fetch_file_info — those really break expectations and best practices).

While a little bit more annoying to distribute, if the size or complexity of the code cannot be brought down, maybe splitting the source code in several files (e.g. one for each major command) could be an option?

Suggestion: Allow an executable version of the python script.

Many people don't really use Python for many things and there are programs that can be used to convert python to an executable.

Would be nice to have an option to download as an executable and would give you something to put in your releases tab. Then people could just have it setup to run off batch files when they want to refresh and update stuff.

Edit: Bonus if it can work well with a GUI like
https://github.com/Twombs/GOGRepo-GUI

When was skippatches removed?

I recently updated from an older version of your script, and when I tried to run it today with my typical py -2 gogrepoc.py download -skipgalaxy -skippatches -lang en -os windows mac I got the unrecognized argument message. Sure enough, searching through the python script, it has been removed. The old version (I don't remember which one, but I have the file downloaded) included it. Is there another way to accomplish the same thing? Why was skippatches removed?

Games that can't be downloaded / have persistent non-fatal warnings though Gogrepoc but can be downloaded though the GOG website

Please use this thread to report games / items, that have problems downloading through gogrepoc but do download through the website (please include the website link you used to verify this (ie the one with /en0game1 or /18102 at the end, not the specific download URI )
Please separate out reports of items that do not download vs those that have persistent non-fatal errors.

Unwanted languages downloaded

On my set-up (and ever since) I have updated the manifest through python gogrepoc update -os windows -lang en.
When downloading the games via python gogrepoc download, other languages beyond english are downloaded

E.g.

  1. Master of Magic Classic: french and german were also downloaded
  2. The Witcher 3 Wild Hunt Game:
    • br, de, fr, jp, pl, ru language packs;
    • bonus packs in french, german, italian, polish, russian & spanish
  3. Project Eden: french manual
  4. Syndicate: french manual and reference card
  5. Forgotten Realms Unlimited Adventures: german manual and reference card
  6. The Witcher: italian, polish & german manual and game guides
    • other like the above examples

Is this the expected behaviour?

Download fails if the manifest, game_item.size is None

Problem

Traceback (most recent call last):
  File "C:\Users\h1435\code\gogrepoc\gogrepoc.py", line 2777, in <module>
    main(process_argv(sys.argv))
  File "C:\Users\h1435\code\gogrepoc\gogrepoc.py", line 2539, in main
    cmd_download(args.savedir, args.skipextras, args.skipids, args.dryrun, args.ids,args.os,args.lang,args.skipgalaxy,args.skipstandalone,args.skipshared, args.skipfiles,args.downloadlimit)
  File "C:\Users\h1435\code\gogrepoc\gogrepoc.py", line 1709, in cmd_download
    work_dict[dest_file] = (game_item.href, game_item.size, 0, game_item.size-1, dest_file,downloading_file)
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

Using the latest build from master, deleted the manifest file and regenerated it via update -full

It looks like during a mass update, GOG servers may not always produce a file size information.
It appears that while updating the local manifest with large amount of changes, like using the -full option, the size information of some files, especially in the extras section, are not always returned. This causes the size property in the manifest to be None

This causes:

work_dict[dest_file] = (game_item.href, game_item.size, 0, game_item.size-1, dest_file,downloading_file) to fail, more precisely game_item.size-1 to fail as game_item.size is None (coming from the manifest)

Manually updating that game_item via the update command appears to be refreshing the missing information and allows further processing.

Workaround attempt Fix/workaround/robustness

Setting 'game_item.size' to 0 (or any other number) in this case allows the process to continue. While in case of download there will be a message in the logs..

1:23:30 | {tom_clancys_ghost_recon}
11:23:30 |      pass       setup_tom_clancys_ghost_recon_1.4.0.0_(19232).exe
11:23:30 |      pass       gr_manual.zip
11:23:30 |      unknown    gr_command_map.zip has no size info.  skipping
11:23:30 |      pass       gr_level_builder.zip
11:23:30 |      pass       gr_wallpaper.zip
11:23:30 |      pass       gr_vehicle_tagging_guide.zip
11:23:30 |      pass       gr_avatars.zip
11:23:30 |      pass       gr_artworks.zip
11:23:30 |      pass       gr_desktop_theme.zip
11:23:30 |      pass       gr_screensaver.zip
11:23:30 | nothing to download

When the actual download starts without chunk_tree around line 1942.

if game_item.size -> end with value of -1; which will cause a HTTP 200 response without the 'Content-Range' header and produces and exception.
if game_item.size is set to a large number it will cause a pre-allocation of a large file; and it will fail the hdr format check a few lines latter.

Suggestion

Not sure if the worker code can be updated to handle this situation. Regardless we should consider adding a log message with the suggested fix (reimport the problematic item) when the problem is encountered in line 1709.

High memory usage when downloading

I was downloading my games using this script, but my linux box was reporting quite high memory usage. Like in the range of 10GB+. Are downloaded files being saved periodically on the disk, or keeped in the RAM? For windows files are split into 4GB chunks by GoG, but some of Linux games are over 20GB in size for single file.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.