magnitus-/gogcli

Client to Interact With the API of GOG.com

About

This is a command line client to interact with the GOG.com api.

The end goal is to automate the backup of game files.

Usage

1. Create a file named cookie with the following format:

sessions_gog_com=<value taken from gog.com>
gog-al=<value taken from gog.com>

To get the values, login to gog.com in your browser, then, get the cookie values.

Here is a guide to look at your cookies in Chrome: https://developers.google.com/web/tools/chrome-devtools/storage/cookies

The above is needed, because GOG.com does not yet have an official api for third-party tools with user-generated api keys.

So, any tool wishing to get some kind of api token or cookie programatically without a lot of user-involvement will need to scrape information from the login page and circumvent the recaptcha. At best, this functionality would be flaky and subject to frequent malfunction, so I opted not to go that direction, at the risk of being less user-friendly.

2. Go download a binary for your platform and put it in the same directory as your cookie file: https://github.com/Magnitus-/gogcli/releases

3. See what commands currently supported.

For Linux, you can run the following on the command prompt:

./gogcli --help

Netscape Cookies

The client also supports Netscape cookie files. By default, it will use the format defined above, so to use the Netscape format, you need to specify it like so:

./gogcli gog-api user-info -y netscape

Supported Storage Solutions

The client supports both the filesystem and s3-compatible object stores (tested with Minio, but should be compatible with Ceph, Swift, Amazon S3, Digital Ocean Spaces and others).

If you use the local filesystem, you just need to provided a path to commands.

If you use an s3 store, you need to provide a path to a configuration file in json format which is as follows:

{
    "Endpoint": "<The S3 endpoint of your object store>",
    "Region": "<The S3 region your bucket should be in>",
    "Bucket": "<The bucket in which your manifest and game files should be stored>",
    "Tls": true|false,
    "AccessKey": "<Your access key>",
    "SecretKey": "<Your secret key>"
}

Building The Binaries Yourself

If you prefer, you can build the binary locally:

• Get golang: https://golang.org/dl/
• Run: go build

Architecture

The documentation below consists of quick howtos.

For a deeper understanding of what the commands do, read the following doc: https://github.com/Magnitus-/gogcli/tree/main/architecture-documentation

Common Use Case Examples

Here, I assume that the gogcli binary is in the PATH so that I just type gogcli without having to use a relative path (ex: ./gogcli).

Furthermore, for Windows, the executable would be gogcli.exe, but the commands are otherwise the same.

Note that I'm using the long form arguments in all the examples to make everything more legible, but if you'd like to type less, all the arguments have a short form version, following the POSIX convention.

Creating an Initial Manifest

If I want to generate an initial manifest for Linux and Windows files, for the French and English language, I would type:

gogcli manifest generate --lang=english --lang=french --os=windows --os=linux

This will take a couple of minutes and will produce a manifest.json manifest file.

If GOG.com had some files where not downloadable (they sometimes forget to delete links to files that are no longer available), it will be indicated in the manifest-404-warnings.json file.

You can get a summary of your manifest by typing:

gogcli manifest summary

If you have a very large manifest and want to look for Master of Orion and Master of Magic entries without having to open the file, you can type:

gogcli manifest search --title="Master of Orion" --title="Master of Magic"

Uploading Games From Your Manifest

So now, you are ready to upload your games in your storage.

If you want to store your games in your filesystem, say in the /home/eric/games directory, you would type:

gogcli storage apply manifest --path=/home/eric/games --storage=fs
gogcli storage execute-actions --path=/home/eric/games --storage=fs

If instead, you have an s3 store whose configuration information is in the file s3.json (see instructions above for how to configure s3 stores), you would type:

gogcli storage apply manifest --path=s3.json --storage=s3
gogcli storage execute-actions --path=s3.json --storage=s3

If you have 1000 games to upload and would like to only upload 50 games for now and do the rest later, you would type:

gogcli storage apply manifest --path=s3.json --storage=s3
gogcli storage execute-actions --path=s3.json --storage=s3 --maximum=50

And if of those 1000 games, you to download your smaller indie games first and Faster Than Light first of all (because it is such a great game), you would first type the following to get FTL's game id:

gogcli manifest search --title=FTL

And then type:

gogcli storage apply manifest --path=s3.json --storage=s3
gogcli storage execute-actions --path=s3.json --storage=s3 --maximum=50 --sort-criterion=size --preferred-ids=1207659102

If you'd like to look at what's left to download, you can download all the actions that have yet to run by typing:

gogcli storage download actions --path=s3.json --storage=s3

The remaining actions will be listed in a file called actions.json.

Then you decide to continue with the next 50 games tomorrow. Lets say you still want to download smaller indie games first, but this time, you'd like to download all the Blackwell games first of all.

First, you will type the following to find out the game id of the Blackwell games:

gogcli manifest search --title=Blackwell

Then, you will type the following to download the next 50 games, with the Blackwell games first

gogcli storage execute-actions--path=s3.json --storage=s3 --maximum=50 --sort-criterion=size --preferred-ids=1207662883 --preferred-ids=1207662893 --preferred-ids=1207662903 --preferred-ids=1207662913 --preferred-ids=1207664393

And so forth...

If, after you have uloaded everything, you get a little paranoid and want to make really sure that the files are still in the state indicated by the manifest, you can type:

gogcli storage validate --path=s3.json --storage=s3

If you don't see any output, you are good.

If you don't trust the tool yet and want feedback that it actually did something, you can type:

gogcli storage validate --path=s3.json --storage=s3 --debug

Be warned, you may get more output than you bargained for.

Copy Your Files to A Secondary Storage

So now, lets say that you opted for the s3 storage in the example above, but you'd also like to copy your games on your local drive. You can type:

gogcli storage copy --source-path=s3.json --source-storage=s3 --destination-path=/home/eric/games --destination-storage=fs

Again, if you are feeling paranoid, you can validate that the files copied properly at the destination, by typing:

gogcli storage validate --path=/home/eric/games --storage=fs

Updating Your Storage with GOG.com Updates

So now, GOG.com released some updates and you would like very much to update your storages.

Lets say you want to update your filesystem storage first.

You have two choices:

Option 1: Trust GOG.com to report Updates Properly

So first, you download your manifest from your storage by typing:

gogcli storage download manifest --path=/home/eric/games --storage=fs

Your manifest will be downloaded in manifest.json.

After that, you get a list of your updated and new games by typing:

gogcli update generate

The game ids of your new and updated games will be listed in updates.json.

After that, you want to update your manifest with your updates, by typing:

gogcli manifest update --update=updates.json

Now, you are ready to apply the modifed manifest. Before that, you may wish to run a plan to look at the actions that will run against your storage by typing:

gogcli storage plan --path=/home/eric/games --storage=fs

Alternatively, if you are ok matching file name and file size couting as the same file when there are no checksums (the gog api doesn't provide it for extras... it should be ok 99.99%+ of the time), you can type:

gogcli storage plan --empty-checksum --path=/home/eric/games --storage=fs

The actions will be in the actions.json file.

You can apply your modifed manifest by typing:

gogcli storage apply manifest --path=/home/eric/games --storage=fs
gogcli storage execute-actions --path=/home/eric/games --storage=fs

Again, if you wish for extras without checksum to count as the same file if the file name and file size match, you type this instead:

gogcli storage apply manifest --empty-checksum --path=/home/eric/games --storage=fs
gogcli storage execute-actions --path=/home/eric/games --storage=fs

Afterwards, if you want to copy the modifications from your filesystem to your s3 store, you can type:

gogcli storage copy --source-path=/home/eric/games --source-storage=fs --destination-path=s3.json --destination-storage=s3

If you are surprised that it runs really fast, don't worry. The gogcli storage copy command doesn't just mindlessly copy files. It actually does a diff between the manifests of both storages and copies only what it must.

Option 2: Don't Trust GOG.com, Just Check Everything

In this case, you'll just generate a new manifest from scratch and apply it.

Generate the manifest by running:

gogcli manifest generate --lang=english --lang=french --os=windows --os=linux

Afterwards, you can look at the actions that will run against your storage by typing:

gogcli storage plan --empty-checksum --path=/home/eric/games --storage=fs

Here, you probably always want to use the empty-checksum flag, because all the extras in your generated manifest won't have a checksum (the gog api doesn't provide it) and it will be a lot of extras to upload (for your entire game collection) if you decide that files with an empty checksum don't count as the same file even when the file name and file size matches.

Anyways, the actions will be listed in the actions.json file.

If you are satisfied with the actions that will run, you can now apply your manifest by typing:

gogcli storage apply manifest --empty-checksum --path=/home/eric/games --storage=fs
gogcli storage execute-actions --path=/home/eric/games --storage=fs

Afterwards, you can copy the changes from your filesystem to your s3 store by typing:

gogcli storage copy --source-path=/home/eric/games --source-storage=fs --destination-path=s3.json --destination-storage=s3

Updating Your Storage with GOG.com When You Have Pending Actions

Ok, so you started storing your games from your manifest using the gogcli storage apply command, but it did not complete, either because there was an error or you used the --maximum flag.

You took a break and in the interim, gog updated some games and now you're stuck with a storage that has some half-uploaded manifest that is no longer valid.

What do you do? The same thing you would if there were no pending actions. To recap below...

First of all, you produce an updated manifest.json manifest using the gogcli storage download manifest + gogcli update generate + gogcli manifest update commands (option 1)... or you just use a gogcli manifest generate command (option 2).

After that, you run:

gogcli storage apply manifest --path=s3.json --storage=s3
gogcli storage execute-actions --path=s3.json --storage=s3

Or if you really don't want to redownload extras that don't have a checksum and trust that the file was not updated if the file name and size matches:

gogcli storage apply manifest --empty-checksum --path=s3.json --storage=s3
gogcli storage execute-actions --path=s3.json --storage=s3 

What just happened? Your storage's manifest got updated and the remaining actions got adjusted to include additional actions from the change in your manifest.

Repair Broken Storage

Ok, so ran storage validate and it returned some errors, maybe you deleted some files per accident or maybe, you generated a storage with a previous version of gogcli, then migrated your manifest and you'd like to make sure your storage is still ok.

Here, you might not be able to run a gogli manifest appy, because this commands only applies a differential between your local manifest and the manifest in the storage. It assumes that the storage's game files reflect the storage's manifest and do not check them separately.

Instead, to reconcile a situation where the game files diverge from what the manifest indicates, assuming your local file manifest.json contains a proper manifest (obtained either directly from your storage if you can manage it or otherwise from GOG.com running a gogli manifest generate command though in the later case, you'll have to redownload most of your extras as they don't have a checksum), you would run (assuming you have an s3 store):

gogcli storage repair --path=s3.json --storage=s3 

After running the command, you'll possibly have a bunch of pending actions in your storage if adjustments were needed.

You will execute those actions just like you would with an apply by running:

gogcli storage execute-actions --path=s3.json --storage=s3 

Dealing With Repeated Download Mismatch

Sometimes, during a download, you might have to deal with repeated errors like this:

addFileAction(gameId=1207659025, fileInfo={Kind=extra, Name=darkstone_avatars.zip, ...}, ...) -> Download file size of 445 does not match expected file size of 184891
addFileAction(gameId=1207659025, fileInfo={Kind=extra, Name=darkstone_artworks.zip, ...}, ...) -> Download file size of 445 does not match expected file size of 6430403

If it occurs only once, it might be a bad download. If it occurs several times, it might mean that the game has changed since your manifest has been generated.

The best way to deal with the problem is to first get your manifest (using an s3 storage as an example here):

gogcli storage download manifest --storage=s3 --path=s3.json

Update that game entry in your manifest:

gogcli manifest update --id=1207659025

And finally, update the actions list in your storage with your updated manifest:

gogcli storage apply manifest --empty-checksum --path=s3.json --storage=s3

Searching Manifest

Sometimes, you want to find games matching certain criteria in your manifest. Gogcli has a search command to help you accomplish this.

You can output the result of your search either in a terminal or in a file.

For example, if you want to find games who title include the string master of orion and output them on the terminal, you would type:

gogcli manifest search --title="master of orion"

If you want to find games containing patch installer files and output them in the terminal, you would type:

gogcli manifest search --has-url="^/downloads/.*/[a-z]{2,2}[0-9]patch[0-9]$"

And if you find there is too much output on the terminal with your last command and would rather output into a file, you would type:

gogcli manifest search --has-url="^/downloads/.*/[a-z]{2,2}[0-9]patch[0-9]$" --terminal=false

If you want to adapt the above command to find games containing a French-only installer, you would type:

gogcli manifest search --has-url="^/downloads/.*/fr[0-9]installer[0-9]$" --terminal=false

Trim Patches

Patches for some games can take a lot of disk space. Gogcli containes a convenience command to trim patches for a given game from your manifest and then you can apply it to your storage normally.

Lets assume I want to trim the patches for the game with id 1488706732. Assume I have an s3 store with storage specifications in a file called s3.json.

I would run the following command to get the manifest from storage:

gogcli storage download manifest -p s3.json -k s3

I would run the following command to trim the patches from the game in the manifest:

gogcli manifest trim-patches --id=1488706732

I would run the following command to look at a plan for the update (outputed in actions.json):

gogcli storage plan --empty-checksum --path=s3.json --storage=s3

And finally, assuming I am satisfied with the plan, I would run the following two commands to apply the update to the storage and delete the files:

gogcli storage apply manifest --empty-checksum --path=s3.json --storage=s3
gogcli storage execute-actions --concurrency=1 --path=s3.json --storage=s3

Manifest Summary

The following command will output a summary of your manifest:

gogcli manifest summary

It will tell you how many games are in your manifest, how many files, how many installer files, how many extra files, the aggregate size of all your game files, the average size of a game in your collection as well as the largest and smallest game in your collection.

Migration

From gogcli version 0.10.x to 0.18.x

With the eventual goal of creating more customizable and decoupled plugin storage solutions, I opted to add game slugs to the manifest file. These were added purely to add customisability to future storage plugins and should not impact existing functionality.

So, if you don't migrate a manifest generated by gogcli between version 0.10.x and 0.18.x, empty slug entries will be added to your manifest.

If you want those slug entries to be properly populated with the right values, you can either regenerate a new manifest or migrate your existing manifest with the command below:

gogcli storage manifest migrate

Note: The above assumes that your manifest is already at version 0.10.x or later. If that is not the case, read the instructions below.

From gogcli version 0.9.x or earlier

The manifest had to be changed in version 0.10.0, because of an unforeseen situation with languages which forced me to change the manifest format.

To migrate your storage's manifest (using an s3 storage as an example here), copy the manifest to migrate in your current path. Then run:

gogcli storage manifest migrate

gogcli storage repair --path=s3.json --storage=s3

Note: You can upgrade a manifest generated by gogcli version 0.9.x with a gogcli binary between versions 0.10.x and 0.18.x only.

So if you want to upgrade your manifest generated in gogcli v0.9.x or earlier to version 0.19.x or later, you should:

• First download gogcli version 0.18.x and do a migration
• Then download gogcli version 0.19.x or later and do another migration

gogcli's Issues

Add some sort of progress bar when downloading

Currently, there is no way by default to see how far your download has gone, this makes it difficult to know when a download is about to complete. I think that adding a Progress bar would be a good idea, alongside a download time calculation.

In addition, this will make it easier to determine if a download has stalled.

Download Games Metadata and Media Files in the Storage

With the ultimate goal of providing an experience similar to the GOG library page and game pages to one's collection offline, the first part of this process would be to download the metadata and accompanying media files (ie, pictures) in the storage.

Because GOG provides youtube videos in its game pages, those can't be downloaded and would have to be linked.

Technically, the videos on Steam can be downloaded, though figuring out a mapping to Steam videos for all games would be labor intensive and there is an ethical consideration to ponder about hitting the Steam servers with a lot of video download requests for content that has been purchased elsewhere.

Allow to use slug instead of id in game directory name

As title says using just id for game directory stinks and it's very hard to find game you are looking for.

Please allow to use just slug (or if you are worried about slug not being uniqe slug-id combination).

Keep Previous Manifest When Applying

Currently, when a manifest is applied, the previous manifest is overwritten and lost.

It would be a good idea to make a copy of the previous manifest and keep it in storage, until the next manifest after that is applied.

Provide Dashboard to Visualise and Download Games from Storage

Provide a web dashboard to list games, view game details and download them from the storage.

This would follow issue #7

Broken redirects prevent proper manifest creation

I can't generate a fresh manifest at the moment, due to a temporary error on GOG's side. When running gogcli.exe manifest generate --lang=english --lang=german --os=windows, I end up with the following manifest.json (this is the entire content of the generated file):

{
  "Errors": [
    "GetDownloadFileInfo(downloadPath=/downloads/imperator_rome/en1patch0) -> Expected response status code of 302, but got 404"
  ]
}

Seems like it aborts after encountering the first problem of this kind, broken HTTP redirects handled by retrieveUrlRedirectLocation() from what I see. It happens for various of the Imperator: Rome files when retrying, the download links for them are pretty unstable right now (also confirmed via manual browser download). No other files are created by gogcli, only this unusable manifest.

I tried running it with debug-level logging, but nothing of interest is logged beyond starting a HTTP GET request. Output stops after the batch of requests containing the broken one was logged and then the program terminates with exitcode 1. Here are the last 10 lines of the log (got lucky the culprit was actually in the very last line, usually it's somewhere else in the batch):

[sdk] GetDownloadFileInfo(downloadPath=/downloads/simulacra_2/en1installer0) -> GET https://www.gog.com/downloads/simulacra_2/en1installer0
[sdk] GetDownloadFileInfo(downloadPath=/downloads/geneforge_3/en1installer0) -> GET https://www.gog.com/downloads/geneforge_3/en1installer0
[sdk] GetDownloadFileInfo(downloadPath=/downloads/cyberpunk_2077_game/en1installer6) -> GET https://www.gog.com/downloads/cyberpunk_2077_game/en1installer6
[sdk] GetDownloadFileInfo(downloadPath=/downloads/pathologic_2/en1patch2) -> GET https://www.gog.com/downloads/pathologic_2/en1patch2
[sdk] GetDownloadFileInfo(downloadPath=/downloads/rebel_galaxy_outlaw/en1installer2) -> GET https://www.gog.com/downloads/rebel_galaxy_outlaw/en1installer2
[sdk] GetDownloadFileInfo(downloadPath=/downloads/not_for_broadcast/en1patch1) -> GET https://www.gog.com/downloads/not_for_broadcast/en1patch1
[sdk] GetDownloadFileInfo(downloadPath=/downloads/layers_of_fear/en1installer1) -> GET https://www.gog.com/downloads/layers_of_fear/en1installer1
[sdk] GetDownloadFileInfo(downloadPath=/downloads/my_time_at_portia/en1patch1) -> GET https://www.gog.com/downloads/my_time_at_portia/en1patch1
[sdk] GetDownloadFileInfo(downloadPath=/downloads/finding_paradise/en1patch2) -> GET https://www.gog.com/downloads/finding_paradise/en1patch2
[sdk] GetDownloadFileInfo(downloadPath=/downloads/imperator_rome/en1patch0) -> GET https://www.gog.com/downloads/imperator_rome/en1patch0

I would have expected the program to create a working manifest and list the error alongside other problems in manifest-warnings.json. The option --tolerate-dangles is implicitly set to true, after all. I've also tried setting it explicitly, just to be absolutely sure.

Add Basic Download Speed Measurement on the gog-api download-url-path Command

After the gog-api download-url-path has ran, it should display the file size, how long the download took and the download speed per second.

Rework Storage Commands

Atm, we have the following 3 storage commands:

1. storage apply: Uploads a manifest in the storage, generates a new action file in the storage and execute the actions in the actions file
2. storage resume: Run the remaining actions for a pre-existing action file in the storage
3. storage update-actions: Uploads a manifest in the storage and update a pre-existing action file in it

I think the apply tries to do too much by also executing the actions file, preventing a more customized workflow (if someone wanted to edit the actions file first for example).

I also think that apply and update-actions as separate commands is needless overhead for the end-user. While the case where there are pending actions in the storage that need to be accounted for when updating the manifest is separate, it can be auto-detected and handled by the same command.

I should have the following commands instead:

1. storage apply manifest: Updates the manifest, generating a new actions file or updating an existing one in the process
2. storage execute actions: Run the actions listed in the actions file on the storage

HTTP 404 terminates manifest update

Unfortunately, GOG continues to throw spanners in your gears.

I ran into two products that prevented me from updating my manifest via gogcli.exe manifest update --update=updates.json. The outputs stopped like this:

[manifest writer] Got all info on game with id 1174117468. 300 games left to process
[manifest writer] Got all info on game with id 1169744036. 299 games left to process
 getDownloadFileInfoWorkaroundWay(downloadPath=/downloads/spiritfarer_ost/en1patch3) -> body download handle retrieval error: did not expect status code of 404

 getDownloadFileInfoWorkaroundWay(downloadPath=/downloads/prison_architect_going_green/en1patch0) -> body download handle retrieval error: did not expect status code of 404

After this, the program terminated immediately (pretty sure it must have been via a non-zero exit code but I didn't check). It created a manifest-update-progress.json, but I couldn't find a way to make gogcli actually use that to resume the update, it restarted from scratch on every retry. I logged the output at the various available levels, but couldn't get anything more useful out of it.

As the problem was perfectly reproducible and happened on the same url every time I tried, I just removed the respective product id from my updates.json file and reran the command, rinse and repeat until I got no terminating error anymore. This might just be a temporary problem and I will retry after my downloads are done, but I wanted to report it anyway so you can decide if it's worth handling.

Broken XMLs may prevent proper manifest creation

Unfortunately I'm experiencing a similar behavior to #16 again. I've run gogcli.exe manifest generate twice and each time it failed handling some broken data related to XMLs. I set the log-level to debug the second time around but didn't get any useful info beyond calling GetDownloadFileInfo() for the problematic file. The version used was v0.19.0.

As creation takes a very long time and uses a lot of network resources ever since the XML problems on GOG began, I'd rather not do a couple more test runs with the current version. Here are the two created manifests I ended up with in their entirety:

{
  "Errors": [
    "GetDownloadFileInfo(downloadPath=/downloads/order_of_battle_us_marines/en1patch1) -> retrieval request error: Get \"https://gog-cdn-lumen.secure2.footprint.net/token=nva=1650461891~dirs=6~token=0bade971fde4080f53a15/secure/offline/1744082009/1192658776/52863558974871196/4509/setup_order_of_battle_u.s._marines_8.3.0_%2832bit%29_%2834403%29.exe.xml\": unexpected EOF"
  ]
}

{
  "Errors": [
    "GetDownloadFileInfo(downloadPath=/downloads/caves_of_qud/en1patch2) -> Could not parse file xml metadata and the first line was not the expected format: EOF"
  ]
}

Thanks for implementing the workaround method by the way, I'm fully aware of the consequences of using it. Getting an almost complete manifest that only misses those couple of problematic entries would be perfect. I would just retry to retrieve their data via gogcli.exe manifest update later then.

Add command to selectively download games and their files

I understand that gogcli acts more like a mirror program but it would be nice if there was a command to download only selected game(s) and possibly only selected files within those games.

For example I may only want to download the extras of a game which I have already downloaded previously, or only download the installer with required patches instead of all patches.

Windows: CRLF newlines in cookie file causes error "net/http: invalid byte '\r' in Cookie.Value"

E:\Apps\Net\gogcli>gogcli update generate
2022/06/26 11:29:50 net/http: invalid byte '\r' in Cookie.Value; dropping invalid bytes
2022/06/26 11:29:51 net/http: invalid byte '\r' in Cookie.Value; dropping invalid bytes
2022/06/26 11:29:51 net/http: invalid byte '\r' in Cookie.Value; dropping invalid bytes
2022/06/26 11:29:51 net/http: invalid byte '\r' in Cookie.Value; dropping invalid bytes
2022/06/26 11:29:51 net/http: invalid byte '\r' in Cookie.Value; dropping invalid bytes
2022/06/26 11:29:53 net/http: invalid byte '\r' in Cookie.Value; dropping invalid bytes
^C

Probably just need to add instructions to the README to save the cookie file using LF line endings. (Could be tricky for non-technical users)

Allow inversion of filter

I use a manifest filtered for English and German windows setups including extras to download my entire library. Additional Languages and OS would only take up space I will never use. However, there are some games that haven't been translated to English or German (or may don't have a Windows version, but I don't know of any). By using my filter, I would miss these completely. In my case, "Kajko i Kokosz" is the only such game. Other examples could be "Anstoss 2 Gold Edition" and "ANSTOSS 3: Der Fußballmanager" which are only available in German, but those of course don't affect me personally.

My current workaround to deal with this problem: I generate a complete unfiltered manifest, then filter it via gogcli search to my preferences and download the big chunk of my games. To catch missed products, I wrote a script that searches the manifest for products that weren't matched by my filter settings at all. I then tag the results on GOG as "FOREIGN", search-filter the full manifest with gogcli again to find only products with that tag, and finally download it in every language and for all OS.

I think it would be neat to offer such a functionality as a native feature in gogcli. Maybe there's a better way to handle this edge case, it was just the best I could come up with in a short time and from what I can tell, it works fine.

No Feedback on if Manifest is Generating

When I type the command to generate a manifest, it just sits there. There's is a blinking cursor under the command but it doesn't appear to be doing anything and no manifest file is added to the gogcli folder.

I suspect this has to do with my cookie file but I'm not 100% sure. I looked everywhere in both Chrome's and Firefox's cookie inspector but couldn't find sessions_gog_com anywhere to get a value from. Any help?

SkipUrls broken in v0.22.0

I tried to generate a manifest and got a few 404 URLs again, so I excluded them via regex pattern in the SkipUrls array of the manifest-generation-progress.json file - as explained here.

But something breaking must have changed between v0.21.0 and v0.22.0, because the new version seemingly ignores this option, while it works fine with the old one, just as before. I tried manifest generate-resume on both versions with the exact same progress file containing the following filter option:

      "SkipUrls": [
        "^/downloads/episode_2_blackwell_unbound/en2installer0$",
        "^/downloads/episode_3_blackwell_convergence/en2installer0$",
        "^/downloads/heart_of_the_woods/en1patch1$",
        "^/downloads/the_blackwell_epiphany/en2installer0$"
      ],

v0.21.0 completed without errors, but v0.22.0 terminated pretty quickly after reaching the point where it actually processes games. It printed the following error before exiting:

 getDownloadFileInfoWorkaroundWay(downloadPath=/downloads/episode_2_blackwell_unbound/en2installer0) -> body download handle retrieval error: did not expect status code of 404

That error should not have happened if SkipUrls were honored. I did several tries and it stopped on episode 3 a few times as well, even though the exact URL was also configured to be skipped. The progress file was initially created by v0.22.0, so a format change can't be the reason (I updated gogcli before doing my quarterly backup).

These are all the broken URLs from my library right now, so things got better in that regard. Just one of the old patch links left and three Mac installers that will probably get resolved.

On a side note, manifest generation is blazingly fast in comparison to before, because there wasn't a single file that needed the workaround method to get its checksums. Seems like their CDN problems and having to switch to their own temporary solution was good for one or two things at least (not sure if it had anything to do with it, but sounds plausible to me). Just ~1.5 hours for a complete manifest on v0.21.0 with no errors beyond these four broken URLs, almost ideal conditions for my library.

Expired auth cookie gives unfriendly error message "json parsing error"

I ran gogcli update generate, and received this error message:

GetOwnedGames(page=1, search=) -> json parsing error: invalid character '<' looking for beginning of value

I guessed that it was failing to parse HTML, which in my experience is often returned by 400 or 401 HTTP responses. Sure enough, updating my cookie values fixed the error.

Would it be possible to give a more meaningful error message?

Add Actions Summary Command

Atm, you can view the summary of a downloaded manifest with the manifest summary command.

It would be nice if there was the same command for actions showing at least:

• The number of added games
• The number of deleted games
• The number of added files
• The number of deleted files

Implement Multiplart Download POC on gog-api download-url-path Command

Multipart downloads have the potential to speedup slow downloads on large files for some users and allow to reuse some of the downloading on the next attempt if it is interrupted.

This should be implemented standalone on the gog-api download-url-path command as a poc before further integration is contemplated in the storage part of the tool.

Right now, I see 3 potential ways to approach this:

1. Just make arbitrary range queries
2. Use the ranges and checksums provided by GOG on the older offline installers endpoints used on the website
3. Investigate if the newer Galaxy endpoints could be used instead

Potential drawbacks of each approach:

1. Very flexible, but because there aren't any pre-computed checksums on arbitrary range queries, bad downloads where the checksum don't match at the end would have to be retried from scratch as there would be no way to know which range request downloaded a bad segment
2. There is at least one game where the xml document indicating the ranges is malformed. Even if we assume its an anomaly for only one game, I believe this document was mainly used by the now deprecated GOG Downloader, so it might be dropped in the future
3. I'm not sure if it is possible to access all the Galaxy endpoints with the cookie data that this client uses. Even if it is, I'm not sure if it is possible to download the offline installers from the Galaxy endpoints. I'd have to investigate that.

Add support for Netscape (curl) cookie format

Hi,

Thanks for making the tool, however it isn't working for me:

TheDcoder@arch /t/gogcli [1]> gogcli gog-api user-info
Email:                   
Username:                
Avatar:                  
UserId:                  
GalaxyUserId:            
PreferredLanguage:       English
SelectedCurrency:        USD
WalletBalance:           0.000000 
WishlistedItems:         0
PurchasedItems:
  Games:                 0
  Movies:                0
Updates:
  Messages:              0
  PendingFriendRequests: 0
  UnreadChatMessages:    0
  Products:              0
Friends:
TheDcoder@arch /t/gogcli> gogcli gog-api owned-games
GetOwnedGames(page=1, search=) -> json parsing error: invalid character '<' looking for beginning of value

I am running your latest linux-amd64 build and I extracted cookies properly from my browser (I am logged into GOG.com). Any idea what might be going wrong?

Include Safe Mode By Default When Applying Manifests

By default, when applying a manifest, if it would result in a game being deleted, the operation should abort.

There should be a special flag to specify that you want to possibly delete some games

Custom Storage Service Support

For extensibility with third parties, there should be a third storage option called custom storage service.

This storage should take a url as a parameter and communicate with an external service following a documented restful api.

This would allow any third-party to implement a custom storage by implementing the api, without having to add code to this project.

Include File Locking

Sometimes, some game files are removed from the GOG.com library by the publishers for good reasons and sometimes, they are dropped to the detriment of the end-user.

A legitimate removal might be removing the previous version of an installer when a game is updated to a new version.

A detrimental removal might be the publisher removing the Linux installer for a game (even though it works fine) or a game extra (ex: the soundtrack).

Given the above case, the tool should allow end-users to lock certain files in the manifest and make them non-removable during future manifest updates, without having to manually edit the manifest and actions file each time.

Add support for resuming partial downloads

I noticed that gogcli currently starts the download from scratch if I interrupt it, it would be nice if there was an option to resume partially downloaded files.

I tested this using the gog-api download-url-path command.

Perhaps an additional command to verify the checksum of an already downloaded file would be nice to have too! 😁

[Question] Remove other language if prefered exists

Hello,
is there a way to remove downloads from manifest for other languages than preferred?

I'm using
gogcli manifest generate --lang=polish --lang=english --os=windows
but some games have both languages available and i'm interested in only one (Polish in this example).
Can I somehow filter other languages out?

Example for The Witcher 3: Wild Hunt - Complete Edition

D:\gogcli>gogcli manifest search --title="The Witcher 3: Wild Hunt - Complete Edition"
{
  "Games": [
    {
      "Id": 1495134320,
      "Slug": "the_witcher_3_wild_hunt_game_of_the_year_edition_game",
      "Title": "The Witcher 3: Wild Hunt - Complete Edition",
      "CdKey": "",
      "Tags": [],
      "Installers": [
...
  ],
  "EstimatedSize": "185.26 GB",
  "VerifiedSize": 196994462426,
  "Filter": {
    "Titles": [
      "The Witcher 3: Wild Hunt - Complete Edition"
    ],
    "Oses": [
      "windows"
    ],
    "Languages": [
      "polish",
      "english"
    ],
    "Tags": [],
    "Installers": true,
    "Extras": true,
    "ExtraTypes": [],
    "SkipUrls": [
      "^/downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/[a-z]{2,2}[0-9]patch[0-9]$"
    ],
    "HasUrls": [],
    "Intersections": []
  },
  "ProtectedFiles": null
}

This would allow to limit download time and size.

Thanks for any tips!

Repair and Validate Should Have an Option to Skip Checksum Validation

Currently, for "storage repair" and "storage validate" commands, the checksum of files are compared against the value in the manifest.

When your collection is large, this can take some time.

There should be a flag to skip the checksum validation, useful in cases where you just want to perform a quick sanity check that the files in storage match the manifest.

Support RESTful API as Commands

While the command-line commands are fine, it would be even better for scripting purposes if the functionality provided by the commands of this tool was also supported via a documented restful api.

Then, any custom logic would more easily be scripted on top of this tool without having to launch an external process and having to deal with process outputs or file outputs.

Where to find cookie "sessions_gog_com"

I have no problem finding the cookie "gog-al" but I just cannot find the cookie "session_gog_com". I want to update my gogcli cookie file and found the value before, but somehow forgot where I got it from. @Magnitus- can you please explain shortly how to get it in a chrome browser (e.g. Edge)? Should it be under "Applications" > "Cookies" too? I do not have it there.

Update: I found it. After clicking on "Forum" it finally appeared.

Not sure if related to this or the GUI created for it but it appeared to be missing some files.

I used this with GOGcli-GUI
https://github.com/Twombs/GOGcli-GUI

Not sure which was the cause, but I noticed with Fallout 4, it refused to pull up all the installer files, I think it was missing part 2 of the multiple parts, didn't matter how many times I refreshed, it still seemed to miss it.