magnitus-/gogcli

Client to Interact With the API of GOG.com

gogcli's Issues

Include Safe Mode By Default When Applying Manifests

By default, when applying a manifest, if it would result in a game being deleted, the operation should abort.

There should be a special flag to specify that you want to possibly delete some games

Broken redirects prevent proper manifest creation

I can't generate a fresh manifest at the moment, due to a temporary error on GOG's side. When running gogcli.exe manifest generate --lang=english --lang=german --os=windows, I end up with the following manifest.json (this is the entire content of the generated file):

{
  "Errors": [
    "GetDownloadFileInfo(downloadPath=/downloads/imperator_rome/en1patch0) -> Expected response status code of 302, but got 404"
  ]
}

Seems like it aborts after encountering the first problem of this kind, broken HTTP redirects handled by retrieveUrlRedirectLocation() from what I see. It happens for various of the Imperator: Rome files when retrying, the download links for them are pretty unstable right now (also confirmed via manual browser download). No other files are created by gogcli, only this unusable manifest.

I tried running it with debug-level logging, but nothing of interest is logged beyond starting a HTTP GET request. Output stops after the batch of requests containing the broken one was logged and then the program terminates with exitcode 1. Here are the last 10 lines of the log (got lucky the culprit was actually in the very last line, usually it's somewhere else in the batch):

[sdk] GetDownloadFileInfo(downloadPath=/downloads/simulacra_2/en1installer0) -> GET https://www.gog.com/downloads/simulacra_2/en1installer0
[sdk] GetDownloadFileInfo(downloadPath=/downloads/geneforge_3/en1installer0) -> GET https://www.gog.com/downloads/geneforge_3/en1installer0
[sdk] GetDownloadFileInfo(downloadPath=/downloads/cyberpunk_2077_game/en1installer6) -> GET https://www.gog.com/downloads/cyberpunk_2077_game/en1installer6
[sdk] GetDownloadFileInfo(downloadPath=/downloads/pathologic_2/en1patch2) -> GET https://www.gog.com/downloads/pathologic_2/en1patch2
[sdk] GetDownloadFileInfo(downloadPath=/downloads/rebel_galaxy_outlaw/en1installer2) -> GET https://www.gog.com/downloads/rebel_galaxy_outlaw/en1installer2
[sdk] GetDownloadFileInfo(downloadPath=/downloads/not_for_broadcast/en1patch1) -> GET https://www.gog.com/downloads/not_for_broadcast/en1patch1
[sdk] GetDownloadFileInfo(downloadPath=/downloads/layers_of_fear/en1installer1) -> GET https://www.gog.com/downloads/layers_of_fear/en1installer1
[sdk] GetDownloadFileInfo(downloadPath=/downloads/my_time_at_portia/en1patch1) -> GET https://www.gog.com/downloads/my_time_at_portia/en1patch1
[sdk] GetDownloadFileInfo(downloadPath=/downloads/finding_paradise/en1patch2) -> GET https://www.gog.com/downloads/finding_paradise/en1patch2
[sdk] GetDownloadFileInfo(downloadPath=/downloads/imperator_rome/en1patch0) -> GET https://www.gog.com/downloads/imperator_rome/en1patch0

I would have expected the program to create a working manifest and list the error alongside other problems in manifest-warnings.json. The option --tolerate-dangles is implicitly set to true, after all. I've also tried setting it explicitly, just to be absolutely sure.

Support RESTful API as Commands

While the command-line commands are fine, it would be even better for scripting purposes if the functionality provided by the commands of this tool was also supported via a documented restful api.

Then, any custom logic would more easily be scripted on top of this tool without having to launch an external process and having to deal with process outputs or file outputs.

Custom Storage Service Support

For extensibility with third parties, there should be a third storage option called custom storage service.

This storage should take a url as a parameter and communicate with an external service following a documented restful api.

This would allow any third-party to implement a custom storage by implementing the api, without having to add code to this project.

Allow to use slug instead of id in game directory name

As title says using just id for game directory stinks and it's very hard to find game you are looking for.

Please allow to use just slug (or if you are worried about slug not being uniqe slug-id combination).

Windows: CRLF newlines in cookie file causes error "net/http: invalid byte '\r' in Cookie.Value"

E:\Apps\Net\gogcli>gogcli update generate
2022/06/26 11:29:50 net/http: invalid byte '\r' in Cookie.Value; dropping invalid bytes
2022/06/26 11:29:51 net/http: invalid byte '\r' in Cookie.Value; dropping invalid bytes
2022/06/26 11:29:51 net/http: invalid byte '\r' in Cookie.Value; dropping invalid bytes
2022/06/26 11:29:51 net/http: invalid byte '\r' in Cookie.Value; dropping invalid bytes
2022/06/26 11:29:51 net/http: invalid byte '\r' in Cookie.Value; dropping invalid bytes
2022/06/26 11:29:53 net/http: invalid byte '\r' in Cookie.Value; dropping invalid bytes
^C

Probably just need to add instructions to the README to save the cookie file using LF line endings. (Could be tricky for non-technical users)

Include File Locking

Sometimes, some game files are removed from the GOG.com library by the publishers for good reasons and sometimes, they are dropped to the detriment of the end-user.

A legitimate removal might be removing the previous version of an installer when a game is updated to a new version.

A detrimental removal might be the publisher removing the Linux installer for a game (even though it works fine) or a game extra (ex: the soundtrack).

Given the above case, the tool should allow end-users to lock certain files in the manifest and make them non-removable during future manifest updates, without having to manually edit the manifest and actions file each time.

Expired auth cookie gives unfriendly error message "json parsing error"

I ran gogcli update generate, and received this error message:

GetOwnedGames(page=1, search=) -> json parsing error: invalid character '<' looking for beginning of value

I guessed that it was failing to parse HTML, which in my experience is often returned by 400 or 401 HTTP responses. Sure enough, updating my cookie values fixed the error.

Would it be possible to give a more meaningful error message?

Add Basic Download Speed Measurement on the gog-api download-url-path Command

After the gog-api download-url-path has ran, it should display the file size, how long the download took and the download speed per second.

Repair and Validate Should Have an Option to Skip Checksum Validation

Currently, for "storage repair" and "storage validate" commands, the checksum of files are compared against the value in the manifest.

When your collection is large, this can take some time.

There should be a flag to skip the checksum validation, useful in cases where you just want to perform a quick sanity check that the files in storage match the manifest.

Allow inversion of filter

I use a manifest filtered for English and German windows setups including extras to download my entire library. Additional Languages and OS would only take up space I will never use. However, there are some games that haven't been translated to English or German (or may don't have a Windows version, but I don't know of any). By using my filter, I would miss these completely. In my case, "Kajko i Kokosz" is the only such game. Other examples could be "Anstoss 2 Gold Edition" and "ANSTOSS 3: Der Fußballmanager" which are only available in German, but those of course don't affect me personally.

My current workaround to deal with this problem: I generate a complete unfiltered manifest, then filter it via gogcli search to my preferences and download the big chunk of my games. To catch missed products, I wrote a script that searches the manifest for products that weren't matched by my filter settings at all. I then tag the results on GOG as "FOREIGN", search-filter the full manifest with gogcli again to find only products with that tag, and finally download it in every language and for all OS.

I think it would be neat to offer such a functionality as a native feature in gogcli. Maybe there's a better way to handle this edge case, it was just the best I could come up with in a short time and from what I can tell, it works fine.

Broken XMLs may prevent proper manifest creation

Unfortunately I'm experiencing a similar behavior to #16 again. I've run gogcli.exe manifest generate twice and each time it failed handling some broken data related to XMLs. I set the log-level to debug the second time around but didn't get any useful info beyond calling GetDownloadFileInfo() for the problematic file. The version used was v0.19.0.

As creation takes a very long time and uses a lot of network resources ever since the XML problems on GOG began, I'd rather not do a couple more test runs with the current version. Here are the two created manifests I ended up with in their entirety:

{
  "Errors": [
    "GetDownloadFileInfo(downloadPath=/downloads/order_of_battle_us_marines/en1patch1) -> retrieval request error: Get \"https://gog-cdn-lumen.secure2.footprint.net/token=nva=1650461891~dirs=6~token=0bade971fde4080f53a15/secure/offline/1744082009/1192658776/52863558974871196/4509/setup_order_of_battle_u.s._marines_8.3.0_%2832bit%29_%2834403%29.exe.xml\": unexpected EOF"
  ]
}

{
  "Errors": [
    "GetDownloadFileInfo(downloadPath=/downloads/caves_of_qud/en1patch2) -> Could not parse file xml metadata and the first line was not the expected format: EOF"
  ]
}

Thanks for implementing the workaround method by the way, I'm fully aware of the consequences of using it. Getting an almost complete manifest that only misses those couple of problematic entries would be perfect. I would just retry to retrieve their data via gogcli.exe manifest update later then.

Download Games Metadata and Media Files in the Storage

With the ultimate goal of providing an experience similar to the GOG library page and game pages to one's collection offline, the first part of this process would be to download the metadata and accompanying media files (ie, pictures) in the storage.

Because GOG provides youtube videos in its game pages, those can't be downloaded and would have to be linked.

Technically, the videos on Steam can be downloaded, though figuring out a mapping to Steam videos for all games would be labor intensive and there is an ethical consideration to ponder about hitting the Steam servers with a lot of video download requests for content that has been purchased elsewhere.

Add some sort of progress bar when downloading

Currently, there is no way by default to see how far your download has gone, this makes it difficult to know when a download is about to complete. I think that adding a Progress bar would be a good idea, alongside a download time calculation.

In addition, this will make it easier to determine if a download has stalled.

HTTP 404 terminates manifest update

Unfortunately, GOG continues to throw spanners in your gears.

I ran into two products that prevented me from updating my manifest via gogcli.exe manifest update --update=updates.json. The outputs stopped like this:

[manifest writer] Got all info on game with id 1174117468. 300 games left to process
[manifest writer] Got all info on game with id 1169744036. 299 games left to process
 getDownloadFileInfoWorkaroundWay(downloadPath=/downloads/spiritfarer_ost/en1patch3) -> body download handle retrieval error: did not expect status code of 404

 getDownloadFileInfoWorkaroundWay(downloadPath=/downloads/prison_architect_going_green/en1patch0) -> body download handle retrieval error: did not expect status code of 404

After this, the program terminated immediately (pretty sure it must have been via a non-zero exit code but I didn't check). It created a manifest-update-progress.json, but I couldn't find a way to make gogcli actually use that to resume the update, it restarted from scratch on every retry. I logged the output at the various available levels, but couldn't get anything more useful out of it.

As the problem was perfectly reproducible and happened on the same url every time I tried, I just removed the respective product id from my updates.json file and reran the command, rinse and repeat until I got no terminating error anymore. This might just be a temporary problem and I will retry after my downloads are done, but I wanted to report it anyway so you can decide if it's worth handling.

Rework Storage Commands

Atm, we have the following 3 storage commands:

1. storage apply: Uploads a manifest in the storage, generates a new action file in the storage and execute the actions in the actions file
2. storage resume: Run the remaining actions for a pre-existing action file in the storage
3. storage update-actions: Uploads a manifest in the storage and update a pre-existing action file in it

I think the apply tries to do too much by also executing the actions file, preventing a more customized workflow (if someone wanted to edit the actions file first for example).

I also think that apply and update-actions as separate commands is needless overhead for the end-user. While the case where there are pending actions in the storage that need to be accounted for when updating the manifest is separate, it can be auto-detected and handled by the same command.

I should have the following commands instead:

1. storage apply manifest: Updates the manifest, generating a new actions file or updating an existing one in the process
2. storage execute actions: Run the actions listed in the actions file on the storage

Add support for resuming partial downloads

I noticed that gogcli currently starts the download from scratch if I interrupt it, it would be nice if there was an option to resume partially downloaded files.

I tested this using the gog-api download-url-path command.

Perhaps an additional command to verify the checksum of an already downloaded file would be nice to have too! 😁

SkipUrls broken in v0.22.0

I tried to generate a manifest and got a few 404 URLs again, so I excluded them via regex pattern in the SkipUrls array of the manifest-generation-progress.json file - as explained here.

But something breaking must have changed between v0.21.0 and v0.22.0, because the new version seemingly ignores this option, while it works fine with the old one, just as before. I tried manifest generate-resume on both versions with the exact same progress file containing the following filter option:

      "SkipUrls": [
        "^/downloads/episode_2_blackwell_unbound/en2installer0$",
        "^/downloads/episode_3_blackwell_convergence/en2installer0$",
        "^/downloads/heart_of_the_woods/en1patch1$",
        "^/downloads/the_blackwell_epiphany/en2installer0$"
      ],

v0.21.0 completed without errors, but v0.22.0 terminated pretty quickly after reaching the point where it actually processes games. It printed the following error before exiting:

 getDownloadFileInfoWorkaroundWay(downloadPath=/downloads/episode_2_blackwell_unbound/en2installer0) -> body download handle retrieval error: did not expect status code of 404

That error should not have happened if SkipUrls were honored. I did several tries and it stopped on episode 3 a few times as well, even though the exact URL was also configured to be skipped. The progress file was initially created by v0.22.0, so a format change can't be the reason (I updated gogcli before doing my quarterly backup).

These are all the broken URLs from my library right now, so things got better in that regard. Just one of the old patch links left and three Mac installers that will probably get resolved.

On a side note, manifest generation is blazingly fast in comparison to before, because there wasn't a single file that needed the workaround method to get its checksums. Seems like their CDN problems and having to switch to their own temporary solution was good for one or two things at least (not sure if it had anything to do with it, but sounds plausible to me). Just ~1.5 hours for a complete manifest on v0.21.0 with no errors beyond these four broken URLs, almost ideal conditions for my library.

Add Actions Summary Command

Atm, you can view the summary of a downloaded manifest with the manifest summary command.

It would be nice if there was the same command for actions showing at least:

• The number of added games
• The number of deleted games
• The number of added files
• The number of deleted files

Add command to selectively download games and their files

I understand that gogcli acts more like a mirror program but it would be nice if there was a command to download only selected game(s) and possibly only selected files within those games.

For example I may only want to download the extras of a game which I have already downloaded previously, or only download the installer with required patches instead of all patches.

Implement Multiplart Download POC on gog-api download-url-path Command

Multipart downloads have the potential to speedup slow downloads on large files for some users and allow to reuse some of the downloading on the next attempt if it is interrupted.

This should be implemented standalone on the gog-api download-url-path command as a poc before further integration is contemplated in the storage part of the tool.

Right now, I see 3 potential ways to approach this:

1. Just make arbitrary range queries
2. Use the ranges and checksums provided by GOG on the older offline installers endpoints used on the website
3. Investigate if the newer Galaxy endpoints could be used instead

Potential drawbacks of each approach:

1. Very flexible, but because there aren't any pre-computed checksums on arbitrary range queries, bad downloads where the checksum don't match at the end would have to be retried from scratch as there would be no way to know which range request downloaded a bad segment
2. There is at least one game where the xml document indicating the ranges is malformed. Even if we assume its an anomaly for only one game, I believe this document was mainly used by the now deprecated GOG Downloader, so it might be dropped in the future
3. I'm not sure if it is possible to access all the Galaxy endpoints with the cookie data that this client uses. Even if it is, I'm not sure if it is possible to download the offline installers from the Galaxy endpoints. I'd have to investigate that.

Add support for Netscape (curl) cookie format

Hi,

Thanks for making the tool, however it isn't working for me:

TheDcoder@arch /t/gogcli [1]> gogcli gog-api user-info
Email:                   
Username:                
Avatar:                  
UserId:                  
GalaxyUserId:            
PreferredLanguage:       English
SelectedCurrency:        USD
WalletBalance:           0.000000 
WishlistedItems:         0
PurchasedItems:
  Games:                 0
  Movies:                0
Updates:
  Messages:              0
  PendingFriendRequests: 0
  UnreadChatMessages:    0
  Products:              0
Friends:
TheDcoder@arch /t/gogcli> gogcli gog-api owned-games
GetOwnedGames(page=1, search=) -> json parsing error: invalid character '<' looking for beginning of value

I am running your latest linux-amd64 build and I extracted cookies properly from my browser (I am logged into GOG.com). Any idea what might be going wrong?

Keep Previous Manifest When Applying

Currently, when a manifest is applied, the previous manifest is overwritten and lost.

It would be a good idea to make a copy of the previous manifest and keep it in storage, until the next manifest after that is applied.

Provide Dashboard to Visualise and Download Games from Storage

Provide a web dashboard to list games, view game details and download them from the storage.

This would follow issue #7