exhumer/pytingen

Google Drive Index Generator for Tinfoil

TinGen

This project has been renamed to pyTinGen. Users who still want to follow the Python 3 version of TinGen can follow this guide to set remote to pyTinGen instead of TinGen.

This fork will no longer accept any new feature. Only bugfixes will be merged into this fork.

This project is being feature deprecated as I plan to replace this project with TinGenQt

Script that will allow you to easily generate an index file (with encryption if needed) for use with Tinfoil.

This project is a based on this project by BigBrainAFK.

NOTE: The token generated by this script can also be used with Tinfoil.

All tinfoil indexes are generated with the following format: Tinfoil-New-Index-Info.md

Requirements

• Python 3
• Modules from requirements.txt found in the root of project directory.
• credentials.json (or any file name if using --credentials flag to pass custom location for the required credential file). It can be obtained from here by clicking the Enable Drive API button in there while being signed in with the user account you want to generate credentials for or from Google's Developer Console.
• Google Drive Folder IDs to scan and index.

Execute the following command in a terminal to install all the required modules.

pip install -r requirements.txt

Usage

Usage instructions can be found here.

usage: TinGen.py [-h] [--credentials CREDENTIALS_FILE_NAME] [--token TOKEN_FILE_PATH] [--headless]
                 [--index-file INDEX_FILE_PATH] [--share-files] [--no-recursion] [--add-nsw-files-without-title-id]
                 [--add-non-nsw-files] [--add-nsw-info-to-success] [--add-update-date-to-success]
                 [--add-update-time-to-success] [--success SUCCESS_MESSAGE] [--encrypt]
                 [--public-key PUBLIC_KEY_FILE_PATH] [--vm-file VM_FILE] [--upload-to-folder-id UPLOAD_FOLDER_ID]
                 [--upload-to-my-drive] [--new-upload-id] [--share-uploaded-index]
                 [--tinfoil-min-ver TINFOIL_MINIMUM_VERSION] [--auth | --generator]
                 [--zstandard | --zlib | --no-compress] [--theme-blacklist [THEME_BLACKLIST ...]]
                 [--theme-whitelist [THEME_WHITELIST ...]] [--theme-error ERROR_MESSAGE]
                 [FOLDER_ID_TO_SCAN ...]

Script that will allow you to generate an index file with Google Drive file links for use with Tinfoil

positional arguments:
  FOLDER_ID_TO_SCAN     Folder IDs of Google Drive folders to scan

optional arguments:
  -h, --help            show this help message and exit
  --credentials CREDENTIALS_FILE_NAME
                        Path to Google Application Credentials
  --token TOKEN_FILE_PATH
                        Path to Google OAuth2.0 User Token
  --headless            Allows to perform Google OAuth2.0 User Token Authentication in headless environment
  --index-file INDEX_FILE_PATH
                        Path to output index file
  --share-files         Share all files inside the index file
  --no-recursion        Scans for files only in top directory for each Folder ID entered
  --add-nsw-files-without-title-id
                        Adds files without valid Title ID
  --add-non-nsw-files   Adds files without valid NSW ROM extension(NSP/NSZ/XCI/XCZ) to index
  --add-nsw-info-to-success
                        Adds simple information of NSW titles added to index
  --add-update-date-to-success
                        Adds index update date to index
  --add-update-time-to-success
                        Adds index update time to index
  --success SUCCESS_MESSAGE
                        Adds a success message to index file to show if index is successfully read by Tinfoil
  --encrypt             Encrypts the resulting index file with AES-ECB-256
  --public-key PUBLIC_KEY_FILE_PATH
                        Path to RSA Public Key to encrypt AES-ECB-256 key with
  --vm-file VM_FILE     Path to VM File
  --upload-to-folder-id UPLOAD_FOLDER_ID
                        Upload resulting index to Folder ID supplied
  --upload-to-my-drive  Upload resulting index to My Drive
  --new-upload-id       Uploads the newly generated index file with a new File ID instead of replacing old one
  --share-uploaded-index
                        Shares the index file that is uploaded to Google Drive
  --tinfoil-min-ver TINFOIL_MINIMUM_VERSION
                        Minimum Tinfoil client version to use index with
  --auth                Run Google User Token authorize task if token doesn't exist
  --generator           Run index generation task
  --zstandard, --zstd   Compresses index with Zstandard compression method
  --zlib                Compresses index with zlib compression method
  --no-compress         Flag to not compress index

  --theme-blacklist [THEME_BLACKLIST ...]
                        Theme IDs to add to index to blacklist
  --theme-whitelist [THEME_WHITELIST ...]
                        Theme IDs to add to index to whitelist
  --theme-error ERROR_MESSAGE
                        Error message to show if theme check fails

Credits

• BigBrainAFK for inital crypto script for index encryption.
• blawar for tinfoil and for early access to details about new index format and supported compression methods and also for helping me with my dumb questions.

pytingen's Issues

Add index chainloading support

To add support, specify secondary index via "directories" key in main index.

This allows user to hide files in index from showing up in file browser.

i got error

i have problem to run tingen
next is error msg. what is problem?
Files scanned: 9.31kfile [00:41, 1.63s/file]
Traceback (most recent call last):
File "C:\rclone\pytingen-main\tingen.py", line 219, in
generator.index_generator(
File "C:\rclone\pytingen-main\TinGen_init_.py", line 205, in index_generator
self.scan_folder(
File "C:\rclone\pytingen-main\TinGen_init_.py", line 113, in scan_folder
old_val_count = self.title_ext_infos[file_ext]["count"]
KeyError: 'zip'
Files scanned: 9.57kfile [00:42, 226file/s]

Execute Permission

Thanks for the hashbang! Can you chmod +x TinGen.py? :) If you can do that, then I won't have to ever change anything after a git pull! Thanks man, you rock! TinGen is the best! And if you can't, no big deal :).

"Could not automatically determine credentials"

This only started happening today (2020-07-21). but when I go to generate the index file, it fails and a get an error: "Could not automatically determine credentials. I'll attach a pastebin with the full output as well.

To replicate the issue:

• Have credentials.json and gdrive.token in TinGen.py's folder
• Run something along the lines of TinGen.py MYGDRIVEID --share-files --success "index.tfl loaded successfully!"
• Fails

Pastebin of full error

File sharing on gdrive

Adding gdrive index.tfl to tinfoil

Sorry but can't seem to make it work...
I generated the index.tfl correctly and i tried adding the gdrive link directly to tinfoil with the output specified in the log
In tinfoil locations.conf I added gdrive:/xxxxXXXXXxxxxxx#index.tfl
(xxxxxxxx being the file id)
But it doesn't list anything

If I manually download the file and save it on the SD card and add the folder of the SD card where it is stored in the scan ==> Success i even get the message i've set up to be certain
If I add a gdrive folder in which i only put the index.tfl ==> fail, it doesn't list anything but if I click on it in file browser, i can see the content of the gdrive folder i.e index.tfl but still doesnt list...

So to sum it up

• gdrive file id directly added with #index.tfl to locations.conf ==> FAIL
  -gdrive folder added to locations.conf ==> FAIL
  -download index.tfl on sd card and add the sd folder to locations.conf ==> SUCCESS

Any help appreciated, maybe i do it wrong in the locations.conf...

{"url":"gdrive:/xxxxxxxxxxxxx#index.tfl","title":"gdrive perso","enabled"1}

Edit : before the question arised, yes i have access to the gdrive within tinfoil (copied credentials and token and using sxos)

Add threading to scan files faster

unretryable error..

Hello,

Been trying to set this up, i successfully created the gdrive.token on my laptop but no way to start the program... (same symptom under windows or under docker on synology NAS with generated gdrive.token file) :

date	stream	content
2020-06-08 15:02:12	stdout	Files scanned: 0.00file [00:00, ?file/s]
2020-06-08 15:02:12	stdout	Exception: Unretryable Error
2020-06-08 15:02:12	stdout	raise Exception("Unretryable Error")
2020-06-08 15:02:12	stdout	File "/tingen/TinGen/init.py", line 99, in _apicall
2020-06-08 15:02:12	stdout	resp = self._apicall(self.drive_service.files().list(
2020-06-08 15:02:12	stdout	File "/tingen/TinGen/init.py", line 106, in _ls
2020-06-08 15:02:12	stdout	return self._ls(
2020-06-08 15:02:12	stdout	File "/tingen/TinGen/init.py", line 124, in _lsf
2020-06-08 15:02:12	stdout	for _file in self._lsf(folder_id):
2020-06-08 15:02:12	stdout	File "/tingen/TinGen/init.py", line 159, in get_all_files_in_folder
2020-06-08 15:02:12	stdout	files = self.gdrive_service.get_all_files_in_folder(folder_id, recursion, files_progress_bar)
2020-06-08 15:02:12	stdout	File "/tingen/TinGen/init.py", line 234, in scan_folder
2020-06-08 15:02:12	stdout	self.scan_folder(folder_id, files_progress_bar, recursion, add_nsw_files_without_title_id, add_non_nsw_files)
2020-06-08 15:02:12	stdout	File "/tingen/TinGen/init.py", line 261, in index_generator
2020-06-08 15:02:12	stdout	generator.index_generator(args.folder_ids, args.recursion, args.add_nsw_files_without_title_id, args.add_non_nsw_files)
2020-06-08 15:02:12	stdout	File "TinGen.py", line 41, in
2020-06-08 15:02:12	stdout	Files scanned: 0.00file [00:00, ?file/s]Traceback (most recent call last):
2020-06-08 15:02:12	stdout	Generating index
2020-06-08 15:02:10	stdout	Already up to date.
2020-06-08 14:59:43	stdout	Files scanned: 0.00file [00:00, ?file/s]
2020-06-08 14:59:43	stdout	Exception: Unretryable Error
2020-06-08 14:59:43	stdout	raise Exception("Unretryable Error")
2020-06-08 14:59:43	stdout	File "/tingen/TinGen/init.py", line 99, in _apicall
2020-06-08 14:59:43	stdout	resp = self._apicall(self.drive_service.files().list(
2020-06-08 14:59:43	stdout	File "/tingen/TinGen/init.py", line 106, in _ls
2020-06-08 14:59:43	stdout	return self._ls(
2020-06-08 14:59:43	stdout	File "/tingen/TinGen/init.py", line 124, in _lsf
2020-06-08 14:59:43	stdout	for _file in self._lsf(folder_id):
2020-06-08 14:59:43	stdout	File "/tingen/TinGen/init.py", line 159, in get_all_files_in_folder
2020-06-08 14:59:43	stdout	files = self.gdrive_service.get_all_files_in_folder(folder_id, recursion, files_progress_bar)
2020-06-08 14:59:43	stdout	File "/tingen/TinGen/init.py", line 234, in scan_folder
2020-06-08 14:59:43	stdout	self.scan_folder(folder_id, files_progress_bar, recursion, add_nsw_files_without_title_id, add_non_nsw_files)
2020-06-08 14:59:43	stdout	File "/tingen/TinGen/init.py", line 261, in index_generator
2020-06-08 14:59:43	stdout	generator.index_generator(args.folder_ids, args.recursion, args.add_nsw_files_without_title_id, args.add_non_nsw_files)
2020-06-08 14:59:43	stdout	File "TinGen.py", line 41, in
2020-06-08 14:59:43	stdout	Files scanned: 0.00file [00:00, ?file/s]Traceback (most recent call last):
2020-06-08 14:59:43	stdout	Generating index
2020-06-08 14:59:41	stdout	Already up to date.

The already up to date is due to git pull inside my sart shell script for docker, but on windows i don't have it but still the same issue.

This is my command line to start
python TinGen.py --headless --add-non-nsw-files --upload-to-folder-id ${FOLDER_ID_TO_UPLOAD_TO} FOLDER_ID_TO_SCAN ${FOLDER_ID_TO_SCAN}

is there a specific way to specify the folder_id_to_scan in the command line ? between [] ? or what ? and if several foldrs how to separate them ?
(for information i tryed with [] same error tho)

Thanks for your help

Index File Contents Not Generating

I was working on setting up an index file using the script. I was able to get the base python environment setup and executed the command. It seemed to have scanned the files as shown below:

[system@vps TinGen]# python3 TinGen.py --credentials creds/credentials.json --token creds/gdrive.token --success "Test" --index-file index.html --share-files --no-compress <gdrive_id>
Generating index
Files scanned: 5.43kfile [00:29, 1.62s/file]Adding success message to index
Creating generated index to index.html
Sharing files in index
File Share Progress: 0.00file [00:00, ?file/s]
Index Generation Completefile [00:00, ?file/s]
Files scanned: 10.8kfile [00:35, 304file/s]

However upon viewing contents of the index file this is all that is displayed:

[system@vps TinGen]# cat index.html
TINFOIL {"files": [], "success": "Test"}

Attempted on both Windows 10 and CentOS. Both operating systems showed the same results when trying to generate an index file.

Theme Blacklist/Whitelist

hi can we make this https://blawar.github.io/tinfoil/custom_index/#theme-blacklist implementation to pyTingen?

encrypt broken with latest update

shinji@casper:~/TinGen$ python3 TinGen.py --success "Loaded" --encrypt --index-file /var/www/public_html/shop/index.html [FOLDER_ID]
Generating index
Files scanned: 22.7kfile [02:00, 19.9file/s]Adding success message to index
Creating generated index to /var/www/public_html/shop/index.html
Traceback (most recent call last):
  File "TinGen.py", line 66, in <module>
    create_tinfoil_index(generator.index, Path(args.index_file), compression_flag, rsa_pub_key_path=public_key, vm_path=vm_file)
  File "/home/shinji/TinGen/TinGen/utils.py", line 61, in create_tinfoil_index
    if rsa_pub_key_path is not None and rsa_pub_key_path.is_file():
AttributeError: 'str' object has no attribute 'is_file'
Files scanned: 22.7kfile [02:01, 186file/s]

If I leave --encrypt off it works but that makes sense since the traceback is suggesting the error happens when it checks rsa_pub_key_path which defaults to public.pem. Last known good commits were from April 30th.

Verified it. Commit ce6b739 breaks it.

Add follow shortcut support and option to disable it if needed

Add check for "mimeType": "application/vnd.google-apps.shortcut". Then check if user wants to follow shortcut or not. If not follow shortcut, skip it. If follow shortcut, check the target details for shortcut. and do stuff again based on that and so on.

Add shortcutDetails(targetId,targetMimeType) to get request to get the shortcutDetails. It looks like this.

 "shortcutDetails": {
  "targetId": "1SqcsGNkkd****************",
  "targetMimeType": "application/vnd.google-apps.folder"
 }

Add SAs support so share files can be multithreaded

It would be awesome to add support for SAs so share files can be perform quickly. After files is scanned then assign each of SAs to each folder to perform the share files. I think this will make the share files perform quickly.

--update-mode flag

Implement Basic GUI

Implement upload to same file ID for index updates