goawaynow/hydrausnao

A script to automatically search SauceNao for images and send results to Hydrus Network.

HydrausNao

A script to automatically search SauceNao for images and send results to Hydrus Network.

Disclaimer: I know basic coding, but have no prior experience with Python. Don't expect anything amazing.

Requirements

• Python 3.9+
• Requests pip install -U requests
• SauceNao-API pip install -U saucenao_api
• Hydrus-API pip install -U hydrus-api

Use

1. In your Hydrus Client, the Client API must be enabled, and a key with the permissions "Add URLs", "Search Files" and "Add Tags" must be created. "Add Tags" is not necessary if tagging is disabled in config.ini.
2. On SauceNAO, create or login to a user account and navigate to the search api page
3. Run the script once to generate a default config.ini
4. Edit config.ini, adding your relevant API keys to the Hydrus and SauceNao sections, and making other changes as necessary
5. Create a new text document in the same folder as hydrus_saucenao_search.py called 'hashes.txt' (or any name you may have changed 'hash_file' to in the General section or config.ini) and paste in sha256 hashes for any files you with to search
6. Run the script

The script will inform you of the status of its searches as well as your remaining SauceNao API use limits. Any URLs found will automatically be added to Hydrus for download, in a new page called "HydrausNao" (unless changed in config.ini)

About config.ini

By default, the API key options will be the only options in a generated config.ini. Any key found in config_default.ini can be copied into config.ini in order to override the default values.

If your config_default.ini becomes somehow damaged, you can delete it and run the script in order to generate a new, clean copy.

Index-specific options are have defaults defined under the [DEFAULT] section, and are overridden by any options defined in sections named after the respective index. Currently, the index-specific option is blacklist_file.

About blacklists

HydrausNao now supports username blacklists. These blacklists are simple text files with one username per line.

By default, a small DeviantArt blacklist with several users known for uploading other's work is included as an example, but not enabled.

To enable it, add this block to your config.ini:

[da]
blacklist_file = blacklist-deviantart.txt

If adding to this blacklist, it's recommended you copy the file and update your configuration to point to that instead. Any changes to the included text file may be removed upon updates to the script.

Credits

This script is based heavily upon Identify and Rename Images 1.1, the API example script provided by SauceNao.

hydrausnao's Issues

Allow for multiple results to be sent to Hydrus

In my other issue (#12), I mentioned a second enhancement that'd help with my efficiency, and I will specify it here.

It would be quite helpful for my workflow to be able to send more than one result to Hydrus, so long as they're still above the similarity threshold. Combined with the enhancement I suggested in #12, this enhancement would allow me to save time, data, and daily search count.

Hopefully this wouldn't be too difficult to implement alongside index namespaces?

Check for and handle 404s returned by SauceNao

Make Status Testing Optional

Don't sleep script if hash file is done processing

Use saucenao-api instead of requests and json

https://pypi.org/project/saucenao-api/

Test for Silent Failures on Boorus

Some boorus, such as Danbooru, return a 200 status for deleted files.

Add function for index-specific namespaces

First of all, I'd like to thank you for this script. I've been using it regularly for a while now, and it's been really helpful for me.

That being said, I'd find it very helpful if the script could allow you to define a unique namespace for specific indices. As is, I am needing to set a namespace while searching only one index, which has been very inefficient as far as data usage and my SauceNao daily limit.

How easy would something like that be to implement?

I do have a second suggestion that would also help in efficiency, especially in tandem with this, but I will post that separately.

Quick edit: I imagine that some (including myself, possibly) might also potentially find it useful to allow something similar for the hit/miss/no_result text as well.

Add a Username Blacklist

Deviantart is not a reliable source due to rampant theft on the platform, a problem often protected by the site's own policies.

A blacklist will help make DeviantArt usable for reliable sourcing.

Script breaks if result has an empty percentage

If a result for a SauceNao search has an empty percent-similarity value (meaning if a result doesn't have a number for the similarity), the script errors out with the message of ValueError: could not convert string to float: ''

Possible solutions:

• Search using the actual file instead of the thumbnail if this occurs, as those tend to work where the thumbnail fails in this way
• Tag the result with an identifier (such as [namespace]:error), but otherwise skip over it.

This issue has come up time and again in my use. Ideally this could be solved on SauceNao's end, but allowing this script to handle this case would make things much easier if it's not able to be solved there.

unexpected keyword argument 'service_name_to_tags'

Got the below error trying to run the script

PS R:\Hydrus Network> python .\hydrus_saucenao_search.py
Processing hash: 00a39d1338be503028b771dcf89c3eb311188d88824043ecc9e9bc1a090a882f
hit! 83.96
Traceback (most recent call last):
File "R:\Hydrus Network\hydrus_saucenao_search.py", line 398, in
handle_results(results)
File "R:\Hydrus Network\hydrus_saucenao_search.py", line 328, in handle_results
tag_file("hit")
File "R:\Hydrus Network\hydrus_saucenao_search.py", line 273, in tag_file
client.add_tags(hashes=line.splitlines(), service_names_to_tags={meta_tag_service: [final_tag]})
TypeError: Client.add_tags() got an unexpected keyword argument 'service_names_to_tags'
PS R:\Hydrus Network>

Add Support for Online Blacklists

Personally-maintained blacklists are nice, but not exactly grab-and-go friendly.

Online blacklists would allow for improved results without the requirement for extra work on behalf of the user, as well as potentially allow other users to submit to the blacklist.

Artstation Returns 403 on Testing

Initially reported via Discord.

Has to be first enabled in config, but even then you run into the issue that artstation file urls gives a 403 response when the url ir processed. Thus they just get considered as a miss. Personally not sure how to get a proper response. Hydrus converts the link to a json (https://hydrusnetwork.github.io/hydrus/help/downloader_parsers_full_example_api.html) to get around this but im sure theres a proper way to get python requests to get a 200 response. I tried updating the header and referer but didnt seem to help. Gallerydl seems to do something with ssl but implementing that is beyond me

Add Config Option for Tag Service Name

I ran into an issue on Line #97 where the script tries to fetch the my tags service. I don't know why but my tag service is named local tags instead, this leads to the script crashing since the tags service can't be found. Simply changing the name to match my tag service name resolved this issue.

I'm no real python developer, but I do think I could make a PR to add this option myself if you would accept the merge.

Dependency issue with pip

Hey! Somewhat new to pip but more experienced with NPM.

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
hydrus-api 5.0.1 requires requests<3.0.0,>=2.28.2, but you have requests 2.23.0 which is incompatible.

It seems hydrus-api needs something newer than saucenao allows:

saucenao-api 2.4.0 requires requests~=2.23.0, but you have requests 2.32.2 which is incompatible.

Any ideas?

Issue with script not working

See below

Make SauceNao API Key Optional

SauceNao doesn't require an API key for basic use.

Allow Hydrus API key to be set via command line