Giter Site home page Giter Site logo

ada-af / dbooru Goto Github PK

View Code? Open in Web Editor NEW
6.0 1.0 0.0 631 KB

Client for different *booru

License: Do What The F*ck You Want To Public License

Python 70.98% HTML 26.57% CSS 2.45%
python cli web client api-client derpibooru-downloader wtfpl python3 e621 e621-api pictures downloader python-requests booru gelbooru derpibooru

dbooru's Introduction

DBooru

Branching

Master

  1. Stable branch
  2. Rare updates
  3. Lots of changes per update

Next or any other branch

  1. Unstable
  2. Lots of commits
  3. Lots of updates
  4. New features
  5. Broken old features
  6. Rare commit messages
  7. If there a commit message then "Minor changes"*
    *Minor changes may include removal of half of all code

Features

Both versions

  1. Search in downloaded images
    1. By tags
    2. By image dimensions
  2. Viewing images
  3. Loading images from *booru
    1. Even with proxy (socks5 only)

Web-Interface

  1. Downloading images
  2. Exporting images
  3. Tag predictions
  4. Change page with ← or → arrows
  5. Scroll through images with ← or → arrows

CLI-Version

¯\_(ツ)_/¯

To be deprecated

Installation

Dependencies

  • Python 3.5+ or PyPy3 5.9.0+
  • pip
  • requests
  • requests[security]
  • pysocks
  • idna
  • cryptography
  • ffmpeg or pillow
  • flask
  • jinja2
  • werkzeug
  • markupsafe
  • click
  • itsdangerous
  • termcolor
  • mysql-connector-python

Or you can just type pip install --user -r requirements.txt in terminal

Configuration

  1. Run python DBooru_web.py or pypy3 DBooru_web.py
  2. Change default settings

or

  1. Set modules (line 11) in settings_file.py
  2. Configure modules (placed in dermod/sitesupport)
  3. (Optionally) Change other settings (View Settings_file.py)

How to run

CLI

  1. Run python main.py or pypy3 main.py
  2. Type in "get images"
  3. Wait for images to download
  4. Use search
  5. ???
  6. PROFIT

Web

  1. Run python DBooru_web.py or pypy3 DBooru_web.py
  2. Press Update DB button
  3. Wait for images to download
  4. Use search
  5. ???
  6. PROFIT

Modules

Why do i need modules?

Modules allows you to configure DBooru for using with multiple *booru without rewriting half of code

I want more modules!

Open issue with site name and module description OR Write your own module

Are you kidding me?

It's not too hard (if site uses api ofc), you can use included modules as example.

Module must contain:

  1. Configurable options
  2. Site domain
  3. Search query
  4. Search endpoint
  5. Paging parameter
  6. Empty page delimiter
  7. Additional parameters (such as API-key parameter)
  8. Hard limit
  9. Parser Configuration

Examples

Configurable options
>> username = "NAME GOES HERE"
>> apikey = "KEY"

Must have # Do not change values after user-configurable options

Site Domain
>> domain = "https://example.com"

Search query [Can be configurable]
>> query = "likedby:{}".format(username) [Preferred variant]
>> query = "sky, ponies" [Well, if you gonna download pictures with this tags only, then it's ok]

Search endpoint (Varies depending on site)
>> endpoint = "/search?q={}".format(query)
>> endpoint = "/tags/{}".format(query)

Paging parameter (Varies depending on site)
>> paginator = "&page={}"
>> paginator = "&p={}"
>> paginator = "/{}"

In case of overly specific pagination can be implemented as class (check gelbooru module)
>> class paginator:
>>    def format(page):
>>        return "/pg/{}".format(page*25)

Empty page delimiter (Varies depending on site)
> RegExp matching page with no images
>> empty_page = "\[\]$"
>> empty_page = "{images:\[\]"

Sleep time (to get rid of rate limiting)
> Value must be int or float
>> slp = 1
>> slp = 0.5

Additional params
> Such as api keys
>> params = '&key={}&username={}'.format(apikey, username)

Hard limit (for when your life is not limited enough)
>Use if api does not allow requesting pages after N
>tip: this is not mandatory, but if you don't want to be banned specify this
>> hard_limit = 750 [notice, this time we're not using quotes]

Parser Configuration
May use whatever python functions/packages you want to use (check gelbooru module)
MUST start with and MUST contain parse(self, string) method
>> class Module:
>>    def __init__(self):
>>        self.tags = []
>>        self.ids = []
>>        self.links = []
>>        self.form = []
>>        self.height = []
>>        self.width = []  


Parse() method
Response from server passed with string argument
If page number required `pg_num` added to input arguments
  Note: pg_num receives raw page number (1, 2, 3, ...)
>>    def parse(self, string): 
>>    # here goes parsing
>>    # ...
>>    # parsed values must be added to lists
>>    for i in parsed_values:
>>      self.tags.append(i[0]) # tags MUST be delimited by `,,`
>>      self.ids.append(i[4])
>>      self.links.append(i[3])
>>      self.form.append(i[5])
>>      self.height.append(i[2])
>>      self.width.append(i[1])

Commands and Web-endpoints

CLI

Main menu

Enter this commands if prompt starts with DB>

Command Description
help Shows in-app help
get images Downloads images that you liked/favorited on *booru
get images --force Downloads images without checking if file exists
total Prints amount of entries in local DB
count <tag> Prints amount of entries tagged with <tag>
show <image_id> Opens image in image viewer or browser if no viewers found
quit (exit) Closes app
<anything> Uses input as list of tags and searches for it

Search

Enter this commands if prompt starts with Search@DB>

Command Description
show <image_id> Opens image in image viewer or browser if no viewers found
quit (exit) Closes app
back Returns to main menu
<any_number> Show page numbered as <any_number>
<anything> Uses input as list of tags and searches for it

Web

Endpoint Method Parameters (Body for POST) Description Returns
"/" GET Main page HTML-page
"/search" GET q=search_query page=int Search images HTML-page
"/image/str" GET View image with tags HTML-page
"/dl/filename" GET Browser-friendly download method Image
"/raw/filename" GET Raw image data Image
"/predict" GET phrase=search_query Tries to predict search query Plain text data
"/next/str" GET Tries to get id of next (newer) image Redirect (302) to /image/*
"/previous/str GET Tries to get id of previous(older) image Redirect (302) to /image/*
"/thumbnail/filename" GET Makes thumbnail (500px) of image (returns full image if thumbnailer disabled) Image
"/json/search" GET q=query page=int Searches images and returns json result of search JSON
"/random" GET Redirects to random image Redirect (302) to /image/*
"/random/tags GET Redirects to random image tagged with tags Redirect (302) to /image/*
"/update" GET Updates DB (Same as CLI: get images) Returns 200 code when update started successfully or 409 in case when there's already update in progress
"/remove/prefix_id" DELETE Used to remove image and its DB entry
"/reload/prefix_id" PATCH Triggers image redownloading
"/settings" GET Settings page HTML-page
"/settings/<option>" POST opt_type=variable_type&<option>_new_opt=value Endpoint for changing settings_file.py variables Redirect (302) to /settings#<option>_form
"/settings/<modulename>/<option>" POST <option>_new_opt=value Endpoint for changing module's variables Redirect (302) to /settings#<modulename>_form

Search basics and syntax

Basic search rules

  1. Wildcard through * (asterisk)

Example: "ti*" will return images where at least one tag starts with ti

  1. Tags must be separated by , (comma)

Example: "safe, princess luna"

  1. Tags are not case sensitive

Example: "SaFe" == "SafE" == "safe"

  1. Search for "tag" will only return images tagged with "tag" not "tag*"

Example: "safe" returns images with tag "safe" and doesn't returns "safezone"

  1. Searching multiple tags will return images matching all the tags

Example: "safe, princess luna" will return images tagged with both "safe" and "princess luna"

  1. Exclude tags by placing - (hyphen-minus) before tag

Example: "-safe" returns all images not tagged with "safe"

  1. OR queries will return images tagged with tag1 or tag2

Example: "(fluttershy|applejack)" returns images tagged with fluttershy, applejack or both tags

  1. OR queries could be mixed with AND queries

Example: "safe,(princess luna|changeling)" returns images tagged princess luna, safe or changeling, safe

  1. Exclusion is not supported for OR queries. It's just pointless.
  2. Rules 1,3,4,5 work almost the same for exclude

Special tags

Works only for filtering searches

Example: width=100 works, while -width=100 doesn't work

  1. height
  2. width
  3. ratio

Syntax

  1. = or == means equal to <value>
  2. != or <> means not equal to <value>
  3. > means bigger than <value>
  4. < means less than <value>
  5. >= means bigger or equal to <value>
  6. <= means less or equal to <value>

Example: 'safe, width>100" will return images tagged with 'safe' tag and image with width bigger than 100px

Settings_file.py

Option Format Description
modules List (['String', 'String']) Enables modules
suppress_errors Bool (True/False) Prints errors and stacktrace in case of happening
ssl_verify Bool (True/False) or String ("Path") Enable/Disable ssl verification or set custom CA Cert
enable_proxy Bool (True/False) Enables/Disables proxy for requests
proxy_type String Sets proxy type
proxy_ip String (IP) Sets proxy IP
proxy_port String (Port) Sets proxy port
BASE_DIR Function Magic for flask to work
web_ip String (IP) Set IP to bind Web interface
web_port Integer (port) Sets port to bind Web interface
thumbnailer String (One of "ffmpeg", "PIL", "disabled") Defines tool to make thumbnails or not to make them at all
conv_format String (ffmpeg output format) Format to use when making thumbnails
ffmpeg_args String (ffmpeg parameters) For situations when you think that default settings suck
gif_to_webp Bool (True/False) Creates webp thumbnails for gifs
disable_mobile Bool (True/False) Should tag prediction be disabled on mobile
predict_tags Integer (number) How many tags to show when predicting input
showing_imgs Integer (number) How many images to show per page
showing_tags Integer (number) How many tags to show per image (CLI-only)
images_path String (Path) Where to store loaded images
export_path String (Path) Where to store exported images
time_wait Integer (seconds) How long thread can stay alive
ids_file String (Path/Filename) Name for tempfile (No need to change)
db_name String Defines name for database file In case of using mysql defines schema name
use_mysql Bool (True/False) Defines if mysql connection should be used
mysql_user String Defines mysql user
mysql_password String Defines mysql password
mysql_host String Defines mysql host
mysql_port Integer (Port) Defines mysql port
thread_cap Integer (number) Defines maximum running threads before blocking creating new threads
sleep_time Integer (seconds) Defines time to wait before creating new thread after thread cap is reached
enable_polling Bool Setting for enabling/disabling polling for changes in settings_file.py
polling_time Integer (seconds) Defines time (in seconds) between checks for changes in settings_file.py
first_run Bool Defines if settings page should be opened on first run
touch_controls Bool Enables swipe controls on touchscreens

/settings autogeneration

DBooru can automatically generate /settings entries for variables in settings_file.py

Options block must start with two newlines and block title on the third

Every text line must start with #, except for variable declaration

Lines with no keywords will be interpreted as description

By default examples are NOT shown on page

Keywords

Keywords must be followed by : , except for # Require example

Keyword Description Notes
Example Defines example Can be used more than once
Format Defines variable type Should be one of string, bool, int. Defaults to string.
Options Defines available options for variable Must be formatted as Python list
Require example Requires examples to be shown on /settings page May or may not end with :

Examples

# Will show examples and dropdown with options
# Require example
# Example: var = "something"
# ExaMplE: var = "nothing"
# Format: var = string
# Options: ['module', 'thing', 'more things']
var = "done"


# Example block

# Won't show example
# Input field will be provided
# description line
# description line
# Format: test_vat = string
# Example: test_var = "test"
# description line
test_var = "test passed"

# var2 desc
# does thing
# Format: var2 = bool
# ...
var2 = True

dermod/aliases.py

Allows creating alias to tag, so you can find one tag using alias

Note: Aliasing alias won't work

Syntax

  aliases = {
      "alias1": "aliased tag1",
      "alias2": "aliased tag1",
      "alias3": "aliased tag2"
  }

dbooru's People

Contributors

ada-af avatar anon-a avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

dbooru's Issues

E621 site issues.

Hey there,

You project currently violates a bunch of the API rules for e621 and has gotten a few people banned from the site.

I'm contacting you in hopes that you will make corrections to your project so that it complies with the API guidelines. The guidelines and other API information can be found here https://e621.net/help/show/api

  • This project impersonates a web browser, which is against the e621 API usage terms. The user agent submitted must comply with the user-agent rules for API usage, which can be seen on the wiki page for the API.
  • This project does not sanity check HTTP response codes such as 404, 403, 503, 500 etc. This causes it to continue to make requests even when it will never properly work. Both 400 and 500 range errors should be an abort condition, because the site is not willing to serve the requests made.
  • The exit condition for your loops is not valid so it makes infinite requests.
  • E621 has a maximum page count of 750 and page access above this will always fail. There is a work around listed on the API documentation wiki.
  • When threaded the delay between requests is not honored properly and we've seen people making as many as 50 requests a second to the server. This needs to be fixed.
  • E621 will always return valid JSON for .json requests unless something has gone terribly wrong and you're hitting a hard error condition. Please use the python json module to parse this data and handle invalid input as an exit condition.

Thank you for your time and I hope that you will make these changes.

[Gelbooru] Getting favorites

Gelbooru requires logging in to access "My favorites" page.
And has no API endpoints to list them nor supports it through search.

Possible fixes:

  • Use browser cookies
  • Wait for gelbooru to implement API endpoint

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.