Giter Site home page Giter Site logo

chillfish8 / lust Goto Github PK

View Code? Open in Web Editor NEW
342.0 4.0 13.0 1.65 MB

A fast, auto-optimizing image server designed for high throughput and caching; Now that is hot.

License: MIT License

Rust 99.45% Dockerfile 0.55%
image-processing image-server rust tokio

lust's Introduction

Lust Logo

๐Ÿ”ฅ Build your own image CDN system your way with lust.

Lust is an auto-optimising image server, designed for high throughput and low latency handling of images, now that is lustful. Re-encode uploaded images into png, jpeg, webp or even into gif based formats!

Resize them to your liking automatically with sizing presets, instantly create small, medium and large variants with just a few line in a config file. Now that's the spirit of lust

And much more like caching, on the fly resizing and diffrent processing modes to name a few.

Getting started

Creating a config file

It's highly advised to take a look at some example config files to get an idea of what a general config file would look like.

Full documentation in markdown form can also be found here, this is also served directly by the server as part of the documentation ui endpoint.

Installation

Building from Source

To building from source, just clone this repo via git clone https://github.com/chillfish8/lust.git and then run cargo build --release.

Installing via Cargo

You can install lust directly via cargo and the git flag:

cargo install lust --git https://github.com/ChillFish8/lust.git

Docker Images

Lust has a set of pre-built, optimised docker images ready to go. Just run it with

docker run -v "my_configs:/var/lust/" chillfish8/lust:latest --config-file "/var/lust/config.yaml"

Note: Assuming there is a folder called my_configs with a config.yaml file in it.

After Installation

Once you're up and running navigate to http://127.0.0.1:8000/ui or /ui of what ever port your server is running on to see the full OpenAPI docs.

Caching

Lust makes use of a Least Recently Used in-memory cache which can be adjusted for your needs via the cache_size key in the configuration file. The larger the number the more images it will cache at once and vice versa. NOTE: With bigger images this can create much higher RAM usage

Scaling

Lust's ability to scale is purely down to the backend you use, so it is worth noting that the file system backend is only really designed for testing. For full scale deployment consider using Scylla or a s3 compatible blob store to serve data from.

If your goal is high-end performance, Scylla DB will be the most performant by a large margin, but this will come with a higher operating cost.

Formats

Lust supports any of the following formats:

  • Png
  • JPEG
  • GIF
  • Webp

Any uploaded images will be given a unique uuid and be re-encoded into all the other enabled formats in all presets. This is especially useful when you want to serve several variants of the same image with different formats.

You can also adjust this based on the processing mode, aot/Ahead of time encoding will follow the old lust behavour by encoding and resizing each image at upload time.

jit/Just in time encoding will only resize and re-encode at request time, storing a base copy of the file to generate new images. This can save on a considerable amount of CPU time and disk space depending on your requirements.

Finally, we have the realtime encoder, this will only store an original copy like the jit encoder but instead will never save the resized and encoded image, this does also enable the ability to do on the fly resizing and is recommended for situations where you're not expecting to serve image to the public network.

Presets

The server can take several sizing presets which can be targeted via the size query parameter when getting an image. These presets will mean every image at upload time will be resized to fit the width and height bounds using the configured resizing filter (defaults to nearest neighbour).

Regardless of presets an original image is always stored and can be accessed via the size=original query. The default preset when served without a size parameter can be set in the configuration file via default_serving_preset key.

Data Efficiency

Lust's data storage efficiency is roughly the same as storing on a plain file system outside any system the database backend employs when storing the data.

lust's People

Contributors

chillfish8 avatar mishaszu avatar mrkomododragon avatar thepanz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

lust's Issues

[ ๐ŸŒ  Feat ] Let backends return (signed) links instead of images

Is your feature request related to a problem? Please describe.
To reduce load on the lust server, I'd like to serve pictures directly from block storage.

Describe the solution you'd like
When the block storage is public, a static link should be returned instead of the image itself. If the block storage is private a signed link should be returned instead.

Describe alternatives you've considered
Only other solutions that are more complicated e.g. imageflow.

CQL performance issues

Currently, Lust supports Cassandra via its scylla DB driver, while this system works logically it is currently prone to timeouts depending on the data and setup. This might be an issue with the testing database or more likely my less than perfect CQL knowledge leading to in-efficient queries.

[ ๐Ÿ› Bug ] After delete a file is not deleted from global cache

Describe the bug
After deleting an image it's possible to get a variant kept in cache. If files are removed manually (tested on filesystem storage type) and a user tries to get an image that was deleted and previously was kept in cache new file is created (as I assume from cache ).
If the global cache option is disabled or the bucket cache is used instead of the global the problem doesn't occur.

To Reproduce
Steps to reproduce the behavior:

  1. set Lust running instance
  2. creating an image
  3. getting the image
  4. deleting the image
  5. getting the image again (it's returned & created)
  6. deleting the image manually
  7. getting the image again (it's returned & created)

Expected behavior

  • Deleting image shouldn't be available with get endpoint
  • most importantly it shouldn't create a new instance of an image if it was deleted

Versioning (please complete the following information):

  • Lust Version = master
  • OS = ubuntu 20.04
  • Rustc Version = 1.70

[ ๐ŸŒ  Feat ] Dynamically load extensions

Currently, Lust is shipped as one big binary with some 500+ dependencies, this is rather large and leads to a lot of dependencies that most people using Lust for will never use if they only use one storage backend.

A solution is to dynamically load backends from DLLs, this would allow a user to build/download their given backend(s) and load them from the main server rather than having to have all the other backends shipped to them.

This would also allow a more pluggable experience for developing extensions etc...

[ ๐Ÿ› Bug ] If default_serving_preset is set to null, delete doesn't remove original size image

Describe the bug
If default_serving_preset is set to null original image in 0 directory is not deleted, and Lust returns 200 status.

To Reproduce
Steps to reproduce the behavior:

  1. create setup with default_serving_preset=null
  2. run Lust with the config
  3. add image
  4. delete image

Expected behavior
It should delete an image from all sizing presets

Versioning (please complete the following information):

  • Lust Version = master
  • OS = ubuntu 20.04
  • Rustc Version = 1.70

[ ๐ŸŒ  Feat ] Prometheus metrics support

At the moment the server has no real way of providing metrics to the user which may be an issue for some who want monitoring, or at least is a large inconvenience to do.

A Prometheus endpoint would allow things like Grafana dashboards to make life much easier to produce monitoring for.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.