haste/image

Image processing for Elixir

Image

Image is an image processing library for Elixir. It is based upon the fabulous vix library that provides a libvips wrapper for Elixir.

Image is intended to provide well-documented common image processing functions in an idiomatic Elixir functional style as a layer above the very comprehensive set of functions in Vix and libvips.

As of version 0.18.0, Image provides optional integration with Bumblebee allowing machine learning models to be deployed as part of an image processing pipeline.

Image is also intended to be an idiomatic API layer over eVision (OpenCV). As of Image verison 0.9.0, the Image.QRcode.decode/1 function is provided to decode QRcodes with the functionality to do so provided by eVision. The eVision integration is optional and currently considered experimental.

In a very simple image resizing benchmark, Image is approximately 2 to 3 times faster than Mogrify and uses about 5 times less memory.

The documentation can be found at https://hexdocs.pm/image.

Installation

Image can be installed by adding image to your list of dependencies in mix.exs:

def deps do
  [
    {:image, "~> 0.18.0"}
  ]
end

Installing Libvips

Installing Vix requires libvips with development headers. Installation is platform dependent however the common platform installations are:

• macOS: Install using homebrew with brew install libvips
• Linux: Install with apt install libvips-dev

For more details see https://www.libvips.org/install.html

In addition the following will be required (and would normally be installed by the steps above):

• pkg-config
• A C compiler

Configuring Libvips

Vix and libvips offer various configuration parameters that affect debug output, image caching, concurrency of imaging operations and memory leak detection. Each of these options has reasonable defaults so no action is required in order to start using the library.

Vix NIF Error Logging

Vix NIF code writes logs to stderr on certain errors. This is disabled by default. To enable logging set the VIX_LOG_ERROR environment variable to true.

GLib Debug Output

The platform upon which Image and Vix stand is libvips, a C library that performs the image manipulation. It's libvips that delivers the speed, memory efficiency and functionality.

libvips uses the GLib library which has configurable debug output. This output depends on the setting of the environment variable G_DEBUG. The initial value will depend on the installation method of libvips for a given system. It can be changed by setting the G_DEBUG environment variable to one of the following:

• fatal-warnings which causes GLib to abort the operation at the first call to g_warning() or g_critical().

• fatal-criticals causes GLib to abort the operation at the first call to g_critical().

• gc-friendly causes newly allocated memory that isn't directly initialized, as well as memory being freed to be reset to 0. The point here is to allow memory checkers and similar programs that use Boehm GC alike algorithms to produce more accurate results.

• resident-modules causes all modules loaded by GModule will be made resident. This can be useful for tracking memory leaks in modules which are later unloaded; but it can also hide bugs where code is accessed after the module would have normally been unloaded.

• bind-now-modules causes all modules loaded by GModule to bind their symbols at load time, even when the code uses %G_MODULE_BIND_LAZY.

To produce debug output for only the most critical issues, set G_DEBUG as follows (in bash and compatible shells):

export G_DEBUG=fatal-criticals

Memory Leak Detection

The environment variable VIPS_LEAK determines whether libvips reports possible memory leaks. To enable leak detection (on bash compatible systems):

export VIPS_LEAK=true

To stop leak detection:

unset VIPS_LEAK

Concurrency

Image (because of Vix and libvips) will execute concurrent image operations using a number of system native threads (not BEAM processes). The number of threads available for concurrent image operations is configurable by either setting the environment variable VIPS_CONCURRENCY or through a call to Image.put_concurrency/1. The current number of configured threads is returned from Image.get_concurrency/0.

The default number of threads is equal to the number of cores detected on the running system. This may create CPU contention with other workloads given that image processing is CPU intensive. Therefore it may be prudent to reduce the number of threads if overall system throughput is being affected.

Installing Nx

Nx provides multi-dimensional arrays (tensors) and numerical definitions for Elixir. These tensors can also be used as an interchange format for binary image data. When Nx is installed and Image is compiled, the functions Image.to_nx/2 and Image.from_nx/1 are defined.

Nx is required in order to support the functions in Image.Classification. It is also required to support eVision and Bumblebee integrations.

1. Add Nx to your mix.exs

def deps do
  [
    ...
    {:nx, "~> 0.4"},
    ...
  ]
end

2. Add a default Nx backend to config.exs. Nx supports a variety of backends that provide CPU and GPU acceleration. The CPU backend that provides the widest platform compatibility is EXLA.Backend.

config :nx,
  default_backend: EXLA.Backend

3. Update dependencies

mix deps.get

Installing Bumblebee

Bumblebee provides pre-trained and transformer Neural Network models in Axon that can be used for a variety of image classification, segmentation and detection operations. When Bumblebee is configured, the functions in Image.Classification become available.

1. Add Bumblebee and exla to your mix.exs. exla is required in order to provide CPU or GPU acceleration for the models so as to deliver acceptable performance. Nx will be installed as a transitory dependency.

def deps do
  [
    ...
    {:bumblebee, "~> 0.1"},
    {:exla, "~> 0.4"},
    ...
  ]
end

2. Add a default Nx backend to config.exs. Nx supports a variety of backends that provide CPU and GPU acceleration. The CPU backend that provides the widest platform compatibility is EXLA.Backend.

config :nx,
  default_backend: EXLA.Backend

3. Update dependencies

mix deps.get

Installing eVision

eVision (OpenCV) provides Elixir bindings to OpenCV, the most well-known computer vision library.

As of Image version 0.9.0, experimental support for eVision (OpenCV) is provided. There is extensive documentation on how to install eVision and the required OpenCV. However to most cases the following should be enough:

1. Add eVision to your mix.exs. exla may be optionally configured too, see the Nx installation section above.

def deps do
  [
    ...
    {:evision, "~> 0.1"},
    ...
  ]
end

2. Update dependencies

mix deps.get

Then proceed as normal. eVision will download a precompiled OpenCV for the appropriate system architecture and compile both the NIF and Elixir code.

Security Considerations

There are several considerations in the use of any image processing library and any NIF-based library:

1. If a NIF crashes it will likely bring down the BEAM virtual machine. libvips is a robust, time-tested library however this risk cannot be eliminated.

2. Image processing is CPU intensive with its concurrent pipelining model and default concurrency level equal to the number of cores in the host machine, CPU starvation for other parts of the application is a possibility. In such cases, reducing the libvips concurrency is recommended.

3. Image processing by its nature operates on external data and there have been exploits based upon maliciously crafted images. The two primary vectors are:

   • An invalid image format that causes the image parser to crash and therefore crash the NIF and the BEAM
   • Executable code embedded in image metadata (such as EXIF data) that if passed un-escaped to a web browser may result in arbitrary code execution.

In comparison to Imagemagick that has a reported 638 CVEs, there have been only 8 CVE's reported for libvips, each resolved in a very timely manner.