0x09/resdet

Detect source resolution of upscaled images

resdet - Detect source resolution of upscaled images.

Dependencies

resdet bundles KISS FFT as its FFT implementation (permissively licensed) but can use FFTW (GPL, potentially faster) if available.
Image loaders are available using any or all of libpng, libjpeg, mjpegtools (for yuv4mpeg), and MagickWand.

libresdet can be used completely standalone provided the client supplies the image data.

Building

The tools and supplied build scripts target a generic *nix environment with bash, pkg-config, GNU make, and a GCC compatible C compiler using an autotools-like build process:

./configure
make
make install

see configure --help for options

Examples

Resize an image with ImageMagick's convert command and detect it:

$ convert blue_marble_2012.png -resize 150% resized.png

$ resdet resized.png
given: 768x768
best guess: 512x512
all width        height
  512 (77.03%)     512 (76.38%)

Just test if an image has been upscaled:

$ resdet -v0 resized.png && echo upscaled
upscaled

See resdet -h for more options.

API

resdet can be used as a library by including resdet.h and linking with libresdet. To install the library, use make install-lib after building.
A pkg-config file is included; linker flags can be gotten via pkg-config --libs --static resdet
API documentation is available in the doc subdirectory.
Though the supplied build script is meant for Unix-likes, libresdet is written in standard C and should be usable on any platform with a C99 libc.

FAQ

This utility takes an image or frames of video which may have been upscaled and attempts to exactly identify the original resolution.

How?

Traditional resampling methods tend to manifest as an odd extension of a signal's frequency domain combined with a low-pass filter (where the shape of the filter differs among the various resampling methods). Conveniently the Discrete Cosine Transform causes the zero-crossings of this extension to fall at exactly the offset the source was upscaled from. resdet works by trying to identify these inversions.

Getting the best results

resdet works best on images that are as close to the source as possible. Filtering and compression artifacts can significantly harm the accuracy of this test. In general, clearer and more detailed images will fare better.

Colorspace

resdet can provide significantly more accurate results if detection is performed in the same colorspace the image was originally resized in. If an image might have been resized in a linear light colorspace before being converted to a non-linear colorspace such as sRGB, it's worth attempting detection in linear light as well.

Here is an example that uses ImageMagick's convert command to convert the colorspace to linear RGB and provide the increased precision result to resdet using the portable floatmap format:

convert image.png -colorspace RGB pfm:- | resdet -t image/x-portable-floatmap -

Video

For compressed video stills, the best results can be gotten by choosing a highly detailed keyframe with a low quantizer. Single-frame yuv4mpeg streams are preferred over png screenshots for videos with chroma subsampling as it preserves the separation of the chroma planes. Some ways to obtain a y4m frame:

FFmpeg/avconv: ffmpeg -i source -ss timestamp -vframes 1 -pix_fmt yuv420p image.y4m

mpv: mpv --start timestamp --frames 1 --vf format=yuv420p -o image.y4m source

Better results should be possible by analyzing multiple frames together. resdet supports this with the PFM, mjpegtools (y4m), and MagickWand image loaders. To obtain multiple frames in the examples above, simply replace the argument to -vframes for FFmpeg or --frames for mpv with the desired number of frames. Note that currently frames will be read in bulk, so choose only a small section of the video to avoid consuming too much memory. This is not an inherent limitation, and will likely change.

JPEG

Moderate to heavily compressed JPEG sources tend to produce false positives at multiples of 1/8 the input resolution, generally with more appearing and in higher ranks as the quality decreases. resdet currently doesn't filter/penalize such results, although this can be mitigated somewhat by applying a deblocking filter to the image before analysis.

Deblocking example with FFmpeg: ffmpeg -i source.jpg -vf pp=ha/va image.png

Caveats

resdet works well on images resampled with traditional methods, but will not work with newer neural network-based resizers.

If you think something might be upsampled but you aren't getting a good result with resdet, install spec and have a look at an absolute value spectrum – it's usually possible to identify by sight. Our example earlier looks like this:

Note the solid black lines indicating zero-crossings 512px in from each dimension.

Can the same method work for downsampling?

There's no direct equivalent. Since downsampling loses information by definition it's likely not possible. Certain spectral features can appear as a result of typical downsampling operations, but the author is unaware of a way to reliably identify or exploit them in a meaningful way.

Source?

Looking at too many spectrograms.
Specifically, this project was born out of a yet-unpublished image deduplication framework, while attempting to identify whether duplicates were scaled versions of one another.
While some resources doing similar things via unspecified methods exist, I don't know of any comparable algorithm to resdet or publication describing something like it (but would be glad to read if they exist).

resdet's Issues

Make an online version

I think this library can be compiled to WASM using Emscripten like how I took svg2ass, compiled it to WASM and created a GUI for it.

It's useful since a lot of people don't like having to building software. There is already an open issue on providing static binaries, but I think a WASM version would be better as it would work in any browser that supports WASM (which are most browsers nowadays except for those that disable it to reduce security risks). A GUI would be pretty much required (as using a terminal in a webpage is not ideal), which have the added bonus of making it usable on mobile devices.

Well, I guess I can include the ffmpeg.js build I use in MKV Extract to extract frames in case some user inputs a video. ffprobe can be used to get a list of keyframes, then it would choose a few of the most highly detailed keyframes based on their size, and return resdet results for each one. Although, since this build does not include any decoder, it would have to rely on browser's native decoding (take input video, remux into a MP4 file with a single frame, pass to a <video>, save the rendered frame to a <canvas>, encode to PNG, pass the result to resdet), which is not ideal.

At the moment I don't have a lot of time to work on this, but I'm posting it here in case someone wants to try those ideas, otherwise I will try it eventually.

git error thrown by configure script

The configure script ends with this error:

...
Generating lib/resdet.pc
fatal: Not a git repository (or any of the parent directories): .git

If i proceed with the make command the executables are built, but resdet crashes with any image i've tried:

*** Error in `./resdet': free(): invalid pointer: 0x099a2050 ***
======= Backtrace: =========
/lib/i386-linux-gnu/libc.so.6(+0x67377)[0xb757c377]
/lib/i386-linux-gnu/libc.so.6(+0x6d2f7)[0xb75822f7]
/lib/i386-linux-gnu/libc.so.6(+0x6dbb1)[0xb7582bb1]
./resdet[0x804c22e]
./resdet[0x8049657]
./resdet[0x804a443]
./resdet[0x8048fa0]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf7)[0xb752d637]
./resdet[0x804938c]
======= Memory map: ========
...

Strange results from a videogame png

A lots of videogames are using now dynamic resolution.
The output is 4K, but the image is upscaled.

I try with a game screenshot of Gears 5, I know that the resolution is variable between 1080p and 2160p.
I think to use resdet to draw a chart of the resolution change but this is the result

given: 3840x2160
best guess: 1278x27
all width        height
 1278 (55.71%)      27 (64.34%)
 1280 (55.17%)      37 (63.60%)
                    59 (61.96%)
                    12 (59.41%)
                    38 (58.70%)
                    69 (58.67%)
                    13 (58.09%)
                    28 (58.08%)
                    70 (57.91%)
                    20 (57.70%)
                    80 (57.68%)
                    51 (57.50%)
                    19 (57.33%)
                    91 (57.24%)
                    48 (57.09%)
                    60 (56.58%)
                    30 (56.50%)
                    29 (56.03%)
                    26 (55.90%)
                   124 (55.75%)
                    58 (55.66%)
                    81 (55.63%)
                   102 (55.57%)
                    62 (55.52%)
                    52 (55.30%)
                   116 (55.07%)

This is the original file

Can this be ported as an FFmpeg filter, similar to cropdetect?

Hello there,

Can this be ported as an FFmpeg filter (dependent on the library implementation libresdet) to offer functionality for resolution up-scaling detection?

FFmpeg has a filter named cropdetect that can automatically detect crop size and print out recommended parameters to the logging system. Perhaps such an example would be a good start.

Interested in writing a Python port of resdet

Hello,

I would be interested writing a python port of your method(s); I wonder if you could share more info on how it works. From the readme files I understand you look on the Discrete Cosine Transform of the image for the location of the zero-crossings. Once you find the location of the black lines in both axis, you know the original size of the picture. Is what i mention accurate?

Can you share any other hints on the technical details of the implementation or any other references?

Also could you share any insights on what resampling filters are identifiable by this technique (Bilinear, Bicubic, Nearest, etc)?
Thanks!

Got Invalid image

For more discussions I open a new issue script.
Here is the problem which i meet.

I already install libpng and libjpeg on my system(Ubuntu 16.04). However when I run resdet test.png it got the errors as bellow :

$ resdet test.png
Invalid image

Here is my config.mak

CC=/usr/bin/cc
AR=/usr/bin/ar
CFLAGS=-Ofast -march=native -mtune=native -std=gnu99 
PREFIX=/usr/local
BINPREFIX=/usr/local/bin
LIBPREFIX=/usr/local/lib
INCPREFIX=/usr/local/include
PCPREFIX=/usr/local/lib/pkgconfig
HAVE_FFTW=3
HAVE_LIBJPEG=1
HAVE_LIBPNG=1
DEFS= -DHAVE_LIBJPEG -DHAVE_LIBPNG=1 -I/usr/include/libpng12
LIBS= -ljpeg -lm -lfftw3f -lpng12

So the root cause may because of the libpng12 is not the resdet target version and the ImageMagick's libraries were not installed properly.

After install ImageMagick's libraries the problem was solved.

Getting absurd "Best guess" values.

I'm getting some absurd "best guess" values. Any ideas what could be wrong?

given: 1920x1080
best guess: 1449x12
all width        height
 1449 (57.05%)      12 (57.03%)
 1396 (56.10%)      16 (55.37%)
  397 (55.72%)      36 (55.37%)
  395 (55.43%)      33 (55.34%)

Dead link

Hi, sorry for creating an issue for this but I am interested in the Upscaled Blue Marble spectrum, I'd like to have a look at it but the link is dead, is there somewhere I can get it from?

Thanks

Invalid image

I only get invalid image no matter what image

Possible to get static binaries?

I installed libpng and libjpeg on my system. Compiled resdet but it still doesn't work with jpegs. Would it be possible for you to provide static binary for Linux? (Or maybe even for Windows?)

Don't mention Lenna.png in docs

https://en.wikipedia.org/wiki/Lenna#Criticism

Lenna comes from a Playboy centerfold. It doesn't seem appropriate to use as a general-purpose example.

y4m invalid image?

Hi. Trying to use with a y4m. Generated one as per example in readme, but when i point resdet at it i get Invalid Image.
image.y4m.zip

I've tried building a few different ways.
Once i built with

./configure --disable-libpng --disable-libjpeg

Testing for cc... ok
Testing for ar... ok
Testing for pkg-config... ok

CC is "/usr/bin/cc"
AR is "/usr/bin/ar"
CFLAGS is "-Ofast -march=native -mtune=native -std=gnu99 "
PREFIX is "/usr/local"
BINPREFIX is "/usr/local/bin"
LIBPREFIX is "/usr/local/lib"
INCPREFIX is "/usr/local/include"
SHAREPREFIX is "/usr/local/share/resdet"
PCPREFIX is "/usr/local/lib/pkgconfig"

Testing for fftw3f... not found
Testing for magic.h... ok
Testing for magic... ok
Testing for mjpegtools... ok
Testing for MagickWand... ok
Testing for m... ok

it would then recognize png files (otherwise, it was giving invalid image for pngs too)

Let me know what other info I can provide.

Using OSX 10.12

Detection of and reversal of upscaling algorithms?

I realize this feature could be horribly more complex but I'm looking for a way to go one or two steps further here and detect the type of upscaling done and reverse it to get the clean source.

Sometimes trying the default kernels available in this software still won't produce a good result: https://github.com/Irrational-Encoding-Wizardry/descale

So this is where detection of a possible best guess custom kernel could come in.

Client Buffer Support

I wanted to see if we could get a thumbs up on adding a feature for reading buffer input for the client and not the file location.