pedrocr/rawloader

rust library to extract the raw data and some metadata from digital camera images

rawloader's Issues

Some images with ambiguous orientation data are not rotated correctly

Some image formats contain orientation information in the Makernote tag. Here's one example where the normal Orientation tag doesn't match the one in the Makernote:

https://raw.pixls.us/getfile.php/346/nice/Sony%20-%20SLT-A99%20-%2014bit%20(3:2).ARW
https://raw.pixls.us/getfile.php/346/exif/RAW_SONY_SLT-A99V.ARW.exif.txt

Exiftool outputs the following data where 270 CW is correct:

0x0112 Orientation                     : Rotate 180
0x0028 Camera Orientation              : Rotate 270 CW

The CameraOrientation tag for this particular image is within Tag9400a (0x9400) in the Makernote. According to http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/Sony.html there are many different places where orientation can be stored.

Expose ljpeg decoder

There's a ljpeg decoder here, but it's not exposed and not used in any way yet
https://github.com/pedrocr/rawloader/blob/master/src/decoders/mod.rs#L73

Is it possible to make it public? This decoder is unique because at the moment it's the only library in pure rust able to decode lossless jpeg. As a result, it's usable in webassembly context.

In addition, there's a decode_ljpeg_2components, but no decode_ljpeg_1components. I made a simple version of decode_ljpeg_1components function, would you mind reviewing it if I submit a pr?

will this project keep going?

I've found this project have not updating for years, so will this project keep going?

Fix clippy warnings

Hi Pedro,

I would like to fix some clippy-warnings in the code, if this is ok for you.

Best regards,
Maximilian

random tile decoding problems

heya,

i'm using rawloader through a thin c-bindings layer (which you can find here).

when running this code on a sequence of dng files, i get random tile decoding issues which manifest themselves like so:

or so:

jumping around the images. looks like a threading issue to me (?). is this a bug? am i calling it wrong? do i have to wait before reading out the buffer after calling rawloader::decode_file()?

Thanks for this!

Hi, I mostly wanted to thank you for having done this raw decoder in Rust! (don't hesitate to close the issue)

In case you have a little more time I'd love to know how you managed to do the NEF decoder. In particular, I've been trying to understand how to decompress the strips without much success. I've detailed my issue in this post where someone redirected me here.

I've found the source code of dcraw very intimidating and not easy to understand for me (I'm not coding much in C also). Did you have other sources of information to help you? I haven't found much on the Web, but maybe just as with this library I didn't search for the right things.

Regarding the licensing of this code, are you set on LGPL?

Best regards

Advice/Documentation for final brightness

Maybe I'm missing something but I do not understand yet how I can determine the final/real brightness.

For example, if a camera only a bit-depth of 12 then 0x0fff is max brightness. If a camera uses 14 bit then 0x3ff is the max brightness of a pixel. How can I find out the final/real brightness with what's given right now by the library?

Panic on D3400 decode

Hello! I tried the sample usage on in the readme with this file from raw.pixls.us and got a panic on decode. Here's the backtrace from rawloader which looks suspiciously like an underflow:

thread 'main' panicked at 'index out of bounds: the len is 4096 but the index is 65535', /home/genbyte/src/rawloader/src/decoders/basics.rs:127:30
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:78
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:59
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1076
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1537
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:62
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:198
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:218
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:486
  11: rust_begin_unwind
             at src/libstd/panicking.rs:388
  12: core::panicking::panic_fmt
             at src/libcore/panicking.rs:101
  13: core::panicking::panic_bounds_check
             at src/libcore/panicking.rs:73
  14: <usize as core::slice::SliceIndex<[T]>>::index
             at /home/genbyte/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/slice/mod.rs:2872
  15: core::slice::<impl core::ops::index::Index<I> for [T]>::index
             at /home/genbyte/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/slice/mod.rs:2732
  16: <alloc::vec::Vec<T> as core::ops::index::Index<I>>::index
             at /home/genbyte/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/liballoc/vec.rs:1942
  17: rawloader::decoders::basics::LookupTable::dither
             at /home/genbyte/src/rawloader/src/decoders/basics.rs:127
  18: rawloader::decoders::nef::NefDecoder::do_decode
             at /home/genbyte/src/rawloader/src/decoders/nef.rs:311
  19: rawloader::decoders::nef::NefDecoder::decode_compressed
             at /home/genbyte/src/rawloader/src/decoders/nef.rs:237
  20: <rawloader::decoders::nef::NefDecoder as rawloader::decoders::Decoder>::image
             at /home/genbyte/src/rawloader/src/decoders/nef.rs:140
  21: rawloader::decoders::RawLoader::decode_unsafe
             at /home/genbyte/src/rawloader/src/decoders/mod.rs:522
  22: rawloader::decoders::RawLoader::decode::{{closure}}
             at /home/genbyte/src/rawloader/src/decoders/mod.rs:530
  23: std::panicking::try::do_call
             at /home/genbyte/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:297
  24: __rust_try
  25: std::panicking::try
             at /home/genbyte/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:274
  26: std::panic::catch_unwind
             at /home/genbyte/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panic.rs:394
  27: rawloader::decoders::RawLoader::decode
             at /home/genbyte/src/rawloader/src/decoders/mod.rs:529
  28: rawloader::decoders::RawLoader::decode_file
             at /home/genbyte/src/rawloader/src/decoders/mod.rs:544
  29: rawloader::decode_file
             at /home/genbyte/src/rawloader/src/lib.rs:104
  30: easyraw::main
             at src/main.rs:14
  31: std::rt::lang_start::{{closure}}
             at /home/genbyte/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67
  32: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:52
  33: std::panicking::try::do_call
             at src/libstd/panicking.rs:297
  34: std::panicking::try
             at src/libstd/panicking.rs:274
  35: std::panic::catch_unwind
             at src/libstd/panic.rs:394
  36: std::rt::lang_start_internal
             at src/libstd/rt.rs:51
  37: std::rt::lang_start
             at /home/genbyte/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67
  38: main
  39: __libc_start_main
  40: _start
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Duplicate camera files for rebel line

The rebel line of canon cameras (T*i) have duplicate camera definition files - one for the official model and another for the localized model name.

I don't think the localized model name file is necessary as the camera is populated from the alias section of the official model file.

Eg: a T2i camera has its properties defined in T2i.toml and 550d.toml

Demosaicing Bits in example from README.md

in the example of the README.md file i had to change line

let pixlow = (pix&0x0f) as u8;

into

let pixlow = (pix&0xff) as u8;

else i got dramatic posterization. i don't know whether this is a typo or has to do with the bit depth of my camera (14-Bit, arw format, Sony IlCE-7SM2).

Shutter speed, ISO and aperture

Hi! Is there any hope that shutter speed, ISO and aperture will appear in RawImage struct? How difficult is it to implement?

CR3 support

First of all - awesome library, thanks for putting it out there!

I'm using rawloader to uniquely identify raw files, even when some of the attributes have been written to the file (e.g. an jpg preview has been updated). I do so by reading the raw image buffer and creating a sha256 hash out of it.

Works perfectly fine for my CR2 files, but it looks like CR3 is not supported yet.

You can find sample files on https://raw.pixls.us by searching for "EOS RP" or "EOS M50".

Let me know if there's anything I can do to assist you here!

Edit: A description of the file format can be found at https://github.com/lclevy/canon_cr3.

Access blackarea values

Thanks for the library!

To learn Rust I'm trying to re-implement the magiclantern cr2hdr tool. One of the methods attempts to calculate the standard deviation of the black noise in the image. It would be useful to be able to access the values of blackareah and blackareav in a loaded raw image for this purpose (perhaps similar to the crops property).

Any advice on how to do this, or why this is a bad idea are appreciated. I can also send you a pull request that will expose it too!

more examples

Hi,

So glad you made this library, thank you!
Can you please add more examples, besides ppm? Maybe also a link to a good learning resource.

Implement std::error::Error for errors

Currently the error type is Err(String) for most (maybe all) operations. This means you can't use this library with something like anyhow.

I've heard good things about thiserror. Maybe that's an alternative?

Apple proraw DNG support

Hi I would like to make a pull request for iphone pro raw DNG support (which is 3-component ljpeg, using predictor 7).

I have already implemented the decoding, very simple, but the design is not finalised. I see a few ways to proceed:

1. Add a separate function like "proraw_decode" to decompressors.rs, just for decoding 3-component predictor 7 data.
   • Bad: It will be 80% identical to decode_ljpeg_3components.
   • Good: Specialising the implementation to proraw will help performance. As proraw is 12 bit, it is possible to do the predictor calculation within u16. But a more general implementation would need to convert to u32 which I guess would be slower (because 16 bit files might exist somewhere??).
2. Same as first option, but a more general implementation (including u32 conversion).
3. Write a whole set of general predictor 7 functions equivalent to the existing decode_ljpeg_(1|2|3|4)components
4. Make predictor be a parameter of decode_ljpeg_(1|2|3|4)components, this will be slower.
5. Combine decode_ljpeg_(1|2|3|4)components in to one generic function - like fn decode_ljpeg<const N: usize>(...), where component number is a compile time parameter. and then have separate functions for each predictor:
   - fn decode_ljpeg_predictor_1<const N: usize>(...)
   - fn decode_ljpeg_predictor_7<const N: usize>(...)
   This will generate separate code paths for different component counts, so should not lose performance compared to the current duplicate implementations (will need testing of course) all while reducing repetition. This option would require upgrading to rust 2021 for const generics though.

Please let me know what would be best as a pull request!

Also, the data seems to be BGR not RGB, but I guess that can be considered later Nevermind, that was an issue with how I was saving test files. The data is indeed RGB.

Error: "Couldn't find tag Tag::Make"

RawLoaderError { msg: "Couldn't find tag Tag::Make" }

Happens during rawloader::decode_file(path).unwrap();, with dng file.

Tile or Scanline loading support

Forgive a noob question, I'm fairly new to rust. Looking at the source, I don't think it is, but figured I would ask. Is it possible to access raw image data without full loading the image into ram? (for example to load the image in tiles or by scanline or something, at least for .nef, .cr2, and .awr). I'm stacking raw files in different ways, and I can only load part of them in ram at once. (due to limited ram) Any help would be appreciated. If it's not possible, is it something that could be done with an easy modification? (I could probably fund some development if its something thats not a huge change) Thanks!

Official IRC channel for project discussion

I usualy idle freenode and thought it would be a good idea to have an IRC channel for implementation discussion so I registered #rawloader. Some of the issues with my pull request (#2) might have been avoided if I discussed them with you beforehand.

Official and localized models

When an image file contains an alias of the camera model, rather than the official camera model, there is no way to relate back to the official model.

For example, with a T3i, I would expect to get back the official model of Canon EOS 600D (from data/cameras/canon/600d.toml) when calling camera.model. However, instead I get back the localized model Canon EOS REBEL T3i.

This can introduce difficulties when you want to branch behaviour based on the camera model as each localized case must be handled, rather than a single official model case.

Proposal would be to modify RawImage.{model, clean_model} to return the official model name, and add two new properties RawImage.{localized_model, clean_localized_model} that return the localized model name.