gordinmitya/yuv2buf

Android CameraApi2 / CameraX Image to ByteBuffer converter.

Android Camera Api2 / CameraX YUV to ByteBuffer

OUTDATED!

7 years passed since CameraApi 2 was introduced in Android 5. So Google decided that it's time to make convenient way to get RGB images from it.

ImageAnalysis from CaemraX now supports setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888) developer.android.com

Hovewer if you do not use CameraX for some reason - welcome!

Motivation:

When you're attempting to get an Image from ImageReader or ImageAnalysis.Analyzer you actually get 3 separate ByteBuffers which you can't pass to further processing. You have to merge them but that also is not easy because they are full of row and pixel strides.

Solution:

This library carefully merges 3 buffers into one with respect to all strides. As a result, you receive YUV type (NV21 or I420) and ByteBuffer to pass into OpenCV or a neural network framework.

The solution was tested with CameraX, CameraApi 2, and MediaCodec (for video files).

The whole library is a single file, you can just copy Yuv.java into your project.

Usage

private var reuseBuffer: ByteBuffer? = null

fun convert(image: ImageProxy): Pair<Bitmap, Long> {

    // val converted = Yuv.toBuffer(image)
    // OR pass existing DirectBuffer for reuse
    val converted = Yuv.toBuffer(image, reuseBuffer)
    reuseBuffer = converted.buffer

    val format = when (converted.type) {
        ImageFormat.YUV_420_888 -> Imgproc.COLOR_YUV2RGB_I420
        ImageFormat.NV21 -> Imgproc.COLOR_YUV2RGB_NV21
        else -> throw IllegalArgumentException()
    }

    // process converted.buffer ByteBuffer with one of converters
}

About android image formats

In android documentation ImageFormat.YUV_420_888 is stated as 'generic YCbCr format', so the actual format depends on u and v order and are these planes interleaved or not (pixelStride=1 or 2).

Most of the time you will get NV21 format (just as in Camera1 api).

And you even may start wondering is there a phone with I420 format (u and v pixelStride==1) – Yes! There is one. Blackview BV6000S Android 7.0. Screenshot. It looks like it's the only one. Other models of Blackview I was able to find in local sellers used NV21.

Converters

1. RenderScriptConverter.kt - built-in, no additional libraries required. Uses Bitmap for rotation.
2. OpenCVRoteterter.kt - preform rotation on yuv image and then converts color. The fastest in total time.
3. OpenCVConverter.kt - the fastest color conversion without rotation. Rotation of a rgb image is slightly harder.
4. MNNConverter.kt - if your goal is futher processing with neural network. Conversion and rotation performed in single operation.

PS: If your goal is to get Mat you may consider this method from OpenCV.

Benchmark

Snapdragon 855 (Xiaomi Mi 9T Pro). Image resolution 480x640.

	OpenCV (YUV rotate)	OpenCV (RGB rotate)	RenderScript	MNN
color	~1ms	~1.6ms	~2.2ms	~21ms
rotate	~3.5ms	~2.8ms	~16.3ms	NA (included in color)

Alternatives

1. (For OpenCV users) Copy private method from OpenCV camera implementation: JavaCamera2View, Mat rgba().
2. Capture image from camera directly to RenderScript Allocation.getSurface();
3. RenderScript implementation in android/camera-samples repo.
4. Switch back to CameraApi 1 (some trade-offs);
5. Manipulate pixels manually as it has done in TFLite demo ImageUtils. However, even with C++ implementation it's ridiculously slow. ~50ms for image about 1280x720 on Snapdragon 855;

TODO

• make the library;
• write unit tests;
• add RenderScript example;
• add OpenCV example;
• add MNN example;
• publish to GooglePlay;
• publish to mavenCentral();
• add TFLite example.

yuv2buf's Issues

Average RGB value

Hello,
Is it possible to calculate average RGB values of the image before it's converted to Bitmap?
I've tried it with whole resulting bitmap, but it so huge and FPS drops to 7 in average. Also, I've tried it with downscaled bitmap, but it also slow on some phones. Is there a better way to solve this?

Btw, I love your yuv to rgb conversion algorithm! Thank you!

Convert Image to NV12 format support

Hi, could you please add an option to convert the Image to NV12 buffer format instead of NV21 ? Because my device's camera output NV12 image. Thanks in advance.

Trying to understand padding issues

Hi. On some android devices I’ve observed that there is some padding added. The row stride is > the image width on all 3 planes (YUV420).

Through trial and error I found I only need to trim the padding from the luma channel but not the chroma channels. Coincidentally the pixel stride is 1 on the luma channel and 2 on the chroma channels.

I’m trying to understand why the padding sometimes exists and why it can only be trimmed when pixel stride is 1.

In my example I observed the following on some devices when streaming YUV420 from camera2:
Image width: 1280
Image height: 720
Y pixel stride: 1
Y row stride: 1536
U pixel stride: 2
U row stride: 1536
V pixel stride: 2
V row stride: 1536