Image::from_bytes factory

[iced-rs]

Image::from_bytes factory

[hecrj]

Thanks for the idea!

Yes, I think this is a basic use case we should satisfy. However, it's not as straightforward as it may seem.

The main challenge here is to avoid decompressing and uploading the image data to the GPU every time we draw. In other words, we need to keep track of the image somehow.

The current renderer uses the path of an Image as the identifier and lazily loads its data on the first draw (this should probably happen in another thread eventually). Then, the renderer keeps a cache that connects each image path with a texture on the GPU. Currently, this cache is never trimmed (if you use a lot of images, you will run out of video memory eventually).

I think we could leverage guillotiere to create texture atlases for images and invalidate parts of them efficiently. This is most likely not an easy task and not completely necessary to implement the feature of this issue, but I wanted to mention it in case anyone feels like going on a journey!

For now, a way to implement this would be to attach a unique id together with the image bytes and use that as a key in the renderer cache. A way to make this explicit would be with an opaque image::Data type or similar:

const EMBEDDED_IMAGE: image::Data = image::Data::from_bytes(
    "my_id",
    include_bytes!("image.png"),
);

const FILE_IMAGE: image::Data = image::Data::from_path("resources/ferris.png");

However, I am not a fan of using stringified global identifiers, as they open the door for many pesky bugs (think what would happen if an identifier is used twice with different bytes).

Overall, I think the problem is telling us that an Image widget is not the best approach for this. I have the feeling a Canvas widget/abstraction will be able to satisfy these needs better.

What are your thoughts?

[ejmahler]

My use case is for fractal art and other procedural art, where there never was an "original image" and it's all procedurally generated. Having a GUI library for procedural art would be amazing, because a lot of procedural art is fiddling with settings. Right now I do that via command line arguments, but it would be great if I could use checkboxes, type numbers into fields, etc and get realtime updates.

There are a few tiers of features I can think of that would make my life easier, ordered from least work to most work.

1. An API to use raw RGBA8 bytes as an image. This would let me click a button, compute the procedural data on a background thread or with rayon, then pass the result to the GUI once it's done
2. An API to apply an arbitrary fragment shader alongside the image, plus arbitrary uniforms. Procedural art usually involves 2 separate steps. The first is a long-running computation, and the second is code that convers the results of that computation into some sort of visualization. If I could use visualization settings to drive some shader uniforms, I could write all the visualization code in a shader, letting me get real-time changes in visualization. I'm imaging dragging a slider back and forth and being able to see updates as I drag. Rayon could theoretically handle something like this for small images, but with shaders I could do something like supersample an 8k image, etc
3. Render targets. Some fractals, like the mandelbrot set, or the abelian sandpile fractal can themselves be computed by fragment shaders that output to an R32 render target. If I could compute these by bouncing them back and forth between 2 render targets, I could completely eliminate the "long-running computation" I mentioned earlier.
4. Readback of render targets. The ultimate goal of procedural art is to share the results with others, and being able to pull data out of a render target and pass it to something like piston::Image to save would let me do that.

It sounds like you're already open to #1, and #2 doesn't seem too crazy imo. I have no idea how much work #3 and #4 would take.

[ejmahler]

This will allow to include the image bytes in an executable binary or load it manually.

Or generate it procedurally.

[hecrj]

As for "Canvas" widget, It's a great idea for both image and mesh data. Do you have any plan for that?

@piaoger Yes! A Canvas widget is in the roadmap. The related issue is #32.

why not use some quick hash on the image bytes as the cache identifier?

@bingo347 I think that could work. We will need to make image::Data own the bytes instead of a reference. This will make it clearer to users that you are supposed to keep it as part of the application state while also satisfying other use cases (like storing remote image data). I will try to play with this and see if it works out well.

Right now I do that via command line arguments, but it would be great if I could use checkboxes, type numbers into fields, etc and get realtime updates.

@ejmahler This reminds me a lot of the mesh example I made in Coffee. Of course, you can hardly call that art, but the idea would be the same: a GUI driving a graphics engine.

I think all the features you list, besides maybe the first one, are meant to be satisfied by some kind of canvas widget that exposes an advanced graphics API for drawing. A widget that would only be available in GPU-accelerated renderers.

I think #32 (or a similar custom widget) should satisfy your use cases in the long run. You seem to be way more experienced than me with graphics programming and I'd love to hear any suggestions you may have about a possible API and its implementation, while learning more about your use cases.

[simlay]

I’ve also wanted this feature and wouldn’t mind implementing. Feel free to assign the task to me.

[piaoger]

I also want something like Image::from_bytes before. My case is providing a remote url as path, and the image will be downloaded and loaded ..

As for "Canvas" widget, It's a great idea for both image and mesh data. Do you have any plan for that?

[bingo347]

@hecrj why not use some quick hash on the image bytes as the cache identifier?
Canvas widget Is good idea for dynamically changing images, but for static images I like the idea with image::Data type

[hecrj]

I have opened #90, which should satisfy some of the use cases discussed here. Let me know what you think.

@ejmahler Would you also be interested in an image::Handle::from_pixmap method?