Memory issue

[metalpetal]

Memory issue

[YuAo]

Looks like something related to CVMetalTextureCacheRef.

If you add these lines after makeCGImage you can get the memory back.

let cgImages = try images.map({ try self.context.makeCGImage(from: $0) })

DispatchQueue.main.asyncAfter(deadline: .now() + 1.1, execute: {
    self.context.coreVideoTextureCache.flush()
})

The magical 1.1 second comes from here https://developer.apple.com/documentation/corevideo/kcvmetaltexturecachemaximumtextureagekey?language=objc It says "By default, textures age out of the cache after one second."

I tried to disable the age-out mechanism, i.e. set a maximum texture age of zero. However it does not work as expected. The texture cache object created with the {kCVMetalTextureCacheMaximumTextureAgeKey: 0} cache attributes still acts as if it is created with nil cache attributes. If you print the texture cache object, it says its cacheAttributes is nil.

Looks like a bug of CVMetalTextureCacheRef. I'll try to talk to an apple engineer and see if he/her can confirm the bug or maybe provide a workaround.

[YuAo]

CIImage may contains compressed or uncompressed image data.
The suggestion is to use compressed image data, like JPEG.

let imageData: [Data] = try images.map({ item in
                                try autoreleasepool(invoking: {
                                    var img: CGImage?
                                    let task = try self.context.startTask(toCreate: &img, from: item, sRGB: false)
                                    task.waitUntilCompleted()
                                    let data = UIImage(cgImage: img!).jpegData(compressionQuality: 0.8)
                                    self.context.reclaimResources()
                                    return data!
                                })
                            })

Or write these files to a temp directory. Passing the file urls around.

[YuAo]

No.

[SerjPridestar]

@YuAo for some reason if I don't reclaim resources after 1.1 seconds after generating the compressed images the memory isn't dropping. However, for the try self.context.render(image, to: pixelBuffer) it seems to be working fine.

[YuAo]

Try adding context.reclaimResources() after makeCGImage?

There are lots of intermediate buffers with different sizes in your render pipeline. So MetalPetal may not be able to reuse these intermediate texture between renderings. For performance reasons MetalPetal do not reclaim resources after each render, though it has a mechanism of reclaiming resources on memory warnings.

[SerjPridestar]

I tried adding it still the problem persists after I finish generating all the CGImages:

let cgImages = try images.map({ try self.context.makeCGImage(from: $0) })
let uiImages = cgImages.map({ UIImage(cgImage: $0) })
self.context.reclaimResources()

I don't think I have different sizes in my render pipeline, since I take all the images and crop and transform them all to the same size and then call the context.render(image, to: pixelBuffer). I did this for the same exact reason you mentioned, I wanted the rendering to be more performant and use up less memory and reuse the resources already allocated. After a lot of experimentation I noticed by having all the images the same size this is the case. However, still after each context.render call I see resources up to 15mb - 20mb. I will try adding context.reclaimResources() after each render call see if I still see this progressive incremental increase in memory.

In general my goal out of using Metal with the AVAssetWriterInput is to be able to generate really high quality videos with no constraints on how many images I could render on the AVAssetWriterInputPixelBufferAdaptor.

[YuAo]

The situation here is a little bit weird, I may need some more time to investigate.

[SerjPridestar]

Alright! If you need anything or I could do anything to help please do inform me! Again thank you very much!

[SerjPridestar]

@YuAo I'll give it a shot right now. Good job digging and finding the problem, I spent a day on it and couldn't find the main problem.

Is it the same with the context.render(image, to: pixelBuffer)? The call to that increments a gradual increase in memory. Maybe I'm wrong, but to me there shouldn't be a limitation on how many images you can draw to the AVAssetWriterInputPixelBufferAdaptor. I'm guessing the process is so fast that the system isn't able to flush any memory needed, thus an overload of images just crashes the app before even getting the chance to clear up memory. I would love to hear your thoughts on this. Thank you very much Yuao your work is very much appreciated!

[YuAo]

Yes, it is the same with context.render(image, to: pixelBuffer).

[SerjPridestar]

@YuAo I also did some extra digging, also having the PHCachingImageManager.requestImage makes a significant difference in memory. It seems that UIImage is an uncompressed version of the Data, so dealing with 4k resolution images this can make a major difference. Now I'm using PHCachingImageManager.requestImageData instead. The downside of this, is you use up more processing time orienting the images to the correct orientation. Its a good trade off in my opinion, much less memory used to a one second more processing.

Moreover, I guess it should be the same when having the exporter complete I should return a compressed version of the images instead of a UIImage.

By the way Yuao I was thinking of adding MetalPetalVideoGenerator to the MetalPetal repository as example on how to use MetalPetal to make videos out of images. What do you think?

[YuAo]

Hi @SerjPridestar

I've implemented something that works like CVMetalTextureCacheRef but without the caching/memory issue. Ref: ad3b54e

MTICVMetalTextureBridge works on iOS 11 and later. I've modified the MTIContext's implementation to make it use MTICVMetalTextureBridge instead of MTICVMetalTextureCache on supported OS. Ref: ad3b54e#diff-54e5c572d59afa060fcb48e0a6689978R129

I'd like you to give it a try.

BTW, there are some other improvements you can make regarding your project.

1. Make your image with .transient cache policy, so MetalPetal does not need to keep the decoded image. MTIImage(cgImage: $0, options: [MTKTextureLoader.Option.SRGB : false], isOpaque: false).withCachePolicy(.transient)

2. As you said, you should return a compressed version of the images instead of CGImage/UIImages. The CGImage array created after export is memory consuming.

MetalPetalVideoGenerator is really a great example of how you can use MetalPetal to make videos out of images. However, I don't think adding MetalPetalVideoGenerator to MetalPetal is a good idea. MetalPetal and its demo should focus on its main feature of image and video processing. The main idea about MetalPetalVideoGenerator is the AVFoundation part, I think.

[SerjPridestar]

Hello Yuao! I tried the improvements and great news the memory drops significantly and everything works as expected! Before the improvements after finishing exporting the memory would stay at around 324 mb, after improvements the memory is reclaimed and the memory is at 115 mb.

I think it is safe to close this issue for now!

Moreover, I agree about your opinion about adding MetalPetalVideoGenerator to the demos as its main focus is AVFoundation as its end goal is to make a video out of different sources of images.

Once again great work with this and thank you for patching a fix so promptly much appreciated!

[SerjPridestar]

Only thing I noticed right now as you suggested instead of returning CGImage/UIImages, I tried CIImage, not sure if it is a compressed version, but memory stays at 800 mb; on the other hand, CGImage after export has its memory stay at 400 mb. I did make sure to reclaim resources after 1.1s after generating the CIImages

[SerjPridestar]

@YuAo do we still need to call self.context.reclaimResources() after 1.1 seconds?

[SerjPridestar]

It's all good now. After a lot of experimentation, I realized keeping a reference to the MTIImage is what was making the memory increment and not being able to reclaimResources(). So when I got done rendering the image to the video and tried to create the compressed image data, there wasn't enough memory to do that. Thus, I realized its best after I'm done with rendering the MTIImage I directly create the image data save it in an array and set the reference to that MTIImage to nil and reclaim the resources. Now it works perfectly! Thank you so much Yuao!!

[YuAo]

A MTIImage's cachePolicy tells the MTIContext how it should handle the rendered buffer of that image. You can keep the MTIImage around, just make sure its cache policy is .transient.

[SerjPridestar]

@YuAo Eventhough I had all the MTIImages using the cache policy .transient having them in memory around they were using up a large amount of memory.

I changed the whole algorithm to be on demand basis. So I only create an MTIImage only when the time comes to render that ImageData.

In my opinion I don't I think it's MTIImage is using up the memory per-say rather than the CIImage. What I noticed while using the MTIImage(ciImage: ciImage, isOpaque: true) constructor is that MTIImage rather the MTICIImagePromise keeps a reference to the CIImage as you mentioned above CIImage may contain compressed or uncompressed image data. In my use case I have 4k images using that constructor the CIImage does definitely occupy a large space in memory and is costly. At the end of the day, acting like resources aren't scare while programming is just lazy programming, so I did change the whole algorithm to take into consideration that resources are scarce. Without you helping realize this, I wouldn't have been able to draw many 4k images to the video. From 6 images limit to now 50 images! 🎉 So thank you for that!