Android Tflite model fails to load on GPU Delegate: CL_OUT_OF_HOST_MEMORY

[filip-halt]

Android Tflite model fails to load on GPU Delegate: CL_OUT_OF_HOST_MEMORY

[sawantkumar]

Hi @filip-halt ,

Can you please provide the tflite model file so that i can replicate the issue?

[filip-halt]

Hi @filip-halt ,

Can you please provide the tflite model file so that i can replicate the issue?

You can find a copy here: https://github.com/filip-halt/tflite_bug It was too large to upload directly into this chat.

[filip-halt]

Turns out that this is most likely due to a conv2dtranspose layer in the model. I was under the impression that conv2dtranpose was supported but I could be wrong.

Another interesting thing that happens is that when you convert the model with:

converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_types = [tf.float16]

the resulting binary is 2x as large as float32 and 20% slower on mobile. When I inspected the graph with Netron, it looks like nothing was converted to float16, not even the conv2d's

[sawantkumar]

Hi @filip-halt ,

I ran your model using GPU delegate on dimensity 9000 and it ran fine without giving any issues. Can you please try it out with a different device and let me know if it worked there. However the list of supported operators for tflite is here and TRANSPOSE_CONV is listed there.

[github-actions]

This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you.

[filip-halt]

I believe that this is a grouped TRANSPOSE_CONV conversion problem. Tensorflow seems to barely support this and is what is breaking when converting onnx to tf. The default conversion creates a large amount of layers that ultimately cause an OOM on the phone when loading.

[sawantkumar]

Hi @filip-halt , we're wondering if you may be able to resolve your issue by using AI-Edge-Torch, you can find more information here: googleblog.

I have actually created a simple script for converting a mobilenet model here:

import torch
import torchvision
import ai_edge_torch

mobilenet_model = torchvision.models.mobilenet_v3_small()
sample_inputs = (torch.randn(1, 3, 224, 224),)

edge_model = ai_edge_torch.convert(mobilenet_model.eval(), sample_inputs)
edge_model.export("mobilenet_v3_small.tflite")

If you want to, you can actually try visualizing the result in model-explorer as well.

Please try them out and let us know if this resolves your issue. If you still need further help, feel free to open a new issue at the respective repo.