In tflite, how to use the same memory to serve different models with exactly the same structure

[captainst]

In tflite, how to use the same memory to serve different models with exactly the same structure

[captainst]

Hi @sushreebarsa

I made some investigations through the subgraph and arena_planner, and found that tflite does not provide no an such interface for the moment. I managed to use interpreter->SetCustomAllocationForTensor to mount an external allocated memory to each tensor. This is a technical viable approach, but in this case we loose the efficiency of memory usage provided by arena_planner. The amount memory we needed to provide externally is 2x to 4x, which does not make practical sense.

A more intelligent solution could be to make arena_planner use a memory pool provided externally. This could help to greatly alleviate the memory fragmentation problem, when we need to load and off-load different models frequently. This involves some modifications to arena.

For the moment I would think that the issue being closed. Probably we could contribute with PR when it is done from our side.

Thank you.

[sushreebarsa]

@captainst TensorFlow Lite (TFLite) doesn't have a built-in mechanism to directly share memory between different models loaded in separate interpreters. Model quantization is the recommended approach for this issue, kindly refer to this guide.
Thank you!

[captainst]

@captainst TensorFlow Lite (TFLite) doesn't have a built-in mechanism to directly share memory between different models loaded in separate interpreters. Model quantization is the recommended approach for this issue, kindly refer to this guide. Thank you!

Thank you! I only need to load one model at a time. So I don't need separate interpreters. Since all the models are of the same structure, I want to use one interpreter to serve all the models, one at a time. If I could find a way to manipulate the weights and quant params inside each tensor.

I'll do some further experiments to see. Thank you again.

[sushreebarsa]

@captainst Is there any update on this issue?
Thank you!

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No