Upload model has a bottleneck on the big model transferring

[wise-aifs]

Upload model has a bottleneck on the big model transferring

[chenjr0719]

Interesting idea, but this might cause a really BIG change. I suggest a new feature to afs-api, make afs-api use user's own blob storage instance. No centralized blob storage also means you don't have to upload models through afs-api. But this might also increase efforts of metadata management. What do you think?

[benchuang11046]

We have a plan to setting user-provide blob function on portal.
And put the blob credential to the user workspace runtime environment variables.
This is why I proposed afs2_model.update_model_to_blob(blob_credential, model_file) the blob_credential can be optional when the user-provide blob appears.

Yes, @chenjr0719 is right. There is a effort to managed the model on blob storage without afs-api if update model directly.
After updating(or uploading) the model, we may notify afs-api the content modified and sync meta data.

However, centralized blob storage architecture in afs-api will be maintained at least a month...

[estherxyz]

Hi @benchuang11046 ,

I have a question about this, "The bottleneck of uploading is about 500-600MB on private cloud.(Maybe less on public cloud)".
The bottleneck is because of network transmission speed, or it is the limit about api process speed... or... others?

[benchuang11046]

Hi @estherxyz
For testing, increasing afs-api memory can improve the upload model size a little bit.(600MB -> 700MB)
Additionally, there are many factors cause the issue, like network transmission slow causes response timeout.
There is a problem that we have a big model to upload depends on afs-api resource enough or not.