Comments (4)
Interesting idea, but this might cause a really BIG change. I suggest a new feature to afs-api
, make afs-api
use user's own blob storage instance. No centralized blob storage also means you don't have to upload models through afs-api
. But this might also increase efforts of metadata management. What do you think?
from afs2-model.
We have a plan to setting user-provide blob function on portal.
And put the blob credential to the user workspace runtime environment variables.
This is why I proposed afs2_model.update_model_to_blob(blob_credential, model_file)
the blob_credential can be optional when the user-provide blob appears.
Yes, @chenjr0719 is right. There is a effort to managed the model on blob storage without afs-api
if update model directly.
After updating(or uploading) the model, we may notify afs-api
the content modified and sync meta data.
However, centralized blob storage architecture in afs-api
will be maintained at least a month...
from afs2-model.
Hi @benchuang11046 ,
I have a question about this, "The bottleneck of uploading is about 500-600MB on private cloud.(Maybe less on public cloud)".
The bottleneck is because of network transmission speed, or it is the limit about api process speed... or... others?
from afs2-model.
Hi @estherxyz
For testing, increasing afs-api
memory can improve the upload model size a little bit.(600MB -> 700MB)
Additionally, there are many factors cause the issue, like network transmission slow causes response timeout.
There is a problem that we have a big model to upload depends on afs-api
resource enough or not.
from afs2-model.
Related Issues (18)
- pip install failed HOT 2
- Integrate with Travis CI and Codecov?
- How can I Know Models number in AFS? HOT 6
- How can I know the run jupyter file command line in AFS? HOT 2
- Switch from requests to aiohttp HOT 2
- models method upload_model support v2 HOT 2
- Requirements without specific version HOT 2
- Add tox support
- Deprecate v1 related operations
- Refactor _create() in models HOT 1
- For public pypi name HOT 5
- Codecov report not show HOT 1
- About afs2-model on PyPI.org
- v2.2 user manual enhanced HOT 2
- model repository API parameters introduction
- models API parameters introduction
- get_model_info() always get the latest model version
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from afs2-model.