Comments (7)
@dcrankshaw
This issue is critical for the product deployment system. We resolved it by applying the 'blue-green' deployment policy to our system. I implemented own SwarmContainerManager inspired by DockerContainerManager.
- we call
python_deployer.deploy_python_closure()
.- [python_deployer] calls
build_and_deploy_model()
of clipper_admin.- [clipper_admin] calls
self.build_model()
. - [clipper_admin] calls
self.deploy_model()
. - [clipper_admin] calls
deploy_model()
of SwarmContainerManager.- [SwarmContainerManager] creates a new swarm service.
- [SwarmContainerManager] checks that,
- all tasks of the new swarm service have some 'running' status, or not.
- the replicas number of the new swarm service is the same with predefined replicas, or not.
- [SwarmContainerManager] adds it to the metric config.
- [clipper_admin] calls
self.register_model()
to register the new model.
- [clipper_admin] calls
- [python_deployer] calls
I experienced that the time to initialize the new swarm service is very variable, so I think that Clipper might need some routines to check a new model's status before registering it to Clipper.
from clipper.
Maybe clipper_manager could have functions to undeploy
old models. I was thinking, when a new model is successfully deployed, clipper can automatically stop the old containers/models.
Or undeploy
can also be called manually from the clipper manager
Maybe a few new APIs
- Undeploy a model
- Remove application
- Pause application
from clipper.
Yeah agreed. Have you found that Swarm's container status is sufficient to indicate whether a container is running yet? We've found that for several deep learning models, especially when running on a GPU, there is a non-trivial amount of time after a container has started to initialize the model and connect to Clipper. Using the underlying container manager to detect when the containers ready as you do would be relatively simple to implement, but I was worried that that was not sufficient.
from clipper.
@chester-leung For a first version of a fix, you should modify the container_manager.deploy_model
function to block until the container is actually running. Right now, we start the container then return immediately, rather than waiting until the container is fully running. You'll need to implement this for both the Kubernetes container manager and the Docker container manager.
from clipper.
@dcrankshaw
Thank you for your advice. I agreed with your opinion. How about use Docker's healthcheck option in DockerFile and then check healthy
status in the our *ContainerManager to decide whether a container is running or not?
According to the reference,
When a container has a healthcheck specified, it has a health status in addition to its normal status. This status is initially starting. Whenever a health check passes, it becomes healthy (whatever state it was previously in). After a certain number of consecutive failures, it becomes unhealthy.
from clipper.
That's a good idea. @chester-leung for a first step, let's get a version working that just looks at the container state. As a second step, we can modify the RPC implementation to write a file somewhere once the container has connected to Clipper as the healthcheck.
from clipper.
So far, I've implemented a fix to force the container manager to sleep until all added containers are deemed ready. This should fix the problem of querying a model in a new container before the new container is fully functional.
from clipper.
Related Issues (20)
- Batch_size issue HOT 5
- Metrics
- Debugging input and output
- Number of predictions mismatch
- Potential deadlock when deleting model container replica queue
- Issue with speed of clipper with custom PyTorch model HOT 1
- Model Selection HOT 4
- Study the source code and development
- Dynamic Batching HOT 2
- discussion question HOT 2
- Failed to retrieve a prediction response within the specified latency SLO HOT 2
- No connected models found for query HOT 1
- all input-type (except double) do not work.
- Kubernetes Connection - the server could not find the requested resource
- Question for python_deployer.deploy_python_closure - case of multiple outputs from func
- About the source code HOT 1
- basic query example HOT 1
- develop test failed on win10: qf startup error *Option ‘thread_pool_size’ does not exist*
- Unregistering an application, and then registering again with different output doesn't change the output
- Documentation website unavailable(expired) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from clipper.