codalab / competitions-v1-compute-worker Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
We basically have 3 docker images for the compute workers:
codalab/competitions-v1-compute-worker:1.1.5
(CPU)codalab/competitions-v1-nvidia-worker:v1.5-compat
(GPU)codalab/competitions-v1-compute-worker:latest
Not sure if latest
is even used.
However, we have many branches in the repository:
master
Updated 6 minutes ago by Didayolo
dependabot/pip/celery-5.2.2
Updated 2 months ago by dependabot[bot]
dependabot/pip/pyyaml-5.4
Updated 12 months ago by dependabot[bot]
17-legacy-nvidia-worker-compat-py3
Updated 14 months ago by Tthomas63
python3
Updated 2 years ago by Tthomas63
17-legacy-nvidia-worker-compat
Updated 2 years ago by ckcollab
162-nvidia-worker
Updated 2 years ago by zhengying-liu
dependabot/pip/psutil-5.6.6
Updated 2 years ago by dependabot[bot]
dependabot/pip/requests-2.20.0
Updated 2 years ago by dependabot[bot]
162-nvidia-worker-monitor
Updated 3 years ago by ckcollab
feature/realtime-detailed-results
Updated 3 years ago by ckcollab
feature/legacy-azure-version-fix
Updated 3 years ago by ckcollab
162-nvidia-worker-celery-4-3-0
Updated 3 years ago by Tthomas63
feature/fix-prune
Updated 3 years ago by Tthomas63
feature/suppress-warning
Updated 3 years ago by Tthomas63
competitions-v1-compute-worker/worker.py
Line 502 in 3d2ff02
The way we implement execution_time_limit in docker command doesn't work as we expected. This option doesn't stop the container after this limit time.
Here is Docker documentation : https://docs.docker.com/engine/reference/commandline/run/#stop-timeout
"The --stop-timeout flag sets the number of seconds to wait for the container to stop after sending the pre-defined (see --stop-signal) system call signal. If the container does not exit after the timeout elapses, it's forcibly killed with a SIGKILL signal.
If you set --stop-timeout to -1, no timeout is applied, and the daemon waits indefinitely for the container to exit.
The Daemon determines the default, and is 10 seconds for Linux containers, and 30 seconds for Windows containers."
This issue concerns the Google Cloud version of compute worker for AutoDL.
If we ssh to some workers (Google Cloud VMs, login required) and check the Docker logs:
gcloud beta compute --project "autodl-221715" ssh --zone "us-west1-a" "gpu-06-25-2019-20-30-04-000"
nvidia-docker logs -f compute_worker
We see a lot of error messages such as:
or in text version:
entr: cannot stat '/tmp/codalab/tmp9CPTvU/run/output/detailed_results.html': No such file or directory
It seems that this issue doesn't completely block the submission handling process but it may be the cause of other issues.
The docker run command for the new docker version seem to be
docker run --gpus all ...
instead of
nvidia-docker run ...
I have tested the former and it worked for me.
If with the latter, there will be the following error:
docker: Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v1.linux/moby/1f97be47849eff264efa7a02e7fb6767ceb51ac432ddc502820f667035d84db1/log.json: no such file or directory): exec: "nvidia-container-runtime": executable file not found in $PATH: unknown.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.