ktaletsk/cucalc

CUDA + CoCalc Docker container

CuCalc = CUDA + CoCalc Docker container

This is Docker container build on top of CoCalc image (https://github.com/sagemathinc/cocalc-docker) to add support of CUDA for GPU programming. My use is running it on my dedicated desktop computer with GPU and access from laptop anywhere.

Also added some useful packages missing from CoCalc Docker but present in the CoCalc.com, such as Anaconda.

Prerequisites:

• Docker
• nvidia-docker (https://github.com/NVIDIA/nvidia-docker).

To build image, type

sudo make build

To run container, type

sudo make run

To stop container, type

sudo make stop

To start again, type

sudo make start

cucalc's Issues

fatal error: cudnn.h: No such file or directory

The following test for GPU functionality from the Theano tutorial:

import os
os.environ["MKL_THREADING_LAYER"] = "GNU"
os.environ["THEANO_FLAGS"] = "device=cuda,floatX=float32"
from theano import function, config, shared, tensor
import numpy
import time

vlen = 10 * 30 * 768  # 10 x #cores x # threads per core
iters = 1000

rng = numpy.random.RandomState(22)
x = shared(numpy.asarray(rng.rand(vlen), config.floatX))
f = function([], tensor.exp(x))
print(f.maker.fgraph.toposort())
t0 = time.time()
for i in range(iters):
    r = f()
t1 = time.time()
print("Looping %d times took %f seconds" % (iters, t1 - t0))
print("Result is %s" % (r,))
if numpy.any([isinstance(x.op, tensor.Elemwise) and
              ('Gpu' not in type(x.op).__name__)
              for x in f.maker.fgraph.toposort()]):
    print('Used the cpu')
else:
    print('Used the gpu')

Produces an error message:

Can not use cuDNN on context None: cannot compile with cuDNN. We got this error:
/tmp/try_flags_cgM6mq.c:4:19: fatal error: cudnn.h: No such file or directory
 #include <cudnn.h>
                   ^
compilation terminated.

Mapped name None to device cuda: GeForce GTX 980 Ti (0000:01:00.0)

Although the output indicates that the GPU was in fact used during the execution

[GpuElemwise{exp,no_inplace}(<GpuArrayType<None>(float32, vector)>), HostFromGpu(gpuarray)(GpuElemwise{exp,no_inplace}.0)]
Looping 1000 times took 1.333771 seconds
Result is [ 1.23178029  1.61879349  1.52278066 ...,  2.20771813  2.29967761
  1.62323296]
Used the gpu

Kubernetes Cluster support

Hi! I'm just wondering if there's any image of CoCalc with both CUDA and Kubernetes Cluster support. I'm aware of cocalc-kubernetes and CuCalc projects, but unfortunately I couldn't find an image that attends to both. Is that a thing?

To use MKL 2018 with Theano you MUST set "MKL_THREADING_LAYER=GNU"

With Jyptyer worksheet anaconda_kernel:

import theano
from theano import tensor

I get the following error message:

RuntimeErrorTraceback (most recent call last)
<ipython-input-1-78067c42f5c0> in <module>()
----> 1 import theano
      2 from theano import tensor
/opt/conda/lib/python2.7/site-packages/theano/__init__.pyc in <module>()
    122 from theano.printing import pprint, pp
    123 
--> 124 from theano.scan_module import (scan, map, reduce, foldl, foldr, clone,
    125                                 scan_checkpoints)
    126 
/opt/conda/lib/python2.7/site-packages/theano/scan_module/__init__.py in <module>()
     39 __contact__ = "Razvan Pascanu <r.pascanu@gmail>"
     40 
---> 41 from theano.scan_module import scan_opt
     42 from theano.scan_module.scan import scan
     43 from theano.scan_module.scan_checkpoints import scan_checkpoints
/opt/conda/lib/python2.7/site-packages/theano/scan_module/scan_opt.py in <module>()
     58 
     59 import theano
---> 60 from theano import tensor, scalar
     61 from theano.tensor import opt, get_scalar_constant_value, Alloc, AllocEmpty
     62 from theano import gof
/opt/conda/lib/python2.7/site-packages/theano/tensor/__init__.py in <module>()
     15 from theano.tensor import opt
     16 from theano.tensor import opt_uncanonicalize
---> 17 from theano.tensor import blas
     18 from theano.tensor import blas_scipy
     19 from theano.tensor import blas_c
/opt/conda/lib/python2.7/site-packages/theano/tensor/blas.py in <module>()
    153 from theano.scalar import bool as bool_t
    154 from theano.tensor import basic as T
--> 155 from theano.tensor.blas_headers import blas_header_text
    156 from theano.tensor.blas_headers import blas_header_version
    157 from theano.tensor.opt import in2out, local_dimshuffle_lift
/opt/conda/lib/python2.7/site-packages/theano/tensor/blas_headers.py in <module>()
    985 
    986 
--> 987 if not config.blas.ldflags:
    988     _logger.warning('Using NumPy C-API based implementation for BLAS functions.')
    989 
/opt/conda/lib/python2.7/site-packages/theano/configparser.pyc in __get__(self, cls, type_, delete_key)
    330             except KeyError:
    331                 if callable(self.default):
--> 332                     val_str = self.default()
    333                 else:
    334                     val_str = self.default
/opt/conda/lib/python2.7/site-packages/theano/configdefaults.pyc in default_blas_ldflags()
   1406             res = try_blas_flag(flags)
   1407             if res:
-> 1408                 check_mkl_openmp()
   1409                 maybe_add_to_os_environ_pathlist('PATH', lib_path[0])
   1410                 return res
/opt/conda/lib/python2.7/site-packages/theano/configdefaults.pyc in check_mkl_openmp()
   1250         import mkl
   1251         if '2018' in mkl.get_version_string():
-> 1252             raise RuntimeError('To use MKL 2018 with Theano you MUST set "MKL_THREADING_LAYER=GNU" in your environement.')
   1253     except ImportError:
   1254         raise RuntimeError("""
RuntimeError: To use MKL 2018 with Theano you MUST set "MKL_THREADING_LAYER=GNU" in your environement.

question: installing additional conda packages after build

What is the correct method to install additional conda packages, e.g. sympy, after build is complete and CuCalc is running? If I use conda install sympy in docker exec -it cucalc bash then sympy is not available to projects in CuCalc due to issues with file privileges.

Modify default .bashrc for project

I have to manually modify .bashrc to update PATH variable for new packages like CUDA and Conda.

I could not identify the location of default .bashrc file in CoCalc. If I would know that location, I would modify the Docker image at build time.

question: how to add a julia kernel ?

I have installed cocalc image on google cloud.
I have tried to add julia 1.4 kernel to jupyter but it failed.
I have set .julia to be here : /usr/local/share/.julia.

Config

julia bin directory : /opt/julia/bin/julia

ln -s /opt/julia/bin/julia /usr/local/bin

IJulia is installed

using Pkg ;
Pkg.add("IJulia");

etc/julia/startup.jl

ENV["JULIA_DEPOT_PATH"] = "/usr/local/share/.julia" ; 
ENV["JUPYTER"] = "/usr/local/bin/jupyter"; 

/usr/local/share/jupyter/kernels/julia-1.4/kernel.json

{
  "display_name": "Julia 1.4.0",
  "argv": [
    "/opt/julia/bin/julia",
    "-i",
    "--startup-file=yes",
    "--color=yes",
    "--project=@.",
    "/usr/local/share/.julia/packages/IJulia/yLI42/src/kernel.jl",
    "{connection_file}"
  ],
  "language": "julia",
  "env": {},
  "interrupt_mode": "signal"
}

Error I get

sh-4.4$ jupyter console --kernel=julia-1.4
ERROR: LoadError: ArgumentError: Package IJulia not found in current path:
- Run `import Pkg; Pkg.add("IJulia")` to install the IJulia package.

Stacktrace:
[1] require(::Module, ::Symbol) at ./loading.jl:892
[2] include(::Module, ::String) at ./Base.jl:377
[3] exec_options(::Base.JLOptions) at ./client.jl:288
[4] _start() at ./client.jl:484

It seems that I don't understand Julia package management for projects (""--project=@." options in kernel.json).

Thanks in advance for your help.

missing dependencies

I had to add the following when building on Linux Mint 19:

diff --git a/Dockerfile b/Dockerfile
index eb6d60d..b3959c7 100755
--- a/Dockerfile
+++ b/Dockerfile
@@ -21,6 +21,8 @@ ENV NCCL_VERSION 2.2.12
 
 RUN apt-get update && apt-get install -y --no-install-recommends \
         cuda-cudart-$CUDA_PKG_VERSION \
+        cuda-cudart-dev-$CUDA_PKG_VERSION \
+        cuda-cupti-$CUDA_PKG_VERSION \
         cuda-libraries-$CUDA_PKG_VERSION \
         cuda-nvtx-$CUDA_PKG_VERSION \
         libnccl2=$NCCL_VERSION-1+cuda9.2 \