agile6v/container_cpu_detection

Calculate the number of available CPUs in the container.

container_cpu_detection

About

This program is used to autuomatically calculate the number of available CPU cores in the container based on the cgroup information by hijacking the sysconf system call. (This method of hijacking uses a Unix system trick LD_PRELOAD to hijack dynamic link library)

I have known the secnario of the usage is that Nginx calculates the number of available CPU cores by sysconf(_SC_NPROCESSORS_ONLN) and starts the corresponding number of worker processes. In addition, there must be many others usage scenarios.

Algorithm

There are three ways that Docker limits CPU resources.

• cpuset
• cpu quota & cpu period
• cpu shares

Docker mounts cgroup information into container starting with version 1.8, so it is possible to determine the appropriate number of CPUs based on the cgroup information in the container.

Assume that the cgroup information is mounted in the /sys/fs/cgroup/:

Step 1 - Get the value of the cpuset

By reading /sys/fs/cgroup/cpuset/cpuset.cpus. By default, this value is equal to the number of the host CPU cores.

Step 2 - Get the value of the cpu quota & cpu period

By reading /sys/fs/cgroup/cpu/cpu.cfs_period_us & /sys/fs/cgroup/cpu/cpu.cfs_quota_us. Calculate the available CPU cores by quota divided by period and rounded up.

Step 3 - Get the value of the cpu shares

By reading /sys/fs/cgroup/cpu/cpu.shares. If you don't specify the cpu shares when running a container then its default value is 1024.

Step 4 - Calculate the number of available CPU cores.

These three ways can be specified at the same time. When they are specfied at the same time, take the minimum value as the number of available CPU cores.

Usage

• make
• make test (Please make sure that docker has already installed.)

Note:

1. Set the environment variable DETECTION_TARGETS in the container to specify which programs you want to hijack. Multiple name can be specified at the same time. (separated by a colon)
2. Set the environment variable LD_PRELOAD in the container to specify the path of the .so file.
3. Running a container with -v parameter to mount .so file into the container.
4. Only for Linux.

For example:

docker run -ti --rm \
--cpuset-cpus 0,1 --cpu-quota 200000 \
-v `pwd`/detection.so:/usr/lib/detection.so \
-v `pwd`/sysconf_test:/tmp/sysconf_test \
-e DETECTION_TARGETS=sysconf_test \
-e LD_PRELOAD=/usr/lib/detection.so \
ubuntu /tmp/sysconf_test

container_cpu_detection's Issues

Issue when loading detection.so

issue below

peter@ubuntu:~/container_cpu_detection$ make
gcc -std=c99 -Wall -shared -g -fPIC detection.c -o detection.so
gcc sysconf_test.c -o sysconf_test
peter@ubuntu:~/container_cpu_detection$ make test
docker run -ti --rm --cpuset-cpus 0,1 --cpu-quota 200000 -v /home/peter/container_cpu_detection/detection.so:/usr/lib/detection.so -v /home/peter/container_cpu_detection/sysconf_test:/tmp/sysconf_test -e DETECTION_TARGETS=sysconf_test -e LD_PRELOAD=/usr/lib/detection.so ubuntu /tmp/sysconf_test
/tmp/sysconf_test: symbol lookup error: /usr/lib/detection.so: undefined symbol: dlsym
Makefile:20: recipe for target 'test' failed
make: *** [test] Error 127

my environment

gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.4.0-1ubuntu1~18.04.1' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)

specific usage？

I want to know how it is actually used.
can give a complete example?

error on ubuntu18.04

I met error when following the guide, can you please help?

docker run -ti --rm
--cpuset-cpus 0,1 --cpu-quota 200000
-v pwd/detection.so:/usr/lib/detection.so
-v pwd/sysconf_test:/tmp/sysconf_test
-e DETECTION_TARGETS=sysconf_test
-e LD_PRELOAD=/usr/lib/detection.so
ubuntu /tmp/sysconf_test
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
473ede7ed136: Pull complete
c46b5fa4d940: Pull complete
93ae3df89c92: Pull complete
6b1eed27cade: Pull complete
Digest: sha256:29934af957c53004d7fb6340139880d23fb1952505a15d69a03af0d1418878cb
Status: Downloaded newer image for ubuntu:latest
/tmp/sysconf_test: symbol lookup error: /usr/lib/detection.so: undefined symbol: dlsym

Linux ubuntu 4.15.0-38-generic #41-Ubuntu SMP Wed Oct 10 10:59:38 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

error while loading shared libraries: libapt-private.so.0.0: cannot open shared object file: No such file or directory

environment:

kubernetes: v.1.11.10
docker: 17.09.1-ce
os: ubuntu 16.04

I use sysconf_test like this:

kubectl create configmap detection --from-file=./detection.so
kubectl create configmap sysconf --from-file=./sysconf_test

and then,deploy a pod：

kind: Deployment
metadata:
  name: test-nginx
spec:
  replicas: 1
  template:
    metadata:
      labels:
        tag: lxcfs
        name: test-nginx
    spec:
      volumes:
        - name: detection
          configMap:
            name: detection
        - name: sysconf
          configMap:
             name: sysconf
      containers:
        - name: test-nginx
          image: openresty/openresty
          volumeMounts:
            - name: detection
              mountPath: /usr/lib
            - name: sysconf
              mountPath: /tmp
          resources:
            requests:
              cpu: "0.2"
              memory: "64Mi"
            limits:
              cpu: "1"
              memory: "128Mi"

when i login to pod, and execute command：

root@k8s-m:/home/www/server/kube-yamls/test# kubectl exec -it test-nginx-5c6bf4fb88-jlh74  /bin/bash 
root@test-nginx-5c6bf4fb88-jlh74:/# apt install procps
apt: error while loading shared libraries: libapt-private.so.0.0: cannot open shared object file: No such file or directory

this sysconf make something wrong

How can I fix it ？