Giter Site home page Giter Site logo

cms-docker's Introduction

These are the docker files for CMS Development Tools services.

Notice that since these are in a public repository you should not expose any details of the CMS infrastructure and have everything configurable via an environment variable. Actual values of the environment variables are maintained somewhere else.

Automatic docker image build

In order to automatically build and deploy images to our Docker Hub, create a sub-directory which should contain ( e.g. see https://github.com/cms-sw/cms-docker/tree/master/slc6 )

  • config.yaml which defines the images you want to build
  • Dockerfile which should contains the details of how to build the container

cms-docker's People

Contributors

aandvalenzuela avatar aledegano avatar amaltaro avatar cmsbuild avatar davidlt avatar ddaina avatar ericvaandering avatar fabioespinosa avatar gartung avatar gudrutis avatar iahmad-khan avatar iarspider avatar kpedro88 avatar ktf avatar mmascher avatar mourtz avatar mrodozov avatar muhammadimranfarooqi avatar rovere avatar saqibhaleem avatar smuzaffar avatar todor-ivanov avatar tulamor avatar vkuznet avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar

Forkers

apfeiffer1 rovere ktf smuzaffar albf roacobb kreczko ericvaandering iahmad-khan rajivreddy jpata gartung mrodozov gudrutis dciangot mourtz makortel fabioespinosa vkuznet ddaina saqibhaleem muhammadimranfarooqi tulamor amaltaro todor-ivanov aandvalenzuela kpedro88 mmascher

cms-docker's Issues

Bootstrapping in SLC5 image fails

Please let me know if I should post this issue somewhere else.

For an upcoming open data release, I'm trying to install CMSSW_4_2_8_lowpupatch1 into an SLC5 container, but the bootstrap script (or actually cmspkg.py with cmspkg_tag = "V00-00-37") fails. Any idea how to solve it? I don't know why the package size returns an invalid literal (see below).

export SCRAM_ARCH=slc5_amd64_gcc434
sh /opt/cms/bootstrap.sh setup -r cms -architecture ${SCRAM_ARCH} -server cmsrep.cern.ch
slc5_amd64

Output:

Downloading driver file slc5_amd64_gcc434...Done driver slc5_amd64_gcc434.
Downloading driver file slc5_amd64_common ...Downloading driver file common_amd64_common ...Downloading bootstrap core packages...
Reading Package Lists...
Get:1 http://cmsrep.cern.ch cmssw/cms/slc5_amd64_gcc434 external+gcc+4.3.4-cms-1-1.slc5_amd64_gcc434.rpm
Get:2 http://cmsrep.cern.ch cmssw/cms/slc5_amd64_gcc434 external+expat+2.0.0-cms2-1-1.slc5_amd64_gcc434.rpm
Get:3 http://cmsrep.cern.ch cmssw/cms/slc5_amd64_gcc434 external+zlib+1.2.3-cms-1-1.slc5_amd64_gcc434.rpm
Get:4 http://cmsrep.cern.ch cmssw/cms/slc5_amd64_gcc434 external+beecrypt+4.1.2-cms2-1-1.slc5_amd64_gcc434.rpm
Get:5 http://cmsrep.cern.ch cmssw/cms/slc5_amd64_gcc434 external+bz2lib+1.0.5-cms-1-1.slc5_amd64_gcc434.rpm
Get:6 http://cmsrep.cern.ch cmssw/cms/slc5_amd64_gcc434 external+db4+4.4.20-cms-1-1.slc5_amd64_gcc434.rpm
Get:7 http://cmsrep.cern.ch cmssw/cms/slc5_amd64_gcc434 external+elfutils+0.131-cms-1-1.slc5_amd64_gcc434.rpm
Get:8 http://cmsrep.cern.ch cmssw/cms/slc5_amd64_gcc434 external+neon+0.26.3-cms2-1-1.slc5_amd64_gcc434.rpm
Get:9 http://cmsrep.cern.ch cmssw/cms/slc5_amd64_gcc434 external+openssl+0.9.8e-cms4-1-1.slc5_amd64_gcc434.rpm
Get:10 http://cmsrep.cern.ch cmssw/cms/slc5_amd64_gcc434 external+libxml2+2.6.23-cms2-1-1.slc5_amd64_gcc434.rpm
Get:11 http://cmsrep.cern.ch cmssw/cms/slc5_amd64_gcc434 external+rpm+4.4.2.2-cms3-1-1.slc5_amd64_gcc434.rpm
Get:12 http://cmsrep.cern.ch cmssw/cms/slc5_amd64_gcc434 external+apt+0.5.15lorg3.2-cms3-1-1.slc5_amd64_gcc434.rpm
Done.
Unpacking core packages...593260 blocks
593260 blocks
4691 blocks
4691 blocks
360 blocks
360 blocks
1318 blocks
1318 blocks
Can't open /opt/cms/bootstraptmp/BOOTSTRAP/inst/slc5_amd64_gcc434/external/beecrypt/4.1.2-cms2/lib/libbeecrypt.la: No such file or directory.
2340 blocks
2340 blocks
44015 blocks
44015 blocks
5887 blocks
5887 blocks
2848 blocks
2848 blocks
46793 blocks
46793 blocks
35411 blocks
35411 blocks
21602 blocks
21602 blocks
49589 blocks
49589 blocks
Done.
Harvesting system for locally available software...Done.
Initializing local rpm database.../opt/cms/slc5_amd64_gcc434/tmp/system-import /opt/cms
/opt/cms
Done.
Installing packages in the local rpm database...Can't open /opt/cms/slc5_amd64_gcc434/external/beecrypt/4.1.2-cms2/lib/libbeecrypt.la: No such file or directory.
Done
cmspkg setup done.
Installing default packages.Reading Package Lists...
Get:1 http://cmsrep.cern.ch cmssw/cms/slc5_amd64_gcc434 cms+cms-common+1.0-1-1107.slc5_amd64_gcc434.rpm
Building Dependency Tree...
The following NEW packages will be installed:
  cms+cms-common+1.0

0 upgraded, 1 newly installed, 0 removed and 0 not upgraded.
Traceback (most recent call last):
  File "/opt/cms/share/cms/cmspkg/V00-00-37/cmspkg.py", line 1207, in ?
    process(args, opts, cache_dir)
  File "/opt/cms/share/cms/cmspkg/V00-00-37/cmspkg.py", line 1088, in process
    repo.install(pkg, reinstall=opts.reinstall, force=opts.force)
  File "/opt/cms/share/cms/cmspkg/V00-00-37/cmspkg.py", line 675, in install
    size_compress, size_uncompress = self.package_size (pk[1])
  File "/opt/cms/share/cms/cmspkg/V00-00-37/cmspkg.py", line 514, in package_size
    return st.st_size, int(out)
ValueError: invalid literal for int():

There was a problem while installing the default packages
Done
Bootstrap environment to be found in:
### /opt/cms/slc5_amd64_gcc434/external/apt/0.5.15lorg3.2-cms3/etc/profile.d/init.sh
/opt/cms/bootstrap.sh: line 1835: cd: /opt/cms/bootstraptmp/BOOTSTRAP: No such file or directory

@katilp

A method to make docker images smaller

Docker does not free up space from deleted files. The ~5GB of rpms downloaded for a release are still counted in the image space usage even after removal. I was able to create a much smaller image this way:
docker run -it -v /tmp:/build ...
Setup as usual in /opt/cms

mv /opt/cms/slc7_amd64_gcc700/var/cmspkg /build
ln -s /opt/cms/slc7_amd64_gcc700/var /build/cmspkg

Run cmspkg install as usual.

After install

rm /opt/cms/slc7_amd64_gcc700/var/cmspkg
cp -rpv /build/cmspkg /opt/cms/slc7_amd64_gcc700/var

Update apptainer version in containers

There are outdated apptainer versions in several current CMS containers:

  • cmssw-el7: 1.2.3-1.el7
  • cmssw-el8: 1.2.4-1.el8

These need to be updated to 1.2.5-1, which includes the resolution to apptainer/apptainer#1779.

(I'm not sure what action needs to be taken for this update - maybe just triggering a new build of the containers? The apptainer version doesn't seem to be pinned anywhere in this repository, as far as I can tell.)

Install tmux for el8/el9 images

@aandvalenzuela , one next update of el8/el9 containers please add tmux too. It is a small package [a] so should not increase the container size too much.

[a]

Installing:
 tmux                                                          x86_64                                                      2.7-3.el8                                                        baseos                                                      315 k
Installing dependencies:
 libevent                                                      x86_64                                                      2.1.8-5.el8                                                      baseos                                                      253 k

Perl issue in el8 container

When I run these commands:

cmssw-el8
source /cvmfs/cms.cern.ch/cmsset_default.sh
cmsenv

I get the following warning:

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

This does not happen in e.g. the cc7 image, so something is probably missing from the el8 image.

Docker hub Team membership restrictions

Docker hub has changed the "Free Team" for organizations. Now they only allow 1 team of max 3 members. Currently we have 3 teams with more than 3 members each (specially dmwm team has 12 members). For now dockerhub has not asked us to cleanup these team but I guess in future they might. Currently we can not add any more teams or members to any of our teams ( we already have 16 seats fill out of 3).

For cms images we are using cmsblddoc, I would suggest that for dmwm you can also start using dmwmbot instead of individual accounts.

FYI @muhammadimranfarooqi @yuyiguo @ericvaandering @bbockelm @vkuznet @fabioespinosa @ddaina @Panos512 @goughes @amaltaro @todorivanov

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.