inveniosoftware/docker-invenio

Docker base images for Invenio.

docker-invenio's Issues

Change in default python in centos 7 makes yum fail

By changing the default python version from 2 to 3.6 yum among other system tools stop working, since they need python2 to work. For example:

[root@localhost]# yum
  File "/usr/bin/yum", line 30
   except KeyboardInterrupt, e:
                        ^

I need yum & co to extend with some extra libraries that cannot be installed via pip, therefore I wanted to extend the image. For now my fix was to revert the default momentarily:

# Set python2 as default to make yum work
RUN alternatives --set python /usr/bin/python2
# Install pre-requisites
RUN yum update -y && \
    yum install -y <libraries>
RUN yum clean -y all

# Fall back to python3 as default
RUN alternatives --set python /usr/bin/python3.6

However, I have to know which python version was setted in the Invenio image and hardcode it to fall back. It cannot be taken from the environment since python3 has a symbolic link to python3.4.

Is there a more elegant solution? Thanks!

centos8-python:3.8 uses python 3.6 as default

Package version (if known):

Describe the bug

A clear and concise description of what the bug is.

Steps to Reproduce

1. Fetch latest invenioRDM instance with python version 3.8
2. Run containerized command invenio-cli containers start --lock --build --setup
3. See errors/warnings

Ignoring warnings

Ignoring celery: markers 'python_version >= "3.7"' don't match your environment
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv
Ignoring ipython: markers 'python_version >= "3.7"' don't match your environment
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv
Ignoring traitlets: markers 'python_version >= "3.7"' don't match your environment
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv

Errors

[pipenv.exceptions.InstallError]: ERROR: Command errored out with exit status 1: /usr/bin/python3.6 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-7ej6ai_k/uwsgi_3e7c7e1254ff4f63bb8706984b88a699/setup.py'"'"'; __file__='"'"'/tmp/pip-install-7ej6ai_k/uwsgi_3e7c7e1254ff4f63bb8706984b88a699/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-wla3jbr2/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.6m/uwsgi Check the logs for full command output.
ERROR: Couldn't install package: uwsgi
 Package installation failed...
ERROR: Service 'web-ui' failed to build: The command '/bin/sh -c pipenv install --deploy --system --pre' returned a non-zero code: 1
Failed to start containerized instance.

Additional context

From debug, i got to see that if you run python -V you get 3.8 while running python3 -V you get version 3.6.
there is something to do with symlink see also comment: https://github.com/inveniosoftware/docker-invenio/pull/40/files#r623840370

docker: xrootd base image

Create a base image from inveniosoftware/centos7-python with XRootD libraries included.

Usage:

FROM inveniosoftware/centos7-python:3.6-xrootd
# ...

Webhooks and cron

• Prototype a webhooks mapper service that will transform a webhook payload, received from Harbor (see doc here), to a version understood by the configured received (e.g. Discord, see doc here).
• Create a repo to host the code, with tests, and deploy.
• Implement some HTTP request that will maintain this GitHub repo crons active, so that the Docker image can be repeatedly built. Eventually, this can be probably done in CERN GitLab.

duplicate of #64

docs: update README with user documentation

1. Change README from RST to MD (seems like docker hub doesn't render RST).
2. Update README with usage documentation for end-users similar to other docker images (https://hub.docker.com/_/centos/)

CI: Check if the repo stays alive

Check if the current CI setup is working in keeping the repo "alive" for the cron pipelines.
The already added GitHub Action step gautamkrishnar/keepalive-workflow@v1 should ensure that.

global: setup automatic builds and publishing

Automatic build of images on docker hub and possible also registry.centos.org

Rebuild released tags

We should probably split the GH Actions to do the following:

• Release on tag. For Almalinux only, it should:

  1. build the image
  2. push with tags latest, major, major.minor, major.minor.patch.

• Rebuild on schedule, both for Almalinux and the 2 CentOS 7:

  1. checkout the repo on the latest push tag
  2. build docker images
  3. push same tags to replace the built images upstream (for CentOS, tags are hardcoded with 3.6 and 3.9)

• Refactor CI auto-release, testing this.

• Run periodic scans on schedule for Almalinux and the 2 CentOS 7:

  1. scan the image on registry.cern.ch
  2. report errors

Base Docker image

• Look into conversation on Discord with Werner about using the RedHat UBI image
• See existing docker-invenio
• [POSTPONED] How to include XRootD distribution in the image
  • Out of scope (because of complexity); we'll use local files instead (common R/W volume on OpenShift)
  • Look into S3-based file API?

Decision: use AlmaLinux as base image.

Findings

Today (20 Oct 2022), apps end of life:

• Python
  • 3.10 in ~4 years
  • 3.9 in ~3 years
  • 3.8 in 2 years
  • 3.7 in 8 months
• NodeJs
  • v18: upcoming
  • v16: ended, only security updates for 10 months
• Almalinux
  • v9: in 4.5 years

Tasks

• Review exiting Docker image with AlmaLinux and Python 3.9
  • for example, why did we pin setuptools?
  • run as normal user, not root? WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager
  • check if any vulnerabilities: https://registry.cern.ch/harbor/projects/1825/
• Decide the naming/tagging convention, for example inveniosoftware/almalinux9:python3.9-node18 vs inveniosoftware/almalinux9-python3.9-node18 vs .... Ask the community.
  • Decided to go for incremental versions instead: inveniosoftware/almalinux:v1.0.0
• Can we host such images in registry.cern.ch, to avoid DockerHub limitations? Is there any other alternative?
• Setup automatic builds

Pipenv error during installation

The base inveniosoftware/centos7-python image has errors due to the recent pipenv release (2020-05-28), where pipenv install --system attempts to use a (non-existent) virtualenv Python instead of the system Python. See e.g. pypa/pipenv#4315

Installing dependencies from Pipfile.lock (876ab0)…
Failed to load paths: /bin/sh: /root/.local/share/virtualenvs/src--FJPeFkS/bin/python: No such file or directory

Needs a version pin here

Incompatible repo in py36 image

I inherit form the docker python3.6 image. However, when I try to update the packages with yum update -y I get a conflict of dependencies:

[Shortened version]

file /usr/lib64/python3.6/xml/parsers/__pycache__/expat.cpython-36.opt-2.pyc from install of python36-libs-3.6.8-1.el7.x86_64 conflicts with file from package python36u-libs-3.6.7-1.ius.centos7.x86_64
  file /usr/lib64/python3.6/xml/parsers/__pycache__/expat.cpython-36.pyc from install of python36-libs-3.6.8-1.el7.x86_64 conflicts with file from package python36u-libs-3.6.7-1.ius.centos7.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/_exceptions.cpython-36.opt-1.pyc from install of python36-libs-3.6.8-1.el7.x86_64 conflicts with file from package python36u-libs-3.6.7-1.ius.centos7.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/_exceptions.cpython-36.opt-2.pyc from install of python36-libs-3.6.8-1.el7.x86_64 conflicts with file from package python36u-libs-3.6.7-1.ius.centos7.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/_exceptions.cpython-36.pyc from install of python36-libs-3.6.8-1.el7.x86_64 conflicts with file from package python36u-libs-3.6.7-1.ius.centos7.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/expatreader.cpython-36.opt-1.pyc from install of python36-libs-3.6.8-1.el7.x86_64 conflicts with file from package python36u-libs-3.6.7-1.ius.centos7.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/expatreader.cpython-36.opt-2.pyc from install of python36-libs-3.6.8-1.el7.x86_64 conflicts with file from package python36u-libs-3.6.7-1.ius.centos7.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/expatreader.cpython-36.pyc from install of python36-libs-3.6.8-1.el7.x86_64 conflicts with file from package python36u-libs-3.6.7-1.ius.centos7.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/handler.cpython-36.opt-1.pyc from install of python36-libs-3.6.8-1.el7.x86_64 conflicts with file from package python36u-libs-3.6.7-1.ius.centos7.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/handler.cpython-36.opt-2.pyc from install of python36-libs-3.6.8-1.el7.x86_64 conflicts with file from package python36u-libs-3.6.7-1.ius.centos7.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/handler.cpython-36.pyc from install of python36-libs-3.6.8-1.el7.x86_64 conflicts with file from package python36u-libs-3.6.7-1.ius.centos7.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/saxutils.cpython-36.opt-1.pyc from install of python36-libs-3.6.8-1.el7.x86_64 conflicts with file from package python36u-libs-3.6.7-1.ius.centos7.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/saxutils.cpython-36.opt-2.pyc from install of python36-libs-3.6.8-1.el7.x86_64 conflicts with file from package python36u-libs-3.6.7-1.ius.centos7.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/saxutils.cpython-36.pyc from install of python36-libs-3.6.8-1.el7.x86_64 conflicts with file from package python36u-libs-3.6.7-1.ius.centos7.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/xmlreader.cpython-36.opt-1.pyc from install of python36-libs-3.6.8-1.el7.x86_64 conflicts with file from package python36u-libs-3.6.7-1.ius.centos7.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/xmlreader.cpython-36.opt-2.pyc from install of python36-libs-3.6.8-1.el7.x86_64 conflicts with file from package python36u-libs-3.6.7-1.ius.centos7.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/xmlreader.cpython-36.pyc from install of python36-libs-3.6.8-1.el7.x86_64 conflicts with file from package python36u-libs-3.6.7-1.ius.centos7.x86_64
  file /usr/lib64/python3.6/zipfile.py from install of python36-libs-3.6.8-1.el7.x86_64 conflicts with file from package python36u-libs-3.6.7-1.ius.centos7.x86_64

This is due to the ius/epel conflict. I was able to fix it by :

yum erase python36u python36u-devel python36u-pip
yum install -y rh-python36 python36-devel python36-pip

Should I provide a PR with this change? Or is there any specific reason why to use the ius libraries?

builds: automatic build to get latest CentOS 7

We need, for security reasons, to be up to date with the base CentOS image, so this are the options ordered by preference:

• Explore the possibility of using Docker Hub repository links so we would link with centos:7 and a rebuild would be triggered when the image changes.
• Add a daily Travis build which will trigger a rebuild on Docker Cloud via webhooks (to create the webhook https://cloud.docker.com/swarm/inveniosoftware/repository/docker/inveniosoftware/[IMG-NAME]/builds/edit go to the bottom of the page).
• Add a daily Travis build which would build the image itself and push it to Docker Cloud.

Dealing with code vulnerability alerts (dependabot)

global: create base images for Invenio

Create Docker base images for Invenio based on work already done in https://gitlab.cern.ch/invenio/base

Suggestion for possible images (please suggest others):

• centos7-python36
• cc7-base-python36
• centos7-python36-xrootd
• cc7-base-python36-xrootd

Ideas could also be taken from the source-to-image (S2I) images: https://hub.docker.com/r/centos/python-36-centos7/ and INSPIRE https://github.com/inspirehep/inspire-docker

Images should be usable on OpenShift (i.e. take care of random user id)

readme: update documentation to reflect python 3.7 and 3.8 images

CentOS 8 images

CentOS 8 is out now so we should create an images similar to our CentOS 7 images.

Add project files

• AUTHORS.rst
• LICENSE (MIT)
• README.rst

Read only HOME directory does not work in OpenShift

The assumption of a writable directory done by pipenv does not work in OpenShift or other deplyments that have no or at least non writable home directories.

There is an official pipenv ticket here.

I opted for the same solution and added to my Dockerfile the following ENV variables:

ENV PIPENV_VENV_IN_PROJECT="/opt/invenio/src/"
ENV PIPENV_CACHE_DIR="/opt/invenio/src/"

However, since pipenv is installed here it might be nice to add those lines and make it match with the WORKING_DIR.

tests: add travis test

See example https://github.com/getsentry/docker-sentry/blob/master/.travis.yml

See also https://github.com/docker-library/official-images

Vulnerability alerts for Docker images

• Docker images scan (e.g. our base image needs to be upgraded) - done in registry.cern.ch

python 3.9 image

docker: upgrade node/npm from v6.x to v10.x

Test with cookiecutter instance and make sure the assets are building with the latest release.

almalinux: InvenioRDM step: RUN pipenv install --verbose --deploy --system fails

Package version (if known): registry.cern.ch/inveniosoftware/almalinux:1

Describe the bug

When building our instance's image with FROM registry.cern.ch/inveniosoftware/almalinux:1 (per https://github.com/inveniosoftware/cookiecutter-invenio-rdm/blob/master/%7B%7Bcookiecutter.project_shortname%7D%7D/Dockerfile#L11), the image fails to build with a large error output that I truncated as follows:

 u''Step 4/12 : RUN pipenv install --deploy --system\n'', u'' ---> Running in dce117f57eeb\n'', u''\x1b[91m\x1b[0m'', u''Installing dependencies from Pipfile.lock (3ee068)...\n'', u"\x1b[91mAn
        error occurred while installing alembic==1.9.3 ; python_version >= ''3.7'' --hash=sha256:8fd6aaea56f5a703a190d25a705dfa91d7c313bb71de2f9c68f5abdcaf5df164 --hash=sha256:ed2f73ea9c986f43af8ad7502c5f60d6bb1400bcd6d29f230e760e08884cb476! Will try again.
        An error occurred while installing amqp==5.1.1 ; python_version >= ''3.6'' --hash=sha256:6f0956d2c23d8fa6e7691934d8c3930eadb44972cbbd1a7ae3a520f735d43359 --hash=sha256:2c1b13fecc0893e946c65cbd5f36427861cffa4ea2201d8f6fca22e2a373b5e2! Will try again.
        An error occurred while installing aniso8601==9.0.1 --hash=sha256:1d2b7ef82963909e93c4f24ce48d4de9e66009a21bf1c1e1c85bdd0812fe412f --hash=sha256:72e3117667eedf66951bb2d93f4296a56b94b078a8a95905a052611fb3f1b973! Will try again.
        An error occurred while installing appdirs==1.4.4 --hash=sha256:7d5d0167b2b1ba821647616af46a749d1c653740dd0d2415100fe26e27afdf41 --hash=sha256:a841dacd6b99318a741b166adb07e19ee71a274450e68237b4650ca1055ab128! Will try again.

[... this goes on for every dependency ...]

\x1b[0m", u''Installing initially failed dependencies...
[pipenv.exceptions.InstallError]: Collecting alembic==1.9.3
[pipenv.exceptions.InstallError]:   Downloading alembic-1.9.3-py3-none-any.whl (210 kB)
[pipenv.exceptions.InstallError]: ERROR: Exception:
[pipenv.exceptions.InstallError]: Traceback (most recent call last):
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
[pipenv.exceptions.InstallError]:     status = run_func(*args)
\x1b[0m'', u''\x1b[91m[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_internal/cli/req_command.py", line 247, in wrapper
[pipenv.exceptions.InstallError]:     return
    func(self, options, args)
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_internal/commands/install.py", line 400, in run
[pipenv.exceptions.InstallError]:     requirement_set = resolver.resolve(
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
[pipenv.exceptions.InstallError]:     result = self._result = resolver.resolve(
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
[pipenv.exceptions.InstallError]:     state = resolution.resolve(requirements, max_rounds=max_rounds)
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_vendor/resolvelib/resolvers.py", line 348, in resolve
[pipenv.exceptions.InstallError]:     self._add_to_criteria(self.state.criteria, r, parent=None)
[pipenv.exceptions.InstallError]:   File
    "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
[pipenv.exceptions.InstallError]:     if not criterion.candidates:
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
[pipenv.exceptions.InstallError]:     return bool(self._sequence)
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
[pipenv.exceptions.InstallError]:     return any(self)
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
[pipenv.exceptions.InstallError]:     return (c for c in iterator if id(c) not in self._incompatible_ids)
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/found_candidates.py",
    line 47, in _iter_built
[pipenv.exceptions.InstallError]:     candidate = func()
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link
[pipenv.exceptions.InstallError]:     self._link_candidate_cache[link] = LinkCandidate(
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/candidates.py", line 301, in __init__
[pipenv.exceptions.InstallError]:     super().__init__(
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/candidates.py", line 163, in __init__
[pipenv.exceptions.InstallError]:     self.dist = self._prepare()
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/candidates.py", line 232, in _prepare
[pipenv.exceptions.InstallError]:     dist
    = self._prepare_distribution()
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/candidates.py", line 312, in _prepare_distribution
[pipenv.exceptions.InstallError]:     return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_internal/operations/prepare.py", line 491, in prepare_linked_requirement
[pipenv.exceptions.InstallError]:     return self._prepare_linked_requirement(req, parallel_builds)
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_internal/operations/prepare.py", line 536, in _prepare_linked_requirement
[pipenv.exceptions.InstallError]:     local_file = unpack_url(
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_internal/operations/prepare.py", line 166, in unpack_url
[pipenv.exceptions.InstallError]:     file
    = get_http_url(
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_internal/operations/prepare.py", line 107, in get_http_url
[pipenv.exceptions.InstallError]:     from_path, content_type = download(link, temp_dir.path)
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_internal/network/download.py", line 147, in __call__
[pipenv.exceptions.InstallError]:     for chunk in chunks:
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_internal/cli/progress_bars.py", line 52, in _rich_progress_bar
[pipenv.exceptions.InstallError]:     with progress:
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_vendor/rich/progress.py", line 1165, in __enter__
[pipenv.exceptions.InstallError]:     self.start()
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_vendor/rich/progress.py",
    line 1156, in start
[pipenv.exceptions.InstallError]:     self.live.start(refresh=True)
[pipenv.exceptions.InstallError]:   File "/usr/local/lib/python3.9/site-packages/pipenv/patched/pip/_vendor/rich/live.py", line 132, in start
[pipenv.exceptions.InstallError]:     self._refresh_thread.start()
[pipenv.exceptions.InstallError]:   File "/usr/lib64/python3.9/threading.py", line 899, in start
[pipenv.exceptions.InstallError]:     _start_new_thread(self._bootstrap, ())
[pipenv.exceptions.InstallError]: RuntimeError: can\''t start new thread

Steps to Reproduce

1. Try to run image build FROM registry.cern.ch/inveniosoftware/almalinux:1

Expected behavior

That the image build like it used to (and does with FROM inveniosoftware/centos7-python:3.9).

build: Python 3.7 build fails

The command '/bin/sh -c curl https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz -O -J -L && tar xzf Python-3.7.0.tgz -C /usr/src/ && /usr/src/Python-3.7.0/configure --enable-optimizations --prefix=/usr/bin/python3 && make altinstall && ln -s /usr/bin/python3/bin/python3.7 /usr/bin/python3.7 && rm Python-3.7.0.tgz' returned a non-zero code: 2

More here.

Vulnerability alerts for services

How to scan for vulnerabilities for Docker images used in the services? (e.g. nginx, rabbitmq, redis, etc...)

Unpin pipenv

Due to #28 Currently fix by downpining in #34

pipenv currently emits a confusing error message when used with --system, so pin it to a version where this wasn't happening. A fix has been committed upstream in pypa/pipenv@cce026e but hasn't been released yet (latest tested 2020.6.2).

Wait for new release and test again.

Upgrade Node.js to v18

Node.js v18 is not yet fully compatible with webpack 4 and the webpack build fails.
When this is solved, upgrade to Node.js v18.

global: migration to github-actions from travis