AWX behind corporate proxy with docker-compose

[XaratosS]

AWX behind corporate proxy with docker-compose

[XaratosS]

I was executing make docker-compose-build and the following output.

make docker-compose-build
[sudo] Passwort für ansible:
Unable to import setuptools-scm, attempting to install now...
Traceback (most recent call last):
  File "/home/ansible/awx/tools/scripts/scm_version.py", line 6, in <module>
    from setuptools_scm import get_version
ModuleNotFoundError: No module named 'setuptools_scm'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ansible/awx/tools/scripts/scm_version.py", line 17, in <module>
    raise Exception(f'\nCommand `{" ".join(cmd)}` failed (rc={result.returncode}).\n\nstdout:\n{result.stdout}\n\nstderr:\n{result.stderr}')
Exception:
Command `/usr/bin/python3 -m ensurepip` failed (rc=1).

stdout:
b''

stderr:
b'/usr/bin/python3: No module named ensurepip\n'
Unable to import setuptools-scm, attempting to install now...
Traceback (most recent call last):
  File "/home/ansible/awx/tools/scripts/scm_version.py", line 6, in <module>
    from setuptools_scm import get_version
ModuleNotFoundError: No module named 'setuptools_scm'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ansible/awx/tools/scripts/scm_version.py", line 17, in <module>
    raise Exception(f'\nCommand `{" ".join(cmd)}` failed (rc={result.returncode}).\n\nstdout:\n{result.stdout}\n\nstderr:\n{result.stderr}')
Exception:
Command `/usr/bin/python3 -m ensurepip` failed (rc=1).

stdout:
b''

stderr:
b'/usr/bin/python3: No module named ensurepip\n'
ansible-playbook tools/ansible/dockerfile.yml \
        -e dockerfile_name=Dockerfile.dev \
        -e build_dev=True \
        -e receptor_image=quay.io/ansible/receptor:devel
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit
localhost does not match 'all'

PLAY [Render AWX Dockerfile and sources] *************************************************************

TASK [Gathering Facts] *******************************************************************************
ok: [localhost]

TASK [dockerfile : Create _build directory] **********************************************************
ok: [localhost]

TASK [dockerfile : Render supervisor configs] ********************************************************
ok: [localhost] => (item=supervisor_web.conf)
ok: [localhost] => (item=supervisor_task.conf)
ok: [localhost] => (item=supervisor_rsyslog.conf)

TASK [dockerfile : Render Dockerfile] ****************************************************************
changed: [localhost]

PLAY RECAP *******************************************************************************************
localhost                  : ok=4    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

DOCKER_BUILDKIT=1 docker build \
        -f Dockerfile.dev \
        -t ghcr.io/ansible/awx_devel:devel \
        --build-arg BUILDKIT_INLINE_CACHE=1 \
        --cache-from=ghcr.io/ansible/awx_devel:devel .
[+] Building 400.5s (13/50)                                                            docker:default
 => [internal] load build definition from Dockerfile.dev                                         0.0s
 => => transferring dockerfile: 7.79kB                                                           0.0s
 => [internal] load .dockerignore                                                                0.0s
 => => transferring context: 56B                                                                 0.0s
 => [internal] load metadata for quay.io/ansible/receptor:devel                                 10.6s
 => [internal] load metadata for quay.io/centos/centos:stream9                                   6.8s
 => ERROR importing cache manifest from ghcr.io/ansible/awx_devel:devel                         30.0s
 => [builder  1/10] FROM quay.io/centos/centos:stream9@sha256:3e26f56ecab0f2c62fb60eec5d963a575  0.0s
 => [internal] load build context                                                                0.0s
 => => transferring context: 47.61kB                                                             0.0s
 => CACHED FROM quay.io/ansible/receptor:devel@sha256:a1dc3d4f76a8a1133a611ac065ab4fe84144e8111  0.0s
 => https://copr.fedorainfracloud.org/coprs/ansible/Rsyslog/repo/epel-9/ansible-Rsyslog-epel-9.  0.5s
 => CACHED [builder  2/10] RUN rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial          0.0s
 => CACHED [stage-1  3/34] ADD https://copr.fedorainfracloud.org/coprs/ansible/Rsyslog/repo/epe  0.0s
 => ERROR [builder  3/10] RUN dnf -y update && dnf install -y 'dnf-command(config-manager)' &  359.8s
 => CANCELED [stage-1  4/34] RUN dnf -y update && dnf install -y 'dnf-command(config-manager)  359.9s
------
 > importing cache manifest from ghcr.io/ansible/awx_devel:devel:
------
------
 > [builder  3/10] RUN dnf -y update && dnf install -y 'dnf-command(config-manager)' &&     dnf config-manager --set-enabled crb &&     dnf -y install     gcc     gcc-c++     git-core     gettext     glibc-langpack-en     libffi-devel     libtool-ltdl-devel     make     nodejs     nss     openldap-devel     patch     postgresql     postgresql-devel     python3-devel     python3-pip     python3-setuptools     swig     unzip     xmlsec1-devel     xmlsec1-openssl-devel:
0.294 /bin/sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
0.335 Failed to set locale, defaulting to C.UTF-8
359.7 CentOS Stream 9 - BaseOS                        0.0  B/s |   0  B     05:59
359.7 Errors during downloading metadata for repository 'baseos':
359.7   - Curl error (28): Timeout was reached for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http&countme=1 [Failed to connect to mirrors.centos.org port 443: Connection timed out]
359.7   - Curl error (28): Timeout was reached for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [Failed to connect to mirrors.centos.org port 443: Connection timed out]
359.8 Error: Failed to download metadata for repo 'baseos': Cannot prepare internal mirrorlist: Curl error (28): Timeout was reached for https://mirrors.centos.org/metalink?repo=centos-baseos-9-stream&arch=x86_64&protocol=https,http [Failed to connect to mirrors.centos.org port 443: Connection timed out]
------
Dockerfile.dev:21
--------------------
  20 |     # Install build dependencies
  21 | >>> RUN dnf -y update && dnf install -y 'dnf-command(config-manager)' && \
  22 | >>>     dnf config-manager --set-enabled crb && \
  23 | >>>     dnf -y install \
  24 | >>>     gcc \
  25 | >>>     gcc-c++ \
  26 | >>>     git-core \
  27 | >>>     gettext \
  28 | >>>     glibc-langpack-en \
  29 | >>>     libffi-devel \
  30 | >>>     libtool-ltdl-devel \
  31 | >>>     make \
  32 | >>>     nodejs \
  33 | >>>     nss \
  34 | >>>     openldap-devel \
  35 | >>>     patch \
  36 | >>>     postgresql \
  37 | >>>     postgresql-devel \
  38 | >>>     python3-devel \
  39 | >>>     python3-pip \
  40 | >>>     python3-setuptools \
  41 | >>>     swig \
  42 | >>>     unzip \
  43 | >>>     xmlsec1-devel \
  44 | >>>     xmlsec1-openssl-devel
  45 |
--------------------
ERROR: failed to solve: process "/bin/sh -c dnf -y update && dnf install -y 'dnf-command(config-manager)' &&     dnf config-manager --set-enabled crb &&     dnf -y install     gcc     gcc-c++     git-core     gettext     glibc-langpack-en     libffi-devel     libtool-ltdl-devel     make     nodejs     nss     openldap-devel     patch     postgresql     postgresql-devel     python3-devel     python3-pip     python3-setuptools     swig     unzip     xmlsec1-devel     xmlsec1-openssl-devel" did not complete successfully: exit code: 1
make: *** [Makefile:582: docker-compose-build] Fehler 1

Also I do have set up the proxy for apt, curl, docker, git and wget. Those cmds work without any issue.

[XaratosS]

Just wanted to let you know, that I had to add the build-args in the Makefile. Then the download of ghcr.io/ansible/awx_devel:devel failed. I did a docker pull on this one and it worked till I got a diffrent error.

40.48   Downloading zope.interface-5.5.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (257 kB)
40.51 Requirement already satisfied: pip==21.2.4 in /var/lib/awx/venv/awx/lib/python3.9/site-packages (from -r /dev/stdin (line 477)) (21.2.4)
40.51 Requirement already satisfied: setuptools==65.6.3 in /var/lib/awx/venv/awx/lib/python3.9/site-packages (from -r /dev/stdin (line 479)) (65.6.3)
40.53 ERROR: Could not find a version that satisfies the requirement django-ansible-base (unavailable) (from versions: 0a1)
40.53 ERROR: No matching distribution found for django-ansible-base (unavailable)
40.53 WARNING: You are using pip version 21.2.4; however, version 23.3.2 is available.
40.53 You should consider upgrading via the '/var/lib/awx/venv/awx/bin/python3.9 -m pip install --upgrade pip' command.
40.93 make: *** [Makefile:153: requirements_awx] Error 1
------
Dockerfile.dev:57
--------------------
  55 |         /tmp/requirements/
  56 |
  57 | >>> RUN cd /tmp && make requirements_awx
  58 |
  59 |     ARG VERSION
--------------------
ERROR: failed to solve: process "/bin/sh -c cd /tmp && make requirements_awx" did not complete successfully: exit code: 2
make: *** [Makefile:582: docker-compose-build] Fehler 1```

[hgfi]

You need to configure your docker client to use the proxy:
https://docs.docker.com/network/proxy/#configure-the-docker-client
You also most likely need to configure the docker service to also use the proxy to pull images.

I don't have a problem with the proxy setup but make docker-compose-build fails on the same step:
40.53 ERROR: Could not find a version that satisfies the requirement django-ansible-base (unavailable) (from versions: 0a1)

I use Debian 11
Docker version 24.0.7
Docker Compose version v2.21.0

I cloned the awx repo with git clone -b 23.6.0

[XaratosS]

@hgfi thanks for the hint. I have configured the proxy for the containers in the file as seen in the docs. It works now. Interestingly I was looking through the traceback and found another issue:

13.94    command: /var/lib/awx/venv/awx/bin/python3.9 /var/lib/awx/venv/awx/lib64/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpxlnfm7k2
13.94        cwd: /tmp/pip-install-_w44_47t/django-ansible-base_3a6924f99ecf4df188e827806e793221
13.94   Complete output (15 lines):
13.94   WARNING setuptools_scm._integration.setuptools pyproject.toml does not contain a tool.setuptools_scm section
13.94   error: Multiple top-level packages discovered in a flat-layout: ['test_app', 'ansible_base'].
13.94
13.94   To avoid accidental inclusion of unwanted files or directories,
13.94   setuptools will not proceed with this build.
13.94
13.94   If you are trying to create a single distribution with multiple packages
13.94   on purpose, you should not rely on automatic discovery.
13.94   Instead, consider the following options:
13.94
13.94   1. set up custom discovery (`find` directive with `include` or `exclude`)
13.94   2. use a `src-layout`
13.94   3. explicitly set `py_modules` or `packages` with a list of names
13.94
13.94   To find more information, look for "package discovery" on setuptools docs.
13.94   ----------------------------------------

Another issue that occured is this:

0.133 python3.9: can't open file '/tmp/tools/scripts/scm_version.py': [Errno 2] No such file or directory
0.143 python3.9: can't open file '/tmp/tools/scripts/scm_version.py': [Errno 2] No such file or directory

When I am run a the python script in the bash the script runs without any issues.

~/awx$ python tools/scripts/scm_version.py
23.6.1.dev7+g42701f32fe.d20240115

[XaratosS]

Okay I figured it out. @hgfi can you add the http and https proxy to the makefile? That should fix the issue. Somehow it is not using the global environments, that I have set in /etc/environment.

Another issue that occured ist, that with make docker-compose I will get the following issue:

ansible-galaxy install --ignore-certs -r tools/docker-compose/ansible/requirements.yml;
Starting galaxy collection install process
Process install dependency map
[WARNING]: Skipping Galaxy server https://galaxy.ansible.com. Got an unexpected error when getting available versions of collection awx.awx: Unknown error when attempting to call Galaxy at
'https://galaxy.ansible.com/api/': <urlopen error [Errno 101] Das Netzwerk ist nicht erreichbar>
ERROR! Unknown error when attempting to call Galaxy at 'https://galaxy.ansible.com/api/': <urlopen error [Errno 101] Das Netzwerk ist nicht erreichbar>
make: *** [Makefile:538: docker-compose] Fehler 1

This points to a proxy issue. Sadly I am not sure, why ansible-galaxy is not taking the environment file and use the http and https proxy.

[XaratosS]

I got it all fixed up. You have to set up the proxy in the makefile before the ansible-galaxy install command.
It would look like this:

http_proxy=http://xxx.xxx.xxx.xxx:<yourPort> ansible-galaxy install....