circleci-public/cimg-base

The CircleCI Base (Ubuntu) Docker Convenience Image.

cimg-base's Issues

Investigate the viability of adding Docker's BuildKit into the image

This will also be considered for the Linux VM images.

Update Dockerize

Problems we need to address:

• the version we're using comes from a CircleCI storage location (not great for keeping up-to-date)
• the source is no longer maintained, there's a fork we can switch to
• lets implement a download hash here
• we need arm64 support
• we might be able to compile a slimmer version that what's available on GitHub

Ubuntu 22.04 becomes default starting with July edge and September current

The July monthly snapshot for this image will be the last one where Ubuntu 20.04 is the default version. Starting with the edge tag in July, Ubuntu 22.04 will become default. By the August monthly snapshot, Ubuntu 22.04 will be default in current.

Create chart graphic to explain LTS support in this image

Something like this one by Ubuntu:

Add Retry Command

Often users need to retry commands due to hiccups, flakiness, etc. It would be helpful if a retry command was added to this base image similar to this one](https://github.com/kadwanev/retry).

Assume -y for Apt

As standard in Ubuntu, in this image installing a package with Apt requires the flag --yes or more commonly, -y. Here's an example:

sudo apt-get update && sudo apt-get install -y git

The legacy Convenience Images as well as the Linux machine image avoids this by pre-configuring it within the image. This image should do the same.

First reported by @glenjamin.

`stable` tag does not have `docker-compose` installed

docker pull cimg/base:stable && docker run --rm -it cimg/base:stable docker-compose --version
stable: Pulling from cimg/base
Digest: sha256:9d94b570bdd6809833d5e12d2ca1cb56de77d71dfe1d7e2fe44070b7fbbed048
Status: Image is up to date for cimg/base:stable
docker.io/cimg/base:stable
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "docker-compose": executable file not found in $PATH: unknown.

Vs2021.11 tag:

docker pull cimg/base:2021.11 && docker run --rm -it cimg/base:2021.11 docker-compose --version
2021.11: Pulling from cimg/base
Digest: sha256:376c1dc2a8a549aa66c03ccd1c5efc8212e40b0d6138a1e465841a9dab8b1630
Status: Image is up to date for cimg/base:2021.11
docker.io/cimg/base:2021.11
docker-compose version 1.29.2, build 5becea4c

The base image is too huge

This base image is almost 1gb. That's crazy! It takes 10-30secs to download and extract this image on CircleCI, which makes jobs much slower than they have to be.

Almost half the space is spent on Docker & associated tools. I don't think these are used by the majority of CircleCI jobs. They should be moved to a -docker variant tag of this image.

Another few hundred megs is spent on C/C++ compiler infrastructure. This must be even less useful, I think you could remove it entirely.

Then there are a lot of silly mistakes. Installing vim instead of vim-tiny costs nearly 100mb. Installing software-properties-common adds a similar amount of disk.

Are you interested in a pull request to drag this image back to a sensible size?

Checkout step failed

Checkout step failed to git clone, because project dir is owned by root and not writable by circleci.

Cloning into '.'...
/home/circleci/project/.git: Permission denied

exit status 1

circleci@9405c3127ad2:~$ ls -ld /home/circleci/project/
drwxr-xr-x 2 root root 4096 Jan  2 00:04 /home/circleci/project/

Image version:

Status: Downloaded newer image for cimg/base:stable
  using image cimg/base@sha256:563d629e330773aae63e800b431500761bf1597a765de5c9d92529ddadb330b7

Where is SSH_CONFIG_DIR set?

I would assume this image sets this value to /home/circleci/.ssh, but it seems to not do so. Where does the default checkout find this env variable then? transparency--

More robust tests than simply `docker build`ing the images`

add some goss tests, etc.

Might need alternative git source

We use the popular/defacto git ppa. This says to support arm64 but so far I'm having trouble. If I continue to have trouble, we might need to find a different source or use system git, which means we'll go back to an older release.

Supporting multiple Ubuntu versions at once

The plan for this image is to always use exclusively Ubuntu Long Term Support (LTS) releases. Right now, this image is based on the current LTS, Ubuntu 18.04. The next LTS, Ubuntu 20.04, will be released in 6 months.

I believe that it's important to support that new release the day it comes out. The current architecture of this image means that to do so would mean dropping 18.04 images that same day. I don't believe that to be ideal either. I'm thinking maybe we support the previous LTS for a year. That gives users 12 months to migrate any package support from the old LTS to the new one. This seems fair especially considering that for smaller and/or more simlistic projects, the LTS change likely won't affect projects at all.

I propose we do this by utilizing "parent variants", which I'm thinking the rest of the images will support as well in order to have the node/browser variant, which will allow us to build both Ubuntu 18.04 and 20.04 images at the same time.

Include `amazon-ecr-credential-helper` in default package list

Hello,

What is the process for having more default packages including in the base image... specifically I would like to include amazon-ecr-credential-helper

Build images for arm64

Currently the image is only available for intel architecture, as the AWS Graviton is now GA having the images ready for arm64 architecture will save a lot of time.

Docker Compose 2.10.1

Blocked by #187

Include psql?

Related to #61

Is it possible to also include postgresql-client package?
psql executable is used under the hood in the bundle exec db:structure:load task in Rails.

It fails with the ruby-2.6-node image:

Try `n` for Node version management

Previously, I thought n wouldn't work for us, since it requires an existing installation of Node, but now that we install Node LTS already, we could layer on n to provide version management. It seems to be simpler, more stable, and more regularly updated than nvm:

https://github.com/tj/n

Upgrade `yq` to `v4.26.1` for CVE fix

Hi all,

would it be possible to upgrade the yq version from currently v4.23.1 to 4.26.1 or newer?
With that version, they have upgraded a vulnerable dep on golang.org/x/net that is reported by our security scanner:

usr/local/bin/yq (gobinary)

Total: 1 (HIGH: 1, CRITICAL: 0)

┌──────────────────┬────────────────┬──────────┬────────────────────────────────────┬───────────────────────────────────┬───────────────────────────────────────────────────────────┐
│     Library      │ Vulnerability  │ Severity │         Installed Version          │           Fixed Version           │                           Title                           │
├──────────────────┼────────────────┼──────────┼────────────────────────────────────┼───────────────────────────────────┼───────────────────────────────────────────────────────────┤
│ golang.org/x/net │ CVE-2021-44716 │ HIGH     │ v0.0.0-20210813160813-60bc85c4be6d │ 0.0.0-20211209124913-491a49abca63 │ golang: net/http: limit growth of header canonicalization │
│                  │                │          │                                    │                                   │ cache                                                     │
│                  │                │          │                                    │                                   │ https://avd.aquasec.com/nvd/cve-2021-44716                │
└──────────────────┴────────────────┴──────────┴────────────────────────────────────┴───────────────────────────────────┴───────────────────────────────────────────────────────────┘

Start using --no-install-recommends

When using Apt, use the --no-install-recommends flag.

This is a breaking change so saving it for the next LTS.

Add `~/.local/bin` to PATH.

Or at least confirm that we're already doing this.

Related: https://github.com/circleci/circleci-images/issues/420

What are the use cases and value of including Power Shell in this image?

Proposed as an idea by Kyle.

Add package `software-properties-common`

So that we can use PPAs properly.

Nana123

The `latest` tag for this image is publishing

The Docker tag latest is publishing for this image which is a bug.

Use compare-url orb to only publish images if Dockerfiles are modified

random commits to master shouldn't result in tags being republished unless Dockerfiles are actually modified

libncurses5-dev in stable-20.4 5 days ago, now is not present

Switch Default Ubuntu version to 20.04

Aiming for the middle of the month. This would immediately hit the edge tag, and then on September 2nd, the next monthly snapshot.

Add Locale/Ecoding Info

This image needs to set the locale info for UTF-8.

Like this: https://github.com/kyrofa/github-snap-builder-docker/blob/master/Dockerfile.core18#L34-L40

Add dotnet sdk convenience images

Aligned with https://hub.docker.com/_/microsoft-dotnet-sdk

DEBIAN_FRONTEND=noninteractive

This needs to be set.

Basic Checkout of build fails on 22.04, works on 20.04

I did a basic run checkout step using cimg/base:2022.06-22.04 and cimg/base:2022.06-20.04....
Why is 22.04 failing? but 20.04 works?

Error Seen when failing

Using SSH Config Dir '/home/circleci/.ssh'
git version 2.36.1
Cloning git repository
Cloning into '.'...
error: cannot create async thread: Operation not permitted
fatal: fetch-pack: unable to fork off sideband demultiplexer

exit status 128

Simple config

# Use the latest 2.1 version of CircleCI pipeline process engine. See: https://circleci.com/docs/2.0/configuration-reference
version: 2.1
# Use a package of configuration called an orb.
orbs:
  # Declare a dependency on the welcome-orb
  welcome: circleci/[email protected]
jobs:
  build:
    working_directory: /home/circleci/project
    docker:
     # - image: cimg/node:12.20.2
      - image: cimg/base:2022.06-22.04
     # - image: cimg/base:2022.06-20.04

    steps:
      - checkout
      - run: echo $(ls -a) 
      
workflows:
  version: 2
  test:
    jobs:
        - build:
            context: 
              - smanley1-XXX17-Sndbx

Include the CircleCI CLI?

@KyleTryon and I had a discussion on the CircleCI CLI within CircleCI builds. Right now, it may look like the CLI is installed on every build but this is a portion of the CLI that is injected at runtime.

Currently none of our images, legacy or next-gen, pre-installs the CircleCI CLI. Should we?

Two cons I can think of:

• more MBs of space taken in the image
• the CLI at times updates frequently, this image will update monthly

A pro:

• orb development - the pack and publish commands are important

Anything else to add @CircleCI-Public/cpeng?

Add apt pkg autoconf

bionic-security is not refreshed prior to image startup

If Ubuntu pulls a package from the distribution for security reasons, then it will become uninstallable. Such is the case with zstd which no longer is installable with the snapshotted APT package repo from when the image was cut.

running /bin/bash -c "sudo" "apt" "install" "zstd"

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package zstd
Error: exit status 100

This error goes away if you explicitly run sudo apt update prior to installing any packages.

The sample won't pass as cowsay is not in the system path

The revised version of config.yml is something like this:

jobs:
  build:
    docker:
      - image: cimg/base:2022.02
    steps:
      - checkout
      # Whatever you want to do
      - run: sudo apt-get update && sudo apt-get install -y cowsay
      - run: /usr/games/cowsay Continuous Integration Rocks!

Ship a `circleci` system user

The Next-Gen images were suppose to ship with just a root user. That was the plan.

The security gains by using a regular user with sudo access are irrelevant in the world of Docker because:

1. With sudo they get root privileges anyway
2. With passwordless sudo, there's no additional security of a password.

Due to legacy image migration concerns, having sudo around would be useful. I'm not sure if useful enough on its own, but there are CircleCI platform features that expect to see a circleci home directory.

With little negative affect to our images and lots of positive affect for migration, we should reintroduce the circleci user to images.

Last two monthly snapshots weren't created

Can you create this image please?

Hi team!

Happy customer here that can be happier if we solve an issue I'm having.

Spin-up time is still 40+ seconds time for me, which it's hella crazy, and can't reduce it further. I need GCP + Node to run my deployments since we're deploying our apps to Google Cloud.

I'm using this image that a guy did because it combined both. Can u guys create an image for gcp + node or provide me any solution? I don't know anything about docker btw lol.

NikolajDL/gcloud-sdk-nodejs#2

Thanks

Include libs for popular databases?

When using the Docker executor, it's commonplace for CircleCI 2.0+ to use secondary/service images for databases such as MariaDB and PostgreSQL rather than having that database installed in the primary image itself. This is how the legacy images operate to this day.

To connect to the database, it's done over TCP, a private IP between the primary and secondary container.

For some apps, such as a Ruby app, connecting to the DB is typically done with the pg gem. This gem requires libpq-dev installed in order to work. Other languages can have similar needs for both PostgreSQL and other DBs.

Some of our legacy images (such as circleci/ruby) has this lib installed and some don't.

How should we handle this?

We can:

1. have a user just install it if they need it
2. pre-install this lib (and similar) in the base image so that all language images get it
3. pre-install it in specific language images on a case-by-case basis
4. take care of it in an orb

Solutions 2-3 cause image sizes to rise. We likely can't just cater to one DB, so then we should add more. More being... all the DBs we officially support in images? That's about 5.

Solution 4 I like because we can also install the CLIs for these DBs as well which some may prefer. Gives up flexibility there and keeps the images lean. The downside is another step up front for a user to setup and it will slightly add to a job's runtime.

Thoughts? Opinions?

Verify Git Version

Check if the git version we include is new enough. We may want to use the semi-official PPA for a newer version.

Remove stable tag from DevHub

This should occur some time after #149 is merged. Perhaps 2 months?

To be clear, this intent is to prevent the tag from appearing. It'll still be a working tag for anyone using it.

Checkout step of build fails after release on 19 May 2022

Our build started failing after the latest release a few hours ago of cimg/base:current.
It is failing on the checkout step:

Using SSH Config Dir '/home/circleci/.ssh'
git version 2.36.1
Cloning git repository
Cloning into '.'...
Warning: Permanently added the ECDSA host key for IP address '140.82.113.3' to the list of known hosts.
remote: Enumerating objects: 43931, done.        
remote: Counting objects: 100% (1363/1363), done.        
remote: Compressing objects: 100% (568/568), done.        
remote: Total 43931 (delta 959), reused 1107 (delta 782), pack-reused 42568        
Receiving objects: 100% (43931/43931), 38.30 MiB | 39.98 MiB/s, done.
Resolving deltas: 100% (30259/30259), done.
Checking out branch
fatal: unsafe repository ('/mnt/ramdisk' is owned by someone else)
To add an exception for this directory, call:

	git config --global --add safe.directory /mnt/ramdisk

exit status 128

Here are the steps in this job that get to this point:

  build-and-push:
    docker:
      - image: cimg/base:current
    working_directory: /mnt/ramdisk
    steps:
      - gcp-gcr/gcr-auth:
          registry-url: gcr.io,us-west1-docker.pkg.dev
      - checkout
      - ......

How do you suggest we resolve this issue?
Are we doing something wrong in our job, or is there a bug in this release?

Environment Details

Environment details from "Spin up environment" step:

Build-agent version 1.0.123804-6379e05c (2022-05-18T21:49:50+0000)
System information:
 Server Version: 20.10.15
 Storage Driver: overlay2
  Backing Filesystem: xfs
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Kernel Version: 5.13.0-1021-aws
 Operating System: Ubuntu 20.04.4 LTS
 OSType: linux
 Architecture: x86_64

Starting container cimg/base:current
Warning: No authentication provided, using CircleCI credentials for pulls from Docker Hub.
  image cache not found on this host, downloading cimg/base:current
current: Pulling from cimg/base
Digest: sha256:087e0c2f4f3702c9080b726e240ec4e649eb3242cc2b453c8a49476a40bc50d1
Status: Downloaded newer image for cimg/base:current
cimg/base:current:
  using image cimg/base@sha256:087e0c2f4f3702c9080b726e240ec4e649eb3242cc2b453c8a49476a40bc50d1
  pull stats: Image was already available so the image was not pulled
  time to create container: 740ms
Time to upload agent and config: 509.777606ms
Time to start containers: 375.07039ms

It worked a few hours ago when based on cimg/base@sha256:a394761a8abe3dbb25170529be8687c5d47eea2cbe118e37a7b49adb48692d45

`docker version` command fails due to daemon connection

As I understand, this image should include both Docker and Docker Compose. When I run docker version in a job, it fails with the following error; it appears the Docker daemon isn't started or is misconfigured.

#!/bin/bash -eo pipefail
docker version
Client: Docker Engine - Community
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfc
 Built:             Thu Aug 29 05:29:11 2019
 OS/Arch:           linux/amd64
 Experimental:      false
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Exited with code exit status 1
CircleCI received exit code 1

Install cmake

This was pre-installed in the legacy Ruby image, and likely a few others such as Python. Installing here instead of one of those as it's a general compiling tool and will likely be needed by some of the buildpack-deps migrants once that image is deprecated.

UID for circleci user mismatch between containers and machines

Hello,

I ran into an issue with the mismatch of the circleci user id inside of the cimg images (explicitly set to UID 3434) and the machine type executor VMs. The useradd command in the Dockerfile template is here. The PR that introduced this change is #38 based on the issue #35, however neither of those issues gives a reason for using 3434 for the cimg userid. I imagine this was to avoid some sort of collision issue; however on the flip side this introduces issues when mounting things from the machine into a docker container running off a cimg based image.

As an example use case: we're using the stackhawk orb to run a hawkscan-local job. This job runs inside of a machine executor and also accepts a list of steps executed inside of containers to allow for setting up a local scan target. I used the cimg/ruby docker image for running our local scan target for consistency, and ran into this issue when mounting the repo that I checked out using a checkout step.

It's simple enough to work around this by adding a chmod step to modify permissions on the mounted directory, but this feels a bit hacky, and prone to trip up other users.

Is it possible to sync the UIDs of the circleci user between machine executors and cimg images?

Thanks,
Tim

Empty HOME var sets path to /bin

I'm digging into why $HOME is empty, but comparing dockerfile template and actual path I am reasonable confident.

Dockerfile
ENV PATH $HOME/bin:$HOME/.local/bin:$PATH

Container

root@ccc0a6f04778:~/repo# echo $PATH
/bin:/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

More robust documentation on release schedule, structure, etc.

Remove the default (and unintentional) python binary

I believe this image has a python binary installed that's getting pulled into downstream images. This was not intentional. Aiming to remove this with the 22.04 based image.

Install git-lfs

I was caught off guard when the first application I published that was built using cimg/node:lts, had users complaining about errors for missing images. I assumed anything CI related would include git and git-lfs so as to pull my entire code base.