pyca / infra Goto Github PK
View Code? Open in Web Editor NEWInfrastructure configuration for pyca projects (mostly dockerfiles)
License: Other
Infrastructure configuration for pyca projects (mostly dockerfiles)
License: Other
As discussed in IRC, we're going to use a single node docker swarm to run containers that need access to secrets. Things to do:
docker swarm init
refs #42
I've been playing with the latest manylinux-related standards and verified that it's fairly easy to create images for manylinux_2_24 and include ppc64le+s390x (these are the only archs PyPA has base images for).
Here's what I've got: https://github.com/orgs/ansible/packages?ecosystem=container&q=pylibssh-
I needed a few minor adjustments to get this working (like using apt for 2_24 b/c it's Debian 9 based). If you're interested, I could contribute the patches back to this repo. Let me know :)
After cryptography 39 is released.
CentOS 6 is EOL since November 30th, 2020. The main repositories appear to be offline:
Error: Cannot find a valid baseurl for repo: base
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
Eg. Invalid release/repo/arch combination/
removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/6/base/mirrorlist.txt
Is fixed upstream in pypa/manylinux#838, so should just be a cache of rebuilding the pyca image with the updated base pypa one.
Upstream bug: pypa/manylinux#836
We have a variety of sensitive credentials we should find a shared, secure storage for. This is lower priority than getting the new CI system fully functional, but once that's done we can talk about approaches.
Supposedly this can be done with --sessionTimeout=<minutes>
when starting jenkins. Maybe we can do this with JENKINS_OPTS
? (https://github.com/jenkinsci/docker/blob/master/jenkins.sh)
Credentials are dependent on #30 but write a bit about temporarily enabling ssh, shutting down VMs, running commands from https://esxi-patches.v-front.de/ESXi-6.5.0.html, rebooting, and booting the VMs back up.
Should also try to capture how to create new VMs I guess.
This is a tracking issue for switching the PR builder job over:
Other outstanding questions:
Whenever we add a new docker container to the pyca/infra
jenkins job we need to create a corresponding repo on docker hub. We should document that requirement.
No longer supported pypa/manylinux#1437 and we removed it in pyca/cryptography for 40.0.
However, we need to keep the image around until we are ready for 40.
https://hub.docker.com/r/pyca now houses our docker builds. I originally set it up as a single repo cryptography-runner
with tags like jessie
and jessie-libressl-2.4.5
, but it seemed like docker wanted us to use separate repos with mostly just latest
tags. Accordingly, I switched it to separate repos, but it's not clear to me this has gained us much. Should we keep down this path or switch back? We need to be able to build every image on push to master and then ideally be able to trigger a jenkins job to pull down the new images/clean out the old ones when they're all done building. The former is possible with either approach and I'm not sure if the latter is possible either way.
One small disadvantage of the separate repo approach is that we have to create the repo on the docker hub side whenever we create a new one (with a single repo we'd just need to add a new config entry to the existing repo).
We need to:
This profile does not have any public repositories
Perhaps this is related to the recent docker hub plan pricing changes?
Use setcap cap_net_bind_service=+ep ./caddy
to allow listening on ports<1024
PR up at pyca/bcrypt#108
Not needed once 41 is released (our 40 CI still builds against it)
sh install_openssl.sh ${ARCH}
is not actually passing ARCH as we expect.
See the output in https://ci.cryptography.io/blue/organizations/jenkins/pyca%2Finfra/detail/PR-120/1/pipeline/50
This should be x86_64 but it's configuring for 686 because it's not seeing an arg provided to the script. It appears that the OpenSSL config script might be "smart" enough to ignore what we tell it and compile for 64-bit anyway (since the 64-bit manylinux1 wheels definitely work), but we should still fix this as it's basically only working due to luck.
registry.access.redhat.com/ubi9/ubi gives us a RHEL9.x base image if we want to build one. If we do that, do we drop FIPS testing on CentOS 9 stream?
They shouldn't do this.
We need systemd unit files to launch caddy & jenkins.
How can we get this under config management in some fashion? @alex you found something that jenkins supports that might help with this right?
tasks.py
release automation (pyca/bcrypt#178)tasks.py
release automation (pyca/pynacl#539)Related to the same server that we run jenkins on:
A subset of jobs are created manually and pointed at pipeline scripts in our git repo. Document their (limited) config.
They're coming soon!
xenial and latest
(which is 17.04 as of a few days ago) would both probably be appropriate. This is a separate task because it's lower priority than getting the other stuff working.
The requirement for this is what RTD needs to over HSTS preloading for custom domains.
This is not an immediate priority but added for tracking: https://jenkins.io/doc/book/pipeline/shared-libraries/
This should allow us to reuse the builder hash -> jobs code.
So I was recovering support for manylinux1 for another project and I wanted to keep the build-arg so I attempted to pass it to install_libffi.sh
. To my surprise, it kept being equal to an empty string. After some digging, I realized that ARG
is declared before FROM
and so it is out of scope for the build (it's vaguely documented in the docs but isn't always obvious). So I fixed this by re-declaring it after that line. So at this point, I had the following in the Dockerfile
:
ARG RELEASE
FROM quay.io/pypa/${RELEASE}
ARG RELEASE
...
RUN sh install_libffi.sh "${RELEASE}"
ADD install_openssl.sh /root/install_openssl.sh
ADD openssl-version.sh /root/openssl-version.sh
RUN sh install_openssl.sh
...
What could possibly go wrong? Well, that RUN sh install_openssl.sh
kept failing with something like this:
--> 45c19e24b84
STEP 12: RUN sh install_openssl.sh manylinux1
+ OPENSSL_URL=https://www.openssl.org/source/
+ source /root/openssl-version.sh
++ export OPENSSL_VERSION=openssl-1.1.1h
++ OPENSSL_VERSION=openssl-1.1.1h
++ export OPENSSL_SHA256=5c9ca8774bd7b03e5784f26ae9e9e6d749c9da2438545077e6b3d
755a06595d9
++ OPENSSL_SHA256=5c9ca8774bd7b03e5784f26ae9e9e6d749c9da2438545077e6b3d755a065
95d9
++ export 'OPENSSL_BUILD_FLAGS_WINDOWS=no-ssl3 no-ssl3-method no-zlib no-share
d no-comp no-dynamic-engine'
++ OPENSSL_BUILD_FLAGS_WINDOWS='no-ssl3 no-ssl3-method no-zlib no-shared no-co
mp no-dynamic-engine'
++ export 'OPENSSL_BUILD_FLAGS=no-ssl3 no-ssl3-method no-zlib no-shared no-com
p no-dynamic-engine enable-ec_nistp_64_gcc_128'
++ OPENSSL_BUILD_FLAGS='no-ssl3 no-ssl3-method no-zlib no-shared no-comp no-dy
namic-engine enable-ec_nistp_64_gcc_128'
+ curl -#O https://www.openssl.org/source//openssl-1.1.1h.tar.gz
####################################################################### 100.0%
+ check_sha256sum openssl-1.1.1h.tar.gz 5c9ca8774bd7b03e5784f26ae9e9e6d749c9da
2438545077e6b3d755a06595d9
+ local fname=openssl-1.1.1h.tar.gz
+ local sha256=5c9ca8774bd7b03e5784f26ae9e9e6d749c9da2438545077e6b3d755a06595d
9
+ echo '5c9ca8774bd7b03e5784f26ae9e9e6d749c9da2438545077e6b3d755a06595d9 open
ssl-1.1.1h.tar.gz'
+ sha256sum -c openssl-1.1.1h.tar.gz.sha256
openssl-1.1.1h.tar.gz: OK
+ rm openssl-1.1.1h.tar.gz.sha256
+ tar zxf openssl-1.1.1h.tar.gz
+ PATH=/opt/perl/bin:/opt/rh/devtoolset-2/root/usr/bin:/usr/local/sbin:/usr/lo
cal/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ pushd openssl-1.1.1h
~/openssl-1.1.1h ~
+ ./config no-ssl3 no-ssl3-method no-zlib no-shared no-comp no-dynamic-engine
enable-ec_nistp_64_gcc_128 --prefix=/opt/pyca/cryptography/openssl --openssldi
r=/opt/pyca/cryptography/openssl
Operating system: x86_64-whatever-Linux
This system (Linux) is not supported. See file INSTALL for details.
Error: error building at STEP "RUN sh install_openssl.sh manylinux1": error while running runtime: exit status 1
WAAAAAAAT?
I had no idea why this was happening so I tried different things like downgrading openssl and comparing to old-and-working dockerfiles.
After a while, it struck me: it seems like some build script internals of OpenSSL itself relied on the ${RELEASE}
env vars somewhere internally, and re-exposing it with the second ARG
made it available and set to some unsupported value during build-time. And that is why it kept failing. ๐คฏ
So I just wanted to document that there's a mysterious connection between ${RELEASE}
and install_openssl.sh
(the make
invocation inside really) that can be implicitly triggered by adding that second ARG
instruction.
I've patched it for myself by unsetting RELEASE
at the beginning of the script:
diff --git a/cryptography-manylinux/install_openssl.sh b/cryptography-manylinux/install_openssl.sh
index d3f7789..d0618b3 100755
--- a/cryptography-manylinux/install_openssl.sh
+++ b/cryptography-manylinux/install_openssl.sh
@@ -1,6 +1,8 @@
#!/bin/bash
set -xe
+unset RELEASE
+
OPENSSL_URL="https://www.openssl.org/source/"
source /root/openssl-version.sh
I figured that you may want to use that build-arg as a conditional in the future and wanted to prevent others from stepping on the same rake. That said, you may want to consider either renaming this var to be something more unique (maybe PYCA_
-prefixed) or injecting those unset
instructions into the scripts urging people not to remove them.
You may also choose not to care about this corner case which is fine. FWIW my job here is done, the main motivation was to document this.
Hi folks,
I occasionally look into PyCA projects and want to borrow some bits related to testing and packaging as they often contain great reusable ideas.
Would you mind putting a LICENSE file into this repo so that I (and others) would know the implications of copying things from here?
Thanks in advance!
Ideally this would be automated, but let's document the steps first.
Next step after #44
Hi folks,
JFYI GH improved the privilege model of the GHCR+GITHUB_TOKEN secret combo two days ago: https://github.blog/changelog/2021-03-24-packages-container-registry-now-supports-github_token/.
I've verified that it actually works in my repo:
Go to each image page Package settings
-> Action access
tab. For example, https://github.com/orgs/pyca/packages/container/cryptography-manylinux2010/settings/actions_access
Click Add repository
-> type in infra
, select it.
Replace s/GHCR_TOKEN/GITHUB_TOKEN/g
in https://github.com/pyca/infra/blob/main/.github/workflows/build-docker-images.yml (this is not the first step to ensure you grant the access first).
3a. I noticed there's secrets.GHCR_TOKEN_USER
but it seems unnecessary, GH docs suggest the event actor so I just have ${{ github.actor }}
in my workflows, you probably should use this too.
Remove GHCR_TOKEN
from https://github.com/pyca/infra/settings/secrets/actions.
Delete this token however it's called from your personal user (or bot?) account at https://github.com/settings/tokens
Windows Server 2016 supports containers. Investigate whether we can build a container with all the Python/Visual Studio versions and replace our server 2012 VMs.
Ideally there'd be some sort of auto-magic webhook thingy here.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.