paritytech/scripts

Collection of Dockerfiles, scripts and related resources

Parity Scripts & Dockerfiles

Nowadays, this repo is mostly the open collection of the company's Dockerfiles that are used by Parity in different ways. If you work on a Polkadot SDK-based project, you could be interested in our CI image (ci-unified) in the dockerfiles directory.

But what about scripts?

scripts mostly is a legacy name. Yes, this repo contains some scripts, but they are kept here for historical reasons. We are not actively maintaining them. If you are looking for something specific, please, open an issue and we will try to help you.

Additional information

• We use the ci-unified image for most of our CI pipelines (including the pipelines for Polkadot SDK-based/Polkadot SDK-related projects and Polkadot SDK itself). You can find more information about the image in the dockerfiles/ci-unified directory.
• Most of the images are published to Docker Hub and could be found here.
• If you have access to Parity's internal GitLab, please have a look at this project's pipeline schedules. You can use them to build Docker/OCI images on demand.

Legacy notes

• Reproduce CI locally

scripts's Issues

Sqash and merge button can be helpful to this repo.

brew linkapps depreciation warning

Ran the install script and Partiy installed without an issue. However, I do get the following warning

Warning: `brew linkapps` has been deprecated and will eventually be removed!

Unfortunately `brew linkapps` cannot behave nicely with e.g. Spotlight using
either aliases or symlinks and Homebrew formulae do not build "proper" `.app`
bundles that can be relocated. Instead, please consider using `brew cask` and
migrate formulae using `.app`s to casks.

Excuting a binary on M1 according to instructions is not working for me

At the very bottom of the README, it says:

⚠️ If you want to execute a binary on OS X pay attention that with docker it is compiled for Linux. So if want to run it you need to use something like:
cargoenvhere paritytech/contracts-ci-linux:production cargo run

Can someone reproduce this or am I doing something wrong?

Build Dockerfiles as part of CI

Today I almost merged a PR with a set of Dockerfiles that wouldn't build. We should run
some checks as part of a CI workflow to ensure that Dockerfiles still build after being
updated.

install-parity.sh: PARITY_DEB_URL does not use HTTPS

line 4:
PARITY_DEB_URL=http://d1h4xl4cr1h0mo.cloudfront.net/v${PARITY_VERSION}/x86_64-unknown-linux-gnu/parity_${PARITY_VERSION}_amd64.deb

later:
curl -L $PARITY_DEB_URL > $file
sudo dpkg -i $file

I'd really rather not install packages form an insecure connection. Changing the url to use https seems to work, so maybe that's a really quick fix.

Parity install script failing for specific version

The Parity client install script fails when using a specific version. Example:
bash <(curl https://get.parity.io -Lk) -r v1.10.6

dpkg-deb: error: '/tmp/tmp.VAwOMY2BP7/parity_1.10.6_ubuntu_amd64.deb.md5' is not a debian format archive

The problem is that the installation script will accidentally download the .md5 file instead of the .deb file.
https://raw.githubusercontent.com/paritytech/scripts/master/get-parity.sh

This began failing when the versioned vanity-service began returning .md5 files.
https://vanity-service.parity.io/parity-binaries?architecture=x86_64&format=markdown&os=linux&version=stable

Simple fix in installation script is to filter out md5 files:
MD=$(curl -Ss ${LOOKUP_URL} | grep -v sha256 | grep deb | grep -v md5)

get-parity.sh installs parity beta version 1.7.3 after stable release of 1.7.5

The stable version 1.7.5 of parity was released yesterday. But the widely popular script still installs older beta version of parity.

Create new image `releng-scripts`

A new image is required for https://github.com/paritytech/releng-scripts/blob/master/Dockerfile
Since the image will have access to secrets such as publishing to S3, the security needs to be checked carefully.

• make it a rootless image
• harden image
• add Dockerfile to this repo

Review get-substrate.sh Script

According to @bkchr, the get-substrate.sh script is likely very out-of-date. It may be worth reevaluating its value and soundness. For starters, notice that the following line installs git twice:
https://github.com/paritytech/scripts/blob/master/get-substrate.sh#L35

install failed on macos

I was following the instructions on the main README and when I got to the point to download and run the install script, I received the following error:

installer: Error the package path specified was invalid: '/parity-1.8.2-macos-installer.pkg'.

I tracked it down to the mktemp -d command returning an empty string in this script.

I found the downloaded installer package in my home directly and was able to run the command for darwin to finish the install.

I'm not sure how to go about trying to fix the script and testing it, but wanted to open the issue in case anyone else has an idea.

This was on OS X Yosemite Version 10.10.5

ci-unified: fix substrate-contracts-node rand-extension integration

The ink rand-extension integration in substrate-contracts-node is broken because frame_support no longer re-exports log.

For now I'm working around that by building a new ci-unified without this integration; will do a proper fix and re-enable it soon.

one-liner get_substrate is broken

When I do
curl https://getsubstrate.io -sSf | bash

the build fails because wasm isn't built anymore:

   Compiling substrate-consensus-aura v1.0.0 (https://github.com/paritytech/substrate#b4464e12)
   Compiling substrate-cli v1.0.0 (https://github.com/paritytech/substrate#b4464e12)
   Compiling substrate-finality-grandpa v1.0.0 (https://github.com/paritytech/substrate#b4464e12)
   Compiling node-cli v1.0.0 (https://github.com/paritytech/substrate#b4464e12)
   Compiling node-executor v1.0.0 (https://github.com/paritytech/substrate#b4464e12)
error: couldn't read /home/abrenzikofer/.cargo/git/checkouts/substrate-7e08433d4c370a21/b4464e1/node/executor/src/../../runtime/wasm/target/wasm32-unknown-unknown/release/node_runtime.compact.wasm: No such file or directory (os error 2)
  --> /home/abrenzikofer/.cargo/git/checkouts/substrate-7e08433d4c370a21/b4464e1/node/executor/src/lib.rs:26:100
   |
26 | native_executor_instance!(pub Executor, node_runtime::api::dispatch, node_runtime::native_version, include_bytes!("../../runtime/wasm/target/wasm32-unknown-unknown/release/node_runtime.compact.wasm"));
   |                                                                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

error: Could not compile `node-executor`.
warning: build failed, waiting for other jobs to finish...
error: failed to compile `substrate v1.0.0 (https://github.com/paritytech/substrate#b4464e12)`, intermediate artifacts can be found at `/tmp/cargo-installkop8aG`

Caused by:
  build failed
Cloning into '/tmp/tmp.2lBrl5k0N3'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 130 (delta 3), reused 8 (delta 3), pack-reused 121
Receiving objects: 100% (130/130), 31.66 KiB | 549.00 KiB/s, done.
Resolving deltas: 100% (70/70), done.
Run source ~/.cargo/env now to update environment
a

The libp2p-js-echo-server image can be removed

It was initially for libp2p's integration tests.
However libp2p is now mature enough that we can test it against an actual IPFS node, and we don't need this image anymore.

cc libp2p/rust-libp2p#817

One-line Installer not able to install homebrew

import openethereum/parity-ethereum#2474

Error in rustup update nightly

I have a Dockerfile whose parent image is paritytech/ci-linux:974ba3ac-20201006.

When trying to build my dockerfile, the command "RUN rustup update nightly" yields the following error:

`=> ERROR [ 8/12] RUN rustup update nightly

[ 8/12] RUN rustup update nightly:
#11 8.878 info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
#11 10.69 info: latest update on 2021-09-13, rust version 1.57.0-nightly (51e514c0f 2021-09-12)
#11 10.73 info: downloading component 'rust-std' for 'wasm32-unknown-unknown'
#11 14.32 info: downloading component 'cargo'
#11 16.20 info: downloading component 'rust-std'
#11 21.47 info: downloading component 'rustc'
#11 32.63 info: removing previous version of component 'rust-std' for 'wasm32-unknown-unknown'
#11 43.67 info: rolling back changes
#11 43.67 error: could not rename component file from '/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/wasm32-unknown-unknown/lib' to '/usr/local/rustup/tmp/p3ccm91_ff8xhqyw_dir/bk'
#11 43.67 error: caused by: other os error
#11 43.68 error: backtrace:
#11 43.68 error: 0: error_chain::backtrace::imp::InternalBacktrace::new
#11 43.68 1: rustup::utils::utils::rename
#11 43.68 2: rustup::dist::component::transaction::Transaction::remove_dir
#11 43.69 3: rustup::dist::component::components::Component::uninstall
#11 43.69 4: rustup::dist::manifestation::Manifestation::update
#11 43.69 5: rustup::dist::dist::update_from_dist_
#11 43.69 6: rustup::install::InstallMethod::install
#11 43.69 7: rustup::toolchain::DistributableToolchain::install_from_dist
#11 43.69 8: rustup::cli::rustup_mode::update
#11 43.69 9: rustup::cli::rustup_mode::main
#11 43.69 10: rustup_init::main
#11 43.69 11: std::rt::lang_start_internal::{{closure}}::{{closure}}
#11 43.69 at rustc/c7087fe00d2ba919df1d813c040a5d47e43b0fe7/src/libstd/rt.rs:52
#11 43.69 std::sys_common::backtrace::__rust_begin_short_backtrace
#11 43.69 at rustc/c7087fe00d2ba919df1d813c040a5d47e43b0fe7/src/libstd/sys_common/backtrace.rs:130
#11 43.69 12: main
#11 43.69 13: __libc_start_main
#11 43.69 14:
#11 43.69

---

executor failed running [/bin/sh -c rustup update nightly]: exit code: 1
`

I'm aware this is a problem of compatibility between rustup and docker (the problem lies in the rename() function, more specifically in the fact that docker's filesystem behavior and rustup filesystem's behavior are not compatible) , as per rust-lang/rustup#2542.

The only suggested workaround so far is uninstalling and reinstalling rustup in its entirety, but it seems that nightly already tries to install the most recent version (1.57.0-nightly (51e514c0f 2021-09-12)), so I'm not sure why that works.

Do you have any idea can I go about this in a different way?

One-line installer sometimes exits 0 after installation fails

The one-line installer is currently failing. But it is exiting zero despite the failure:

$ bash <(curl https://get.parity.io -L)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   170    0   170    0     0    421      0 --:--:-- --:--:-- --:--:--   420
100  3854  100  3854    0     0   5201      0 --:--:-- --:--:-- --:--:--  5201
Release selected is: beta
Upgrading parity from 0.0.0 to 2.1.2
curl: no URL specified!
curl: try 'curl --help' or 'curl --manual' for more information
/usr/bin/sha256sum: /tmp/tmp.87X5NkX5zl/parity: No such file or directory
/dev/fd/63: line 84: sudo: command not found
rm: cannot remove '/tmp/tmp.87X5NkX5zl/*': No such file or directory
$ echo $?
0

The current failure is because vanity-service.parity.io has not been updated to include v2.0.7 or v2.1.2. Once that gets updated, you can reproduce this issue by running get-parity.sh locally and manually changing VERSION_BETA to "10000.0.0".

Update `ci-linux` to Rust 1.64

Currently it sits on Rust version 1.62.1 which is quite outdated. Apparently wasmi requires Rust version 1.64 so GitLab runners currently fail. This probably just requires a Docker image rebuild.

One liner should use `*-release` from vanity service

Currently it fetches for instance from stable which may contain a newer, yet unreleased version.

Supported Ubuntu Version

Script supports Ubuntu versions which already reached end of life, but not 16.4 LTS.

https://github.com/paritytech/scripts/blob/master/install-parity.sh#L220

One-liner downloads SHA256 file instead of actual installer on MacOS

When I try to install parity with the one-liner, the actual file downloaded to my machine is parity_1.9.3_macos_macos.pkg.sha256 instead of the actual pkg file, so the installer is failing. On this line, the script selects the second index from the cut command, but based on the output I'm getting in my console for the value of $MD, it should be selecting the 3rd index via -f3.

get-parity.sh fails to upgrade to 2.0.0 beta

Linux deb files are no longer published in 2.0.0 so it fails to set $DOWNLOAD_FILE. Here's the output with echoed variables:

Release selected is: beta
LOOKUP: 'https://vanity-service.parity.io/parity-binaries?architecture=x86_64&format=markdown&os=linux&version=beta-release'
LIBSSL: '10'
DOWNLOAD_FILE: ''
basename: missing operand
Try 'basename --help' for more information.
Existing version of parity: 1.11.6 is newer than the version you attempting to install: 

So it evaluates this branch https://github.com/paritytech/scripts/blob/master/get-parity.sh#L49 but the deb file does not exist: https://vanity-service.parity.io/parity-binaries?architecture=x86_64&format=markdown&os=linux&version=beta-release

Should install `libssl-dev` on Debian/Ubuntu

New or Modified get-substrate script specifically for the node template

I would like to reopen #218 for a discussion on an update to this script (substrate-developer-hub/substrate-node-template#129)

I would propose we change or add behaviour to more like the many framework's default init or create app flow (react for example).
I suggest that we modify so that this can be a "one liner" to get from a virgin OS (no dependencies) to:

• Attempt to Identify the environment and OS it is running in (a container {docker, VM or otherwise}, or bare metal) and configure the rest of the script accordingly
  • Error/Warning messages if it cannot be sure or is blocked that are informative of what to do (i.e. "this script doesn't work on Windows, use WSL. Here is a how-to (link)")
• **(Extra/option, highly encouraged when ready!!) Pull a compiled docker image and use it as a base to greatly reduce dependencies **
  • ideally useable by podman by default?
  • Thus dependencies would be already taken care of and the script could skip a few steps below.
  • Sets up IDE to use the container as a development environment (in VSCode for example, we already have this in the node template)
• rust installed & configured (like the present template MakeFile does)
• Clone a detached HEAD state of a release of the node template
  • (optional/extra) Use the template github feature to create a new repo for you based on the most recent release base node template as the remote for the git repo cloned (so you can push immediately, and the repo shows that our template was used)
• cd into the working directory of the new project
• Run make init && make build

Open to help build out a script for these features if others think this is useful! Would be great to get some collaborators on it with me as well to get polish and add features.

If we did do this, we should have the getsubstrate.io or some other HTTPS endpoint to pull this script from automatically update to use the latest version

apt-get update has insufficient permissions on ubuntu

## Populating apt-get cache...

 + apt-get update
W: chmod 0700 of directory /var/lib/apt/lists/partial failed - SetupAPTPartialDirectory (1: Operation not permitted)
E: Could not open lock file /var/lib/apt/lists/lock - open (13: Permission denied)
E: Unable to lock directory /var/lib/apt/lists/
W: Problem unlinking the file /var/cache/apt/pkgcache.bin - RemoveCaches (13: Permission denied)
W: Problem unlinking the file /var/cache/apt/srcpkgcache.bin - RemoveCaches (13: Permission denied)
E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission denied)
E: Unable to lock the administration directory (/var/lib/dpkg/), are you root?
Error executing command, exiting

Can't seem to see why. The script clearly prefixes it with sudo.

Generalize one-line installer

AFTER: openethereum/parity-ethereum#8887

We have to rethink the one-line installer logic after we remove debian/centos pipelines.

Now, that libssl is not an issue anymore, I would suggest just looking:

IF system IS windows THEN
    GET parity.exe FROM windows
ELSIF system IS macos THEN
    GET parity FROM macos
ELSIF system IS linux THEN
    CHECK architecture 
    GET parity FROM linux AND architecture

I know you want to take this @fevo1971 :D

Why not use cachepot instead of sccache

Cachepot is under paritytech already and i am using it as a sccache replacement and works OK. Why still keep the sccache in the build process and docker images?

One-line installer doesn't detect package installation failure

http://pastebin.com/MmmWeyRs

Says Parity has been installed

Inconsistencies between docs and get-parity.sh

In the setup documentation it states you won’t get the cutting edge features from the bash script however it appears to be installing the latest beta version while I would expect it to be installing a stable release.

It also goes on to say this script will also offer to install the Netstats client and connect it to ethstats.net but this appears to be implemented in the install-deps.sh script which is not referenced in the get-parity.sh script and no such prompt is given. Working on confirming the install-deps.sh in addition to the get-parity.sh would successfully setup netstats.

Image `awscli`: check for scheduled build

The image awscli returns a rather old version so I guess we don't rebuild/publish on schedule.

Our image currently return:

aws-cli/1.19.81 Python/3.6.13 Linux/5.15.49-linuxkit botocore/1.20.81

The latest aws cli reports:

aws-cli/2.10.3 Python/3.11.2 Darwin/22.3.0 source/x86_64 prompt/off