A statically linked, native, platform agnostic Git-based Package Manager written in Rust.
- 1. Install
- 2. Background
- 3. Features
- 4. Security
- 5. Build
- 6. Getting started
- 7. Authentication
- 8. Package reference notation
- 9. Matching package references
- 10. Working with multiple package repositories
- 11. Logging
- 12. Commands
- 13. Integrations
- 14. FAQ
- 15. Contributing
- 16. License
- Linux:
curl -Ls https://github.com/aerys/gpm-packages/raw/master/gpm-linux64/gpm-linux64.tar.gz | tar xvz
- Windows:
- cmd.exe:
curl -Ls https://github.com/aerys/gpm-packages/raw/master/gpm-windows64/gpm-windows64.tar.gz | tar xvzf -
- PowerShell:
$tmp = New-TemporaryFile ; Invoke-WebRequest -OutFile $tmp https://github.com/aerys/gpm-packages/raw/master/gpm-windows64/gpm-windows64.tar.gz ; tar xf $tmp
- cmd.exe:
As a software company, we use Git to manage our source code. Thus, we wanted to use the same Git features we know and love to manage the packages built from that source code.
We also use Git-enabled collaborative platforms (GitLab, GitHub, Gitea...) to:
- Build, package and deploy our code using CI/CD.
- Authenticate and authorize clients to report issues and manage projects.
And we wanted to leverage those collaborative features to effortlessly distribute packages to authorized users.
Introducing GPM: the Git-based Package Manager.
GPM is a platform-agnostic package manager leveraging Git capabilities to store, install and update packages. Thanks to GPM:
- Any Git repository becomes a package repository.
- Any Git-powered collaborative platform (GitLab, GitHub, Gitea...) becomes a package management and distribution platform.
- Free, open source and decentralized package management
- 100% backed by the Git protocol
- Fully compatible with the Git ecosystem (GitLab, GitHub, Gitea...):
- commit and manage packages via your favorite Git client
- authentication (password, private key, deploy key, deploy token...)
- commit packages via Git in CI pipelines
- protected package version branches/tags
- release page integration (ex: GPM binary package releases)
- Git LFS support
- release notes via Git tag messages
- Distribute any package for any platform as a simple
*.tar.gz
archive - Semver support
- Public package repositories
- Private package repositories with authentication:
- username/password, deploy token
- SSH private key, deploy key
- Modern, intuitive and clean CLI interactive mode
- Non-interactive mode for automation/deployments
- Transparent SSH config (
~/.ssh/config
) discovery for authentication - Support for all the common Git protocols (HTTP(s), SSH...)
- Security backed by standard libraries (libgit2, libssh2, OpenSSL...)
- Lightweight (<2,5Mo) fully static 0 dependency binary
- Cross-platform: Windows, Linux, Android (Termux), macOS (untested)
- One-liner installation
- Local cache to speedup package discovery and matching
GPM leverages other standard well maintained open source libraries for all sensitive operations:
- All Git operations (clone, pull...) are performed using the Rust bindings for libgit2.
- All SSH operations (Git LFS authentication) are performed using the Rust bindings for libssh2.
- All SSL operations (HTTPS, SSH key management) are performed using the Rust bindings for OpenSSL.
Dependencies:
- OpenSSL
cargo build
Dependencies:
- Docker
docker run \
--rm -it \
-v "$(pwd)":/home/rust/src \
-v "/home/${USER}/.cargo":/home/rust/.cargo \
ekidd/rust-musl-builder \
cargo build --release --target x86_64-unknown-linux-musl
- Create a Git LFS enabled Git repository, for example a GitHub or GitLab repository.
- Install Git LFS on your local computer.
- Clone the newly created repository on your local computer:
git clone ssh://path.to/my/package-repository.git
cd package-repository
- Enable Git LFS tracking for
*.tar.gz
files:
git lfs track "*.tar.gz"
- Add, commit and push
.gitattributes
:
git add .gitattributes
git commit .gitattributes -m "Enable Git LFS."
git push
Voilà! You're all set to publish your first package!
In this example, we're going to create a simple hello-world
package and publish it.
- Make sure you are at the root of the package repository created in the previous section.
- Create and enter the package directory:
mkdir hello-world && cd hello-world
- Create the
hello-world.sh
script:
echo "#/bin/sh\necho 'Hello World!'" > hello-world.sh
- Create your package archive:
tar -cvzf hello-world.tar.gz hello-world.sh
- Add and commit your package archive:
git add hello-world.tar.gz
git commit hello-world.tar.gz -m "Publish hello-world version 0.1.0"
- Tag your package release with a specific version number:
git tag hello-world/0.1.0
- Push your new package:
git push
git push --tags
Your hello-world/0.1.0
package is now stored in your package repository and can be installed using gpm
!
- Install (or build)
gpm
. - Add your package repository to the
gpm
sources:
mkdir -p ~/.gpm
echo "ssh://path.to/my/package-repository.git" >> ~/.gpm/sources.list
- Update the
gpm
cache:
gpm update
- Install your package:
gpm install hello-world=0.1.0 --prefix ~/
The version 0.1.0
of your hello-world
package is now installed and you can run it with sh ~/hello-world.sh
.
gpm
will behave a lot like git
regarding authentication.
If the repository is "public", then no authentication should be required.
Otherwise, the following authentication methods are supported:
- URL encoded HTTP basic authentication (ex:
https://username:[email protected]/repo.git
); - SSH public/private key.
If URL encoded HTTP basic authentication is used, no additional authentication is required.
Otherwise, gpm
will assume SSH public/private key authentication is used.
If SSH public/private key authentication is used:
- If the
GPM_SSH_KEY
environment variable is set to a path that exists/is a file, then its value is used as the path to the SSH private key. - Otherwise, if
gpm
can find the~/.ssh/config
file, parse it and find a matching host with theIndentityFile
option; then the corresponding path to the SSH private key will be used. - Otherwise, if
gpm
can find the~/.ssh/id_rsa
file, it is used as the SSH private key. - Otherwise,
gpm
will continue without authentication.
If the SSH private key requires a passphrase, then:
- If the
GPM_SSH_PASS
environment variable is set/not empty, it is used as the passphrase. - Otherwise,
gpm
will prompt the user to type his passphrase.
This is the most trivial, obvious and simple notation: simply use the package name.
Example:
gpm install my-package
gpm
will search by name for the specified package in all the available package
repositories. Thus, for such package reference to be found, you must make sure:
- The corresponding package repository remote is listed in
~/.gpm/sources.list
(see Working with multiple package repositories). - The cache has been updated by calling
gpm update
.
The complete URI notation for a package is as follow:
${remote-uri}#${package}
remote-uri
: the full URI to the Git remote.package
: a shorthand or${name}=${revision}
package reference.
Example:
gpm install ssh://github.com/my/awesome-packages.git#my-package
In this case, gpm
will clone the corresponding Git repository and look for the package there.
gpm
will look for the specified package only in the specified repository.
The version of a package can be specified using the SemVer version requirement notation:
${package}${semver_req}
package
: the name of the package (ex:my-package
).semver_req
: the semver version requirement (ex:^0.4.2
). If not specified, then the latest version will be installed.
It also allows parsing of ~x.y.z
and ^x.y.z
requirements as defined
at https://www.npmjs.org/doc/misc/semver.html.
Tilde requirements specify a minimal version with some updates:
~1.2.3 := >=1.2.3 <1.3.0
~1.2 := >=1.2.0 <1.3.0
~1 := >=1.0.0 <2.0.0
Caret requirements allow SemVer compatible updates to a specified verion,
0.x
and 0.x+1
are not considered compatible, but 1.x and
1.x+1 are.
0.0.x is not considered compatible with any other version. Missing minor and patch versions are desugared to 0 but allow flexibility for that value.
^1.2.3 := >=1.2.3 <2.0.0
^0.2.3 := >=0.2.3 <0.3.0
^0.0.3 := >=0.0.3 <0.0.4
^0.0 := >=0.0.0 <0.1.0
^0 := >=0.0.0 <1.0.0
Wildcard requirements allows parsing of version requirements of the formats
*
, x.*
and x.y.*
.
* := >=0.0.0
1.* := >=1.0.0 <2.0.0
1.2.* := >=1.2.0 <1.3.0
Example:
gpm install my-package>=2.0.0
A package version can also be set to a valid
Git refspec,
such as a specific branch or a tag, using the @
operator:
${package}@${refspec}
package
is the name of the package (ex:my-package
).refspec
is a valid Git refspec (ex:refs/heads/my-branch
orrefs/tags/my-tag
).
The following section explains how gpm
finds the package archive for a
package named ${package_name}
at version ${package_version}
.
The following pseudo code explains how GPM will find packages for a specific version:
if ${package_ref} includes remote URL
git clone URL
if ${package_ref} does not include version
${package_version} is set to "@master"
for each ${repository} in cache
git checkout master
git reset --hard
if ${package_version} is refspec
git checkout ${package_version}
else # assume ${package_version} is semver
for each ${tag} in ${repository}
if ${tag} matches semver
git checkout ${tag}
if file ${package_name}/${package_name}.tgz exists
if ${package_name}/${package_name}.tgz is LFS link
download ${package_name}/${package_name}.tgz
extract ${package_name}/${package_name}.tgz
Specifying a full package URI might not be practical. It's simpler to specify
a package refspec and let gpm
find it. But where should it look for it?
When you specify a package using a refspec, gpm
will have to find the proper
package repository. It will look for this refspec in the repositories listed
in ~/.gpm/sources.list
.
The following command lines will fill sources.list
with a few (dummy)
package repositories:
echo "ssh://path.to/my/package-repository.git" >> ~/.gpm/sources.list
echo "ssh://path.to/my/another-repository.git" >> ~/.gpm/sources.list
echo "ssh://path.to/my/yet-another-repository.git" >> ~/.gpm/sources.list
# ...
After updating sources.list
, don't forget to call gmp update
to update the
cache.
You can then install packages using their refspec.
Logs can be enable by setting the GPM_LOG
environment variable to one of the
following values:
trace
debug
info
warn
error
Example:
GPM_LOG=info gpm update
Logs can be very verbose. So it's best to keep only the gpm
and gitlfs
module logs. For example:
GPM_LOG="gpm=debug,gitlfs=debug" gpm install hello-world/1.0.0
Update the cache to feature the latest revision of each repository listed in
~/.gpm/sources.list
.
Example:
# first add at least one remote
echo "ssh://github.com/my/awesome-packages.git" >> ~/.gpm/sources.list
echo "ssh://github.com/my/other-packages.git" >> ~/.gpm/sources.list
# ...
# then you can run an update:
gpm update
Clean the cache. The cache is located in ~/.gpm/cache
.
Cache can be rebuilt using the update
command.
gpm clean
Download and install a package.
Example:
# install the "app" package at version 2.0.0 from repository
# ssh://github.com/my/awesome-packages.git in the /var/www/app folder
gpm install ssh://github.com/my/awesome-packages.git#app/2.0.0 \
--prefix /var/www/app
# assuming the repository ssh://github.com/my/awesome-packages.git is in
# ~/.gpm/sources.list and the cache has been updated using `gpm update`
gpm install app/2.0.0 --prefix /var/www/app
Download a package in the current working directory.
Example:
# install the "app" package at version 2.0.0 from repository
# ssh://github.com/my/awesome-packages.git in the /var/www/app folder
gpm download ssh://github.com/my/awesome-packages.git#app/2.0.0 \
--prefix /var/www/app
# assuming the repository ssh://github.com/my/awesome-packages.git is in
# ~/.gpm/sources.list and the cache has been updated using `gpm update`
gpm download app/2.0.0 --prefix /var/www/app
Here is a working Bash script to publish a package from Travis CI: script/publish.sh.
Here is a working PowerShell script to publish a package from AppVeyor: script/publish.ps1.
Here is a template to publish a package from GitLab CI:
.gpm_publish_template: &gpm_publish_template
stage: publish
image:
name: alpine/git:v2.24.1
entrypoint: ["/bin/sh", "-c"]
only:
- tags
resource_group: ${PACKAGE_REPOSITORY}
before_script:
- apk add git-lfs
- git lfs install --skip-repo
script:
- cd ${PACKAGE_ARCHIVE_ROOT} && tar -zcf /tmp/${PACKAGE_NAME}.tar.gz ${PACKAGE_CONTENT} && cd -
- mkdir -p ~/.ssh && echo -e "Host *\n StrictHostKeyChecking no\n IdentityFile /tmp/package-repository-key" > ~/.ssh/config
- GIT_LFS_SKIP_SMUDGE=1 git clone ${PACKAGE_REPOSITORY} /tmp/package-repository
- mkdir -p /tmp/package-repository/${PACKAGE_NAME}
- mv /tmp/${PACKAGE_NAME}.tar.gz /tmp/package-repository/${PACKAGE_NAME}
- cd /tmp/package-repository/${PACKAGE_NAME}
- git config --global user.email "${GITLAB_USER_EMAIL}"
- git config --global user.name "${GITLAB_USER_NAME}"
- git add ${PACKAGE_NAME}.tar.gz
- git commit ${PACKAGE_NAME}.tar.gz -m "Publish ${PACKAGE_NAME} version ${PACKAGE_VERSION}."
- git tag -F "${PACKAGE_CHANGELOG}" "${PACKAGE_NAME}/${PACKAGE_VERSION}"
- git push
- git push --tags
and an example of how to use this template:
publish:
<<: *gpm_publish_template
variables:
PACKAGE_VERSION: ${CI_COMMIT_TAG} # the version of the package
PACKAGE_REPOSITORY: ssh://my.gitlab.com/my-packages-repository.git # the package repository to publish to
PACKAGE_NAME: ${CI_PROJECT_NAME} # the name of the package
PACKAGE_ARCHIVE_ROOT: ${CI_PROJECT_DIR}/bin # the folder containing the files to put in the package archive
PACKAGE_CONTENT: * # the globbing pattern/file list to put in the package archive
PACKAGE_CHANGELOG: ${CI_PROJECT_DIR}/CHANGELOG.md # the path to the changelog file
This template relies on the PACKAGE_REPOSITORY_KEY
GitLab CI variable.
It must contain a passphrase-less SSH deploy key (with write permissions) to
your package repository.
Each package can have its changelog available via the GitLab Releases page of the corresponding GPM package repository.
The following GitLab CI job templates can be used to create a GitLab release for a package:
.gitlab_release_template: &gitlab_release_template
image: alpine:3.11.5
only:
- tags
dependencies:
- changelog
before_script:
- apk add jq curl
script: |
curl --fail --retry 5 \
-X POST \
-H 'Content-Type: application/json' \
-H "PRIVATE-TOKEN: ${RELEASE_AUTH_TOKEN}" \
https://gitlab.example.com/api/v4/projects/${RELEASE_PROJECT_ID}/releases
-d "$(
jq -n \
--arg name "${RELEASE_NAME}" \
--arg tag_name "${RELEASE_TAG}" \
--arg description "$(cat ${RELEASE_NOTES})" \
{
name: $name,
tag_name: $tag_name,
description: $description
}
)"
RELEASE_AUTH_TOKEN
: the GitLab Personal Access Token used for authentication.RELEASE_PROJECT_ID
: the ID of the GitLab package repository project to create the release on. (ex:42
)RELEASE_TAG
: the tag of the released GPM package (ex:my-awesome-package/0.42.3
).RELEASE_NAME
: the name of the released GPM package (ex:My Awesome Package 0.42.3
).RELEASE_NOTES
: the path to the changelog file of the released GPM package.
Here is an example of a CI job using that template:
release:
<<: *gitlab_release_template
variables:
RELEASE_AUTH_TOKEN: ${GITLAB_TOKEN}
RELEASE_TAG: my-awesome-package/${CI_COMMIT_TAG}
RELEASE_PROJECT_ID: 42
RELEASE_NAME: My Awesome Package ${CI_COMMIT_TAG}
RELEASE_NOTES: ${CI_PROJECT_DIR}/CHANGELOG.md
Each package can have its changelog available via the GitHub Releases page of the corresponding GPM package repository.
The gpm-packages release page is a good example of such integration.
The following curl
command can be used to create a release for a package:
curl --fail --retry 5 \
-X POST \
-H "Content-Type:application/json" \
-H "Authorization: token ${RELEASE_AUTH_TOKEN}" \
https://api.github.com/repos/${RELEASE_PROJECT}/releases \
-d "$(
jq -n \
--arg tag_name "${RELEASE_TAG}" \
--arg name "${RELEASE_NAME}" \
--arg body "$(cat ${RELEASE_NOTES})" \
'{
tag_name: $tag_name,
name: $name,
body: $body,
draft: false,
prerelease: false
}'
)"
RELEASE_AUTH_TOKEN
: the GitHub Personal Access Token used for authentication.RELEASE_PROJECT
: the namespaced name of the GitHub package repository to create the release on (ex:aerys/gpm-packages
).RELEASE_TAG
: the tag of the released GPM package (ex:my-awesome-package/0.42.3
).RELEASE_NAME
: the name of the released GPM package (ex:My Awesome Package 0.42.3
).RELEASE_NOTES
: the path to the changelog file of the released GPM package.
Such release creation mechanism can easily be integrated in the CI.
GPM's own GitLab CI configuration features a reusable job template to do exactly that. The result is visible on the release page of the gpm-packages repository.
GPM means "Git-based Package Manager".
The main motivation is to have a platform-agnostic package manager, mainly aimed at distributing binary packages as archives. GPM can be used to leverage any Git repository as a package repository.
Platforms like GitLab and GitHub are then very handy to manage such package archives, permissions, etc...
GPM is also available as an all-in-one static binary. It can be leveraged to download some packages that will be used to bootstrap a more complex provisioning process.
GPM aims at leveraging the Git ecosystem and features.
Git is great to manage revisions. So it's great at managing package versions! For example, Git is also used by the Docker registry to store Docker images.
Git also has a safe and secured native authentication/authorization strategy through SSH. With GitLab, you can safely setup deploy keys to give a read-only access to your packages.
Yes. Cloning a repository full of large binary files can take a lot of time and space. You certainly don't want to checkout all the versions of all your packages everytime you want to install one of them.
That's why you should use Git LFS for your GPM repositories.
Thanks to Git LFS, GPM will download the a actual binary package only when it is required.
Vanilla Git will compress objects. But Git LFS doesn't store objects in the actual Git repository: they are stored "somewhere else".
To make sure we don't use too much disk space/bandwidth "somewhere else", the package archive is stored compressed.
MIT