openkilt/openrepo

Open Source repository management for deb, rpm, and generic packages

openrepo

OpenRepo is a web-based server for managing and hosting repositories containing Debian apt/deb, Redhat rpm, and generic package files.

The server supports:

• RPM, Deb, Generic repository generation and hosting compatible with Debian/Ubuntu apt-get and RedHat yum tools
• Package upload, deletion, copying, and promotion (e.g., for easily moving packages through dev, QA, beta, production repos)
• PGP signing key creation and management
• Version management
• User read/write access control for each repo
• REST API
• CLI app to integrate with CI

Getting Started

The preferred method for running OpenRepo is with Docker using the provided docker-compose.yml configuration file. This will run the necessary services as well as instantiate a PostgreSQL database. All persistent files (i.e., the database, cache data, PGP keys, and the package files) are stored in a relative folder named openrepo-data.

First ensure that you have installed Docker and the Docker Compose plugin

To start the server:

wget https://raw.githubusercontent.com/openkilt/openrepo/master/docker-compose.yml
docker-compose up -d

You can now navigate to the server on http://localhost:7376

The default credentials are:

username: admin
password: admin

If desired, it is possible to point to an alternative PostgreSQL server by updating the "OPENREPO_PG" environment variables in the docker-compose file.

CI Integration

A common requirement is to automatically upload package files produced via Continuous Integration. Please see the OpenRepo Command-Line-Interface documentation for more details.

The CLI program (or REST API) can be used to push new packages to a repo, and can also be used to promote or copy packages to other repos.

Users and Permissions

There are two levels of users:

1. Super User - Has read/write access to all repositories as well as administrative access to add/remove users, keys, and permissions
2. Regular User - Has read access to all repositories. Write access must be granted explicitly for each repository

Two add a new user:

1. As the super user, click on "System Admin" from the menu in the top-right
2. Click on the "Add" button next to the Users link
3. Add a username and password and click "Save" - An API key is automatically created. This can be deleted to disallow API access
4. To enable write access, click on the "Repositories" link, then click the repository where you wish this user to have write access. Add the user to this list and save.

REST API

Repo actions:

Repo UID is created when a new repo is created.

# list names of repos along with IDs
GET /api/repos/

# Show details for a particular repo
GET /api/<repo>/

# Create a new repo
POST /api/repos/

# Delete a repo
DELETE /api/<repo>/

Package actions:

Package UID is created when a new package is uploaded or copied

# List packages for a particular repo
GET /api/<repo>/packages/

# Upload a package to a repo
POST /api/<repo>/upload/

# Delete a package
DELETE /api/<repo>/pkg/<package>/

# Show details for a particular package
GET /api/<repo>/pkg/<package>/

# Copy a package to another repo
POST /api/<repo>/pkg/<package>/copy/

Signing Key actions:

The signing key ID is the fingerprint of the PGP key and is created when the key is uploaded or created

# List all signing keys
GET /api/signingkeys/

# Create a new signing key
POST /api/signingkeys/

# Delete a signing key
DELETE /api/signingkeys/<signingkey>/

Development

Architecture

OpenRepo consists of four running processes:

Nginx web server

The web server hosts the static file content. This includes the "frontend" generated content (Vue/Vuetify) as well as the images and repo files.

The web server also serves as a proxy for the Django endpoints. These are primarily the REST API and the admin interface.

The Nginx web port is the only port that should be exposed to network traffic.

The Django app server

The app server hosts the REST API which is the primary way for the frontend and CLI to interact with the application. There are also a few static pages (e.g., the admin interface, password change forms, etc) that are proxied through to Django.

The Django worker

The worker runs as a background process and communicates exclusively with the database server. The Django worker is responsible for generating metadata when the repos are updated (i.e., packages are uploaded or deleted). This process uses OS tools to create the repos and symlinks the files to their appropriate locations. Some repo generating tools may make use of a cache to store things such as hash information to speed up subsequent repo updates.

The Database

By default OpenRepo uses PostgreSQL. Using other databases are possible (e.g., SQLite to simplify development), however PostgreSQL is recommended for production.

Dev Env Setup

Running the above components individually is the best way to test modifications to the source code.

The first step is to add a file named web/openrepo/settings_local.py and apply any environment variable overrides for development.

For example, the following settings_local.py file will configure your environment to use developer-friendly settings.

import os

os.environ["OPENREPO_VAR_DIR"] = "/var/tmp/openrepo/"
os.environ["OPENREPO_DEBUG"] = "TRUE"
os.environ["OPENREPO_DB_TYPE"] = "sqlite"
os.environ["OPENREPO_LOGLEVEL"] = "DEBUG"

Next, open four separate tabs and run the following commands:

Tab 1: cd web; ./manage.py runserver
Tab 2: cd web; ./manage.py runworker
Tab 3: cd frontend; npm run dev
Tab 4: nginx -c /storage/projects/openrepo/deploy/nginx/nginx.conf.dev

Next, navigate to http://localhost:5173/ to see your code updates. Both the Vue.js dev server and the Django dev server support live updates on code changes.

openrepo's Issues

Can't change hardcoded port 8080

Hi, I was modify docker-compose to expose port 80

version: "3.9"
services:
  nginx:
    image: openkilt/openrepo:latest
    command: nginx
    restart: unless-stopped
    ports:
      - "80:80"
    depends_on:
      - "django"
    volumes:
      - ./openrepo-data:/var/lib/openrepo

But when I'm try to curl repo without trailing slash, OpenRepo return url with port number

√ ~ % curl -v http://openrepo.opentech.local/infra-el8  <--- request without port number
*   Trying 100.100.101.22:80...
* Connected to openrepo.opentech.local (100.100.101.22) port 80 (#0)
> GET /infra-el8 HTTP/1.1
> Host: openrepo.opentech.local
> User-Agent: curl/7.87.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.18.0 (Ubuntu)
< Date: Thu, 11 May 2023 13:06:34 GMT
< Content-Type: text/html
< Content-Length: 178
< Location: http://openrepo.opentech.local:8080/infra-el8/  <-------- problem here
< Connection: keep-alive
<
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>
* Connection #0 to host openrepo.opentech.local left intact

I was try to resolve this via definition env OPENREPO_DOMAIN to host:80 - seems this option isn't working for this case as well

Interested in Pulp Collaboration?

Hi Matt,

I work on the Pulp project for Red Hat, which also has Debian and Redhat support, yum/dnf integration, etc. Is there any interest in discussing combining these efforts? The software stack seems very similar, Python, Django, nginx, etc. One thing we're interested in is having a UI for Pulp, which openrepo has.

If the answer isn't 100% no, would you be willing to have a 30 minute call? You can email me at bmbouter at redhat com if you're willing to meet.

The project looks great so far.

-Brian

Cannot login behind reverse proxy

Hi,

I have found your project and wanted to test it out. I deployed using docker-compose and can access the website and login just fine using http://myLocalIP:7376. I have HAProxy running as reverse proxy for HTTPS SSL offloading. The website is accessible fine, however when I try to login at https://apt.mydomain.com, I get a 403 error:

The log shows:
WARNING log:241: Unauthorized: /api/whoami
WARNING log:241: Unauthorized: /api/repos/
WARNING log:241: Unauthorized: /api/signingkeys/
WARNING log:241: Forbidden (Origin checking failed - https://apt.mydomain.com does not match any trusted origins.): /admin/login/

Searching online, I have found this qestion on StackOverflow, which seems to be related. However, I am not a programmer.

If you need further info or testing, please let me know.

[Suggestion] Update Repo Instructions to make them more 'copy&paste-able'

Hi Matt, it's me again 😅

I have a small suggestion, like the title says. Currently the Repo Instructions look like this:

apt update && apt install -y curl gnupg
curl https://mydomain.com/repo/public.gpg | gpg --yes --dearmor -o /usr/share/keyrings/openrepo-repo.gpg
echo "deb [arch=any signed-by=/usr/share/keyrings/openrepo-repo.gpg] https://mydomain.com/repo/ stable main" > /etc/apt/sources.list.d/openrepo-repo.list
apt update

I would modify them to add sudo in some choice places, so that the commands can be copy-pasted into a non-root terminal, like so:

sudo apt update && sudo apt install -y curl gnupg
curl https://mydomain.com/repo/public.gpg | sudo gpg --yes --dearmor -o /usr/share/keyrings/openrepo-repo.gpg
echo "deb [arch=any signed-by=/usr/share/keyrings/openrepo-repo.gpg] https://mydomain.com/repo/ stable main" | sudo tee /etc/apt/sources.list.d/openrepo-repo.list
sudo apt update

Since redirecting echo into privileged files doesn't work in a non-root terminal, I pipe to sudo tee instead. For similar reasons I added sudo to the gpg command on line 2. apt should be self-explanatory.

Please let me know what you think and thanks for an awesome project. Already getting some nice use out of it 😊

Repo Builds are not sorted by newest first

On the build status screen the builds are sorted alphabetically, instead of being sorted by build number. Due to that the alphabetically smallest build is shown by default when opening the screen, instead of the newest.

Add a REST API option to sign package with repo's private key on package upload

Hi! I've been playing around with openrepo tonight and I'm not sure exactly how I'm supposed to use the signing key with an rpm repo.

I can create a new signing key in the UI and associate it with the repo.
I can upload a (unsigned) rpm using the openrepo CLI.
When I dnf install the rpm, dnf complains that the rpm is not signed.

My expectation was that openrepo upload would sign the rpm as part of the upload, or alternatively, provide a means for me to obtain the signing key so that I can sign the rpm as part of the build process.

I don't see how to do either of those things.

What is the flow that you recommend for this?
My ultimate goal is to publish rpms to the repo from a GH actions CI workflow (and also debs at a later date).

Thanks in advance!

debian based operating system OS upgrade

We tried for the applications to update it's working fine. Please let me know the steps to create a repository for an upgraded OS, such as Linux mint 20 to Linux 21.1,

CLI unable to upload new package

Hi Matt,

I'm trying to upload a new package to my openrepo via CLI like so:

openrepo_cli_x86_64 -k <KEY> -s https://openrepo.my-doman.com upload -r repo package.deb

That yields the following error:

Error.  Received response code 400
{"detail":"Error processing uploaded file"}

The logs from openrepo-django-1 show:
WARNING log:241: Bad Request: /api/repo/upload/

This has worked in the past. I think it stopped working around March-ish. I have already tried the 'new' openrepo_cli_x86_64 from March 6, same behavior. All containers are up-to-date.

Any idea what is going on?

Ability to sort packages in the repo view by name

Currently on the repo screen packages are sorted by upload order. It would be very helpful if the column headers where clickable and would then sort packages by that column

rpm packages with same version but different releases are not distinguished and conflict

RPM packages have a version and a release - generally within the name: <version>-<release>. But openrepo only considers the version, so if you upload same version but newer release, it will conflict with the older release, and unless you specify overwrite then it will fail. In our use-case we want to retain some older releases in the repository so we can rebuild prior versions. We can adopt to using only version (adding a new .<microversion>) but there are lots of pre-existing RPMs with the <version>-<release> convention and supporting this would be a good thing IMO.

cli: lack of 'overwrite' param for upload command

{
  "detail":"Package intel-ethernet-722series version 5.30 already exists in destination repo openrepo and 'overwrite' is not specified"
}

multiple versions of packages

I have an issue where with an apt repo with contents like this:

a simple apt install kumomta will install version 2023.03.30.180934.427c74f0 instead of the latest 2023.06.22.135051.51b72a83.

is there some additional configuration needed for this to work?