Giter Site home page Giter Site logo

pulp_deb's Introduction

pulp_deb

Pulp Deb Release Pipeline Pulp Deb Nightly CI Pulp Deb CI Update python versions

A Pulp plugin to host APT repositories.

Visit the Pulp project homepage or see the plugin documentation for more information. If you want to report a bug, see the issue tracker.

The most important places:

pulp-deb-client

API bindings for python and ruby for talking to the pulp_deb REST API. These are automatically generated for each pulp_deb release using the pulp-openapi-generator.

The most important places:

pulp-cli-deb

pulp-cli-deb Release Pipeline pulp-cli-deb Nightly CI python versions

A pulp-cli plugin to add the commands for using pulp_deb via Pulp CLI.

The most important places:

pulp_deb's People

Contributors

bmbouter avatar daviddavis avatar dkliban avatar dralley avatar ekohl avatar fao89 avatar goosemania avatar hstct avatar ipanova avatar jaymzh avatar jschmid1 avatar jshen28 avatar lubosmj avatar m-bucher avatar manisha15 avatar masselstine avatar maximiliankolb avatar mdellweg avatar mikedep333 avatar n-pochet avatar pulpbot avatar quba42 avatar sdherr avatar thikaxer avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pulp_deb's Issues

Refactor the debian test fixtures building

Author: @quba42 (quba42)

Redmine Issue: 7442, https://pulp.plan.io/issues/7442


The idea is to use best Makefile practices.

Currently, we are building all packages multiple times for each repository.

Instead, packages should have there own make targets which are dependent targets of the repositories that need them.

Ideally all built repositories could share a common pool folder (either directly or using symlinks), so we do not build an unnecessarily large fixtures container.
This might be prohibitively hard to achieve with the current repo building tooling.

Add support "Source" indices and source level packages (*.orig.tar.gz, *.debian.tar.xz, *.dsc)

Author: markawr (markawr)
Date: 2021-05-18

Redmine Issue: 8775, https://pulp.plan.io/issues/8775


The Debian Repository Format (https://wiki.debian.org/DebianRepository/Format) allows for "Sources" indices allowing source level packages to exist alongside deb packages. This allows for 'deb-src' lines to be included in a system's sources.list or equivalent files and allows a user to use 'apt-get source <package_name>' to then inspect a package's source, make modifications and use tools such as pbuilder to build a custom version of a package.

The current Pulp Deb plugin does not handle "Source" indices and using generic_content is not really viable.

Having the Pulp Debian plugin support "sources" indices and source level packages would be a welcome addition to the plugin.

(I have reviewed past work items and didn't see previous requests or discussions on this topic, hopefully I didn't miss anything. I am willing to contribute as needed along with some colleagues but want to gauge first if this support would be accepted and desired by folks maintaining this plugin)

Failure to upload packages compressed with zstd

Version
I'm on the main branch (3055a86).

Describe the bug
When I try to upload a package into pulp_deb, it fails.

To Reproduce
I've tried with various packages but they all seem to fail. Here's an example using vim:

wget http://security.ubuntu.com/ubuntu/pool/main/v/vim/vim_8.2.2434-3ubuntu3_amd64.deb
https --form :/pulp/api/v3/content/deb/packages/ file@vim_8.2.2434-3ubuntu3_amd64.deb

Expected behavior
Package uploads successfully.

Additional context
I get a failure (ErrorDetail(string='Unable to read Deb Package', code='invalid')).

As a user I want advanced copy workflows

Author: @quba42 (quba42)

Redmine Issue: 7661, https://pulp.plan.io/issues/7661


pulp_rpm provides a special API endpoint for advanced copy operations with dependency resolution:
https://pulp-rpm.readthedocs.io/en/latest/workflows/copy.html#advanced-copy-workflow

Since APT packages have dependencies of similar complexity to RPM packages, pulp_deb users could benefit from an analogous feature.

The Katello repo filtering mechanism presumably requires this feature.

We need to support new style APT repo signatures

Author: @quba42 (quba42)

Redmine Issue: 8953, https://pulp.plan.io/issues/8953


See the following specification: https://wiki.debian.org/Teams/Apt/Spec/AptSign

See the following excerpt from IRC:

Conan Kudo
quba42: judging by what's going on, I expect this to launch in Debian 11
it definitely will be part of Ubuntu 22.04
I'm also having a conversation with the RPM folks about whether we might want to consider this for the RPM ecosystem in the future too
because not having an independent implementation from OpenBSD's signify was the main problem for me
now that two exist now (one in C++ and another in Python), I think this is something worth pursuing in the RPM ecosystem
Conan Kudo
quba42: my understanding is that Debian 11 and Ubuntu 22.04 will have both, but immediately afterward GPG will be dropped
quba42: apt-key(8) has already been marked for removal after Debian 11 release
(which is where the gpg key management is in apt)
https://www.mankier.com/8/apt-key
I guess this means I'm packaging up the new python-aptsign package for Fedora...
which means packaging python-apt too

Thanks Conan Kudo, for alerting me to this!

duplicate key value violates unique constraint "core_artifact_sha256_key"

Author: knzivid (knzivid)

Redmine Issue: 8840, https://pulp.plan.io/issues/8840


The sync REST call returns successfully

completed: 
[3 Downloading Artifacts] 
[1 Update ReleaseFile units] 
[1 Update PackageIndex units] 
[0 Associating Content] 
[0 Un-Associating Content]

In the database server log, I see

ERROR:  duplicate key value violates unique constraint "core_artifact_sha256_key"
DETAIL:  Key (sha256)=(addfd22c48ca7f6e67197a63d6ba78dc706678e4b198c901de1bfd99060629cc) already exists.
STATEMENT:  INSERT INTO "core_artifact" ("pulp_id", "pulp_created", "pulp_last_updated", "file", "size", "md5", "sha1", "sha224", "sha256", "sha384", "sha512") VALUES ('76c5f18d-0597-4781-bde3-82cf9455b546'::uuid, '2021-05-28T15:27:04.541372+00:00'::timestamptz, '2021-05-28T15:27:04.541385+00:00'::timestamptz, 'artifact/ad/dfd22c48ca7f6e67197a63d6ba78dc706678e4b198c901de1bfd99060629cc', 4645, NULL, NULL, '3c9ad31f31074b93fb2ca51d8618f851ff9edd67ae509da53eb9a6e2', 'addfd22c48ca7f6e67197a63d6ba78dc706678e4b198c901de1bfd99060629cc', '682032a010aacda85558a346cff70840ff3f0882942b6f22ae5651faca9171f4751574341c285fbc9c4201e693f92c0c', 'b504ff76b0ab65b4e894949a9ce65ced4dc820e4cca961c5926744c453fca0d24ec82bc1e2134c765d2916a9f0d173380d5d7d67cb64da4c5bc22f55b2940dd9'), ('65508db8-5726-474b-8ea4-9c96d1d477f5'::uuid, '2021-05-28T15:27:04.541972+00:00'::timestamptz, '2021-05-28T15:27:04.541981+00:00'::timestamptz, 'artifact/88/89e025a6e18c2a0e31edd2f430ee2a7be8a88c48f156706ec672cc87cb832d', 5527, NULL, NULL, 'd58236d07ed20cf31c96eba8148b733e891967f71b6c418759c3b7b2', '8889e025a6e18c2a0e31edd2f430ee2a7be8a88c48f156706ec672cc87cb832d', 'f026b266558924947a1bc796296f9ee6b15033df275d05a576a09505bb66a3870bc61ce166c405887864ae940f08a9a4', 'b8eb88c5b37e13e53fa549831758dbb7564197323f7fd3d2a19b3ba8a7a9f10718939cb8d94db47987b7e461b4f29012ee2479a7c2bc3d3a73bd1fb8746e338c'), ('c0394932-c14a-4c4c-9d0a-774f28f60be3'::uuid, '2021-05-28T15:27:04.542179+00:00'::timestamptz, '2021-05-28T15:27:04.542187+00:00'::timestamptz, 'artifact/b7/4740e506eec14a271e74ee7096b0ce6194a4810f568727da9557127f8db4b9', 833, NULL, NULL, '3d42a1ff9d8d9caaa5a9526be54d786af61a3d16d00f45f0a740639a', 'b74740e506eec14a271e74ee7096b0ce6194a4810f568727da9557127f8db4b9', '2d3b9145ff8fef94ea470404c3b2327a76347b40e1b23ee301d16146bbf139d6bef83e8a04b49af5d65d4f6a6c73be72', '9008fd1c21bc112726f9687536e2dd8c2a5c5d3133338eff8605b4268bf0dc5fb7a974ea4509c5040e9cf8e2676ff7da29a5da100187f9d0fd8f565626594036') RETURNING "core_artifact"."pulp_id"
ERROR:  duplicate key value violates unique constraint "core_artifact_sha256_key"
DETAIL:  Key (sha256)=(addfd22c48ca7f6e67197a63d6ba78dc706678e4b198c901de1bfd99060629cc) already exists.

This does not happen on the first time sync but only from the second time onwards.

Remote:

{
    "pulp_href": "/pulp/api/v3/remotes/deb/apt/f1d18e33-e8d7-4efc-9168-99c17c91d4aa/",
    "pulp_created": "2021-05-28T15:25:58.211906Z",
    "name": "bionic.apt.llvm.org",
    "url": "http://apt.llvm.org/bionic",
    "pulp_last_updated": "2021-05-28T15:25:58.211916Z",
    "download_concurrency": 2,
    "policy": "immediate",
    "distributions": "llvm-toolchain-bionic-11",
    "components": null,
    "architectures": "amd64",
    "sync_sources": false,
    "sync_udebs": false,
    "sync_installer": false,
    "gpgkey": null,
    "ignore_missing_package_indices": false
}

Pulp versions:

    "versions": [
        {
            "component": "core",
            "version": "3.13.0"
        },
        {
            "component": "file",
            "version": "1.7.0"
        },
        {
            "component": "deb",
            "version": "2.13.0"
        }
    ]

I don't notice anything wrong when I am fetching the packages but I am not sure if I should be worried about this error.

Add examples for common remote options to the sync workflow docs

Author: hostedpower (hostedpower)

Redmine Issue: 9222, https://pulp.plan.io/issues/9222


pulp deb remote create --name sury_php --url https://packages.sury.org/php/ --distributions stretch,buster

Apparently this doesn't work, it sees it as one distribution.

Do I need any special parameter in order to have multiple distributions in one remote? Say I need to support Buster, Stretch and Bullseye I would already need 3 remotes which doesn't seem to make sense.

Am I missing something here?

Very slowly publication and progress report

Author: wilful (wilful)

Redmine Issue: 9237, https://pulp.plan.io/issues/9237


Today i started mirroring for http://archive.ubuntu.com

1. pulp task list --state running | jq
[
  {
    "pulp_href": "/pulp/api/v3/tasks/d41cd931-9d7b-4ead-a381-d6aae8c709e0/",
    "pulp_created": "2021-08-12T07:50:28.476170Z",
    "state": "running",
    "name": "pulp_deb.app.tasks.publishing.publish",
    "logging_cid": "60c1f76bef074040943b9ed4558a8732",
    "started_at": "2021-08-12T07:50:28.621880Z",
    "finished_at": null,
    "error": null,
    "worker": "/pulp/api/v3/workers/b64aee98-0684-4daf-9b67-6f8a4aa36370/",
    "parent_task": null,
    "child_tasks": [],
    "task_group": null,
    "progress_reports": [],
    "created_resources": [
      null
    ],
    "reserved_resources_record": [
      "/pulp/api/v3/repositories/deb/apt/a1a61370-650c-4d9f-9c28-79e444110dda/"
    ]
  }
]

And they still running, i can't understand how long me need waiting to complete.

  "versions": [
    {
      "component": "core",
      "version": "3.14.3"
    },
    {
      "component": "rpm",
      "version": "3.14.0"
    },
    {
      "component": "python",
      "version": "3.4.0"
    },
    {
      "component": "file",
      "version": "1.8.1"
    },
    {
      "component": "deb",
      "version": "2.14.1"
    },
    {
      "component": "container",
      "version": "2.7.0"
    }
  ],

I think we have data for specify progress (count of contents and count packages in repository), please, if it can, add more information about publication process to output

Katello 4.3.0: Bullseye re-sync takes more than 5 hours

Version
Katello 4.3.0, tfm-pulpcore-python3-pulpcore-3.16.6-1.el7.noarch, tfm-pulpcore-python3-pulp-deb-2.16.1-1.el7.noarch
updated from 4.2.2 with pulpcore 3.14

Describe the bug
Since upgrading syncing bullseye main repository takes extremely long, more than 5 hours instead of 22 minutes before the upgrade. Tried twice, each sync more than 5 hours. During sync one postgres process is at almost 100% most of the time.

Buster main repository went from 13 minutes to 42 minutes.

To Reproduce
Upstream URL http://deb.debian.org/debian bullseye main amd64

Expected behavior
Faster sync.

Additional context
Foreman community Katello 4.3.0: Bullseye sync takes more than 5 hours

Add tests to ensure all control file fields present in an upstream package index will be present in pulp_deb

Author: @quba42 (quba42)

Redmine Issue: 7266, https://pulp.plan.io/issues/7266


We should have test packages/assertions etc. to ensure that each control file field present in an upstream repo makes it into our published repos.

Should be ensured for both simple and structured publish.

Note that "Description-md5" and "Build-Essential" are pure metadata fields that may be present in (upstream) Package indices but are not present in the package control files.
We are ignoring such fields for now.
Any attempt to support them would be a new feature.

As a user i want to upload package to a repository and have it available immediately in distribution (skip publication step)

Author: mntzn (mntzn)

Redmine Issue: 7390, https://pulp.plan.io/issues/7390


Current workflow is to:

  1. upload content (package)

  2. add content to repo

  3. create publication from repo

  4. create/update distribution with latest publication

Steps 3, 4 can be omitted and package that have been uploaded to a repository should be immediately available in distribution (as alternative to current workflow)

sync fails under certain conditions due to python3-apt not being installed

Author: @jlsherrill ([email protected])

Redmine Issue: 8173, https://pulp.plan.io/issues/8173


A katello user here: https://projects.theforeman.org/issues/31701 reported this. When syncing a deb repo (unsure which one), it errors with:

/usr/lib/python3.6/site-packages/debian/deb822.py:727 UserWarning: Parsing of Deb822 data with python3-apt's apt_pkg was requested but this package is not importable. Is python3-apt installed?

Looking at the code, the default value for use_apt_pkg is True: https://salsa.debian.org/python-debian-team/python-debian/-/blob/master/lib/debian/deb822.py#L2331

and pulp isn't overriding it:

for package_paragraph in deb822.Packages.iter_paragraphs(package_index.main_artifact.file):

so its defaulting to True.

A full traceback isn't available, nor what repository they were syncing, but we could ask for that info if needed.

Release file version changes require manual confirmation in APT

Version
Affects all versions of pulp_deb, when used with APT version 1.5 and greater.

Describe the bug
pulp_deb publishes the Pulp repo version as the Release file version.
Starting with APT 1.5 and greater, APT asks for user confirmation in order to keep using a repo with a changed Release file version, unless apt-get update --allow-releaseinfo-change is used.

This can break automation not using this flag, whenever the Release file version changes.

Expected behavior
Do not use the Release file version field in a way that clashes with basic APT design.
We should check if there are other release file fields that require manual confirmation when they change as well!

Additional context
See also: https://community.theforeman.org/t/pulp-deb-incorrect-version-in-release-and-inrelease/27416/3

As a user I want to create arbitrary structured publications

Author: @quba42 (quba42)

Redmine Issue: 7287, https://pulp.plan.io/issues/7287


Right now the only way to create structured publications is to sync a remote repository (the metadata structure of which will be recreated by pulp_deb).

Instead, I want the ability to:

  1. manually create a release from a list of one or more components.
  2. add/remove components from existing releases.
  3. manually create a component from a list of zero or more packages.
  4. add/remove packages from existing components.
  5. add/remove releases from a repository.

It is crucial that any of the above actions automatically create a consistent repository. For example: If I add a release to a repository, then any components, packages, etc. part of that release must also be added to the repository automatically

Add option to publish using "No-Support-for-Architecture-all: Packages" repo format

Currently pulp_deb always publishes only separate Architecture = all package indices. This is considered the best practice format, but it is not a format used by any official Debian repos. Instead those use the No-Support-for-Architecture-all: Packages format for more backwards compatibility (for example for Debian bullseye).

I consider it dangerous to exclusively use a format not used by anyone in the wild.

The proposal would be to add a option to publish in the "No-Support-for-Architecture-all: Packages" style.

create publication with signing: invalid hyperlink - object does not exist

Author: igagis (igagis)

Redmine Issue: 9032, https://pulp.plan.io/issues/9032


I'm trying to create a publication with metadata siging via REST api and I get the following error:

HTTP code = 400

{"signing_service":["Invalid hyperlink - Object does not exist."]}

I have created the signing service via pulpcore_manager add-signing-service and there is a GPG key in the keyring and signing shell script, as described here: https://docs.pulpproject.org/pulpcore/workflows/signed-metadata.html

I am able to get the signing service via REST api using its pulp_href, here is the response from REST request:

{
  "pulp_href": "/pulp/api/v3/signing-services/f204bdc9-2c5c-4963-b5a9-75dae6f4960d/",
  "pulp_created": "2021-07-08T12:14:33.021734Z",
  "name": "sign-metadata",
  "public_key": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n\nmQINBGDm6n0BEADJ4jRZGZOfkgXJlbKL2TqhdStTmOedfVYi5VjxbmKVq1bsLqHO\nLb8/7u70M+iz18ERnRVG/F0EziKVOouXVdXKEsZhG2SPkOb9gKT1KlVp33k/bNyr\nJDyAVh3uYTeodCgMyHucFwkI8j5ums48rmIS/GzDxMDNLdzKLWspoQs6956F0mf+\nmmuVglfS0l8RncjXIFsh9JMlDeiK3trrbMGdhBo6U/QadxROYtfTdAZ6a10IW7iL\nMBAPe+pXtTHnCa43cJrqGXV9Y2FJ54GJVV3Kbr9T13M7zjj1ygAzm/eBnm9OqdgD\n55k0ZAyzLcpIdK//0ZLMgLIkcFQ+2BzQXLsi6xAIkzcycfXnvgSY7vUi6uwCLyLf\nyGbPoHd8Fz3q/9XJpa+3dw/q/p96d6dN+o/QmI82dy1I+D4iJ1NtLzH/lTX2ge5K\nh4FIIwGbfzJmgBSGb/2XAqBdh4FFIemmnIu2qq2Yk5SnR6wdw7FvMm3st3ww0PQZ\nnr3udNZ8MlG3AaATODa0Y/UPpUnN6sjNfLjC5sAwIzZw3LerCo6Q70AjVQOhm/x8\nqh0Smz2uupbCR7NAceHlxqWA+4I6r094rHnab4BNntUh/IE2ZQaL02fqAaCgaGjU\nlhQyuBM6Z/oZjrXTiyyC+Oh/b8hjrGbKWgs4ifgyHpuVNRnpubR1R4qPywARAQAB\ntEFJdmFuIEdhZ2lzIChyZXBvc2l0b3JpZXMgbWV0YWRhdGEgc2lnbmluZyBrZXkp\nIDxpZ2FnaXNAZ21haWwuY29tPokCTgQTAQoAOBYhBN8FDZUx8V9VxKhNOlkz8F4w\nKnUCBQJg5up9AhsDBQsJCAcCBhUKCQgLAgQWAgMBAh4BAheAAAoJEFkz8F4wKnUC\n2IMQAKstqHrbj3pG4kVJl2VgYB+HujoDqhf2q95XI+gbLS3A9vE6UOP7Le/2WxVc\nWtaSbqxv9lPZwXL7krhtAPVfNHOs11N4SloBxFnCyrgmgC/xZxAPRD0DofA4Numt\nVYq5TvIjZ9CUFb0NhAE5TuuWljlk76DwZeiZ4zbKRrmJcnatOqPBv5RnYlGjIAVj\nyyCCOCSEYDmAQNwQCyBqTlP+TzsXj/x95lWU54zAn+DbwckKAavbYbowVhlKev/+\nmZpK5Vnpp9Taea2rsh9mWC2tAExLE9nPBP4ed2BKYHQEJzCkYavp9vO9xC2/f02N\nYZpmzMwWP0KFrGTx+Fk89uyVd89ML1zZs5Rysf2jgDbBAcbHJb5rXG9y4CZm0wUZ\nGumohuomz5ywzkE/RSjmdhJJmOylqylvcs2+eTbIkWSvWQjd8pE535QSWeItJNup\nfSc8x+GV1Ln31kPNpClcAjnR31/Xoak694soYEc7uEzVrkBF9UiLXfzmfhXyV0Su\nHZ+py+8zf6XXJ3APi1y/MdnP1KilBbosC3mTIHdVq/oAx8BRxm4FJlb+j4pjnMgl\nVIgDVNiHZCSZIhwj0ZqMlLaDRjGc30iYmA7sn3gnbrpDG3uwc7/7WDHg/NHY41WQ\nOePcoL//9zsx3AtgsCb6ZzkIi51MoyyOnwLTmFDIbQQ32ZZEuQINBGDm6n0BEACf\nnKQtcxqQadDZ0UadqLwFyWuSi8K5khoBFcaaNK6nQc/eZLmj6FjcPvqBzSA/TBg+\nptvwPjD2yRWZKfOrsQTVL9Bijdcu5KF4BEMqtpntqhbtoSnQXqwa8u1aEs0zt/vw\nU5nf6g6G+kX9/whTetFIsGg9eLcUZvETuWrovOKsffF+nkznB89fbSK7MznoOTO5\nu86jc3PRdYMzFjWLT+5N+90HO9ZoGzaegOEx8EraYyeIMXoXwEtR0Yy7vaIG3YaM\nua6UvPRbXh7dtMXSKUJmDsdw5puwE7jez2GQ05898gRtJE5HcRAWkS0yD4UnPp0t\nG2Pe1GRv65lshMa8XTufzzddK8dZOyrxQ5e71La02ZWK5pWNi3CLP8WsSF/jc0Ej\n9K+6YY3iTZSZmU6vYJ17KUFRjN33xFYRvKbBp0S7TVWx9cqWYPCyavG3iDyKsyyI\nLZB8w1t4Ao3p/2uYSYUOu7ifCd9iL1KZ0o3eFa+YxFZigaTtMiYaE5HUaU8akZG8\nReAR2OSyj43GtgrO92lLMFoN/XXg7nnrAiTfmcwH6TKior5ZZQjsot69uSgbKaSj\nxnrGGUPe3yPSxoxEGMYdi7nNqB3VX4d+W/McSK5etNme5OPC4TU7yaEhgxyHhJ4x\nvn3e2EVa+8Hwb8Jh5pd4FGaCsWuaCoWfE0vZ8NaiJQARAQABiQI2BBgBCgAgFiEE\n3wUNlTHxX1XEqE06WTPwXjAqdQIFAmDm6n0CGwwACgkQWTPwXjAqdQKA4RAAjfnI\nCbFd9r893tw5j98TncD7iIfnBt0ESUJ3iGq8KA37q4WFvtdE+zvs7Wq+9cmB0/z+\nt30IATNdnAkLjbON2AQP9KjXQ20KtcXONVuf2hjEogY7L3s4gEwvVeA4hLaLprlj\nETQdWYjiWilD4WQfdWeAEmTDbQLlbFqxCJgz3vT+ImJFh89SePFZgbzUMyTgII+V\n+8cS3FCeQLF/IXxLuNRnM951zat8XnALfi9swbRVMp/ZhQMuHfPJxUBmQ1H4YE46\ny6t6PqVBRrGPSov82vX39vJjoN97JaCSE7ib+bRKrKIZIBk2Cy+j00cusm8HI2xy\nP0pEC9NYXQtV1k0kOiDTdBjYS/9vmSEC/eXa99w3+ScwerRnS61fSNXsAn0RQ6WS\nbitMd3rfhL6Mh3kyGf5pzkBv1iSFtfJFGdKxiDPKKeLLCr0u6/YTmhzaGRy0XwtW\n7u86efHFVneHM9uR6tWthNNy0VHNJcRip9I6Ghfa28TIDKpHsGC221Kvdh5VudkO\n/1mogvFDLTSq1RpBWV5EEVBbymwcmvSrToGCFl7R1PozyHLLG61jt+ofuanku3Ux\n+DqgGnoLJRZtLtMBV/3rd/ywdkRDWzwRQsID6sQV7/ECQvo5Giu0d3+IbsfGcTWf\n9iUCxpwW8haBtIj8FRNx3fczSxGetUodQN6QjUc=\n=tgAd\n-----END PGP PUBLIC KEY BLOCK-----\n",
  "pubkey_fingerprint": "DF050D9531F15F55C4A84D3A5933F05E302A7502",
  "script": "/var/lib/pulp/scripts/sign-metadata.sh"
}

I'm using pulp v3 installation made via ansible.

Add support for Ubuntu comand-not-found (CNF) indices

Author: wilful (wilful)

Redmine Issue: 9239, https://pulp.plan.io/issues/9239


Hi! I trying create mirror for official ubuntu repository: http://archive.ubuntu.com/ubuntu/
After creation on my test Ubuntu server i see error:

/pulp/content/archive.ubuntu.com/dists/focal/main/cnf/Commands-amd64: 404  Not Found [IP: 172.24.160.15 80]
E: Some index files failed to download. They have been ignored, or old ones used instead.

It content (CNF) have in source but not in my repository
I use verbatim publisher, on_demand policy

      "pulp_href": "/pulp/api/v3/remotes/deb/apt/25c10483-64e2-492a-ac30-6efb5bbe3683/",
      "pulp_created": "2021-08-13T13:58:19.364006Z",
      "name": "archive.ubuntu.com",
      "url": "http://archive.ubuntu.com/ubuntu/",
      "ca_cert": null,
      "client_cert": null,
      "tls_validation": true,
      "proxy_url": null,
      "pulp_labels": {},
      "pulp_last_updated": "2021-08-13T13:58:19.364027Z",
      "download_concurrency": null,
      "max_retries": null,
      "policy": "on_demand",
      "total_timeout": null,
      "connect_timeout": null,
      "sock_connect_timeout": null,
      "sock_read_timeout": null,
      "headers": null,
      "rate_limit": null,
      "distributions": "trusty focal",
      "components": null,
      "architectures": "amd64",
      "sync_sources": false,
      "sync_udebs": false,
      "sync_installer": false,
      "gpgkey": null,
      "ignore_missing_package_indices": false
    },

Can i add it is cnf folder?

Servers supporting Acquire-By-Hash may break verbatim publications

Author: @quba42 (quba42)

Redmine Issue: 7819, https://pulp.plan.io/issues/7819


If a server supports Acquire-By-Hash (in a Release file), the verbatim publisher will still claim to do so (since it will republish the Release file verbatim). However, the verbatim publisher will not publish the Acquire-By-Hash locations for package indecies, since we do not sync them.

This should be easy to fix, if we add the Acquire-By-Hash to our sync. Note that this will be the same files under a different path which probably relates to the whole relative_path corner case problem? (I have forgotten what exactly the latest state of that problem/discussion was).

See

file Release is not signed for deb repository

Author: overlord (overlord)

Redmine Issue: 8896, https://pulp.plan.io/issues/8896


good afternoon, Comrades. Please tell me, maybe someone has encountered a problem with synchronization of the deb repository. It syncs without the Release file.gpg, as a result, when executing apt-get update on the target machine, the error "bionic-updates Releaseโ€™ is not signed " appears.
". I found this instruction on the forum Foreman Katello - Deb Sync no Release.gpg but it still doesnโ€™t work. Duma. this is related to the pulp version. I use pulp3

Synchronize translation files

Author: @quba42 (quba42)

Redmine Issue: 8734, https://pulp.plan.io/issues/8734


This feature has been partially implemented, but I don't think it has ever actually worked and contains various sync breaking bugs.
As a result I will disable translation file synchronization completely for now.

The feature can then be re-added once fully completed. The following is a list of issues I have identified with the current incomplete implementation:

  • Debian references extracted versions of translation file archives in the Release file even if not actually there. This messes with the current implementation, but can be fixed by handling translation files the same way we do this for package indices.
  • We are currently synchronizing "translation diff file indexes", but not the files referenced from those indexes.
  • We are storing translation files as GenericContent, but the corresponding serializer cannot deal with multi artifact content.

Duplicate key violates Unique-Constraint core_artifact_sha256_key when syncing debian bullseye-backports

Author: pasche (pasche)

Redmine Issue: 9576, https://pulp.plan.io/issues/9576


I have manually installed Foreman with Katello following the official documentation on a CentOS 7 system.

Created a "Debian" product with 4 repositories "bullseye" (main), "bullseye-backports", "bullseye-security" and "bullseye-updates".

3 of them work (currently without issues). While trying to sync "bullseye-backports", I get a unique-contraint violation error.

See dynaflow output.

---
repo_id: 3
smart_proxy_id: 1
options: {}
remote_user: admin
remote_cp_user: admin
current_request_id: 72734708-ee6a-4eee-8bbb-7bb88d6c1d30
current_timezone: Berlin
current_organization_id: 1
current_location_id: 
current_user_id: 5
Output:

---
pulp_tasks:
- pulp_href: "/pulp/api/v3/tasks/a4eab034-3c3a-4104-a030-8340ad6c1ebe/"
  pulp_created: '2021-11-12T08:44:58.078+00:00'
  state: failed
  name: pulp_deb.app.tasks.synchronizing.synchronize
  logging_cid: 72734708-ee6a-4eee-8bbb-7bb88d6c1d30
  started_at: '2021-11-12T08:45:25.561+00:00'
  finished_at: '2021-11-12T08:45:26.792+00:00'
  error:
    traceback: |2
        File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task
          result = func(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py", line 124, in synchronize
          DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
          loop.run_until_complete(pipeline)
        File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
          return future.result()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
          await asyncio.gather(*futures)
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
          await self.run()
        File "/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py", line 378, in run
          da.artifact.save()
        File "/usr/lib/python3.6/site-packages/pulpcore/app/models/content.py", line 143, in save
          super().save(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 134, in save
          save(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 744, in save
          force_update=force_update, update_fields=update_fields)
        File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 782, in save_base
          force_update, using, update_fields,
        File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 873, in _save_table
          result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
        File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 911, in _do_insert
          using=using, raw=raw)
        File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
          return getattr(self.get_queryset(), name)(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert
          return query.get_compiler(using=using).execute_sql(return_id)
        File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
          cursor.execute(sql, params)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
          return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
          return executor(sql, params, many, context)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
          return self.cursor.execute(sql, params)
        File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
          raise dj_exc_value.with_traceback(traceback) from exc_value
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
          return self.cursor.execute(sql, params)
    description: |
      FEHLER:  doppelter Schlรผsselwert verletzt Unique-Constraint ยปcore_artifact_sha256_keyยซ
      DETAIL:  Schlรผssel ยป(sha256)=(759174ddf14a5995c5764fb9ef3f67645494fc56e640f4d7bee181d534c172eb)ยซ existiert bereits.
  worker: "/pulp/api/v3/workers/8dc5b0a3-bcba-4b07-a13c-8ab8731ae04c/"
  child_tasks: []
  progress_reports:
  - message: Update PackageIndex units
    code: update.packageindex
    state: failed
    done: 4
  - message: Downloading Artifacts
    code: sync.downloading.artifacts
    state: canceled
    done: 21
  - message: Update ReleaseFile units
    code: update.release_file
    state: canceled
    done: 1
  - message: Associating Content
    code: associating.content
    state: canceled
    done: 10
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/remotes/deb/apt/45358d31-6b93-46f9-a517-a73794579b95/"
  - "/pulp/api/v3/repositories/deb/apt/6c967d88-e92c-4c69-bd66-1b52e6355c96/"
create_version: true
task_groups: []
poll_attempts:
  total: 18
  failed: 1
Error:

Katello::Errors::Pulp3Error

FEHLER: doppelter Schlรผsselwert verletzt Unique-Constraint ยปcore_artifact_sha256_keyยซ DETAIL: Schlรผssel ยป(sha256)=(759174ddf14a5995c5764fb9ef3f67645494fc56e640f4d7bee181d534c172eb)ยซ existiert bereits.

---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.0.1/app/lib/actions/pulp3/abstract_async_task.rb:108:in
  `block in check_for_errors'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.0.1/app/lib/actions/pulp3/abstract_async_task.rb:106:in
  `each'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.0.1/app/lib/actions/pulp3/abstract_async_task.rb:106:in
  `check_for_errors'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.0.1/app/lib/actions/pulp3/abstract_async_task.rb:139:in
  `poll_external_task'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action/polling.rb:100:in
  `poll_external_task_with_rescue'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action/polling.rb:22:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action/cancellable.rb:14:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.0.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:572:in
  `block (3 levels) in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:32:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.0.1/app/lib/actions/middleware/remote_action.rb:16:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.0.1/app/lib/actions/middleware/remote_action.rb:40:in
  `block in as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.0.1/app/models/katello/concerns/user_extensions.rb:21:in
  `cp_config'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.0.1/app/lib/actions/middleware/remote_action.rb:27:in
  `as_cp_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.0.1/app/lib/actions/middleware/remote_action.rb:39:in
  `as_remote_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.2.0.1/app/lib/actions/middleware/remote_action.rb:16:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.0/app/lib/actions/middleware/rails_executor_wrap.rb:14:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in
  `wrap'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.0/app/lib/actions/middleware/rails_executor_wrap.rb:13:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action/progress.rb:31:in
  `with_progress_calculation'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action/progress.rb:17:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.0/app/lib/actions/middleware/load_setting_values.rb:20:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in
  `restore_current_request_id'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in
  `restore_curent_timezone'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in
  `restore_current_taxonomies'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:32:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware.rb:19:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.0/app/lib/actions/middleware/keep_current_user.rb:15:in
  `block in run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.0/app/lib/actions/middleware/keep_current_user.rb:54:in
  `restore_curent_user'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.1.0/app/lib/actions/middleware/keep_current_user.rb:15:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/stack.rb:23:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/middleware/world.rb:31:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:571:in
  `block (2 levels) in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:570:in
  `catch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:570:in
  `block in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:473:in
  `block in with_error_handling'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:473:in
  `catch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:473:in
  `with_error_handling'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:565:in
  `execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/action.rb:286:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in
  `block (2 levels) in execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/execution_plan/steps/abstract.rb:167:in
  `with_meta_calculation'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in
  `block in execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in
  `open_action'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/director.rb:94:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in
  `block (2 levels) in perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors.rb:18:in
  `run_user_code'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in
  `block in perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in
  `with_telemetry'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in
  `perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.5.0/lib/dynflow/executors/sidekiq/serialization.rb:27:in
  `perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in
  `execute_job'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in
  `block (2 levels) in process'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in
  `block in invoke'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in
  `invoke'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in
  `block in process'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in
  `block (6 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in
  `local'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in
  `block (5 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in
  `block in <module:Sidekiq>'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in
  `block (4 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in
  `stats'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in
  `block (3 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in
  `block (2 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in
  `global'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in
  `block in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in
  `with_context'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in
  `with_job_hash_context'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in
  `dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in
  `process'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in
  `process_one'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in
  `watchdog'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in
  `block in safe_thread'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in
  `block in create_with_logging_context'"

Unable to create ReleaseArchitecture or ReleaseComponent via the API

Author: lieter (lieter)

Redmine Issue: 6538, https://pulp.plan.io/issues/6538


After creating a release

http -j -a 'admin:password' POST http://pulp3.home.plexis.eu/pulp/api/v3/content/deb/releases/ codename=foo distribution=foo suite=foo

it isn't possible to create a ReleaseArchitecture or ReleaseComponent:

$ http -j -a 'admin:password' :24817/pulp/api/v3/content/deb/releases/
HTTP/1.1 200 OK
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 236
Content-Type: application/json
Date: Tue, 21 Apr 2020 15:47:45 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "codename": "foo",
            "distribution": "foo",
            "pulp_created": "2020-04-17T16:09:02.264431Z",
            "pulp_href": "/pulp/api/v3/content/deb/releases/624b65f8-c8f6-486b-a75a-318591552009/",
            "suite": "foo"
        }
    ]
}

$ http -j -a 'admin:password' POST :24817/pulp/api/v3/content/deb/release_architectures/ architecture=amd64 release='/pulp/api/v3/content/deb/releases/624b65f8-c8f6-486b-a75a-318591552009/'
HTTP/1.1 400 Bad Request
Allow: GET, POST, HEAD, OPTIONS
Connection: close
Content-Length: 56
Content-Type: application/json
Date: Tue, 21 Apr 2020 15:53:02 GMT
Server: gunicorn/20.0.4
Vary: Accept, Cookie
X-Frame-Options: SAMEORIGIN

{
    "release": [
        "Invalid hyperlink - Incorrect URL match."
    ]
}

Being able to do this should enable 'structured publishing' for non-mirrored debian repositories (I'm told)

pulp fails while syncing a http repository of its own

Author: tp-hlawatsch (tp-hlawatsch)

Redmine Issue: 9290, https://pulp.plan.io/issues/9290


Hello everybody ,

I try to implement a staged repository workflow at the moment . I created a repository dv with a distant remote and synced with success.
Now I created a repository in with the distribution of dv as remote ( http://$(hostname):8080/pulp/content/repo/deb/dv )
I switched of tls validation but when I try to sync this new remote I run into the following error

(failed; Cannot connect to host localhost:8080 ssl:default [Connect call failed ('127.0.0.1', 8080)])

In the logs I see the following error [ยน]

the pulp is dockerized with the following version: pulp/pulp:3.14

Can it be that pulp is expecting alwasy a https repository even when http is configured ?

Kind regards
Stephan

[ยน]
pulp [4d4c93f49ea94ab68f733fc22def7e53]: pulpcore.tasking.pulpcore_worker:INFO: Task 05ef9319-9854-4228-a459-4cc24eaafb45 failed (Cannot connect to host localhost:8080 ssl:default [Connect call failed ('127.0.0.1', 8080)])
pulp [4d4c93f49ea94ab68f733fc22def7e53]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/local/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 272, in _perform_task
result = func(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py", line 124, in synchronize
DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()
File "/usr/local/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
loop.run_until_complete(pipeline)
File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
return future.result()
File "/usr/local/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
await asyncio.gather(*futures)
File "/usr/local/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in call
await self.run()
File "/usr/local/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 181, in run
pb.done += task.result() # download_count
File "/usr/local/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 207, in _handle_content_unit
await asyncio.gather(*downloaders_for_content)
File "/usr/local/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py", line 137, in download
await super().download()
File "/usr/local/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py", line 89, in download
download_result = await downloader.run(extra_data=self.extra_data)
File "/usr/local/lib/python3.6/site-packages/pulpcore/download/http.py", line 258, in run
return await download_wrapper()
File "/usr/local/lib/python3.6/site-packages/backoff/_async.py", line 133, in retry
ret = await target(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/pulpcore/download/http.py", line 256, in download_wrapper
return await self._run(extra_data=extra_data)
File "/usr/local/lib/python3.6/site-packages/pulpcore/download/http.py", line 277, in _run
self.url, proxy=self.proxy, proxy_auth=self.proxy_auth, auth=self.auth
File "/usr/local/lib64/python3.6/site-packages/aiohttp/client.py", line 1117, in aenter
self._resp = await self._coro
File "/usr/local/lib64/python3.6/site-packages/aiohttp/client.py", line 521, in _request
req, traces=traces, timeout=real_timeout
File "/usr/local/lib64/python3.6/site-packages/aiohttp/connector.py", line 535, in connect
proto = await self._create_connection(req, traces, timeout)
File "/usr/local/lib64/python3.6/site-packages/aiohttp/connector.py", line 892, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "/usr/local/lib64/python3.6/site-packages/aiohttp/connector.py", line 1051, in _create_direct_connection
raise last_exc
File "/usr/local/lib64/python3.6/site-packages/aiohttp/connector.py", line 1032, in _create_direct_connection
client_error=client_error,
File "/usr/local/lib64/python3.6/site-packages/aiohttp/connector.py", line 975, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc

Rework No-Support-for-Architecture-all format sync handling

Many upstream repos declare the No-Support-for-Architecture-all: Packages option in their release file. For example, most official Debian bullseye repos.

This format results in a number of edge cases requiring special handling.
For example, we should not warn about Architecture: all packages in architecture specific indices for such repos.

Even more problematically, deb-mirror may mirror the metadata indicating No-Support-for-Architecture-all: Packages format, while failing to mirror the binary-all package indices (which actually results in "old style repo format" while declaring "mixed style repo format"). This two requires special handling, or the Architecture: all packages will either not be synced or cause outright sync failures for such mirrors! (which depends on the value of ignore_missing_package_indices).

allow fetching obsolete artifacts from snapshot.debian.org

Author: evgeni (evgeni)

Redmine Issue: 4866, https://pulp.plan.io/issues/4866


Debian removes old versions of packages that have been updated from their mirrors. Because of that using lazy sync against official Debian mirrors might end up with incomplete repos when Pulp syncs metadata on day 1, on day 2 a package gets updated/replaced and on day 3 pulp tries to actually get that package.

Debian offers a snapshot service at https://snapshot.debian.org/, where you can get almost every file ever seen in the Debian archive by accessing https://snapshot.debian.org/file/HASH (where HASH is currently the SHA1 hash of the file). The API docs of snapshot can be found at https://salsa.debian.org/snapshot-team/snapshot/raw/master/API

The idea would be to allow a secondary source in pulp_deb that is tried when the primary mirror replied with a 404.

This is probably also useful for other backends, not just Debian.

Snapshot currently only supports sha1 hashes, but the authors promised to accept patches for sha256 if anyone would write one (the source of the service is in the same repo as the api doc linked above).

Repository sync will fail if Debian components are defined on the remote

Author: simon_t (simon_t)

Redmine Issue: 8692, https://pulp.plan.io/issues/8692


pulpcore version: 3.12.1

pulp_deb version: 2.11.1

Attempts to sync a repository with a remote that has Debian components defined consistently fail with the error "Response payload is not completed".

Failing remote config:

{
    "architectures": "amd64",
    "ca_cert": null,
    "client_cert": null,
    "components": "main contrib non-free",
    "connect_timeout": null,
    "distributions": "bullseye",
    "download_concurrency": 10,
    "gpgkey": null,
    "headers": null,
    "ignore_missing_package_indices": false,
    "name": "remote_debian_bullseye",
    "policy": "immediate",
    "proxy_url": null,
    "pulp_created": "2021-04-29T11:17:38.824675Z",
    "pulp_href": "/pulp/api/v3/remotes/deb/apt/18600e5b-4914-4bbd-ae2a-bc0488c30e07/",
    "pulp_labels": {},
    "pulp_last_updated": "2021-04-30T00:47:55.224174Z",
    "rate_limit": null,
    "sock_connect_timeout": null,
    "sock_read_timeout": null,
    "sync_installer": false,
    "sync_sources": false,
    "sync_udebs": false,
    "tls_validation": true,
    "total_timeout": null,
    "url": "http://ftp.au.debian.org/debian/"
}

Error reported to API:

{
            "child_tasks": [],
            "created_resources": [],
            "error": {
                "description": "Response payload is not completed",
                "traceback": "  File \"/usr/local/lib/pulp/lib/python3.6/site-packages/rq/worker.py\", line 1008, in perform_job\n    rv = job.perform()\n  File \"/usr/local/lib/pulp/lib/python3.6/site-packages/rq/job.py\", line 706, in perform\n    self._result = self._execute()\n  File \"/usr/local/lib/pulp/lib/python3.6/site-packages/rq/job.py\", line 729, in _execute\n    result = self.func(*self.args, **self.kwargs)\n  File \"/usr/local/lib/pulp/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py\", line 122, in synchronize\n    DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()\n  File \"/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py\", line 149, in create\n    loop.run_until_complete(pipeline)\n  File \"/usr/lib/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n    return future.result()\n  File \"/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 174, in run\n    pb.done += task.result()  # download_count\n  File \"/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py\", line 200, in _handle_content_unit\n    await asyncio.gather(*downloaders_for_content)\n  File \"/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py\", line 89, in download\n    download_result = await downloader.run(extra_data=self.extra_data)\n  File \"/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/download/base.py\", line 241, in run\n    return await self._run(extra_data=extra_data)\n  File \"/usr/local/lib/pulp/lib/python3.6/site-packages/backoff/_async.py\", line 133, in retry\n    ret = await target(*args, **kwargs)\n  File \"/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/download/http.py\", line 218, in _run\n    to_return = await self._handle_response(response)\n  File \"/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/download/http.py\", line 185, in _handle_response\n    chunk = await response.content.read(1048576)  # 1 megabyte\n  File \"/usr/local/lib/pulp/lib/python3.6/site-packages/aiohttp/streams.py\", line 380, in read\n    await self._wait(\"read\")\n  File \"/usr/local/lib/pulp/lib/python3.6/site-packages/aiohttp/streams.py\", line 306, in _wait\n    await waiter\n"
            },
            "finished_at": "2021-04-29T22:03:38.950771Z",
            "logging_cid": "2367609e386d4f3b8c94830deea2053e",
            "name": "pulp_deb.app.tasks.synchronizing.synchronize",
            "parent_task": null,
            "progress_reports": [
                {
                    "code": "update.release_file",
                    "done": 1,
                    "message": "Update ReleaseFile units",
                    "state": "canceled",
                    "suffix": null,
                    "total": null
                },
                {
                    "code": "update.packageindex",
                    "done": 3,
                    "message": "Update PackageIndex units",
                    "state": "canceled",
                    "suffix": null,
                    "total": null
                },
                {
                    "code": "associating.content",
                    "done": 10017,
                    "message": "Associating Content",
                    "state": "canceled",
                    "suffix": null,
                    "total": null
                },
                {
                    "code": "unassociating.content",
                    "done": 0,
                    "message": "Un-Associating Content",
                    "state": "canceled",
                    "suffix": null,
                    "total": null
                },
                {
                    "code": "sync.downloading.artifacts",
                    "done": 200,
                    "message": "Downloading Artifacts",
                    "state": "failed",
                    "suffix": null,
                    "total": null
                }
            ],
            "pulp_created": "2021-04-29T22:02:51.216445Z",
            "pulp_href": "/pulp/api/v3/tasks/c6ca7102-7b9a-4774-9455-5eb486b527b3/",
            "reserved_resources_record": [
                "/pulp/api/v3/remotes/deb/apt/2889a55e-8ece-40ae-a811-d39d89deda4e/",
                "/pulp/api/v3/repositories/deb/apt/35664f88-071a-4bec-a1be-b7e0befe369e/"
            ],
            "started_at": "2021-04-29T22:02:51.391384Z",
            "state": "failed",
            "task_group": null,
            "worker": "/pulp/api/v3/workers/ebece415-1026-44a0-8668-0de82c2ac9e9/"
        }

When failing in this manner multiple attempts will progress the sync to a point beyond which it will not progress, in the example above that point was when the "associating.content" value was 10017.

As a further test the remote was recreated with a null value for "components":

{
    "architectures": "amd64",
    "ca_cert": null,
    "client_cert": null,
    "components": null,
    "connect_timeout": null,
    "distributions": "bullseye",
    "download_concurrency": 10,
    "gpgkey": null,
    "headers": null,
    "ignore_missing_package_indices": true,
    "name": "remote_debian_bullseye",
    "policy": "immediate",
    "proxy_url": null,
    "pulp_created": "2021-05-04T02:08:55.188864Z",
    "pulp_href": "/pulp/api/v3/remotes/deb/apt/3f95927d-b39c-4bda-a24e-f850d73f2095/",
    "pulp_labels": {},
    "pulp_last_updated": "2021-05-04T02:08:55.188881Z",
    "rate_limit": null,
    "sock_connect_timeout": null,
    "sock_read_timeout": null,
    "sync_installer": false,
    "sync_sources": false,
    "sync_udebs": false,
    "tls_validation": true,
    "total_timeout": null,
    "url": "http://ftp.au.debian.org/debian/"
}

While the sync attempts for this configuration still failed multiple times it is always with a "Server disconnected" error and the sync progressed further st each attempt until a complete sync was achieved.

{
   "child_tasks": [],
   "created_resources": [
       "/pulp/api/v3/repositories/deb/apt/dbd1bbf7-a5ef-46b6-922a-0ddf62b92ebc/versions/4/"
   ],
   "error": null,
   "finished_at": "2021-05-04T06:24:54.428619Z",
   "logging_cid": "1121eb0f04d742aa80b0a1cb70b0cedb",
   "name": "pulp_deb.app.tasks.synchronizing.synchronize",
   "parent_task": null,
   "progress_reports": [
       {
           "code": "sync.downloading.artifacts",
           "done": 3157,
           "message": "Downloading Artifacts",
           "state": "completed",
           "suffix": null,
           "total": null
       },
       {
           "code": "update.release_file",
           "done": 1,
           "message": "Update ReleaseFile units",
           "state": "completed",
           "suffix": null,
           "total": null
       },
       {
           "code": "update.packageindex",
           "done": 3,
           "message": "Update PackageIndex units",
           "state": "completed",
           "suffix": null,
           "total": null
       },
       {
           "code": "associating.content",
           "done": 119400,
           "message": "Associating Content",
           "state": "completed",
           "suffix": null,
           "total": null
       },
       {
           "code": "unassociating.content",
           "done": 1,
           "message": "Un-Associating Content",
           "state": "completed",
           "suffix": null,
           "total": null
       }
   ],
   "pulp_created": "2021-05-04T06:09:38.553551Z",
   "pulp_href": "/pulp/api/v3/tasks/c9674fbd-5d30-4932-bbc1-1c50e8d6e595/",
   "reserved_resources_record": [
       "/pulp/api/v3/remotes/deb/apt/3f95927d-b39c-4bda-a24e-f850d73f2095/",
       "/pulp/api/v3/repositories/deb/apt/dbd1bbf7-a5ef-46b6-922a-0ddf62b92ebc/"
   ],
   "started_at": "2021-05-04T06:09:38.704737Z",
   "state": "completed",
   "task_group": null,
   "worker": "/pulp/api/v3/workers/a25d6c8a-b4ea-4500-ab08-92104d08bca7/"
}

Full log of a sync failing with the "Response payload is not completed" error:

Apr 30 00:29:19 pulphost.domain.local gunicorn[1381]: pulp [28ac6dceafad4e07933ddfbeead891e7]:  - admin [30/Apr/2021:00:29:19 +0000] "POST /pulp/api/v3/repositories/deb/apt/dbd1bbf7-a5ef-46b6-922a-
0ddf62b92ebc/sync/ HTTP/1.0" 202 67 "-" "HTTPie/0.9.8"
Apr 30 00:29:19 pulphost.domain.local rq[1374]: pulp [None]: rq.worker:INFO: [email protected]: 69878f8d-7e56-4322-8f4c-5687c9ae7aeb
Apr 30 00:29:19 pulphost.domain.local rq[1383]: pulp [28ac6dceafad4e07933ddfbeead891e7]: rq.worker:INFO: resource-manager: Job OK (46efd531-9651-48db-8aa3-73521c79de42)
Apr 30 00:29:19 pulphost.domain.local rq[1374]: pulp [28ac6dceafad4e07933ddfbeead891e7]: pulp_deb.app.tasks.synchronizing:INFO: Downloading Release file for distribution: "bullseye"
Apr 30 00:29:20 pulphost.domain.local rq[1374]: pulp [28ac6dceafad4e07933ddfbeead891e7]: pulp_deb.app.tasks.synchronizing:INFO: Parsing Release file at distribution="bullseye"
Apr 30 00:29:20 pulphost.domain.local rq[1374]: pulp [28ac6dceafad4e07933ddfbeead891e7]: pulp_deb.app.tasks.synchronizing:INFO: Downloading: contrib/binary-amd64/Packages
Apr 30 00:29:20 pulphost.domain.local rq[1374]: pulp [28ac6dceafad4e07933ddfbeead891e7]: pulp_deb.app.tasks.synchronizing:INFO: Downloading: non-free/binary-amd64/Packages
Apr 30 00:29:20 pulphost.domain.local rq[1374]: pulp [28ac6dceafad4e07933ddfbeead891e7]: pulp_deb.app.tasks.synchronizing:INFO: Downloading: main/binary-amd64/Packages
Apr 30 00:29:25 pulphost.domain.local rq[1374]: /usr/local/lib/pulp/lib/python3.6/site-packages/debian/deb822.py:740: UserWarning: Parsing of Deb822 data with python3-apt's apt_pkg was requested bu
t this package is not importable. Is python3-apt installed?
Apr 30 00:29:25 pulphost.domain.local rq[1374]:   warnings.warn(msg)
Apr 30 00:29:25 pulphost.domain.local rq[1374]: pulp [28ac6dceafad4e07933ddfbeead891e7]: pulp_deb.app.tasks.synchronizing:INFO: Artifact digest not matched. Ignored
<snip>
Apr 30 00:29:28 pulphost.domain.local rq[1374]: pulp [28ac6dceafad4e07933ddfbeead891e7]: pulp_deb.app.tasks.synchronizing:INFO: Artifact digest not matched. Ignored
Apr 30 00:29:30 pulphost.domain.local gunicorn[1381]: pulp [953559dc9eb149849798a84899c1fa04]:  - admin [30/Apr/2021:00:29:30 +0000] "GET /pulp/api/v3/tasks/cb2c2cb4-c3ca-4758-be66-faa930487dad/ HT
TP/1.0" 200 1391 "-" "HTTPie/0.9.8"
Apr 30 00:30:03 pulphost.domain.local gunicorn[1381]: pulp [c2c1187c01de41d3bd0ea2ca6ff96079]:  - admin [30/Apr/2021:00:30:03 +0000] "GET /pulp/api/v3/tasks/8f2574c3-56f4-40f3-b0c3-31227bef42fd/ HT
TP/1.0" 200 602 "-" "HTTPie/0.9.8"
Apr 30 00:30:37 pulphost.domain.local rq[1388]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: [email protected]
Apr 30 00:30:47 pulphost.domain.local rq[1374]: pulp [28ac6dceafad4e07933ddfbeead891e7]: asyncio:ERROR: Task was destroyed but it is pending!
Apr 30 00:30:47 pulphost.domain.local rq[1374]: task: <Task pending coro=<Event.wait() running at /usr/lib/python3.6/asyncio/locks.py:283> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object
 at 0x7d07428b8558>()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.6/asyncio/tasks.py:380]>
Apr 30 00:30:47 pulphost.domain.local rq[1374]: pulp [28ac6dceafad4e07933ddfbeead891e7]: asyncio:ERROR: Task was destroyed but it is pending!
Apr 30 00:30:47 pulphost.domain.local rq[1374]: task: <Task pending coro=<Event.wait() running at /usr/lib/python3.6/asyncio/locks.py:283> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object
 at 0x7d07428b8498>()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.6/asyncio/tasks.py:380]>
Apr 30 00:30:47 pulphost.domain.local rq[1374]: pulp [28ac6dceafad4e07933ddfbeead891e7]: asyncio:ERROR: Task was destroyed but it is pending!
Apr 30 00:30:47 pulphost.domain.local rq[1374]: task: <Task pending coro=<Event.wait() running at /usr/lib/python3.6/asyncio/locks.py:283> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object
 at 0x7d07428b83d8>()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.6/asyncio/tasks.py:380]>
Apr 30 00:30:47 pulphost.domain.local rq[1374]: pulp [28ac6dceafad4e07933ddfbeead891e7]: asyncio:ERROR: Task was destroyed but it is pending!
Apr 30 00:30:47 pulphost.domain.local rq[1374]: task: <Task pending coro=<Event.wait() running at /usr/lib/python3.6/asyncio/locks.py:283> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object
 at 0x7d07428b8318>()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.6/asyncio/tasks.py:380]>
Apr 30 00:30:47 pulphost.domain.local rq[1374]: pulp [28ac6dceafad4e07933ddfbeead891e7]: asyncio:ERROR: Task was destroyed but it is pending!
Apr 30 00:30:47 pulphost.domain.local rq[1374]: task: <Task pending coro=<Event.wait() running at /usr/lib/python3.6/asyncio/locks.py:283> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object
 at 0x7d07428b8138>()]>>
Apr 30 00:30:47 pulphost.domain.local rq[1374]: pulp [28ac6dceafad4e07933ddfbeead891e7]: asyncio:ERROR: Task was destroyed but it is pending!
Apr 30 00:30:47 pulphost.domain.local rq[1374]: task: <Task pending coro=<Event.wait() running at /usr/lib/python3.6/asyncio/locks.py:283> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object
 at 0x7d07428b8258>()]> cb=[_wait.<locals>._on_completion() at /usr/lib/python3.6/asyncio/tasks.py:380]>
Apr 30 00:30:48 pulphost.domain.local rq[1374]: pulp [28ac6dceafad4e07933ddfbeead891e7]: rq.worker:ERROR: Traceback (most recent call last):
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/rq/worker.py", line 1008, in perform_job
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     rv = job.perform()
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/rq/job.py", line 706, in perform
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     self._result = self._execute()
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/rq/job.py", line 729, in _execute
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     result = self.func(*self.args, **self.kwargs)
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py", line 122, in synchronize
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     loop.run_until_complete(pipeline)
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     return future.result()
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     await asyncio.gather(*futures)
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     await self.run()
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 174, in run
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     pb.done += task.result()  # download_count
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 200, in _handle_content_unit
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     await asyncio.gather(*downloaders_for_content)
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py", line 89, in download
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     download_result = await downloader.run(extra_data=self.extra_data)
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/download/base.py", line 241, in run
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     return await self._run(extra_data=extra_data)
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/backoff/_async.py", line 133, in retry
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     ret = await target(*args, **kwargs)
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/download/http.py", line 218, in _run
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     to_return = await self._handle_response(response)
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/download/http.py", line 185, in _handle_response
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     chunk = await response.content.read(1048576)  # 1 megabyte
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/aiohttp/streams.py", line 380, in read
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     await self._wait("read")
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/aiohttp/streams.py", line 306, in _wait
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     await waiter
Apr 30 00:30:48 pulphost.domain.local rq[1374]: aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
Apr 30 00:30:48 pulphost.domain.local rq[1374]: Traceback (most recent call last):
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/rq/worker.py", line 1008, in perform_job
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     rv = job.perform()
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/rq/job.py", line 706, in perform
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     self._result = self._execute()
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/rq/job.py", line 729, in _execute
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     result = self.func(*self.args, **self.kwargs)
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulp_deb/app/tasks/synchronizing.py", line 122, in synchronize
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 149, in create
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     loop.run_until_complete(pipeline)
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     return future.result()
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     await asyncio.gather(*futures)
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     await self.run()
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 174, in run
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     pb.done += task.result()  # download_count
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 200, in _handle_content_unit
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     await asyncio.gather(*downloaders_for_content)
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/plugin/stages/models.py", line 89, in download
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     download_result = await downloader.run(extra_data=self.extra_data)
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/download/base.py", line 241, in run
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     return await self._run(extra_data=extra_data)
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/backoff/_async.py", line 133, in retry
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     ret = await target(*args, **kwargs)
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/download/http.py", line 218, in _run
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     to_return = await self._handle_response(response)
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/pulpcore/download/http.py", line 185, in _handle_response
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     chunk = await response.content.read(1048576)  # 1 megabyte
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/aiohttp/streams.py", line 380, in read
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     await self._wait("read")
Apr 30 00:30:48 pulphost.domain.local rq[1374]:   File "/usr/local/lib/pulp/lib/python3.6/site-packages/aiohttp/streams.py", line 306, in _wait
Apr 30 00:30:48 pulphost.domain.local rq[1374]:     await waiter
Apr 30 00:30:48 pulphost.domain.local rq[1374]: aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed

Make various release file fields user configurable

Is your feature request related to a problem? Please describe.

This feature is intended to provide a long term solution for #443 while also empowering users to use pulp_deb repos in combination with "APT pinning" and "unattended upgrade" workflows.

Describe the solution you'd like

  • The upstream Origin, Label, and Version fields should be stored on the release content during sync (if present).
  • Users should be able to define their own Origin, Label, and Version values on repositories and/or publications.
  • User supplied values trump upstream repo values during publish

If possible, consider the possibility of respecting arbitrary upstream and user supplied release file fields?

Additional context

#443 Should provide a backportable bugfix solution for the same problem, while this ticket provides a new design approach as a long term solution.

FORCE_IGNORE_MISSING_PACKAGE_INDICES=True Where does it go?

No suitable Package index file found in 'dists/focal/main/binary-arm64'. If you are syncing from a partial mirror, try setting ignore_missing_package_indices to 'True' on your remote.

I'm trying my best to figure out where exactly apply this variable - or preferably its global version FORCE_IGNORE_MISSING_PACKAGE_INDICES=True - in order to synchronize my Ubuntu 20.04.4 LTS repository with my Foreman+Katello setup. If someone - anyone - could please elaborate comprehensively where I need to put this variable and how to enable it properly, I would greatly appreciate it.

Foreman+Katello is running on Rocky Linux 8.5 and is able to synchronize with its repository just fine. The only repository I'm having issue with is Ubuntu.

Software and Versions

pulp --version returns the following:
pulp3 command line interface, version 0.14.0
pulpcore-selinux-1.3.0-1.el8.x86_64
python38-pulp-ansible-0.10.1-1.el8.noarch
python38-pulp-certguard-1.5.1-1.el8.noarch
python38-pulp-cli-0.14.0-1.el8.noarch
python38-pulp-container-2.9.2-1.el8.noarch
python38-pulp-deb-2.16.1-1.el8.noarch
python38-pulp-file-1.10.1-1.el8.noarch
python38-pulp-python-3.5.2-1.el8.noarch
python38-pulp-rpm-3.17.3-2.el8.noarch
python38-pulpcore-3.16.6-1.el8.noarch
Katello 4.5.0
Foreman 3.3.0

Add test coverage for per repository version uniqueness constraints

Author: @quba42 (quba42)

Redmine Issue: 7441, https://pulp.plan.io/issues/7441


Provide test coverage for: https://pulp.plan.io/issues/6429

I deferred the test coverage in the interest of getting this issue merged, but I still think test coverage would be useful (and at this point I have a fairly detailed idea of what would be needed):

Minimal test:

Create a repository
Upload https://fixtures.pulpproject.org/debian/pool/asgard/o/odin/odin_1.0_ppc64.deb to it.
Also upload https://fixtures.pulpproject.org/debian-complex-dists/pool/asgard/o/odin/odin_1.0_ppc64.deb to it.
Assert that the second repository version has only one package in it, and that the first package was removed from the version.

If we want to be fancy:

Add both packages to an empty repository in a single API call.
Assert that the relevant error is thrown.

Add both packages to the repository from the minimal test using a single API call.
Assert that the new version still only has one package in it.

Extra fancy:

Build a third colliding package in the fixtures.
Add all three to the repository from the minimal test in a single API call.
Assert that the relevant error is thrown.

Support the debug .ddeb package type

Author: lieter (lieter)

Redmine Issue: 6757, https://pulp.plan.io/issues/6757


From Ubuntu Bionic onward, Canonical has patched debuild to change the suffix for debug ($package-dbgsym_$version) package from .deb to .ddeb.

When attempting to upload packages with this suffix, the tasks has the following error:

  File "/home/lieter/.virtualenvs/pulp3-devel/lib/python3.8/site-packages/rq/worker.py", line 886, in perform_job
    rv = job.perform()
  File "/home/lieter/.virtualenvs/pulp3-devel/lib/python3.8/site-packages/rq/job.py", line 664, in perform
    self._result = self._execute()
  File "/home/lieter/.virtualenvs/pulp3-devel/lib/python3.8/site-packages/rq/job.py", line 670, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/home/lieter/src/Pulp/pulpcore/pulpcore/app/tasks/base.py", line 40, in general_create
    serializer.is_valid(raise_exception=True)
  File "/home/lieter/.virtualenvs/pulp3-devel/lib/python3.8/site-packages/rest_framework/serializers.py", line 243, in is_valid
    raise ValidationError(self.errors)

This is because the plugin (rightfully) assumes that all packages are called .deb (or .udeb for installer packages). I believe adding a 'DebugPackage` model that inherits from Package (changing only SUFFIX) would be a good addition to pulp_deb.

The workaround here would be to upload the .ddeb with a relatve_path ending with .deb, but I don't know if Ubuntu's apt/dpkg would like that (afaik, it accepts it).

KeyError: 'amd64' when creating publications during 2to3 migration

Author: @quba42 (quba42)

Redmine Issue: 9165, https://pulp.plan.io/issues/9165


Reported here: https://community.theforeman.org/t/katello-column-deb-aptremote-ignore-missing-package-indices-does-not-exist/24524/3

Pastebin excerpt: https://pastebin.com/h10qnbA4

Jul 25 11:23:49 foreman.com01.comcorp.lan pulpcore-api[50733]: - - [25/Jul/2021:09:23:49 +0000] "GET /pulp/api/v3/task-groups/0b552ed5-49b4-4f6d-9c42-429219d4e016/ HTTP/1.1" 200 439 "-" "OpenAPI-Generator/3.7.1/ruby"
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: pulp: rq.worker:ERROR: Traceback (most recent call last):
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: rv = job.perform()
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: self._result = self._execute()
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: return self.func(*self.args, **self.kwargs)
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 231, in complex_repo_migration
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: migrated_repo.pulp3_repository_version
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 478, in migrate_repo_distributor
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: pulp2dist, repo_version)
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/deb/repository.py", line 74, in migrate_to_pulp3
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: publish(repo_version.pk, simple=True, structured=True)
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: File "/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/publishing.py", line 156, in publish
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: prc.package
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: File "/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/publishing.py", line 195, in add_package
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: self.package_index_files[package.architecture][0]
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: KeyError: 'amd64'
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: Traceback (most recent call last):
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: rv = job.perform()
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: self._result = self._execute()
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: return self.func(*self.args, **self.kwargs)
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 231, in complex_repo_migration
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: migrated_repo.pulp3_repository_version
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 478, in migrate_repo_distributor
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: pulp2dist, repo_version)
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/deb/repository.py", line 74, in migrate_to_pulp3
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: publish(repo_version.pk, simple=True, structured=True)
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: File "/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/publishing.py", line 156, in publish
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: prc.package
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: File "/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/publishing.py", line 195, in add_package
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: self.package_index_files[package.architecture][0]
Jul 25 11:24:04 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: KeyError: 'amd64'
Jul 25 11:24:05 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: pulp: rq.worker:INFO: [email protected]: 647d4983-0cb4-4da0-a29d-d2af0227b772
Jul 25 11:24:05 foreman.com01.comcorp.lan pulpcore-worker-3[50804]: pulp: rq.worker:INFO: [email protected]: Job OK (647d4983-0cb4-4da0-a29d-d2af0227b772)
Jul 25 11:24:05 foreman.com01.comcorp.lan pulpcore-api[50733]: - - [25/Jul/2021:09:24:05 +0000] "GET /pulp/api/v3/tasks/f03587fe-492f-473c-9ec9-8f0700917150/ HTTP/1.1" 200 8183 "-" "OpenAPI-Generator/3.7.1/ruby"
Jul 25 11:24:05 foreman.com01.comcorp.lan pulpcore-api[50733]: - - [25/Jul/2021:09:24:05 +0000] "GET /pulp/api/v3/task-groups/0b552ed5-49b4-4f6d-9c42-429219d4e016/ HTTP/1.1" 200 439 "-" "OpenAPI-Generator/3.7.1/ruby"

Add test to check validity of distributed repos

Author: @quba42 (quba42)

Redmine Issue: 6901, https://pulp.plan.io/issues/6901


We should have a test that synchronizes a fully valid repository from the fixtures, creates, a publication, and finally a distribution, and finally tests that distribution for validity.

Ideally we would use an actual APT client to do this, and possibly even test installing a package from the test repository.

Handle upstream package indices containing packages with the wrong architecture

Minimal reproducer:

pip install pulp-cli-deb

# Constants:
ENTITIES_NAME='test'
REMOTE_OPTIONS='--url=https://apt.releases.hashicorp.com --distribution=bionic --component=main --architecture=amd64 --policy=on_demand'

# Steps using CLI:
pulp deb remote create --name=${ENTITIES_NAME} ${REMOTE_OPTIONS}
pulp deb repository create --name=${ENTITIES_NAME} --remote=${ENTITIES_NAME}
pulp deb repository sync --name=${ENTITIES_NAME}
APT_PUBLICATION_HREF=$(pulp deb publication create --repository=${ENTITIES_NAME} --simple=True --structured=True | jq -r '.pulp_href')
# Creating the publication fails!
pulp deb distribution create --name=${ENTITIES_NAME} --base-path=${ENTITIES_NAME} --publication=${APT_PUBLICATION_HREF}

Error message:

    "error"=>
     {"traceback"=>
       "  File \"/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 317, in _perform_task\n" +
       "    result = func(*args, **kwargs)\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/publishing.py\", line 168, in publish\n" +
       "    prc.package\n" +
       "  File \"/usr/lib/python3.6/site-packages/pulp_deb/app/tasks/publishing.py\", line 207, in add_package\n" +
       "    self.package_index_files[package.architecture][0]\n",
      "description"=>"'arm64'"},

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.