This is the pulp_rpm
Plugin for Pulp Project
3.0+. This plugin provides support for RPM family
content types, similar to the pulp_rpm
plugin for Pulp 2.
For more information, please see the documentation or the Pulp project page.
RPM support for Pulp Platform
Home Page: https://pulpproject.org/pulp_rpm/
License: GNU General Public License v2.0
This is the pulp_rpm
Plugin for Pulp Project
3.0+. This plugin provides support for RPM family
content types, similar to the pulp_rpm
plugin for Pulp 2.
For more information, please see the documentation or the Pulp project page.
Author: @goosemania (ttereshc)
Date: 2019-07-29
Redmine Issue: 5110, https://pulp.plan.io/issues/5110
An a epic ticket which should define a feature design here.
Author: @fao89 (fao89)
Redmine Issue: 6427, https://pulp.plan.io/issues/6427
Author: @fao89 (fao89)
Redmine Issue: 6343, https://pulp.plan.io/issues/6343
Note: kickstart repos also have sub-repos
Reference:
#1268
OrphansTestCase - https://github.com/pulp/Pulp-2-Tests/blob/master/pulp_2_tests/tests/rpm/api_v2/test_orphan_remove.py#L33
Fixtures:
https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-kickstart/
https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-signed/
Author: kersom (kersom)
Redmine Issue: 5261, https://pulp.plan.io/issues/5261
Add more tests around support for modular content.
Author: @fao89 (fao89)
Redmine Issue: 6350, https://pulp.plan.io/issues/6350
https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/tests/functional/api/test_copy.py
Author: @dkliban ([email protected])
Redmine Issue: 6464, https://pulp.plan.io/issues/6464
Some RHEL 5 repositories do not contain the variant
field in their "treeinfo" files. Can pulp 3 handle that?
Author: @goosemania (ttereshc)
Redmine Issue: 5744, https://pulp.plan.io/issues/5744
Currently there is no enforcement at the db level to require certain fields for the majority of content models.
Package needs NEVRA, pkgId, checksum_type at minimum.
UpdateRecord - id, issued_date, updated_date, digest
UpdateCollectionPackage - NEVRA
Modulemd - NSVCA
Module-defaults - module, digest
PackageGroup/Environment/Category - id, digest
PackageLangpacks - matches, digest
RepoMetadataFile - data_type, checksum
DistributionTree - header_version, release_name, release_version, arch, build_timestamp
Checksum - path, checksum
Image - name, path
Addon - addon_id, uid, name, type, packages
Variant - variant_id, uid, name, type, packages
Author: kersom (kersom)
Redmine Issue: 4215, https://pulp.plan.io/issues/4215
When users upload an invalid RPM such as one that has non-utf8 characters in their headers, they should get a 4xx response and a user friendly error message. Currently they receive a 500.
Steps to reproduce.
http POST http://localhost:8000/pulp/api/v3/repositories/ name=foo
export REPO_HREF=$(http :8000/pulp/api/v3/repositories/ | jq -r '.results[] | select(.name == "foo") | ._href')
wget https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-with-non-utf-8/rpm-with-non-utf-8-1-1.fc25.noarch.rpm
http --form POST http://localhost:8000/pulp/api/v3/artifacts/ file@./rpm-with-non-utf-8-1-1.fc25.noarch.rpm
http POST http://localhost:8000/pulp/api/v3/content/rpm/packages/ relative_path=rpm-with-non-utf-8-1-1.fc25.noarch.rpm artifact="/pulp/api/v3/artifacts/1/" filename=rpm-with-non-utf8.rpm
Traceback
HTTP/1.1 500 Internal Server Error
Content-Length: 20215
Content-Type: text/plain; charset=utf-8
Date: Thu, 29 Nov 2018 15:41:48 GMT
Server: WSGIServer/0.2 CPython/3.7.1
Vary: Cookie
X-Frame-Options: SAMEORIGIN
UnicodeDecodeError at /pulp/api/v3/content/rpm/packages/
'utf-8' codec can't decode byte 0x80 in position 168: invalid start byte
Request Method: POST
Request URL: http://localhost:8000/pulp/api/v3/content/rpm/packages/
Django Version: 2.1.3
Python Executable: /home/vagrant/.virtualenvs/pulp/bin/python3
Python Version: 3.7.1
Python Path: ['/home/vagrant/.virtualenvs/pulp/bin', '/usr/lib64/python37.zip', '/usr/lib64/python3.7', '/usr/lib64/python3.7/lib-dynload', '/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages', '/home/vagrant/devel/pulp/pulpcore', '/home/vagrant/devel/pulp/plugin', '/home/vagrant/pulp_rpm', '/home/vagrant/.virtualenvs/pulp/lib/python3.7/site-packages']
Server time: Thu, 29 Nov 2018 15:41:48 +0000
Installed Applications:
['django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django_filters',
'drf_yasg',
'rest_framework',
'pulpcore.app',
'pulp_file.app.PulpFilePluginAppConfig',
'pulp_rpm.app.PulpRpmPluginAppConfig',
'crispy_forms',
'django_extensions']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware']
Traceback:
File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/core/handlers/exception.py" in inner
34. response = get_response(request)
File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/core/handlers/base.py" in _get_response
126. response = self.process_exception_by_middleware(e, request)
File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/core/handlers/base.py" in _get_response
124. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
54. return view_func(*args, **kwargs)
File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/viewsets.py" in view
116. return self.dispatch(request, *args, **kwargs)
File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/views.py" in dispatch
495. response = self.handle_exception(exc)
File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/views.py" in handle_exception
455. self.raise_uncaught_exception(exc)
File "/home/vagrant/.virtualenvs/pulp/lib64/python3.7/site-packages/rest_framework/views.py" in dispatch
492. response = handler(request, *args, **kwargs)
File "/usr/lib64/python3.7/contextlib.py" in inner
74. return func(*args, **kwds)
File "/home/vagrant/pulp_rpm/pulp_rpm/app/viewsets.py" in create
96. package = Package.createrepo_to_dict(cr_pkginfo)
File "/home/vagrant/pulp_rpm/pulp_rpm/app/models.py" in createrepo_to_dict
243. 'description': getattr(package, CREATEREPO_PACKAGE_ATTRS.DESCRIPTION) or '',
Exception Type: UnicodeDecodeError at /pulp/api/v3/content/rpm/packages/
Exception Value: 'utf-8' codec can't decode byte 0x80 in position 168: invalid start byte
Request information:
USER: admin
GET: No GET data
POST: No POST data
FILES: No FILES data
COOKIES: No cookie data
META:
CONTENT_LENGTH = '137'
CONTENT_TYPE = 'application/json'
[vagrant@p3 ~]$ cat /etc/redhat-release
Fedora release 29 (Twenty Nine)
Author: kersom (kersom)
Redmine Issue: 5271, https://pulp.plan.io/issues/5271
When a package is created without a relative path set, that package causes an exception during publishing.
https://www.redhat.com/archives/pulp-list/2019-May/msg00018.html
Author: @pavelpicka (ppicka)
Redmine Issue: 6605, https://pulp.plan.io/issues/6605
If advisory has same ID, version but update_date missing sync will fail.
If updated_date missing in advisory issued_date is not take into consideration in advisory conflict resolution time.
How to reproduce:
git clone https://github.com/pulp/pulp-fixtures
cd pulp-fixtures
1. create fixtures
make fixtures/rpm-unsigned
make fixtures/rpm-advisory-no-update-date
"error" {
"description": "'<' not supported between instances of 'datetime.datetime' and 'NoneType'"
...}
Author: @dkliban ([email protected])
Redmine Issue: 6629, https://pulp.plan.io/issues/6629
When I tried to delete a kickstart repository I got the following error in my Task:
(pulp) [vagrant@pulp2-nightly-pulp3-source-centos7 ~]$ http :/pulp/api/v3/tasks/c59c1a92-1ce5-4424-a948-7fbf1124b4f5/
HTTP/1.1 200 OK
Allow: GET, PATCH, DELETE, HEAD, OPTIONS
Connection: Keep-Alive
Content-Length: 1755
Content-Type: application/json
Date: Fri, 01 May 2020 20:13:22 GMT
Keep-Alive: timeout=5, max=10000
Server: gunicorn/20.0.4
Vary: Accept,Cookie
X-Frame-Options: SAMEORIGIN
{
"child_tasks": [],
"created_resources": [],
"error": {
"description": "(\"Cannot delete some instances of model 'Repository' because they are referenced through a protected foreign key: 'Variant.repository'\", <QuerySet [<Variant: BaseOS>]>)",
"traceback": " File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/worker.py\", line 886, in perform_job\n rv = job.perform()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 664, in perform\n self._result = self._execute()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/rq/job.py\", line 670, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/home/vagrant/devel/pulpcore/pulpcore/app/tasks/repository.py\", line 22, in delete\n models.Repository.objects.filter(pk=repo_id).delete()\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/query.py\", line 710, in delete\n collector.collect(del_query)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/deletion.py\", line 224, in collect\n field.remote_field.on_delete(self, field, sub_objs, self.using)\n File \"/usr/local/lib/pulp/lib64/python3.6/site-packages/django/db/models/deletion.py\", line 27, in PROTECT\n sub_objs\n"
},
"finished_at": "2020-05-01T20:13:12.201492Z",
"name": "pulpcore.app.tasks.repository.delete",
"parent_task": null,
"progress_reports": [],
"pulp_created": "2020-05-01T20:13:11.926923Z",
"pulp_href": "/pulp/api/v3/tasks/c59c1a92-1ce5-4424-a948-7fbf1124b4f5/",
"reserved_resources_record": [
"/pulp/api/v3/repositories/rpm/rpm/9581879e-4b4c-4f53-86e9-90319e835ffb/"
],
"started_at": "2020-05-01T20:13:12.082540Z",
"state": "failed",
"task_group": null,
"worker": "/pulp/api/v3/workers/82124443-d497-4889-9075-bed24a6cb29c/"
}
Author: kersom (kersom)
Redmine Issue: 5408, https://pulp.plan.io/issues/5408
As a user, I can sync modulemd content
As a user, I can sync modulemd_defaults content
As a user, I can sync modular erratum content
Author: @fao89 (fao89)
Redmine Issue: 6349, https://pulp.plan.io/issues/6349
RPM_LARGE_METADATA
can be uploaded into the repo without errors.RPM_LARGE_METADATA = https://repos.fedorapeople.org/pulp/pulp/rpm_large_metadata/nodejs-babel-preset-es2015-6.6.0-2.el6.noarch.rpm
Author: @bmbouter (bmbouter)
Redmine Issue: 4203, https://pulp.plan.io/issues/4203
This machinery should run when a release tag is pushed indicating a new release, similar to how plugins publish to PyPI already. When activated it should:
1. Install the plugin on top of the generic, most recent core container.
2. start that container and run pulp-smash tests against it
3. If all the tests pass, publish the container to quay.io with credentials
The credentials need to be configured somehow in the config files.
Author: iballou (iballou)
Bugzilla: https://bugzilla.redhat.com/buglist.cgi?quicksearch=1854966
Redmine Issue: 7105, https://pulp.plan.io/issues/7105
Related issue: https://pulp.plan.io/issues/6851
I had a number of Katello users fix their applicability issues by applying the patch that resulted from the issue above. However, a couple users are now reporting that there is always one client that isn't reporting applicability properly. The users report that the applicability changes over time. One host might start showing applicability again, but then another fails. Full details are in the following thread, starting at the linked message: https://community.theforeman.org/t/katello-3-15-doesnt-show-available-package-updates-anymore/18948/89
I suspect the issue is in Pulp because the package profiles and bound repositories are correct on the consumers that are exhibiting the applicability issue.
Pulp versions in use:
pulp-server-2.21.2-1.el7.noarch
pulp-rpm-plugins-2.21.2-1.el7.noarch
(Plus pulp/pulp#3990 patched in)
Author: @fao89 (fao89)
Redmine Issue: 6348, https://pulp.plan.io/issues/6348
1. Ensure there is only one Pulp worker.
2. Create a repository. Let its feed reference a large repository. (For example, EPEL.)
3. Start a sync. Immediately restart the ``pulp_workers`` service. (It's
important that ``pulp_workers`` be restarted, not started and stopped.
For details, see #2835 ) This should cause the first sync to abort.
4. Update the repository. Let its feed reference a small repository. (For
example, ``https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-unsigned/``.)
5. Start a sync. Verify that it completes. If `Pulp #2835`_ still affects
Pulp, then the worker will be broken, and the sync will never start.
Reference:
#2835
Author: bherring (bherring)
Redmine Issue: 5055, https://pulp.plan.io/issues/5055
There is currently a 'policy decision' that modules in the base repo (of rhel) don't rely on modular rpms, but that it's not an engineering limitation, and that fedora and rhel will likely do so at some point in the future. This is already a consideration for fedora and fedora-updates repos.
At least in Fedora, you can expect that there will be content in the updates-testing repo depending on content in updates-modular and/or fedora-modular.
This means Pulp will eventually need to support multiple source repos for copy.
As pulp is delivered today, Ursine RPM deps on modular RPMs are NOT covered by test.
Normal, non-modular (ursine) [0] RPMs can depend on module packages.
As an example, the postgres language RPMs depend on the default stream of the postgresql module.
When this is the case, the default module and all artifacts related to that module and the URSINE RPM are ALL copied to the target repo.
At this time, to account for this scenario, a hybrid repo containing URSINE and MODULAR RPMs with modules will be required.
default module: module-FOO: [foo-1.0.rpm, bar-1.0.rpm, baz-1.0.rpm]
repo A
|
|---- chicken-1.1.rpm
|---- zebra-1.0.rpm
|----module-FOO
|----foo-1.0.rpm
|----bar-1.0.rpm
|----baz-1.0.rpm
repo B
|
|----bar-0.7.rpm
Result of copying ursine RPM zebra-1.0.rpm from repo A to repo B:
repo B
|
|---- zebra-1.0.rpm
|----module-FOO
|----foo-1.0.rpm
|----bar-1.0.rpm
|----baz-1.0.rpm
|----bar-0.7.rpm
All available artifacts are copied, always. There is no way to copy just module on its own,
if any of its artifacts are present in a source repo (repo A).
[0] - https://docs.fedoraproject.org/en-US/modularity/architecture/consuming/dnf-behavior/
[1] - https://docs.pulpproject.org/plugins/pulp_rpm/user-guide/features.html?highlight=modularity
Author: jcabrera (jcabrera)
Redmine Issue: 4879, https://pulp.plan.io/issues/4879
Add the possibility to query the file name ("location_href"?) of a package in the contents.
This simplifies the use of jq as we get directly the good href with the use of a GET query
http :24817/pulp/api/v3/content/rpm/packages/?filename=name-version-release.arch.rpm
I prefer the query version than using jq
http :24817/pulp/api/v3/content/rpm/packages/ | jq -r '.results[] | select( .location_href == "name-version-release.arch.rpm")'
There could be a limitation when there are to much contents because the pagination prevents to look for all contents.
Author: @pavelpicka (ppicka)
Redmine Issue: 7253, https://pulp.plan.io/issues/7253
Test to 'copy' with depsolving
create fixture repository with one or more package environments (may extend rpm-unsigned)
test package environment is copied with deps
test package category is copied with deps (rpm-unsigned fixture can be used)
Hi, I'm syncing a CentOS 5 repository for network installations and found a problem, RHEL5 family doesn't support 'sha256' hashing in xml repodata. Then I force to generate metadata with simple 'sha' with --checksum-type=sha1
in rpm create repo
command.
But the group metadata is ignoring this parameter and generating metadata with sha256 checksum and provocates installation fail. Or yum grouplist
in a running machine also fails.
<data type="group">
<locationhref="repodata/734915cb5f5a01aaccbcd072383947abf6a5644d52e092f9497b03e9d5270f73-group.xml.gz"/>
<checksum type="sha256">
734915cb5f5a01aaccbcd072383947abf6a5644d52e092f9497b03e9d5270f73
</checksum>
<timestamp>
1362049622.33
</timestamp>
<open-checksum type="sha256">
ed7ba6f88964212c9003b6f9e8f7cebf57a53a3b3a50f86bf943c1b092f546da
</open-checksum>
</data>
Author: @pavelpicka (ppicka)
Redmine Issue: 7077, https://pulp.plan.io/issues/7077
Test sync repository with auth token
As this is test about authentication the repository to use will be https://fixtures.pulpproject.org/rpm-unsigned/ (as all our fixtures can be used with [only one] auth token)
Author: CodeHeeler (CodeHeeler)
Redmine Issue: 5685, https://pulp.plan.io/issues/5685
Currently our fixtures do not include the PackageEnvironment type from comps.xml so our tests lack coverage for them.
Author: bherring (bherring)
Redmine Issue: 4824, https://pulp.plan.io/issues/4824
Pulp uses revision numbers in repomd.xml to determine if contents need to be updated on sync. However "modifyrepo" does not generate new revision numbers for non rpm data.
Steps:
1) Setup the following repo
$ mkdir /tmp/my-data
$ cd /tmp/my-data
$ wget https://partha.fedorapeople.org/test-repos/rpm-with-productid/elephant-0.3-0.8.noarch.rpm
$ createrepo .
$ echo "100000" >> productid
$ modifyrepo --mdtype=productid productid repodata
$ grep revision repodata/repomd.xml
<revision>1554217257</revision>
2) Sync this repo
3) Now update the repo
$ cd /tmp/my-data
$ echo "100001" >> productid
$ modifyrepo --mdtype=productid productid repodata
$ grep revision repodata/repomd.xml
<revision>1554217257</revision>
Notice that the revision number did not change even though a metadata file got updated. Try resyncing this change and notice that the productid change will get ignored.
Author: kersom (kersom)
Redmine Issue: 5276, https://pulp.plan.io/issues/5276
Simple content copy means that content units can be copied from one repository into another repository without considering content-type-specific invariants, such as making sure dependencies are satisfied.
The most straightforwards current design is to model this after one-shot upload. A new URL endpoint will be defined at /api/v3/rpm/copy/. POSTing to that endpoint with valid parameters will yield an asynchronous task, which will result in a new repository version in the target repository.
The user workflow should look like this:
http --form POST http://localhost:24817/pulp/api/v3/rpm/copy/ source_repository=${S_REPO_HREF} destination_repository=${D_REPO_HREF}
Both the repository parameters are required. When called, all content will be copied.
The additions to Django boilerplate will look something like this (see also: copy.py for an example of the function that does the work, and OneShotUploadSerializer in serializers.py)
urls.py
urlpatterns = [
url(r'rpm/upload/$', OneShotUploadView.as_view()),
url(r'rpm/copy/$', CopyView.as_view())
]
Author: carlgeorge (carlgeorge)
Redmine Issue: 1686, https://pulp.plan.io/issues/1686
I think it would be useful for pulp to be able to handle RPM signing.
If there is already a better alternative for this please let me know.
Thanks!
Author: @goosemania (ttereshc)
Redmine Issue: 4552, https://pulp.plan.io/issues/4552
In Fedora 30, Zchunk metadata is going to be present alongside with older compression formats.
Pulp3 should handle this type of metadata.
Ideally, it should be ignored during sync and optionally generated during publish (similar to sqlite dbs).
Repo with this new metadata type: http://mirror.vutbr.cz/fedora/development/rawhide/Everything/x86_64/os/
Author: @bmbouter (bmbouter)
Redmine Issue: 5354, https://pulp.plan.io/issues/5354
Some tests run long for example a full EPEL sync test takes 50 minutes. Each plugin iteslf could handle this, but it would be much better to have the plugin_template provide facilities.
Specifically a way to have a subset of tests run on Travis at PR time (instead of all tests) would be ideal.
Author: @goosemania (ttereshc)
Redmine Issue: 7175, https://pulp.plan.io/issues/7175
It's very confusing to list images (see section images
) and see only some of them, while having other images (from stage2 of treeinfo) on the main model,
{
"pulp_href": "/pulp/api/v3/content/rpm/distribution_trees/b94bb596-4aa7-4920-938e-66360cbba652/",
"header_version": "1.2",
"release_name": "CentOS Linux",
"release_short": "CentOS",
"release_version": "8",
"release_is_layered": false,
"base_product_name": null,
"base_product_short": null,
"base_product_version": null,
"arch": "x86_64",
"build_timestamp": 1591654006.0,
"instimage": null, <------------ instimage
"mainimage": "images/install.img", <------------ mainimage
"discnum": null,
"totaldiscs": null,
"addons": [],
"checksums": [
{
"path": "images/pxeboot/vmlinuz",
"checksum": "sha256:151fb7b594cb096e25769688cc748eefafeea133894878271bf9c428dba2cc71"
},
{
"path": "images/pxeboot/initrd.img",
"checksum": "sha256:dbffac7f93e35b936dfecc73cbf47e79f7c304fbd84b5df3aaf896ed695ec28e"
},
{
"path": "images/install.img",
"checksum": "sha256:4807e94ec4bd55b02a0f217dd19d34a4824ab04466092dd3ff9ca4bb67e8c3b6"
},
{
"path": "images/efiboot.img",
"checksum": "sha256:39659e208d746e1afc88784f12d8bbd58b074aafca4a65b70be877e22afb3936"
},
{
"path": "images/boot.iso",
"checksum": "sha256:c67876a5602faa17f68b40ccf2628799b87454aa67700f0f57eec15c6ccdd98c"
}
],
"images": [
{
"name": "kernel",
"path": "images/pxeboot/vmlinuz",
"platforms": "x86_64, xen",
"artifact": null
},
{
"name": "initrd",
"path": "images/pxeboot/initrd.img",
"platforms": "x86_64, xen",
"artifact": null
},
{
"name": "efiboot.img",
"path": "images/efiboot.img",
"platforms": "x86_64",
"artifact": null
},
{
"name": "boot.iso",
"path": "images/boot.iso",
"platforms": "x86_64",
"artifact": null
}
],
"variants": [
{
"variant_id": "BaseOS",
"uid": "BaseOS",
"name": "BaseOS",
"type": "variant",
"packages": "Packages",
"source_packages": null,
"source_repository": null,
"debug_packages": null,
"debug_repository": null,
"identity": null
},
{
"variant_id": "AppStream",
"uid": "AppStream",
"name": "AppStream",
"type": "variant",
"packages": "AppStream/Packages",
"source_packages": null,
"source_repository": null,
"debug_packages": null,
"debug_repository": null,
"identity": null
}
]
}
To reproduce sync http://mirror.linux.duke.edu/pub/centos/8/BaseOS/x86_64/kickstart/
The images/install.img
is not present among images but it's a part of a DistributionTree model itself. It is also present in the contentartifact table and is published correctly.
pulp=> select * from rpm_image;
pulp_id | pulp_created | pulp_last_updated | name | path | platforms
| distribution_tree_id
--------------------------------------+-------------------------------+-------------------------------+-------------+---------------------------+-------------
+--------------------------------------
6fed6e01-494a-4c77-9e1c-c85c07c9e48e | 2020-07-20 18:08:35.447812+00 | 2020-07-20 18:08:35.447826+00 | boot.iso | images/boot.iso | x86_64
| b94bb596-4aa7-4920-938e-66360cbba652
593b55a2-9474-437c-83ce-34b87885fa55 | 2020-07-20 18:08:35.447857+00 | 2020-07-20 18:08:35.447865+00 | efiboot.img | images/efiboot.img | x86_64
| b94bb596-4aa7-4920-938e-66360cbba652
82677b85-e707-4a28-affe-7a933cc48dbe | 2020-07-20 18:08:35.447889+00 | 2020-07-20 18:08:35.447896+00 | initrd | images/pxeboot/initrd.img | x86_64, xen
| b94bb596-4aa7-4920-938e-66360cbba652
1d8d8515-3fc0-410d-ac02-6dd317dac708 | 2020-07-20 18:08:35.44792+00 | 2020-07-20 18:08:35.447927+00 | kernel | images/pxeboot/vmlinuz | x86_64, xen
| b94bb596-4aa7-4920-938e-66360cbba652
(4 rows)
pulp=> select * from core_contentartifact where content_id='b94bb596-4aa7-4920-938e-66360cbba652';
pulp_id | pulp_created | pulp_last_updated | relative_path | artifact_id
| content_id
--------------------------------------+-------------------------------+-------------------------------+---------------------------+---------------------------
-----------+--------------------------------------
10ecfd6e-2046-4653-9313-508d4b112976 | 2020-07-20 18:08:35.414974+00 | 2020-07-20 18:08:35.41498+00 | .treeinfo | f8d285ea-c394-4da7-a0a4-40
e61b30f443 | b94bb596-4aa7-4920-938e-66360cbba652
0bdfcf83-6d44-43aa-a299-1844d2ccee44 | 2020-07-20 18:08:35.414996+00 | 2020-07-20 18:08:35.415002+00 | images/boot.iso |
| b94bb596-4aa7-4920-938e-66360cbba652
52cd8187-02e5-4c3a-a56e-7bbb12b064f0 | 2020-07-20 18:08:35.415016+00 | 2020-07-20 18:08:35.415022+00 | images/efiboot.img |
| b94bb596-4aa7-4920-938e-66360cbba652
f960003a-9cb5-4d19-a4b2-a72a0bcaa07d | 2020-07-20 18:08:35.415036+00 | 2020-07-20 18:08:35.415042+00 | images/pxeboot/initrd.img |
| b94bb596-4aa7-4920-938e-66360cbba652
78d19586-663a-4de3-81d7-811d8490eaad | 2020-07-20 18:08:35.415056+00 | 2020-07-20 18:08:35.415062+00 | images/pxeboot/vmlinuz |
| b94bb596-4aa7-4920-938e-66360cbba652
cd5c5408-f76e-482f-8cd3-8b941c412588 | 2020-07-20 18:08:35.415076+00 | 2020-07-20 18:08:35.415081+00 | images/install.img |
| b94bb596-4aa7-4920-938e-66360cbba652
(6 rows)
mainimage
which are not tight to a platform.Data migration is needed to alter the schema and to create missing images.
Author: kersom (kersom)
Redmine Issue: 4758, https://pulp.plan.io/issues/4758
Checksum in a filename helps with caching, allows to have old repodata in the directory, is a default way of publishing in Fedora/Centos/SUSE repos.
Pulp2 publishes with a checksum in the filenames.
<?xml version="1.0" encoding="UTF-8"?>
<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm">
<revision>1537430242</revision>
<data type="primary">
<checksum type="sha256">7d053b4c5880ecec3b8d5c54e0f436860953fbf69d936f1c54e10346c775e40b</checksum>
<open-checksum type="sha256">7ff98c3c80e3a82b6addc196de712f5744af8283bb51d39d053c9000fae03e57</open-checksum>
<location href="repodata/primary.xml.gz"/> <------------------------------------ no checksum prefix
<timestamp>1537430242</timestamp>
<size>4164</size>
<open-size>34974</open-size>
</data>
<data type="filelists">
<checksum type="sha256">761a77c037257b989cbb0c65413419e55c5ee79ef4b0b6a33d262f1eb011443c</checksum>
<open-checksum type="sha256">cbfd64a483c9888d3aa184938f92dd2427c6d6dd298cf83ce9d5f08359ad608d</open-checksum>
<location href="repodata/filelists.xml.gz"/> <------------------------------------ no checksum prefix
<timestamp>1537430242</timestamp>
<size>2110</size>
<open-size>6912</open-size>
</data>
<data type="other">
<checksum type="sha256">484e0c3f42959669ae5e627db74b944c160e71097be714d0394b255b8f92c27f</checksum>
<open-checksum type="sha256">1a66e06a6a6a45722b7e712c05ce780744006c582f89ecbec17271d42e7cd815</open-checksum>
<location href="repodata/other.xml.gz"/> <------------------------------------ no checksum prefix
<timestamp>1537430242</timestamp>
<size>1933</size>
<open-size>5830</open-size>
</data>
<data type="updateinfo">
<checksum type="sha256">2db62d73ee56ed9cb5e6c4b5d2012f7b1625cbf530b63e48eafbf8d9ec4f1949</checksum>
<open-checksum type="sha256">c9a07b3c82c402ff60d5f47b29ef10f780794d074c2ec968324a8485ca55acbd</open-checksum>
<location href="repodata/updateinfo.xml.gz"/> <------------------------------------ no checksum prefix
<timestamp>1537430242</timestamp>
<size>906</size>
<open-size>32964</open-size>
</data>
...
</repomd>
Author: @fao89 (fao89)
Redmine Issue: 6340, https://pulp.plan.io/issues/6340
comps.xml
.Reference:
#1787
similar test: ValidateNoChecksumTagTestCase - https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/tests/functional/api/test_publish.py#L232
Fixture: https://repos.fedorapeople.org/pulp/pulp/fixtures/srpm/
Author: @lubosmj (lmjachky)
Redmine Issue: 5805, https://pulp.plan.io/issues/5805
The commit merged via #1517 resolved the issue https://pulp.plan.io/issues/5699. It seems like we are missing tests which validate a single upload and its publication.
Therefore, the following test scenario is proposed:
1. Create an artifact:
http --form POST :24817/pulp/api/v3/artifacts/ file@./giraffe-0.67-2.noarch.rpm
2. Create an RPM package:
http POST :24817/pulp/api/v3/content/rpm/packages/ artifact="/pulp/api/v3/artifacts/ce225ae1-9cbd-4c20-8f31-c16ab2f0a6c6/" relative_path=giraffe-0.67-2.noarch.rpm
3. Add the created content to an empty repository:
http POST :24817/pulp/api/v3/repositories/rpm/rpm/75c0c59f-d453-42d3-8d6d-a53561b5d516/modify/ add_content_units:="[\"/pulp/api/v3/content/rpm/packages/3c2ac34c-470e-41b7-85b3-b3830efd4722/\"]"
4. Create a publication and check if the corresponding task was finished with a success:
http :24817/pulp/api/v3/publications/rpm/rpm/ repository=/pulp/api/v3/repositories/rpm/rpm/75c0c59f-d453-42d3-8d6d-a53561b5d516/
Author: @fao89 (fao89)
Redmine Issue: 6426, https://pulp.plan.io/issues/6426
delete all artifacts available, no error if not all artifacts are present
Author: @fao89 (fao89)
Redmine Issue: 6341, https://pulp.plan.io/issues/6341
Reference:
#4930
CheckIsModularFlagAfterSyncTestCase - https://github.com/pulp/Pulp-2-Tests/blob/master/pulp_2_tests/tests/rpm/api_v2/test_modularity.py#L146
Fixture: https://repos.fedorapeople.org/repos/pulp/pulp/fixtures/rpm-with-modular/
Author: daviddavis (daviddavis)
Redmine Issue: 3942, https://pulp.plan.io/issues/3942
Currently, we have a digest field that captures a hash digest of the update record's xml. The problem is that this is very fragile and could change based on stuff like whitespace or character encoding.
Instead, store the hash digest of UpdateRecord's fields along with the its UpdateCollections and UpdateCollectionPackages. using a detereministic data structure like an ordered dictionary.
Author: lieter (lieter)
Redmine Issue: 6719, https://pulp.plan.io/issues/6719
The pulpcore API supports uploading content for an artifact in a chunked way (https://docs.pulpproject.org/en/3.3/restapi.html#operation/uploads_update). It would be great if the RPM plugin could do this for a Package as well. This will allow a user to upload an RPM in a chunked way, without having to go through the motions of making a Package from the Artifact.
Extend SingleArtifactContentUploadSerializer
to accept a chunked upload id. This upload id can be used instead of a file to create a content unit.
Author: @goosemania (ttereshc)
Redmine Issue: 5433, https://pulp.plan.io/issues/5433
It should be uploaded using a dedicated endpoint and include a possibility to add it to a repo version.
Author: @fao89 (fao89)
Redmine Issue: 6783, https://pulp.plan.io/issues/6783
Currently, it is not possible to test:
https://github.com/pulp/pulp_rpm/blob/master/docs/_scripts/docs_check_install_with_signed_repodata.sh
because it needs a dnf install to test the installation:
https://github.com/pulp/pulp_rpm/blob/master/docs/_scripts/install_from_signed_repository.sh
As travis runs on ubuntu, for executing this workflow we will need to run docs tests inside the single container
Author: @fao89 (fao89)
Redmine Issue: 6408, https://pulp.plan.io/issues/6408
https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/shared_utils.py#L35
is_previous_revision is utilized on sync optimization, we need to test if it could handle revisions in many formats (timestamp, string, None).
Compare revisions in many formats
is_previous_revision("3.0.0", "2.0.0") -> False
is_previous_revision("2.0.0", "3.0.0") -> True
is_previous_revision("2.0.0", "2.0.0") -> True
is_previous_revision(1234, 1235) -> True #timestamp
is_previous_revision(1235, 1234) -> False #timestamp
is_previous_revision(1235, "2.0.0") -> False
is_previous_revision("2.0.0", 1235) -> False
is_previous_revision("3.1.0", "2.15.0") -> False
is_previous_revision("2.15.0", "3.1.0") -> True
is_previous_revision(None, "3.1.0") -> False
is_previous_revision("3.1.0", None) -> False
is_previous_revision(None, 1234) -> False
is_previous_revision(1235, None) -> False
pytest parametrize: http://doc.pytest.org/en/latest/example/parametrize.html#parametrizing-conditional-raising
Author: daviddavis (daviddavis)
Redmine Issue: 5556, https://pulp.plan.io/issues/5556
Previously, we were exposing plugin_managed repos in core but since that was removed, we need a way in pulp_rpm to expose these repos.
Author: @fao89 (fao89)
Redmine Issue: 6476, https://pulp.plan.io/issues/6476
on repomd we have:
primary.xml.gz
update_info.xml.gz
When testing xml.gz behaves like it is not compressed
On the other hand, it behaves as compressed on S3 testing
https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/tests/functional/api/test_publish.py#L51-L67
def read_xml_gz(content):
"""
Read xml and xml.gz.
Tests work normally but fails for S3 due '.gz'
Why is it only compressed for S3?
"""
with NamedTemporaryFile() as temp_file:
temp_file.write(content)
temp_file.seek(0)
try:
content_xml = gzip.open(temp_file.name).read()
except OSError:
# FIXME: fix this as in CI primary/update_info.xml has '.gz' but it is not gzipped
content_xml = temp_file.read()
return content_xml
Line 123 in comps_utils.py. I seem to get a SON object from obj['conditional_package_names'], which you can not iterate through that way (you get Exception: Too many values to unpack).
Adding .items() fixes the issue , which then returns a proper list. But I guess I'm not suppose to get a SON-object inside the obj['conditional_package_names'] am I right?
This is what obj['conditional_package_names'] looks like: SON([(u'libreoffice-langpack-en', u'libreoffice-core')])
Repository that I'm trying to sync is: rhel-6-x86_64-optional
Author: @dralley (dalley)
Redmine Issue: 6271, https://pulp.plan.io/issues/6271
A user should be able to upload a module metadata YAML file to Pulp, which will create modulemd and modulemd-default content units.
When new modules are created, they should be automatically associated with any referenced RPMs already present or which are added in the future.
In case modulemd-deafults, there is restriction that only one modulemd-defaults can exist for a repository, so we will either need to have a total replacement strategy, a merge strategy like with Errata, or require the user to delete the existing ones first. Validation will be very important.
Author: kersom (kersom)
Redmine Issue: 4149, https://pulp.plan.io/issues/4149
I used the auto generated docs in order to create an errata, and still not clear. Attempt was using the endpoint ` /pulp/api/v3/content/rpm/errata/` .
Add documentation about Errata in Pulp3, and how users can interact with Erratas in Pulp 3, what features are available, and so on.
Author: @fao89 (fao89)
Redmine Issue: 6345, https://pulp.plan.io/issues/6345
Currently, we don't have a good fixture for testing kickstarts:
https://repos.fedorapeople.org/pulp/pulp/fixtures/rpm-kickstart/
As it does not have content in its sub-repositories
Options:
Goal:
Currently, the tests are just checking the added contents for the main repo,
we should test the sub-repos also
Author: @goosemania (ttereshc)
Redmine Issue: 6765, https://pulp.plan.io/issues/6765
Every now and then users report issues which are specific to RHEL content but currently we are testing only CentOS and Fedora.
We also need to test certificates, see #6735.
Get a test cert for CDN and configure it in Travis, see https://pulp.plan.io/issues/6735#How-to-add-these-tests.
Add tests to sync and publish the latest RHEL6, RHEL7 and RHEL8 (BaseOS and AppStream) repos. Those tests should run daily.
Author: @fao89 (fao89)
Redmine Issue: 6344, https://pulp.plan.io/issues/6344
If two repositories have some identical content, then Pulp should be able to re-use that content. This is true even if the content is placed in differing locations in the two repositories. Do the following:
Scenario 1:
Scenario 2:
References:
#6229
#6303
Author: @goosemania (ttereshc)
Redmine Issue: 7288, https://pulp.plan.io/issues/7288
Sync a repo which has a distribution tree with sub repos (more than 1 variant or addon).
This one https://fixtures.pulpproject.org/rpm-distribution-tree/ has 1 main and 4 sub repos. Thus the same code
will be seen up to 5 times.
E.g.:
{
"child_tasks": [],
"created_resources": [
"/pulp/api/v3/repositories/rpm/rpm/be452987-051d-40ac-bcf3-6188678a3fce/versions/1/"
],
"error": null,
"finished_at": "2020-08-06T12:39:51.557658Z",
"name": "pulp_rpm.app.tasks.synchronizing.synchronize",
"parent_task": null,
"progress_reports": [
...
{
"code": "parsing.packages",
"done": 5,
"message": "Parsed Packages",
"state": "completed",
"suffix": null,
"total": 5
},
...
{
"code": "parsing.packages",
"done": 35,
"message": "Parsed Packages",
"state": "completed",
"suffix": null,
"total": 35
},
{
"code": "parsing.packages",
"done": 2,
"message": "Parsed Packages",
"state": "completed",
"suffix": null,
"total": 2
},
{
"code": "parsing.packages",
"done": 3,
"message": "Parsed Packages",
"state": "completed",
"suffix": null,
"total": 3
},
{
"code": "parsing.packages",
"done": 1,
"message": "Parsed Packages",
"state": "completed",
"suffix": null,
"total": 1
},
...
],
"pulp_created": "2020-08-06T12:36:27.046600Z",
"pulp_href": "/pulp/api/v3/tasks/970fe05a-4599-4a34-aafc-6204c0bee823/",
"reserved_resources_record": [
"/pulp/api/v3/remotes/rpm/rpm/ce3e77e3-44eb-4039-a198-32da85ac6fee/",
"/pulp/api/v3/repositories/rpm/rpm/be452987-051d-40ac-bcf3-6188678a3fce/"
],
"started_at": "2020-08-06T12:36:27.172623Z",
"state": "completed",
"task_group": null,
"worker": "/pulp/api/v3/workers/c3b1489e-90b5-48e1-9e03-3d7336f1e318/"
}
The optimization progress report code
can be seen multiple times as well if the main repo has changes while sub repos haven't changed.
{
"child_tasks": [],
"created_resources": [
"/pulp/api/v3/repositories/rpm/rpm/be452987-051d-40ac-bcf3-6188678a3fce/versions/3/"
],
"error": null,
"finished_at": "2020-08-06T12:41:52.933645Z",
"name": "pulp_rpm.app.tasks.synchronizing.synchronize",
"parent_task": null,
"progress_reports": [
{
"code": "optimizing.sync",
"done": 1,
"message": "Optimizing Sync",
"state": "completed",
"suffix": null,
"total": null
},
{
"code": "optimizing.sync",
"done": 1,
"message": "Optimizing Sync",
"state": "completed",
"suffix": null,
"total": null
},
{
"code": "optimizing.sync",
"done": 1,
"message": "Optimizing Sync",
"state": "completed",
"suffix": null,
"total": null
},
{
"code": "optimizing.sync",
"done": 1,
"message": "Optimizing Sync",
"state": "completed",
"suffix": null,
"total": null
},
{
"code": "parsing.comps",
"done": 3,
"message": "Parsed Comps",
"state": "completed",
"suffix": null,
"total": 3
},
{
"code": "parsing.advisories",
"done": 0,
"message": "Parsed Advisories",
"state": "completed",
"suffix": null,
"total": 0
},
{
"code": "parsing.packages",
"done": 1,
"message": "Parsed Packages",
"state": "completed",
"suffix": null,
"total": 1
},
{
"code": "downloading.metadata",
"done": 5,
"message": "Downloading Metadata Files",
"state": "completed",
"suffix": null,
"total": null
},
{
"code": "associating.content",
"done": 1,
"message": "Associating Content",
"state": "completed",
"suffix": null,
"total": null
},
{
"code": "downloading.artifacts",
"done": 1,
"message": "Downloading Artifacts",
"state": "completed",
"suffix": null,
"total": null
},
{
"code": "unassociating.content",
"done": 0,
"message": "Un-Associating Content",
"state": "completed",
"suffix": null,
"total": null
}
],
"pulp_created": "2020-08-06T12:40:49.725072Z",
"pulp_href": "/pulp/api/v3/tasks/50cb4c4d-ca4a-448c-a3f3-2709b9495df5/",
"reserved_resources_record": [
"/pulp/api/v3/remotes/rpm/rpm/ce3e77e3-44eb-4039-a198-32da85ac6fee/",
"/pulp/api/v3/repositories/rpm/rpm/be452987-051d-40ac-bcf3-6188678a3fce/"
],
"started_at": "2020-08-06T12:40:49.845731Z",
"state": "completed",
"task_group": null,
"worker": "/pulp/api/v3/workers/4194635d-e1f1-4224-a687-0d6705181ec4/"
}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.