duplicate key value violates unique constraint "core_artifact_sha256_key"

[pulp]

duplicate key value violates unique constraint "core_artifact_sha256_key"

[ggainey]

@quba42 No fix from me - however, I will observe that there are sync-paths that "ignore conflicts" - two threads try to create-or-update the same Thing at the same time, both try to create(), one wins, one gets the UQ-violation and switches to update instead. Postgres logs the error/collision - but it's harmless.

Unless the task fails, or the resulting repo is missing content, this is not an error.

[pulpbot]

From: knzivid (knzivid)
Date: 2021-05-28T16:46:12Z

---

Interestingly

When I run pulp/pulp official image with these package versions, I do not see this error from postgres.

pulp-ansible                    0.7.3
pulp-certguard                  1.3.0
pulp-container                  2.6.0
pulp-deb                        2.12.0
pulp-file                       1.7.0
pulp-maven                      0.2.0
pulp-python                     3.2.0
pulp-rpm                        3.12.0
pulpcore                        3.12.2

When I run the same in https://pulpproject.org/related-tooling/#docker-pulp-pulp-docker-images with these versions, I see this error in the log.

pulp-deb                        2.12.0
pulp-file                       1.7.0
pulpcore                        3.12.2

Then I looked at the logs in /var/lib/pgsql/data/log in the pulp/pulp docker image and I found similar logs!

2021-05-28 16:16:52.436 UTC [696] ERROR:  duplicate key value violates unique constraint "core_artifact_sha256_key"
2021-05-28 16:16:52.436 UTC [696] DETAIL:  Key (sha256)=(addfd22c48ca7f6e67197a63d6ba78dc706678e4b198c901de1bfd99060629cc) already exists.
2021-05-28 16:16:52.436 UTC [696] STATEMENT:  INSERT INTO "core_artifact" ("pulp_id", "pulp_created", "pulp_last_updated", "file", "size", "md5", "sha1", "sha224", "sha256", "sha384", "sha512") VALUES ('d437ca8a-0d13-4c7a-ba23-911316364809'::uuid, '2021-05-28T16:16:52.435103+00:00'::timestamptz, '2021-05-28T16:16:52.435116+00:00'::timestamptz, 'artifact/ad/dfd22c48ca7f6e67197a63d6ba78dc706678e4b198c901de1bfd99060629cc', 4645, NULL, NULL, '3c9ad31f31074b93fb2ca51d8618f851ff9edd67ae509da53eb9a6e2', 'addfd22c48ca7f6e67197a63d6ba78dc706678e4b198c901de1bfd99060629cc', '682032a010aacda85558a346cff70840ff3f0882942b6f22ae5651faca9171f4751574341c285fbc9c4201e693f92c0c', 'b504ff76b0ab65b4e894949a9ce65ced4dc820e4cca961c5926744c453fca0d24ec82bc1e2134c765d2916a9f0d173380d5d7d67cb64da4c5bc22f55b2940dd9'), ('18b3ee57-eb6b-404c-8e81-a7763c165c29'::uuid, '2021-05-28T16:16:52.435669+00:00'::timestamptz, '2021-05-28T16:16:52.435678+00:00'::timestamptz, 'artifact/88/89e025a6e18c2a0e31edd2f430ee2a7be8a88c48f156706ec672cc87cb832d', 5527, NULL, NULL, 'd58236d07ed20cf31c96eba8148b733e891967f71b6c418759c3b7b2', '8889e025a6e18c2a0e31edd2f430ee2a7be8a88c48f156706ec672cc87cb832d', 'f026b266558924947a1bc796296f9ee6b15033df275d05a576a09505bb66a3870bc61ce166c405887864ae940f08a9a4', 'b8eb88c5b37e13e53fa549831758dbb7564197323f7fd3d2a19b3ba8a7a9f10718939cb8d94db47987b7e461b4f29012ee2479a7c2bc3d3a73bd1fb8746e338c'), ('6fb3b30f-7d58-43db-8a48-2482e160c7b3'::uuid, '2021-05-28T16:16:52.435871+00:00'::timestamptz, '2021-05-28T16:16:52.435880+00:00'::timestamptz, 'artifact/b7/4740e506eec14a271e74ee7096b0ce6194a4810f568727da9557127f8db4b9', 833, NULL, NULL, '3d42a1ff9d8d9caaa5a9526be54d786af61a3d16d00f45f0a740639a', 'b74740e506eec14a271e74ee7096b0ce6194a4810f568727da9557127f8db4b9', '2d3b9145ff8fef94ea470404c3b2327a76347b40e1b23ee301d16146bbf139d6bef83e8a04b49af5d65d4f6a6c73be72', '9008fd1c21bc112726f9687536e2dd8c2a5c5d3133338eff8605b4268bf0dc5fb7a974ea4509c5040e9cf8e2676ff7da29a5da100187f9d0fd8f565626594036') RETURNING "core_artifact"."pulp_id"

Is this harmless?

[pulpbot]

From: @quba42 (quba42)
Date: 2021-06-02T12:00:39Z

---

Not sure if this is harmless. I will try to reproduce this when I get to it.

If it is possible to install packages from the synced repository in question, it may well be harmless...

[quba42]

I cannot synchronize this repository at all, since some of the packages cannot be downloaded. This happens through browser as well as through Pulp, so it does not appear to be a Pulp issue. As a result, I am closing this ticket "cannot reproduce". I am happy to re-open if there is further user input on how to reproduce this.

The error as encountered by my Pulp instance:

pulpcore-worker[13000]: pulp [1ec10a9ffbe14b569d7edb730bca4902]: pulpcore.tasking.pulpcore_worker:INFO: Task cacb278c-041e-468c-9816-3c6c52cca121 failed (503, message='first byte timeout', url=URL('https://apt.llvm.org/bionic/pool/main/l/llvm-toolchain-11/libclang-cpp11_11.1.0~++20211011094159+1fdec59bffc1-1~exp1~20211011214614.8_amd64.deb'))
pulpcore-worker[13000]: pulp [1ec10a9ffbe14b569d7edb730bca4902]: pulpcore.tasking.pulpcore_worker:INFO:   File "/home/vagrant/devel/pulpcore/pulpcore/tasking/pulpcore_worker.py", line 362, in _perform_task
pulpcore-worker[13000]:     result = func(*args, **kwargs)
pulpcore-worker[13000]:   File "/home/vagrant/devel/pulp_deb/pulp_deb/app/tasks/synchronizing.py", line 141, in synchronize
pulpcore-worker[13000]:     DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()
pulpcore-worker[13000]:   File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/declarative_version.py", line 161, in create
pulpcore-worker[13000]:     loop.run_until_complete(pipeline)
pulpcore-worker[13000]:   File "/usr/lib64/python3.10/asyncio/base_events.py", line 641, in run_until_complete
pulpcore-worker[13000]:     return future.result()
pulpcore-worker[13000]:   File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
pulpcore-worker[13000]:     await asyncio.gather(*futures)
pulpcore-worker[13000]:   File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/api.py", line 43, in __call__
pulpcore-worker[13000]:     await self.run()
pulpcore-worker[13000]:   File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py", line 183, in run
pulpcore-worker[13000]:     pb.done += task.result()  # download_count
pulpcore-worker[13000]:   File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/artifact_stages.py", line 209, in _handle_content_unit
pulpcore-worker[13000]:     await asyncio.gather(*downloaders_for_content)
pulpcore-worker[13000]:   File "/home/vagrant/devel/pulpcore/pulpcore/plugin/stages/models.py", line 89, in download
pulpcore-worker[13000]:     download_result = await downloader.run(extra_data=self.extra_data)
pulpcore-worker[13000]:   File "/home/vagrant/devel/pulpcore/pulpcore/download/http.py", line 253, in run
pulpcore-worker[13000]:     return await download_wrapper()
pulpcore-worker[13000]:   File "/usr/local/lib/pulp/lib64/python3.10/site-packages/backoff/_async.py", line 133, in retry
pulpcore-worker[13000]:     ret = await target(*args, **kwargs)
pulpcore-worker[13000]:   File "/home/vagrant/devel/pulpcore/pulpcore/download/http.py", line 249, in download_wrapper
pulpcore-worker[13000]:     return await self._run(extra_data=extra_data)
pulpcore-worker[13000]:   File "/home/vagrant/devel/pulpcore/pulpcore/download/http.py", line 274, in _run
pulpcore-worker[13000]:     self.raise_for_status(response)
pulpcore-worker[13000]:   File "/home/vagrant/devel/pulpcore/pulpcore/download/http.py", line 185, in raise_for_status
pulpcore-worker[13000]:     response.raise_for_status()
pulpcore-worker[13000]:   File "/usr/local/lib/pulp/lib64/python3.10/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
pulpcore-worker[13000]:     raise ClientResponseError(

[ipanova]

@ggainey might have a PR for this pulp/pulpcore#2103

[quba42]

Thanks @ipanova. If it is being fixed within pulpcore, then I guess it is nevertheless fine to keep this pulp_deb issue closed (since it is not a pulp_deb but a pulpcore issue).