Issue with package upload to GitLab Package Registry after upgrade to 8.x.x: UserWarning: upload_dists is not supported by Gitlab

[rosaLux161]

Issue with package upload to GitLab Package Registry after upgrade to 8.x.x: UserWarning: upload_dists is not supported by Gitlab

[rosaLux161]

Is this related to the removal of Twine? Is it because GitLab Package Registries are the same as PyPi Simple?

But then why were only PyPi and GitLab removed while the rest remained functional? It was nice that there was one config file to rule them all (Poetry, Semantic Release and uploading packages). Now when I have to manually tinker with Twine, define additional variables or create more configurations, it feels like a step backwards to me.

[codejedi365]

Is this related to the removal of Twine?

I believe so. Please review the Migration Doc v7 to v8 and it explains why and what is different. In short, it was becoming cumbersome to provide every config option for a twine option pass through. The design decision was to leave twine usage to the user.

Is it because GitLab Package Registries are the same as PyPi Simple?

I would not say this but effectively python semantic-release doesn't handle publishing to registries anymore.

But then why were only PyPi and GitLab removed while the rest remained functional? It was nice that there was one config file to rule them all (Poetry, Semantic Release and uploading packages). Now when I have to manually tinker with Twine, define additional variables or create more configurations, it feels like a step backwards to me.

Other people required additional manipulation of twine which was not easily manageable. Python Semantic Release's true effort is in version determination and ultimately twine did all the publishing. That's why this was a breaking change. It's a change in design that would help keep it maintainable. It is not seen that twine is that much of a configuration change which is why the trade-off was made. GitLab provides documentation examples on how to complete twine publishing.

[codejedi365]

Inside your GitLab Job, you need to add a call to twine between version and publish.

diff a/.gitLab-ci.yml b/.gitlab-ci.yml

 variables:
     GITLAB_CI: 'true'
     GITLAB_TOKEN: ${GL_TOKEN}
     hvcs: 'gitlab'
+    TWINE_PASSWORD: ${CI_JOB_TOKEN}
+    TWINE_USERNAME: gitlab-ci-token
   script:
     - git config user.name xxx
     - git config user.email ${REPOSITORY_USERNAME}@noreply.gitlab.xxx
     - git branch --force $CI_COMMIT_BRANCH $CI_COMMIT_SHA
     - git checkout $CI_COMMIT_BRANCH
     - semantic-release -vv version
+    - twine upload --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/*
     - semantic-release -vv publish

[rosaLux161]

I would not say this but effectively python semantic-release doesn't handle publishing to registries anymore.

But why is there a config option for artifacts upload if publishing to registries is removed? Is this abandoned?

https://python-semantic-release.readthedocs.io/en/latest/configuration.html#upload-to-vcs-release-bool

Also I get an error with twine. This seems to be a GitLab issue, but it was working with the old version of semantic versioning. Is there a possiblity to generate the twine command executed by the old version?

$ twine upload ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi/simple dist/*
ERROR    InvalidDistribution: Cannot find file (or expand pattern):             
         'https://gitlab.xxx/api/v4/projects/408/packages/pypi'

[codejedi365]

But why is there a config option for artifacts upload if publishing to registries is removed?

@rosaLux161, Python Semantic Release does not define artifacts upload as the same as registry publishing. Our definition comes from the fact the Version Control Systems (VCS) consider a registry and the release assets separately. It's easier to think about as the Releases page is for humans and registries are for computers.

1. For the Releases page, humans use it to read changelogs, download assets manual and review available versions like https://github.com/python-semantic-release/python-semantic-release/releases. As you can see there are 2 downloadable assets of the source code for you to click and download. If we wanted to, we could use upload_dists to add other assets to this release and we would do this through the semantic-release publish command. Release definitions are modifiable through the GitHub API. Because we are talking python, many people don't use the assets section to publish their wheel files, but instead use pypi since pip exists and relies on pypi and ultimately pip does not know how to read GitHub Release Assets.

2. Uploading to Pypi, GitHub Packages, or the GitLab Package Registry, are registries that save specific types of distribution assets that are defined by the package managers they support. This is in separate saved location than the repository release and requires different tools/protocol to upload (in our case it's twine). Registry publishing was deprecated with v8.

Please note this project was designed around GitHub and has increased support over the years for alternative VCS's but does not have all functionality implemented either due to time/effort or actual non-compatibility issues. The compatibility for uploading assets to a GitLab Release (the publish command) is not implemented at this time.

All in all, the artifacts upload config option is still available and valid even though registry publishing is deprecated.

[codejedi365]

$ twine upload ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi/simple dist/*
ERROR InvalidDistribution: Cannot find file (or expand pattern):

This error is because twine cannot find any files to upload under dist/*. https://github.com/pypa/twine/blob/fe1885f2bf896c1852dedea7733a582c5718bbbc/twine/commands/__init__.py#L49

I glanced back at your config. You seemed to have commented out your build_command which means your distributions were not built by poetry via python-semantic-release which might be why your dist/ directory is empty.

Also I get an error with twine. This seems to be a GitLab issue, but it was working with the old version of semantic versioning. Is there a possiblity to generate the twine command executed by the old version?

Not really... I looked and we had imported the twine.commands.upload module and passed parameters into the upload() function dynamically. Parameters were dist_path, repository_name, repository_url, username, password, dists, cacert, non_interactive, verbose, skip_existing and combining with the Twine defaults from twine.settings.Settings.

[rosaLux161]

Thanks. But this is not the problem, in my current pyproject the build command is set and I still get the error. Thats my actual pyproject.toml:

[tool.poetry]
name = "xxx"
version = "1.1.2"
packages = [{include = "xxx"}]

[tool.poetry.dependencies]
python = "~3.11"

[tool.poetry.group.dev.dependencies]
pytest = "^7.4.4"
python-semantic-release = "^8.7.0"
twine = "^4.0.2"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[tool.semantic_release]
version_variable = [
    "xxx/__init__.py:__version__",
    "pyproject.toml:version"
]

branch = "main"
#upload_to_release = false
build_command = "poetry build"
commit_version_number = true
commit_subject = "doc: Changelog {version}"
version_source = "commit"

[tool.semantic_release.remote]
name = "origin"
type = "gitlab"
ignore_token_for_push = false
hvcs = "gitlab"
hvcs_domain = "gitlab.xxx"
hvcs_api_domain = "gitlab.rxxx/api/v4"


[tool.semantic_release.remote.token]
env = "GITLAB_TOKEN"

[tool.semantic_release.publish]
dist_glob_patterns = ["dist/*"]
upload_to_vcs_release = false

Thats the pipeline output with ls:

$ poetry install
[...]
Installing the current project: xxx (1.1.2)
$ git config user.name xxx
$ git config user.email ${REPOSITORY_USERNAME}@noreply.xxx.net
$ git branch --force $CI_COMMIT_BRANCH $CI_COMMIT_SHA
$ git checkout $CI_COMMIT_BRANCH
Switched to branch 'main'
$ git fetch --unshallow --prune --prune-tags --tags
From https://gitlab.xxx/xx/xxx
 - [deleted]         (none)     -> v0.0.1
$ semantic-release version
1.2.0
The next version is: 1.2.0! 🚀
🛠 Running build command: poetry build
Building xxx (1.1.2)
  - Building sdist
  - Built xxx-1.1.2.tar.gz
  - Building wheel
  - Built xxx-1.1.2-py3-none-any.whl
$ ls -lah dist/
total 16K
drwxr-xr-x 2 root root 4.0K Jan 15 09:24 .
drwxrwxrwx 7 root root 4.0K Jan 15 09:24 ..
-rw-r--r-- 1 root root 1.7K Jan 15 09:24 xxx-1.1.2-py3-none-any.whl
-rw-r--r-- 1 root root 1.2K Jan 15 09:24 xxx-1.1.2.tar.gz
$ twine upload ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/*
ERROR    InvalidDistribution: Cannot find file (or expand pattern):             
         'https://gitlab.xxx/api/v4/projects/408/packages/pypi'       
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: exit code 1

So the dist is created and at the correct place.

[rosaLux161]

Inside your GitLab Job, you need to add a call to twine between version and publish.

diff a/.gitLab-ci.yml b/.gitlab-ci.yml

 variables:
     GITLAB_CI: 'true'
     GITLAB_TOKEN: ${GL_TOKEN}
     hvcs: 'gitlab'
+    TWINE_PASSWORD: ${CI_JOB_TOKEN}
+    TWINE_USERNAME: gitlab-ci-token
   script:
     - git config user.name xxx
     - git config user.email ${REPOSITORY_USERNAME}@noreply.gitlab.xxx
     - git branch --force $CI_COMMIT_BRANCH $CI_COMMIT_SHA
     - git checkout $CI_COMMIT_BRANCH
     - semantic-release -vv version
+    - twine upload ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/*
     - semantic-release -vv publish

I just copied this without rechecking twine docs. It must be:

>      - semantic-release -vv version
> +    - twine upload --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/*
>      - semantic-release -vv publish
> 

Thanks for your clarifications and help!

[codejedi365]

@rosaLux161, sorry about the typo, should have just copied and pasted rather than retyped from memory. I'm glad it's working for you now.

I updated the code snippet above too.

[rosaLux161]

But why is there a config option for artifacts upload if publishing to registries is removed?

@rosaLux161, Python Semantic Release does not define artifacts upload as the same as registry publishing. Our definition comes from the fact the Version Control Systems (VCS) consider a registry and the release assets separately. It's easier to think about as the Releases page is for humans and registries are for computers.

1. For the Releases page, humans use it to read changelogs, download assets manual and review available versions like https://github.com/python-semantic-release/python-semantic-release/releases.  As you can see there are 2 downloadable assets of the source code for you to click and download.  If we wanted to, we could use upload_dists to add other assets to this release and we would do this through the `semantic-release publish` command.  Release definitions are modifiable through the GitHub API.  Because we are talking python, many people don't use the assets section to publish their wheel files, but instead use pypi since pip exists and relies on pypi and ultimately pip does not know how to read GitHub Release Assets.

2. Uploading to Pypi, GitHub Packages, or the GitLab Package Registry, are registries that save specific types of distribution assets that are defined by the package managers they support.  This is in separate saved location than the repository release and requires different tools/protocol to upload (in our case it's `twine`).  Registry publishing was deprecated with `v8`.

Please note this project was designed around GitHub and has increased support over the years for alternative VCS's but does not have all functionality implemented either due to time/effort or actual non-compatibility issues. The compatibility for uploading assets to a GitLab Release (the publish command) is not implemented at this time.

All in all, the artifacts upload config option is still available and valid even though registry publishing is deprecated.

There is one more point I still don't understand:

I now have a sematic-release version and a twine upload. I've removed the sematic-release publish.

  script:
    - semantic-release --strict version || NO_RELEASE_MADE=
    - if [ ! -v "$NO_RELEASE_MADE" ]; then TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token twine upload --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/*; fi
    # - semantic-release publish

Upload to registry is working through twine, fine. But I still have a release of the new version on the GitLab releases page. Why this? I thought this should happen in semantic-release publish?

[codejedi365]

There is one more point I still don't understand:
...
Upload to registry is working through twine, fine. But I still have a release of the new version on the GitLab releases page. Why this? I thought this should happen in semantic-release publish?

I don't have an answer for you. I was not around for the migration to v8. I do find it a bit misplaced to have the --vcs-release as part of the version command but that is currently how it is set up. After I went through the migration for my own projects and looked at the code of publish, I felt like the entire command should of been deprecated as it barely does anything now (basicallyversion does everything now).

If you want to turn this functionality off, add --no-vcs-release to the version command. You can then create the release entry (to include release notes) separately with semantic-release changelog --post-to-release-tag $version.

[rosaLux161]

I felt like the entire command should of been deprecated as it barely does anything now (basicallyversion does everything now).

But what does publish do now? Apart from reporting that upload_dists is not supported by GitLab?

$ semantic-release publish
/venv/lib/python3.11/site-packages/semantic_release/hvcs/_base.py:88: UserWarning: upload_dists is not supported by Gitlab
  _not_supported(self, "upload_dists")

As I understand it, it's just an empty construct?

[codejedi365]

But what does publish do now?
As I understand it, it's just an empty construct?

No, it's not an empty construct. Publish uploads file assets to a VCS release, if supported by the VCS and implemented by our code. Since you use GitLab, nothing happens.