Packages and feedstocks we should test this

[conda]

Packages and feedstocks we should test this

[ocefpaf]

The gsw recipe needs a compiler and the conda-style numpy pin. We don't need to add support for that right now but we should open issues to track the development of those features.

As far as I know there is no metadata to declare that yet, only ideas. If we will try to be smart we should look for numpy header include in the setup.py, and the use of the Extension class for compilers.

[marcelotrevisani]

I might do add an injection with proxy just to verify if they are using build_ext. But it will work fine for just the packages which use setup.py

[ocefpaf]

If you have other packages that would be nice to test the presence of a compiler please let me know.

numpy, scipy, and matplotlib. Note that the latter is a split recipe in conda-forge, so handling the recipe updated will be different from creating a new recipe from scratch. Do not worry about recipes updates at the moment.

[ocefpaf]

Here are a few more we can use to test:

pybind11, c, cxx:

• scikit-geometry
• openmesh
• xmsstamper

Fortran

• pyspharm
• xlayers
• pymc

cython/numpy

• ciso
• cythongsl

(Please ignore those that are not on PyPI.)

[ocefpaf]

At least, for now, I generate recipes for these packages:

• hypothesis: gist.github.com/marcelotrevisani/31c45c94ed2305056cdd0caa508cce79
• boto: gist.github.com/marcelotrevisani/c5385a33da4e966de48542040e73845e
• botocore: gist.github.com/marcelotrevisani/a57a09007128923e549b8b8c758f7516
• gsw: gist.github.com/marcelotrevisani/52c87047f7af20fe78f1b854d0a386de
• matplotlib: gist.github.com/marcelotrevisani/aa0c83dad03829ef3ffc9ae4e78cb263
• pytest: gist.github.com/marcelotrevisani/fc4aff4de383703f23f74dbed1b39f9b
• requests: gist.github.com/marcelotrevisani/28f63677009685eb865449ece713f1cd
  Those projects which they provided a proper pypi metadata (such as requests, hypothesis and pytest) the recipe generated is pretty close to the current on conda-forge

Indeed. The main missing feature is the license file, but those are rarely in the metadata. We could incorporate @CJ-Wright's bot code though, to heuristically find the license file.

From time to time I will try to update those gist when I changed the code

👍

Some comments on the results:

• noarch recipes cannot have the skip statement. We need add python >= python_requires on host and run instead.

• docutils>=0.10,<0.16 should be docutils >=0.10,<0.16, we need that space or it will fail to build. (The linter should catch that.)

• The matplotlib recipe would require a major overhaul in how python packages declare external dependencies. Just ignore it for now.

• the pytest recipe failed to add setuptools at run time. No big deal b/c we fail that already, we never know when pkg_resources is used to addd it as a run-time depenceny.

[ocefpaf]

To be fair, I think pytest does not need setuptools, does it? Because, I searched their requirements, pypi metadata and setup about it and I didn't find anything. Maybe @nicoddemus can clarify that for us here smile

Most packages that use pkg_resources do not declare setuptools as a runtime dependency. It is rarely a problem b/c 99% of the time people already have setuptools installed. It is a problem for slim conda-envs though. If you install a single package, that uses pkg_resources, in an env that won't pull setuptools (an env without pip for example), then you have a broken env.

[nicoddemus]

Maybe @nicoddemus can clarify that for us here 😄

Indeed we use that, but only in this single location:

https://github.com/pytest-dev/pytest/blob/9631b3c16633eacd2c91dbb9cdc6a2177c693048/src/_pytest/monkeypatch.py#L251-L260

Most packages that use pkg_resources do not declare setuptools as a runtime dependency. It is rarely a problem b/c 99% of the time people already have setuptools installed. It is a problem for slim conda-envs though. If you install a single package, that uses pkg_resources, in an env that won't pull setuptools (an env without pip for example), then you have a broken env.

Never realized that but it makes sense. @ocefpaf do you think packages (like pytest) should explicitly declare setuptools as a runtime dependency?

[marcelotrevisani]

I just updated/created the recipes on gists for

* [boto](https://gist.github.com/marcelotrevisani/c5385a33da4e966de48542040e73845e)

* [botocore](https://gist.github.com/marcelotrevisani/a57a09007128923e549b8b8c758f7516)

* [ciso](https://gist.github.com/marcelotrevisani/5874ef0d49226edf51332c4d5d0a7405)

* [cythongsl](https://gist.github.com/marcelotrevisani/87df0cc02638d385068866199589f42e)

* [gsw](https://gist.github.com/marcelotrevisani/52c87047f7af20fe78f1b854d0a386de)

* [hypothesis](https://gist.github.com/marcelotrevisani/31c45c94ed2305056cdd0caa508cce79)

* [matplotlib](https://gist.github.com/marcelotrevisani/aa0c83dad03829ef3ffc9ae4e78cb263)

* [openmesh](https://gist.github.com/marcelotrevisani/c4a03ca139a78329db280eab13559747)

* [pymc](https://gist.github.com/marcelotrevisani/a21e6318b4792e19042c0e2511cb85f9)

* [pytest](https://gist.github.com/marcelotrevisani/fc4aff4de383703f23f74dbed1b39f9b)

* [requests](https://gist.github.com/marcelotrevisani/28f63677009685eb865449ece713f1cd)

* [xlayers](https://gist.github.com/marcelotrevisani/c7090eae7da943df12c6505f46ab772e)

The recipe xlayers does not seem to be right, I will create some tests with that one.
The matplotlib recipe also needs some investigation

I just updated all the recipes on gists. Just one of the recipes grayskull didn't find the license file but it added the correct spdx short identifier for it

[ocefpaf]

Some comments:

• loved the PLEASE_ADD_LICENSE_FILE
• The short SPDX version Apache 2.0 is lower case, right?
• We probably need a blacklist of things that are not real build dependencies, like setupext and versioneer (Also, you can drop the matplotlib and openmesh recipes from your tests)

The compilers and numpy pinning looks good!

[marcelotrevisani]

At least, for now, I generate recipes for these packages:

• hypothesis: https://gist.github.com/marcelotrevisani/31c45c94ed2305056cdd0caa508cce79
• boto: https://gist.github.com/marcelotrevisani/c5385a33da4e966de48542040e73845e
• botocore: https://gist.github.com/marcelotrevisani/a57a09007128923e549b8b8c758f7516
• gsw: https://gist.github.com/marcelotrevisani/52c87047f7af20fe78f1b854d0a386de
• matplotlib: https://gist.github.com/marcelotrevisani/aa0c83dad03829ef3ffc9ae4e78cb263
• pytest: https://gist.github.com/marcelotrevisani/fc4aff4de383703f23f74dbed1b39f9b
• requests: https://gist.github.com/marcelotrevisani/28f63677009685eb865449ece713f1cd

For now, the project is supporting just the metadata which comes from pypi, but I will start the development to do the injection in the distutils and with that, I will be able to recover some information when the pypis` metadata is missing.

Those projects which they provided a proper pypi metadata (such as requests, hypothesis and pytest) the recipe generated is pretty close to the current on conda-forge

From time to time I will try to update those gist when I changed the code

[marcelotrevisani]

After the integration of #32 it will improve those recipes generated

[marcelotrevisani]

One thing which I saw we can do at least for now is to verify if the build_ext from distutils is being used. Because if it is, there a high probability that is using compilers.
For example, in the case of the gws that is very true
https://github.com/TEOS-10/GSW-Python/blob/1b2ccab2dab3acad7531cd499dd77785f723f88c/setup.py#L13

[marcelotrevisani]

I added a new commit to the PR #32 and it is recognizing the compilers at least for gsw
As you can see by this test it is testing the presence of a C compiler for gsw
https://github.com/marcelotrevisani/grayskull/pull/32/files#diff-5f2e7a5de5a335475f5a23b3621e8639R171

If you have other packages that would be nice to test the presence of a compiler please let me know.

But it is not generating the recipe with the metadata from setup.py, I still need to add a method to merge the metadata from pypi and sdist

[marcelotrevisani]

* `noarch` recipes cannot have the skip statement. We need add `python >= python_requires` on `host` and `run` instead.

Indeed! I had a regression here, because it was working, my mistake. I will add a regression tests for it to not happen again!
If you open the old revision
https://gist.github.com/marcelotrevisani/31c45c94ed2305056cdd0caa508cce79/revisions

You will see that it was being generated beautifully. I didn't pay attention to it, I will fix it in the upcoming days. Thanks for spotting it! :)

* `docutils>=0.10,<0.16` should be `docutils >=0.10,<0.16`, we need that space or it will fail to build. (The linter should catch that.)

Another regression here.. geez... haha. Sorry about that, I will fix it! I thought I had fixed it, maybe the gist it is a bit old, I will recheck it, thanks!

* The `matplotlib` recipe would require a major overhaul in how python packages declare external dependencies. Just ignore it for now.

The matplotlib is a bit tricky because there is no much data in the setup.py and also in the pypi metadata. I will need to investigate it a bit further.

* the pytest recipe failed to add `setuptools` at run time. No big deal b/c we fail that already, we never know when `pkg_resources` is used to addd it as a run-time depenceny.

To be fair, I think pytest does not need setuptools, does it? Because, I searched their requirements, pypi metadata and setup about it and I didn't find anything. Maybe @nicoddemus can clarify that for us here 😄

[marcelotrevisani]

Thanks for all those comments! It will help me to fix and improve grayskull

[marcelotrevisani]

I think it would be nice to explicitly add it as a dependency to run the project.
😃

So in that way the PyPI metadata will have that information

[ocefpaf]

Never realized that but it makes sense. @ocefpaf do you think packages (like pytest) should explicitly declare setuptools as a runtime dependency?

That is a good question. As a packager? Yes! As a Python programmer used to the bad practices that are most of the time harmless? No.

Ideally conda should ship pip, and hence setuptools, with its python so at least we expected the same behavior. But that is another discussion...

[marcelotrevisani]

I just updated/created the recipes on gists for

• boto
• botocore
• ciso
• cythongsl
• gsw
• hypothesis
• matplotlib
• openmesh
• pymc
• pytest
• requests
• xlayers

The recipe xlayers does not seem to be right, I will create some tests with that one.
The matplotlib recipe also needs some investigation

[marcelotrevisani]

Yes, I think I fixed it, but I will double-check!

A blacklist would be good indeed. We might let in plain text somewhere in the project so in that way it would be easy to modify

[marcelotrevisani]

Indeed, it is still putting apache in all capital letters, I will take a look on that because gryaskull get it usually from opensource.org
maybe as the name is equal to that one on opensource.org it is using the same in the pypi metadata, I will take a look at that. Thanks!

[marcelotrevisani]

* The [short SPDX version](https://spdx.org/licenses/Apache-2.0.html) Apache 2.0 is lower case, right?

I just fixed it with #62

https://gist.github.com/marcelotrevisani/a57a09007128923e549b8b8c758f7516

[ocefpaf]

I think we can close this one 😄