conda/grayskull

Grayskull :skull: - Recipe generator for Conda

Grayskull

---

---

"Skeletonr's main goal is to conquer the mysterious fortress of Castle Grayskull, from which He-Man draws his powers. If he succeeds, Skeletor would be able to conquer not only Eternia, but the whole universe."
Adapted from Wikipedia. Image credits: https://he-man.fandom.com

---

Introduction

Grayskull is an automatic conda recipe generator.
The main goal of this project is to generate concise recipes for conda-forge. The Grayskull project was created with the intention to eventually replace conda skeleton.
Presently Grayskull can generate recipes for Python packages available on PyPI and also those not published on PyPI but available as GitHub repositories. Grayskull can also generate recipes for R packages published on CRAN. Future versions of Grayskull will support recipe generation for packages of other repositories such as Conan and CPAN etc..

Installation

It is possible to install this project using pip:

pip install grayskull

or conda, using the conda-forge channel:

conda install -c conda-forge grayskull

It is also possible to clone this repo and install it using pip:

git clone https://github.com/conda/grayskull.git
cd grayskull
pip install -e .

Usage

It is pretty simple to use grayskull. Just call it, pass the repository (pypi or cran) and the package name.

• Example:

grayskull pypi pytest

After that grayskull will create a folder with the same name as the package and inside this folder the generated recipe will be present (meta.yaml).

• Example with pytest (grayskull pypi pytest):

If your package is a GitHub repository just replace the package name with the GitHub repository URL.
For example:

grayskull pypi https://github.com/confluentinc/confluent-kafka-python

You can also generate a recipe from a local sdist archive:

grayskull pypi ./pytest-5.3.5.tar.gz

Note that such a recipe isn't really portable as it will depend on the local path of the sdist file. It can be useful if you want to automatically generate a conda package.

Online Grayskull

It is also possible to use Grayskull without any installation. You can go to this website marcelotrevisani.com/grayskull and inform the name and the version (optional) of the package and it will create the recipe for you.

License

Copyright Marcelo Duarte Trevisani and contributors, 2020-2023.

Distributed under the terms of the Apache 2.0 license, grayskull is free and open source software.

grayskull's Issues

Pragmmatically discover License type

Update license according to package and version. Possibility to search on tarballs, repo, and pypi metadata.

Code already created for conda-forge, it would be interesting to base in this implementation:
https://github.com/regro/cf-scripts/pull/637/files#diff-65ad1c9390fe432dca7ca211091e3287R101

Add tests for entry_points - PyPI

as suggested by @ocefpaf

For example in pytest recipe which has the following entry points

    - pytest = pytest:main
    - py.test = pytest:main

it would be nice to test the entry points adding in the test section the following commands:

test:
  commands:
    - pytest --help
    - py.test --help

Give a better message when executing grayskull without any options

It was raised by @step21_gitlab
When executing just grayskull without any options it is raising an exception.

A proposal fix is to return the help when grayskull is called without any option.

Missed mypy entrypoints

Mypy has 3 entry-points and the new recipe generated with grayskull missed those. Not sure why, investigating. (I used version 0.2.2.)

Keep license file name from source distribution

I noticed when creating a recipe that the upstream repo named the license file LICENSE.md and grayskull made a copy called LICENSE. It might be useful to match the naming convention in the upstream repo so that when the license file is eventually included in the upstream repo, there is no need to make changes to the meta.yaml in addition to deleting the manually packaged copy.

Add option to drop sdist package in a specific folder

Add option (maybe --sdist-download-folder) to drop sdist package in a specific folder

cc: @ocefpaf

Format output colors based on terminal configuration

Generate Package on PyPi and Conda package

Add possibility to disable output

Add possibility to disable output

Add azure pipelines to run the tests

Add entry_points

Add pip check

Add pip check to the tests

Cover corner cases for pkgs which need compilers

• cython
• pybind11

Packages and feedstocks we should test this

Pretty much any Python package in https://github.com/regro/cf-graph-countyfair/blob/master/status/could_use_help.json, bonus point if this creates a recipe that passes the CI!

• hypothesis (almost daily updates and high change of changing metadata)
• boto/botocore (one is tied to another and with the proper metadata boto's PR should fail until botocore is merged)
• gsw (compiler)
• matplotlib (messy outputs)
• pytest (lots of new syntax for package metadata

[Design] Remove coupling between AbstractRecipeModel and other classes

It would be interesting to use composition instead of inheritance.
For that purpose, it is possible to extract the whole AbstractRecipeModel to a normal class and after that, the PyPI class would use it as an internal object to be able to handle the recipe metadata.
The PyPI class should inherit just from a pure abstract class which will define the methods the interface (PyPI, CRAN, ...) should implement.

Add support for setup.cfg

It would be nice to also add support for setup.cfg

Generate PyPi recipes

Generate recipe from PyPi.

Recipes which are interesting to test with:

• pytest
• pandas
• matplotlib
• botocore
• boto
• numpy
• scipy
• altair
• vega
• gsw
• requests

license_family is not generated or stubbed out

Many recipes are now being submitted to conda-forge without this field.

Fail if PyPi package does not contain sdist

The new skeleton should fail if the package does not contain sdist on PyPi.

We may try to get the source code from GitHub tarballs if possible.

Example of a package that just has wheels. (PyQt5)
https://pypi.org/project/PyQt5/#files

Add functional tests

Generate recipe from Cran

Generate recipe based on the Cran repository for R packages.
Bioconda already has its own implementation of skeleton. It would be interesting to discuss with them to adapt their code to this new structure.

Add support to fortran

Add support to fortran compilers

Adding incorrect selectors

grayskull generated a recipe for requests which added the following dependency

- win-inet-pton                 # [win and py==27]

which came from pypi metadata

"win-inet-pton ; (sys_platform == \"win32\" and python_version == \"2.7\") and extra == 'socks'"

It needs a few improvements in the parser

Bug - adding tags to the deps

I found this problem when I was reviewing some recipes on staged-recipes for this one in particular zipp.

On the host requirements it added
setuptools_scm [toml] >=3.4.1
instead of just
````setuptools_scm >=3.4.1``

Grayskull should remove those [toml]

Other recipes to take a look

• telegram-send
• perspective-python

Wrong license discovered

• pytest-django: found BSD-4 and it should be 3
• humanize found the wrong license
• adlfs found BSD-1-Clause and it should be BSD-3-Clause

cc: @ocefpaf

Speed up generation of recipes for some packages

It needs investigation to understand why grayskull pypi web3 is taking about 1min to finish

Issue found when testing with a few recipes

• djangorestframework-xml had the wrong import test and license
• nbdime also got the wrong license

Are you getting the import test from the package name or the modules list?

Improve CLI

It would be nice to add terminal commands to be able to load and update some sections of the recipe.
Something like
--update all --load PATH
Or
--update requirements --load PATH
And add possibly to choose the version of the package
--version 5.1.2

Don't use deprecated SPDX identifiers

For eg: GPL-2.0 is deprecated, but there's a test

get_short_license_id("GPL 2.0") == "GPL-2.0"

Move the license module to a separated package

Wrong recipes generated

as explained by @ocefpaf

• importlib-metadata
  • wrong license
  • added setuptool-scm twice
• keyring
  • added importlib-metadata twice
• clikit
  • wrong indentation
  • check selectors (it probably needs a parenthesis to group them)

Track different pkg names on PyPI and conda-forge

Track different pkg names on PyPI and conda-forge

For example:

• python-opencv is just opencv on conda-forge
• tables on PyPI is pytables on conda-forge (and there is a package on pypi with the name pytables which is not the same pytables on conda-forge)

cc: @ocefpaf

Review the home in about

Try to get the home key from pypi, for some reason is always prioritizing the pypi webpage

Use ruamel yaml instead of PyYaml

That would be useful to add, modify or delete eol comments.
It would be interesting to create an interface in the base class to abstract the manipulation regarding delimiters and selectors

Matplotlib-base rather than matplotlib as default

For most packages we can use matplotlib-base rather than matplotlib.

Create docs

• Create Wiki
• Create Docs
• Update README

disable compilers when dependencies are extra

For the following package tornado_sqlalchemy_login there is a dependency for pybind11 but it is just an extra and in dev mode. However, grayskull is adding the compilers anyway. It should not add it

Add CI to run linters

Remove comments from dependencies

Package to test:

• kubernetes_asyncio

cc: @ocefpaf

Improvements on requirements discory for host

It would be interesting to add a way to recover the modules loaded after the execution of the setup.py. So grayskull will be able to analyse all the modules loaded to install the package

Add support for toml

Create custom Dumper for yaml

Taking too long to generate nipy

Grayskull is taking too much time to generate the recipe for nipy
https://nipy.org/
https://pypi.org/project/nipy/

cc: @ocefpaf

enable handling selectors

Per our gitter discussion, duplicate keys with selectors present special challenges.

Improve logging and user feedback

It is necessary to improve the logs and it would be interesting to add user feedback in a nice output format

Exclude hidden folders when searching for the license

Exclude hidden folders when searching for the license

exclude modules in the same folder as setup.py from host

Grayskull needs to exclude modules from the same level as setup.py
one of the problems is because everything in the same level as setup.py will be seen by python as a module, any new .py file or any folder with a __init__.py

Recipes to test with this problem:

• bokeh : adding _setup_support to host section
• ipycytoscape : adding setupbase to host section

thanks @ocefpaf for reporting it

Enable pytest-xdist

Injection on disutils to be able to recover more metadata

That is important especially when the pypi metadata is missing

add possibility to deal with multiple licenses