Giter Site home page Giter Site logo

thoth-station / prescriptions Goto Github PK

View Code? Open in Web Editor NEW
17.0 3.0 10.0 57.12 MB

⚕️💊 Prescriptions to heal your applications and application dependencies 💊⚕️

Home Page: https://thoth-station.ninja/docs/developers/adviser/prescription.html

License: GNU Affero General Public License v3.0

Python 100.00%
thoth dependency-analysis dependency-management python odbl hacktoberfest

prescriptions's Introduction

Prescriptions for Thoth's adviser

⚕️💊⚕️ ----

Prescriptions to heal your applications and application dependencies.

Why we created prescriptions?

We wanted to create a repository that keeps a database of known issues in Python open-source eco-system, as well as suggestions for Python libraries and runtime environments they can run in. The database is used in Thoth to resolve high quality Python software stacks. The linked blog post describes prescriptions more in-depth.

When using OpenShift or Kubernetes, one provides manifest files that state how the desired state of a cluster should look like. Prescriptions might be seen analogous to this - prescriptions provide a way to declaratively state how the desired dependency resolution should look like considering the prescribed rules. Then, it’s up to the reinforcement learning algorithm implemented in Thoth's adviser to find a solution in the form of a lockfile respecting the prescribed rules, requirements for the application and other inputs to the Thoth's cloud resolver.

See the linked presentation or YouTube video for more info.

How to write a prescription?

If you would like to write a prescription for resolver, check the following docs.

Need help with a prescription?

If you spotted an issue in Python dependencies or Python ecosystem, just let us know by openning an issue and we will help you with writing a prescription.

Data sources used for automatically generated prescriptions

Currently implemented handlers in Thoth's weekly cronjob allow to auto-generate prescriptions for the given data:

  • CVE present in a package, from the PyPA advisory-database
  • Project maintenance and development practices as evaluated by the OSSF Security Scorecards
  • Information on package maintainance obtained via the GitHub API: if the given project is marked as archived, is forked from another project, hosts release notes, its number of maintainers, stars, contributors.
  • The package size, number of downloads, maintainers and last release date from PyPI.

Release Details

Prescriptions are released as a blob. During the tag release session of thoth-adviser, the s2i process clones the prescriptions repo with the latest tag information.

Licensing

The prescription database is released under the terms of GNU Affero General Public License v3.0 or later. See the LICENSE file for more info.

Copyright © 2021 AICoE Project Thoth; Red Hat Inc.

prescriptions's People

Contributors

codificat avatar fridex avatar goern avatar gregory-pereira avatar harshad16 avatar khebhut[bot] avatar mayacostantini avatar merwok avatar sesheta avatar vannten avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar

Forkers

fridex pacospace harshad16 codificat gkrumbach07 mayacostantini gregory-pereira vannten shism2 8forceacpo

prescriptions's Issues

Have docs on how to include new prescriptions in adviser

Is your feature request related to a problem? Please describe.
As Thoth Maintainer,

I would like to know how to include new prescriptions in adviser.

Describe the solution you'd like
Have docs on how to include new prescriptions in adviser.

Describe alternatives you've considered

Additional context

Acceptance Criteria

  • docs are provided in this repo.

No dependency management found

No dependency management found for this repository. If you want to keep your dependencies managed, please submit Pipfile or requirements.in or requirements-dev.in file.

To generate a Pipfile, use:

$ pipenv install --skip-lock --code ./
$ git add Pipfile
$ git commit -m 'Add Pipfile for dependency management'

Make sure your Pipfile or requirements.in or requirements-dev.in is placed in the root of your Git repository.

jupyterlab does not render images with plotly and fails raising error

Is your feature request related to a problem? Please describe.
I'm working with notebooks from jupyterlab and I'm using plotly to plot, but images are not shown in the jupyterlab notebook.

It appears to be a problem of missing dependency (nbformat) in the kernel created for my notebook. plotly/plotly.py#2159.

ValueError: Mime type rendering requires nbformat>=4.2.0 but it is not installed

This dependency is not required by the stack itself, but by jupyterlab to show the images. Adding nbformat to the kernel seems to solve the problem.

Describe the solution you'd like
I would like to receive a recommendation when I use plotly that warns me about missing nbformat if I'm working with jupyterlab.

Describe alternatives you've considered

Additional context

Adjust directory structure so it is browsable on GitHub

Describe the bug

Currently, GitHub does not show all the prescriptions for packages. The reason is:

 Sorry, we had to truncate this directory to 1,000 files. 680 entries were omitted from the list.

To Reproduce
Steps to reproduce the behavior:

  1. Go to prescriptions/prescriptions
  2. See the truncated output

Resolution

We can group packages into directory structure that would capture first two or three laters of projects listed:

- re
  - requests
  - ...

Some links to RPM packages are broken

Describe the bug

Check for example following RPM package and the corresponding pipeline unit that tries to link to the RPM:

prescriptions/prescriptions/cm_/cmigemo/f34_rpm.yaml

Lines 19 to 20 in 4c3a243

message: "Package 'cmigemo' is packaged as an RPM in Fedora 34 as 'python3-cmigemo'"
link: https://src.fedoraproject.org/rpms/python3-cmigemo

The pipeline unit correctly detects cmigemo package from PyPI being packaged as an RPM and available as python3-cmigemo package:

(f34) $ dnf search python3-cmigemo
============================ Name Exactly Matched: python3-cmigemo ============================
python3-cmigemo.noarch : A pure python binding for C/Migemo

However, the link to fedora RPM is broken:

https://src.fedoraproject.org/rpms/python3-cmigemo

The correct one is (note without "3"):

https://src.fedoraproject.org/rpms/python-cmigemo

This can be seen in some other packages as well.

Link to StackOverflow tags for specific packages

Is your feature request related to a problem? Please describe.

As a developer and consumer of Thoth recommendations, I would like to follow recommended StackOverflow link so that I can easily browse stack overflow questions specific to a package recommended.

Describe the solution you'd like

Download tags that are available on stack overflow and derive prescription units out of them that would navigate users to specific SO topic.

Add project administrators/contributors to CODEOWNERS file

Is your feature request related to a problem? Please describe.

As an open-source project maintainer and/or contributor, I would like to be notified about new prescriptions generated/created by Thoth.

For this purpose, I would like to be stated in CODEOWNERS file so that GitHub automatically asks me for a review.

Describe the solution you'd like

Scan all the projects we have for top contributors and maintainers and add them to the CODEOWNERS file.

Create prescriptions on projects quality using GitHub Insights

Is your feature request related to a problem? Please describe.

We could use the information present in the Insights section of projects GitHub repositories to get information about a project's activity and assess its quality.

Describe the solution you'd like

  • Get Insights information such as pulse, contributions, etc via the GitHub API
  • Generate prescriptions for projects with Insights computing a score for the project on each category and eventually giving the project an overall score

Automatically aggregate data for GitHub release notes

Describe the feature

Following GitHub release notes wrap introduced in thoth-station/adviser#1821, we could automatically construct a prescription file holding configuration for open-source packages that host release notes on GitHub

Implementation

Let's create a tool that:

  • goes through the solver documents available on Ceph and extracts GitHub URL out of them for projects that Thoth analyzed - https://github.com/thoth-station/prescriptions-gh-release-notes-job
    • based on the extracted GitHub URL, checks if the given project has release information published on GitHub
    • if release info is published on GitHub, create a GitHubReleaseNoteWrap configuration entry (following docs) that can be consumed by the adviser to give info on the release
  • make prescriptions generated available on ceph and supply them to adviser container - thoth-station/thoth-application#1251
  • automate the prescription generation

Related: thoth-station/adviser#1821
Related: #17

Automatically validate prescriptions on a pull request

Is your feature request related to a problem? Please describe.

I, as Thoth contributor, would like to submit an adjustment to the prescription.yaml file based on my observation on software I use so that Thoth will make better recommendations on the software. To make sure the change is applicable and can be accepted, CI should run thoth-adviser validate-prescription prescription.yaml and report back any issues spotted in the prescription file. This way we will make sure the prescription file is valid and can be used in a deployment.

Describe the solution you'd like

Have a CI run that runs thoth-adviser validate-prescription prescription.yaml with the most recent adviser container image built by AICoE-CI.

Related: thoth-station/adviser#1781

Provide a script that creates markdown report out of prescriptions for a package

Is your feature request related to a problem? Please describe.

As discussed at the TF SIG build community meeting, let's provide a way to generate a markdown file for TensorFlow. This file should turn prescriptions into a README that can be put in a repository. It will keep knowledge about issues/improvements to TensorFlow application stack in a human-readable form.

Missing links to CVE information for container images

Describe the bug

2022-03-23 09:50:25,483 3259502 ERROR    thoth.adviser.prescription.v1.prescription:302: Failed to load prescription from '../prescriptions/prescriptions/_containers/s2i_thoth_f31_py37/quay_security.yaml'
2022-03-23 09:50:25,485 3259502 CRITICAL root:105: Traceback (most recent call last):
  File "/home/fpokorny/git/thoth-station/adviser/thoth/adviser/prescription/v1/prescription.py", line 197, in from_dict
    PRESCRIPTION_SCHEMA(prescription)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 272, in __call__
    return self._compiled([], data)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 594, in validate_dict
    return base_validate(path, iteritems(data), out)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 386, in validate_mapping
    cval = cvalue(key_path, value)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 817, in validate_callable
    return schema(data)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 272, in __call__
    return self._compiled([], data)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 594, in validate_dict
    return base_validate(path, iteritems(data), out)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 386, in validate_mapping
    cval = cvalue(key_path, value)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 635, in validate_sequence
    cval = validate(index_path, value)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 817, in validate_callable
    return schema(data)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 272, in __call__
    return self._compiled([], data)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 594, in validate_dict
    return base_validate(path, iteritems(data), out)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 386, in validate_mapping
    cval = cvalue(key_path, value)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 817, in validate_callable
    return schema(data)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 272, in __call__
    return self._compiled([], data)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 594, in validate_dict
    return base_validate(path, iteritems(data), out)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 386, in validate_mapping
    cval = cvalue(key_path, value)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 635, in validate_sequence
    cval = validate(index_path, value)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 817, in validate_callable
    return schema(data)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 272, in __call__
    return self._compiled([], data)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 594, in validate_dict
    return base_validate(path, iteritems(data), out)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 386, in validate_mapping
    cval = cvalue(key_path, value)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/voluptuous/schema_builder.py", line 817, in validate_callable
    return schema(data)
  File "/home/fpokorny/git/thoth-station/adviser/thoth/adviser/prescription/v1/schema.py", line 204, in _justification_link
    if v.startswith(("https://", "http://")):
AttributeError: 'NoneType' object has no attribute 'startswith'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./thoth-adviser", line 1011, in <module>
    __name__ == "__main__" and cli()
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/fpokorny/.local/share/virtualenvs/adviser-1eaKppR3/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "./thoth-adviser", line 979, in validate_prescription
    prescription = Prescription.validate(prescriptions)
  File "/home/fpokorny/git/thoth-station/adviser/thoth/adviser/prescription/v1/prescription.py", line 116, in validate
    prescription_instance = cls.load(prescriptions)
  File "/home/fpokorny/git/thoth-station/adviser/thoth/adviser/prescription/v1/prescription.py", line 295, in load
    prescription_instance = cls.from_dict(
  File "/home/fpokorny/git/thoth-station/adviser/thoth/adviser/prescription/v1/prescription.py", line 203, in from_dict
    raise PrescriptionSchemaError(str(exc))
thoth.adviser.exceptions.PrescriptionSchemaError: 'NoneType' object has no attribute 'startswith'

To Reproduce
Steps to reproduce the behavior:

  1. Run thoth-adviser validate-prescriptions prescriptions/
  2. See the error reported

Expected behavior

Schema validation should succeed.

Optimize prescriptions loading

Is your feature request related to a problem? Please describe.

As the database of known Python packaging issues will grow, it might be better to create a binary file holding prescriptions and load it on adviser startup. We can still maintain YAML files describing prescriptions, but create a binary file only for deployment to optimize loading and adviser startup.

Describe the solution you'd like

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.