davhau/pypi-deps-db

Dependency DB for python packages on pypi

⚠️ This project is deprecated ⚠️

For further information, see mach-nix

A dependency database for python packages on pypi

The data is updated twice each day via a github cron action. You can fork this repo and it should continue to update itself.

This data allows deterministic dependency resolution which is required by mach-nix to generate reproducible python environments.

The data contains dependencies for sdist and wheel packages, though the sdist dependencies are not fully complete, since there is no fixed standard for declaring dependencies in sdist packages. Most sdist packages which use setuptools/setup.py are contained.

---

Maintaining your own fork

On Github

It will still have the cron action and you probably don't need to do anything.

Elsewhere

Use the included nix flake app to keep the data updated:

• install nix: https://nixos.org/download.html
• get a flakes capable shell

     nix-shell -p nixFlakes

• execute

     nix run .#job-sdist-wheel

pypi-deps-db's Issues

Updates have stopped?

The last commit is from 5 days ago.

And the actions are completing suspiciously fast.

How do I search this database?

If I get the mach-nix message that my pacakge "is not available from any of the selected providers {'sdist', 'nixpkgs', 'wheel'} for the selected python version", how do I search this database to see for which python versions the database does have the package, if any? When I use the github "Search or jump to" box, I get no hits in code. For example, both "keras" and "keras extension:json" return no hits. Thanks.

Edit: the best solution for now, as far as I can tell, is to download the repo as a zip or clone it, then use a search function local to my computer. It's a slightly bothersome step but not big enough to be worth fixing. Will close the issue.

Older pip being used for sdist crawl?

I notice a lot of missing setup.py errors from the sdist crawl and I'm wondering if maybe an older version of pip is being used? pip 19.3.1 and greater should be able to construct wheels without setup.py if there's a pyproject.toml in the sdist with build instructions.

Host full database as a static file

Hey, thanks for the great work!

Have you seen this?

https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/

The basic idea is that it's possible to host an entire SQLite database on GitHub Pages and query it using HTTP Range requests (so there's no need to download the entire database).

This might be a great way to make the data more accessible and especially relevant to mach-nix since querying the database can be made significantly faster for slower connections. Also relevant to #1.

What do you think?

We're going to run into the size quote

Just received 'remote: warning: this repository is approaching its size quota' when pushing to my fork.

404 when downloading `pypi-deps-db` tarball

I have a default.nix that contains:

let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix/";
    # place version number with the latest one from the github releases page
    ref = "refs/tags/3.5.0";
  }) {
    pypiDataRev = "45774a0649b7ccaf9563d9f44d293a62aa23f0f5";
    pypiDataSha256 = "sha256:05mpbqrl3kavsyjf3jain3ksp561j86c058rnmjm5khd5b2lc8qw";
  };
in
mach-nix.mkPython {
  # contents of a requirements.txt (use builtins.readFile ./requirements.txt alternatively)
  requirements = builtins.readFile ./requirements.txt;
  packagesExtra = [
  ];
}

And when running nix-shell I get:

error:
       … while calling the 'derivationStrict' builtin

         at //builtin/derivation.nix:9:12: (source not available)

       … while evaluating derivation 'python3-3.9.9-env'
         whose name attribute is located at /nix/store/5n402azp0s9vza4rziv4z5y88v2cv1mq-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:205:7

       … while evaluating attribute 'passAsFile' of derivation 'python3-3.9.9-env'

         at /nix/store/5n402azp0s9vza4rziv4z5y88v2cv1mq-nixpkgs/pkgs/build-support/trivial-builders.nix:75:7:

           74|       inherit buildCommand;
           75|       passAsFile = [ "buildCommand" ]
             |       ^
           76|         ++ (derivationArgs.passAsFile or []);

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: unable to download 'https://github.com/DavHau/nix-pypi-fetcher/tarball/db3593377fcb56c3808383c8aaa1165c143dacc9': HTTP error 404

       response body:

       404: Not Found

I'm on MacOS, but that hasn't caused problems previously.

How to use this pypi-deps-db

For example,when I find the requests's dependencies.I don't kown to use this tool,and I find not illustration.

miss some package version in sdist

ex: package ontobio version 2.8.3

It's cloud be found in pypi

and I found package ontobio in sdlist but version 2.8.3 not exists.

Having issues Using Python311

I was Planning on using mach-nix for my project but i found out that i am not able to use Python311, which i assume is because of this line
lib.supportedPythonVersions = [ "27" "36" "37" "38" "39" "310" ];
Im not entirely sure how to work with this large repo locally as git on my end is not liking it.

 git commit -m "Added Python 3.11"
[master remote: Enumerating objects: 780, done.
remote: Counting objects: 100% (780/780), done.
remote: Compressing objects: 100% (746/746), done.
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
fatal: could not fetch 3e07082e191c66274a62314fa2201df350f39c86 from promisor remote

Error Message in question

error: The pypiDataRev seems to be older than the nixpkgs which is currently used.
       Because of this, mach-nix might lack dependency information for some python packages in nixpkgs.
       This can degrade the quality of the generated environment or result in failing builds.
       It is recommended to pass a newer pypiDataRev to mach-nix during import.
       For flakes users: Update the `pypi-deps-db` input of mach-nix.
       You can ignore this error by passing 'ignoreDataOutdated = true' to mk* or build* functions

bcrypt 4.0.0 and 4.0.1 need setuptools_rust

• for sdist, setuptools_rust is missing:

  pypi-deps-db/sdist-errors/d8.json

  Line 927 in 9932388

  "36": "ModuleNotFoundError: No module named 'setuptools_rust'",

• and a wheel is not provided for cp38 https://pypi.org/project/bcrypt/4.0.1/#files

blacklist keyword filter?

It seems like someone is spamming pypi quite hard with libraries related to roblox and robux.

see d0a9f29

Does it make sense to implement a filter to reduce storage requirements?

The actions keep failing

Every action keeps failing.

remove whitespace

lets save some space and time (parsing time grows linear with file size)

$ du -sh pypi-deps-db*
1.2G	pypi-deps-db-master
832M	pypi-deps-db-master-noindent
778M	pypi-deps-db-master-nowhitespace

$ du -sh *.zip
82M	pypi-deps-db-master.zip
72M	pypi-deps-db-master-noindent.zip
71M	pypi-deps-db-master-nowhitespace.zip

after running

#! /bin/sh
# noindent
set -e
find . -name '*.json' | while read f; do
  sed -i -E 's/^\s+//' $f
done

and

#! /bin/sh
# nowhitespace
set -e
find . -name '*.json' | while read f; do
  jq -c . $f | sponge $f
done

as expected ...

$ head -c100 pypi-deps-db-master/sdist/aa.json
{
  "aa-statistics": {
    "0.2.1": {
      "27": {
        "install_requires": [
          "allianc

$ head -c100 pypi-deps-db-master-noindent/sdist/aa.json 
{
"aa-statistics": {
"0.2.1": {
"27": {
"install_requires": [
"allianceauth>=2.8.0"
]
},
"36": "27",

$ head -c100 pypi-deps-db-master-nowhitespace/sdist/aa.json
{"aa-statistics":{"0.2.1":{"27":{"install_requires":["allianceauth>=2.8.0"]},"36":"27","37":"27","38

to read the files with whitespace

$ jq -r . pypi-deps-db-master-compact/sdist/aa.json | head -c100
{
  "aa-statistics": {
    "0.2.1": {
      "27": {
        "install_requires": [
          "allianc

Package version missing

Judging by the date, d79d523 should have included mautrix 0.20.0 (released at June 25) but it doesn't – the deps db goes up only to version 0.19.0. There's a wheel (mautrix-0.20.0-py3-none-any.whl ) available at https://pypi.org/project/mautrix/0.20.0/#files but it hasn't been picked up.

Keep getting Invalid version: 'unstable-2022-01-20'

I keep getting this error packaging.version.InvalidVersion: Invalid version: 'unstable-2022-01-20' when trying to start up my nix environment. Below is a more complete stack trace of the error:

Stack trace

building '/nix/store/911dvip5w647kdq0pxiwsahlgg01wqa7-mach_nix_file.drv'...
Traceback (most recent call last):
  File "/nix/store/bl1hg4jv6m0xqaqzlfifs4ykqqd0g9jv-596icq9xnd5b4qipna0rv458y4vgrq5r-mach-nix-src/mach_nix/generate.py", line 110, in <module>
    main()
  File "/nix/store/bl1hg4jv6m0xqaqzlfifs4ykqqd0g9jv-596icq9xnd5b4qipna0rv458y4vgrq5r-mach-nix-src/mach_nix/generate.py", line 103, in main
    do()
  File "/nix/store/bl1hg4jv6m0xqaqzlfifs4ykqqd0g9jv-596icq9xnd5b4qipna0rv458y4vgrq5r-mach-nix-src/mach_nix/generate.py", line 46, in do
    nixpkgs = NixpkgsIndex(nixpkgs_json)
  File "/nix/store/bl1hg4jv6m0xqaqzlfifs4ykqqd0g9jv-596icq9xnd5b4qipna0rv458y4vgrq5r-mach-nix-src/mach_nix/data/nixpkgs.py", line 32, in __init__
    version = parse_ver(nix_data["version"])
  File "/nix/store/ixxrk9r5yvdmqfcp4axk9nb6gzjaby5y-python3-3.9.16-env/lib/python3.9/site-packages/packaging/version.py", line 52, in parse
    return Version(version)
  File "/nix/store/ixxrk9r5yvdmqfcp4axk9nb6gzjaby5y-python3-3.9.16-env/lib/python3.9/site-packages/packaging/version.py", line 197, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
packaging.version.InvalidVersion: Invalid version: 'unstable-2022-01-20'
error: builder for '/nix/store/911dvip5w647kdq0pxiwsahlgg01wqa7-mach_nix_file.drv' failed with exit code 1;
       last 10 log lines:
       >     do()
       >   File "/nix/store/bl1hg4jv6m0xqaqzlfifs4ykqqd0g9jv-596icq9xnd5b4qipna0rv458y4vgrq5r-mach-nix-src/mach_nix/generate.py", line 46, in do
       >     nixpkgs = NixpkgsIndex(nixpkgs_json)
       >   File "/nix/store/bl1hg4jv6m0xqaqzlfifs4ykqqd0g9jv-596icq9xnd5b4qipna0rv458y4vgrq5r-mach-nix-src/mach_nix/data/nixpkgs.py", line 32, in __init__
       >     version = parse_ver(nix_data["version"])
       >   File "/nix/store/ixxrk9r5yvdmqfcp4axk9nb6gzjaby5y-python3-3.9.16-env/lib/python3.9/site-packages/packaging/version.py", line 52, in parse
       >     return Version(version)
       >   File "/nix/store/ixxrk9r5yvdmqfcp4axk9nb6gzjaby5y-python3-3.9.16-env/lib/python3.9/site-packages/packaging/version.py", line 197, in __init__
       >     raise InvalidVersion(f"Invalid version: '{version}'")
       > packaging.version.InvalidVersion: Invalid version: 'unstable-2022-01-20'
       For full logs, run 'nix log /nix/store/911dvip5w647kdq0pxiwsahlgg01wqa7-mach_nix_file.drv'.

This is my sources.json file:

sources.json

{
    "gitignore": {
        "branch": "master",
        "description": "Nix functions for filtering local git sources",
        "homepage": "",
        "owner": "hercules-ci",
        "repo": "gitignore.nix",
        "rev": "bff2832ec341cf30acb3a4d3e2e7f1f7b590116a",
        "sha256": "0va0janxvmilm67nbl81gdbpppal4aprxzb25gp9pqvf76ahxsci",
        "type": "tarball",
        "url": "https://github.com/hercules-ci/gitignore.nix/archive/bff2832ec341cf30acb3a4d3e2e7f1f7b590116a.tar.gz",
        "url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz"
    },
    "mach-nix": {
        "branch": "3.5.0",
        "description": "Create highly reproducible python environments",
        "homepage": "",
        "owner": "DavHau",
        "repo": "mach-nix",
        "rev": "7e14360bde07dcae32e5e24f366c83272f52923f",
        "sha256": "185qf6d5xg8qk1hb1y0b5gggr71vdz8v9d5ga4zg7dmcb1aypxcg",
        "type": "tarball",
        "url": "https://github.com/DavHau/mach-nix/archive/7e14360bde07dcae32e5e24f366c83272f52923f.tar.gz",
        "url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz"
    },
    "niv": {
        "branch": "master",
        "description": "Easy dependency management for Nix projects",
        "homepage": "https://github.com/nmattia/niv",
        "owner": "nmattia",
        "repo": "niv",
        "rev": "82e5cd1ad3c387863f0545d7591512e76ab0fc41",
        "sha256": "090l219mzc0gi33i3psgph6s2pwsc8qy4lyrqjdj4qzkvmaj65a7",
        "type": "tarball",
        "url": "https://github.com/nmattia/niv/archive/82e5cd1ad3c387863f0545d7591512e76ab0fc41.tar.gz",
        "url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz"
    },
    "nixpkgs": {
        "branch": "nixos-23.05",
        "description": "Nix Packages collection",
        "homepage": "",
        "owner": "NixOS",
        "repo": "nixpkgs",
        "rev": "4ecab3273592f27479a583fb6d975d4aba3486fe",
        "sha256": "10wn0l08j9lgqcw8177nh2ljrnxdrpri7bp0g7nvrsn9rkawvlbf",
        "type": "tarball",
        "url": "https://github.com/NixOS/nixpkgs/archive/4ecab3273592f27479a583fb6d975d4aba3486fe.tar.gz",
        "url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz"
    },
    "pypi-deps-db": {
        "branch": "master",
        "description": "Probably the most complete python dependency database",
        "homepage": "",
        "owner": "DavHau",
        "repo": "pypi-deps-db",
        "rev": "570d3543eb53dad7d1eb0bb88ecbcf450bc69847",
        "sha256": "1xhk812r208ppz325wxhzksqjalka6n1sdqgag5x8ilfj506pgr6",
        "type": "tarball",
        "url": "https://github.com/DavHau/pypi-deps-db/archive/570d3543eb53dad7d1eb0bb88ecbcf450bc69847.tar.gz",
        "url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz"
    }
}

This began to happen after I updated the revs and sha sums of both nixpkgs and pypi-deps-db. The previous revs were e64df9c5c8ffa0c9a82006e410ad630e32e6063d and c6a6acdbc1320cbb36d0fee0be00a0c5baf76ad9, respectively. Any idea what's going on or suggestions I could try?

setup.py not needed anymore

With recent setuptools, setup.py is no longer needed, but only a setup.cfg. An example for such a package:

https://gitlab.com/ternaris/rosbags/-/blob/88bbf1b5e3c05e6db3c4b9445a744b1a6c17b1e4/setup.cfg

There are currently many packages affected by this.

With https://github.com/DavHau/pypi-deps-db/blob/99323880924a90acd689a4f23b56551d06d3f780/sdist-errors

$ grep -c -R "No such file or directory: 'setup.py'" /nix/store/zwj5iqlnh22nc8dnwb6kv58jhpiswfl8-pypi-deps-db/sdist-errors/ |cut -d: -f2 |paste -sd+ | bc
42006

wrong pypi-fetcher commit id

the commit id is using the nix-pypi-fetcher-2, but it download from nix-pypi-fetcher. I'm using the latest commit id 4a3ad85df648f98bc371b7dceeb55d042d0c5c51

i assume because of the update you did in 91adfffe522d3571e6549c77695adaa5ef3fecc5 changed to nix-pypi-fetcher-2, but the automated script havent change yet

typing-extensions==4.4.0 not part of the DB

Some requirements could not be resolved.
Top level requirements: 
  typing-extensions==4.4.0  ecdsa==0.18.0  pysha3==1.0.2  eth-keys==0.3.3  rlp  web3==5.22.0  solana==0>
Providers:
  {'_default': ['wheel', 'sdist', 'nixpkgs'],
   'gdal': ['nixpkgs'],
   'pip': ['nixpkgs', 'sdist'],
   'setupmeta': ['wheel'],
   'setuptools': ['nixpkgs'],
   'typing-extensions': ['wheel', 'sdist', 'nixpkgs'],
   'wheel': ['nixpkgs', 'sdist']}
Mach-nix version: master
Python: 3.10.7
Cause: Requirements conflict: typing-extensions (<SpecifierSet('==4.4.0')>,)
The requirements which caused the error:
  typing-extensions (<SpecifierSet('==4.4.0')>,)

The given requirements might contain package versions which are not yet part of the dependency DB
currently used. The DB can be updated by specifying 'pypiDataRev' when importing mach-nix.
For examples see: https://github.com/DavHau/mach-nix/blob/master/examples.md

Do yanked and deleted packages are considered in this db?

Since some packages are yanked and deleted by maintainers, this part of packages would deleted from db or stay forever?
For example, a package named "airflow-sbf-utils", it has a row in this db like:
airflow-sbf-utils: { py3: { 0.1.1: { airflow_sbf_utils-0.1.1-py3-none-any.whl: { requires_dist: [ "python-decouple", "pymssql", "google-cloud-storage", "SQLAlchemy" ], requires_python: ">=3.9" } }, 0.1.2: { airflow_sbf_utils-0.1.2-py3-none-any.whl: "0.1.1@airflow_sbf_utils-0.1.1-py3-none-any.whl" }, 0.1.3: { airflow_sbf_utils-0.1.3-py3-none-any.whl: "0.1.1@airflow_sbf_utils-0.1.1-py3-none-any.whl" }, 0.1.4: { airflow_sbf_utils-0.1.4-py3-none-any.whl: "0.1.1@airflow_sbf_utils-0.1.1-py3-none-any.whl" }, 0.1.5: { airflow_sbf_utils-0.1.5-py3-none-any.whl: "0.1.1@airflow_sbf_utils-0.1.1-py3-none-any.whl" } } },

However, I find this package metadata through API from pypi like this: https://pypi.org/pypi/airflow_sbf_utils/json, it returns not found.

Why are we parsing ourselves

There's probably a good reason that I don't know about,
but whey are we parsing like every sdist and wheel,
when we could ask the pypi json api - at least in the first place, with a fallback to 'parse it yourself'?

Invalid versions in the database file

I'm trying to use mach to provide idf-component-manager, but it fails with an UnsupportedVersionError: Not a valid version: 2.7.*. This is probably related to a line like this, from sdist/b7.json:

  "idf-component-manager": {
    <snip>
    "1.2.0": {
  <snip>
          ":python_version == \"2.7.*\"": [
  <snip>
    },
}

Is there a way to work around this? Maybe the scraper should be fixing invalid x.x.* version constraints?

error

error: experimental Nix feature 'nix-command' is disabled; use '--extra-experimental-features nix-command' to override
How to solute this inssue?