python-packaging/honesty

sdist/bdist validator

honesty's Issues

simple-index reading code never freshens

It looks like there's a bug in is_index_filename -- should be checking for empty string, not None.

deps --flat lists some deps twice with extras

PyJWT and urllib3 both show up twice, once with and once without extras.

(py37) jreese@jreese-mbp ~/pyfi » honesty deps azure-cli --flat | sort
MarkupSafe==1.1.1
PyJWT==1.7.1
PyJWT[crypto]==1.7.1
adal==1.2.3
antlr4-python3-runtime==4.7.2
applicationinsights==0.11.9
argcomplete==1.11.1
azure-batch==9.0.0
azure-cli-command-modules-nspkg==2.0.3
azure-cli-core==2.6.0
azure-cli-nspkg==3.0.4
azure-cli-telemetry==1.0.4
azure-cli==2.6.0
azure-common==1.1.25
azure-core==1.5.0
azure-cosmos==3.1.2
azure-datalake-store==0.0.48
azure-functions-devops-build==0.0.22
azure-graphrbac==0.60.0
azure-keyvault==1.1.0
azure-loganalytics==0.1.0
azure-mgmt-advisor==2.0.1
azure-mgmt-apimanagement==0.1.0
azure-mgmt-appconfiguration==0.4.0
azure-mgmt-applicationinsights==0.1.1
azure-mgmt-authorization==0.52.0
azure-mgmt-batch==7.0.0
azure-mgmt-batchai==2.0.0
azure-mgmt-billing==0.2.0
azure-mgmt-botservice==0.2.0
azure-mgmt-cdn==4.1.0rc1
azure-mgmt-cognitiveservices==5.0.0
azure-mgmt-compute==12.0.0
azure-mgmt-consumption==2.0.0
azure-mgmt-containerinstance==1.5.0
azure-mgmt-containerregistry==3.0.0rc12
azure-mgmt-containerservice==9.0.1
azure-mgmt-core==1.0.0
azure-mgmt-cosmosdb==0.14.0
azure-mgmt-datalake-analytics==0.2.1
azure-mgmt-datalake-nspkg==3.0.1
azure-mgmt-datalake-store==0.5.0
azure-mgmt-datamigration==0.1.0
azure-mgmt-deploymentmanager==0.2.0
azure-mgmt-devtestlabs==4.0.0
azure-mgmt-dns==2.1.0
azure-mgmt-eventgrid==2.2.0
azure-mgmt-eventhub==3.0.0
azure-mgmt-hdinsight==1.4.0
azure-mgmt-imagebuilder==0.2.1
azure-mgmt-iotcentral==3.0.0
azure-mgmt-iothub==0.12.0
azure-mgmt-iothubprovisioningservices==0.2.0
azure-mgmt-keyvault==2.2.0
azure-mgmt-kusto==0.3.0
azure-mgmt-loganalytics==0.5.0
azure-mgmt-managedservices==1.0.0
azure-mgmt-managementgroups==0.2.0
azure-mgmt-maps==0.1.0
azure-mgmt-marketplaceordering==0.2.1
azure-mgmt-media==2.1.0
azure-mgmt-monitor==0.9.0
azure-mgmt-msi==0.2.0
azure-mgmt-netapp==0.8.0
azure-mgmt-network==10.1.0
azure-mgmt-nspkg==3.0.2
azure-mgmt-policyinsights==0.4.0
azure-mgmt-privatedns==0.1.0
azure-mgmt-rdbms==2.2.0
azure-mgmt-recoveryservices==0.4.0
azure-mgmt-recoveryservicesbackup==0.6.0
azure-mgmt-redhatopenshift==0.1.0
azure-mgmt-redis==7.0.0rc1
azure-mgmt-relay==0.1.0
azure-mgmt-reservations==0.6.0
azure-mgmt-resource==9.0.0
azure-mgmt-search==2.1.0
azure-mgmt-security==0.1.0
azure-mgmt-servicebus==0.6.0
azure-mgmt-servicefabric==0.4.0
azure-mgmt-signalr==0.3.0
azure-mgmt-sql==0.18.0
azure-mgmt-sqlvirtualmachine==0.5.0
azure-mgmt-storage==9.0.0
azure-mgmt-trafficmanager==0.51.0
azure-mgmt-web==0.46.0
azure-multiapi-storage==0.3.2
azure-nspkg==3.0.2
azure-storage-blob==1.5.0
azure-storage-common==1.4.2
bcrypt==3.1.7
certifi==2020.4.5.1
cffi==1.14.0
chardet==3.0.4
colorama==0.4.3
cryptography==2.9.2
fabric==2.5.0
humanfriendly==8.2
idna==2.9
importlib-metadata==1.6.0
invoke==1.4.1
isodate==0.6.0
javaproperties==0.5.1
jinja2==2.11.2
jmespath==0.10.0
jsmin==2.2.2
jsondiff==1.2.0
knack==0.7.1
mock==4.0.2
msal-extensions==0.1.3
msal==1.0.0
msal==1.3.0
msrest==0.6.14
msrestazure==0.6.3
oauthlib==3.1.0
paramiko==2.7.1
pkginfo==1.5.0.1
portalocker==1.7.0
pycparser==2.20
pygments==2.6.1
pynacl==1.3.0
pyopenssl==19.1.0
python-dateutil==2.8.1
pytz==2019.1
pyyaml==5.3.1
requests-oauthlib==1.3.0
requests==2.23.0
scp==0.13.2
six==1.15.0
sshtunnel==0.1.5
tabulate==0.8.7
urllib3==1.25.9
urllib3[secure]==1.25.9
vsts-cd-manager==1.0.2
vsts==0.1.25
websocket-client==0.56.0
xmltodict==0.12.0
zipp==3.1.0

Get 'deps' and 'revs' commands into a release

Better color for deps --have reuse

Configuration file

The ways you configure operation today are either either by environment variable or commandline flag; we should have a config file to have machine-specific defaults.

it also may make sense to read index-url from pip.conf.

Add separate json metadata base

Right now the json url is derived from the simple url, but it doesn't appear that devpi mirrors or proxies this data in the same relative place (perhaps related to devpi/devpi#313 ). Adding a second configurable value which is by default the same derivation we have today, but overridable, would solve this while still fetching archives through devpi.

The change would be near references to DEFAULT_HONESTY_INDEX_URL in honesty/cache.py.

'deps' doesn't work with pre-only releases

repro: honesty deps odoo12-addon-partner-contact-birthplace

  File "/home/tim/code/honesty/honesty/cmdline.py", line 340, in deps
    deptree = DepWalker(
  File "/home/tim/code/honesty/honesty/deps.py", line 122, in walk
    (package, v) = self._pick_a_version(req, cache)
  File "/home/tim/code/honesty/honesty/deps.py", line 205, in _pick_a_version
    v = _find_compatible_version(package, req.specifier, self.python_version)
  File "/home/tim/code/honesty/honesty/deps.py", line 369, in _find_compatible_version
    raise ValueError(
ValueError: odoo has no None compatible release with constraint <12.1dev,>=12.0a

Textual notes instead of colors for deps

download/extract automatic index freshening

Right now you have two choices for download/extract:

1. Don't pass --fresh, which will error out if we don't have a cached index with an sdist for that version
2. Pass --fresh, and always pay a roundtrip to update an index, even if the cached version has an sdist for that/

What we have now requires the caller to decide up front, and the api.download_* functions require you to already have a Package object. A stopgap would be a pick_sdist(pkg, ver) function that just has the list comprehension that keeps getting repeated:

sdists = [
    f for f in package.releases[version].files if f.file_type == FileType.SDIST
]

Add json output for deps

`download` command to fetch sdists from PyPI

Ideal command invocation:

honesty download [--index-url=...] [--dest=<dir>] <package>[==<version>]

Suggested API for fetching from Python code:

def download(
    package: str,
    version: packaging.Version,
    dest: Path,
    *,
    index_url: str = "https://pypi.org/simple/"
) -> Path:
    ...

>>> from honesty import download

>>> download("aiosqlite", Version("0.10.0"), Path("/tmp/sdists/"))
PosixPath('/tmp/sdists/aiosqlite-0.10.0.tar.gz')

Add support for keke traces

Depends on #26 for context probably

Support for pip.conf

Would be neat if honesty respected the configured pypi mirror, which is typically in /etc/pip.conf or ~/.config/pip/pip.conf (and more places if you want to be exhaustive... but these are the 2 main ones).

An alternative way would be to get it via pip config get global.index-url

Improve 'download' error message when no sdist

This case is handled properly on 'extract' but not 'download'

Convenient way of specifying python version

Would be very convenient if a short form for --python-version was available, like -p, and if major.minor was accepted, such that we could run:

honesty deps -p3.7 requests

Standardize --verbose, --use-json, etc w/ context

Sort releases that come back from json api

It appears that the releases from the json api are string-sorted, so if you use something like honesty extract foo without a version, it picks the last (not necessarily highest version).

API and download/extract don't normalize versions

This happens for functools32, which has a version "3.2.3-2" which parses as "3.2.3-post2" and that's what Poetry gives for a solve. That alternate spelling of the version also works ok in pip. Honesty should support the same.

os.rename doesn't overwrite on windows

This happens inside cache fetches, for index.html or json mostly. If they're fetched once, then change, we get a traceback.

honesty depends on click, buit doesn't list it as a dependency

(venv37) jreese@jreese-mbp ~/scratch/aiosqlite-0.10.0 ‹130› » pip install -U honesty
Collecting honesty
  Downloading https://files.pythonhosted.org/packages/e2/ea/ad91b23740c5103f15422f1344b92ce4ccff7986ab2837ea0b4bae40dc80/honesty-0.1.1-py3-none-any.whl
Installing collected packages: honesty
Successfully installed honesty-0.1.1
(venv37) jreese@jreese-mbp ~/scratch/aiosqlite-0.10.0 » rehash
(venv37) jreese@jreese-mbp ~/scratch/aiosqlite-0.10.0 » honesty --help
Traceback (most recent call last):
  File "/Users/jreese/scratch/venv37/bin/honesty", line 5, in <module>
    from honesty.cmdline import main
  File "/Users/jreese/scratch/venv37/lib/python3.7/site-packages/honesty/cmdline.py", line 7, in <module>
    import click
ModuleNotFoundError: No module named 'click'

deps --flat should be ordered (toplogical?)

Extraction dir doesn't come from appdirs

On Windows, this means it's not in the right place, but also has forward slashes in it.

Wrong entrypoint in setup.py

(venv37) jreese@jreese-mbp ~/scratch/aiosqlite-0.10.0 » honesty --help
Traceback (most recent call last):
  File "/Users/jreese/scratch/venv37/bin/honesty", line 5, in <module>
    from honesty.cmdline import main
ImportError: cannot import name 'main' from 'honesty.cmdline' (/Users/jreese/scratch/venv37/lib/python3.7/site-packages/honesty/cmdline.py)

Deps read metadata from pep 658/714 metadata links if they exist

Be more consistent about naming like "package"

Most instances of "package" should be changed to "project" per the glossary

early versions of flit have requires_python="3"

This breaks the in-progress deps subcommand. Is this even valid?