fredrikaverpil/creosote

Identify unused dependencies and avoid a bloated virtual environment.

Hi there 👋

I am a passionate, enthusiastic and professional software developer, continuously striving to evolve. I prefer to build scalable and reliable backend microservice systems in public cloud environments. I've got a more verbose profile over at LinkedIn.

📫 How to reach me: Mastodon, LinkedIn

Latest blog posts

• Export repo data from GitHub organization
• Decoupling the ORM class from the data model class
• Access Python dict using dot notation
• Building Python wheels targeting different interpreter versions
• Using GitHub merge queue to ease the Dependabot churn

creosote's Issues

Add a way to only check the dependency file (without checking the site-packages)

What feature request would you like to suggest?

I would like to be able to only compare the dependency file against the code without checking site-packages. I don't mind if my site-packages are bloated on my machine, what I care about is not shipping unnecessary dependencies.

Will the feature request break backwards compatibility?

The default could remain the same, this being an optional flag, so no backwards compatibility issues.

Pin ruff

What happened?

Ruff linting keeps failing inbetween PRs because the version isn't pinned...

Reproduction steps

N/A

Relevant log output

N/A

Ability to exclude some paths (not deps) from checking

What feature request would you like to suggest?

I have a Django project

Unfortunately in the static files there is a JS package that has a Python 2 script in it, so when I run creosote I get:

  File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "apps/static/assets/vendors/ionicons-npm/builder/generate.py", line 29
    print "Generate Fonts"
    ^^^^^^^^^^^^^^^^^^^^^^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(...)?

I have a file structure like:

app/
  a/
    ...py files to check...
  b/
    ...py files to check...
  c/
    ...py files to check...
  static/
    assets/
      ...somewhere in here a bad JS lib...
  t/
    ...py files to check...
  u/
    ...py files to check...
core/
    ...py files to check...
scripts/
    ...py files to check...
tests/
    ...py files to check...

so I am running creosote like:

creosote --p scripts -p core -p tests -p apps --verbose --use-feature v3-args

it looks like my only option is instead of -p apps to manually specify all the relevant subdirs of apps/ as a dozen more -p args

so I'd love to have a flag that is the opposite of -p to exclude dirs

Will the feature request break backwards compatibility?

not if it's just adding a new cli arg

Crash when specifying excluded dependencies

What happened?

Creosote crashed.

Reproduction steps

$ creosote --exclude-deps python-multipart uvicorn --verbose

Creosote version: 2.6.1
Command: creosote --exclude-deps python-multipart uvicorn --verbose
Arguments: Namespace(verbose=True, format='default', paths=['src'], sections=['project.dependencies'], exclude_deps=['python-multipart', 'uvicorn'], deps_file='pyproject.toml', venvs=['.venv'], features=[])
Imports found in code:
Parsing pyproject.toml for dependencies...
['project.dependencies']: []
Detected PEP-621 toml section in pyproject.toml
No dependencies found in section project.dependencies
Found dependencies in pyproject.toml: 
Traceback (most recent call last):
  File "/Users/USER/product/.venv/bin/creosote", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/USER/.venv/lib/python3.11/site-packages/creosote/cli.py", line 194, in main
    excluded_deps_and_not_installed = parsers.get_excluded_deps_which_are_not_installed(
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/USER/.venv/lib/python3.11/site-packages/creosote/parsers.py", line 228, in get_excluded_deps_which_are_not_installed
    if excluded_dep_name not in get_installed_dependency_names(venv):
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/USER/.venv/lib/python3.11/site-packages/creosote/parsers.py", line 212, in get_installed_dependency_names
    site_packages = Path(venv).glob("**/site-packages").__next__()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopIteration

Relevant log output

See previous section.

It does mention a .venv directory that doesn't exist yet in the current directory, but it uses the one from the parent directory.

New, more grokable, test setup

What feature request would you like to suggest?

Would like to see some new pytest fixtures, making it easier to set up integration tests and also grok them. Also, would like to see new tests which can be used during development and for newcomers to the project. Pretty much; increase grokability through better tests.

Will the feature request break backwards compatibility?

No

Unintended behavior when using multiple source code files/folders

What happened?

As reported by @sanmai-NL in #128 (comment):

Can you please document that consecutive --paths parameters override the previous?

Single --paths: an unused dependency is one that isn't referenced at least once in all the Python source code under the specified paths. Multiple --paths: an unused dependency is one that isn't referenced at least once in each of the subsets of source code under each specified path.

This was actually never my intent. Instead my intent is:

An unused dependency is one that isn't referenced at least once in all the Python source code, defined with the -p/--paths argument.

Reproduction steps

Needs steps to reproduce here.

Relevant log output

-

ast.walk instead of ast.iter_child_nodes

Should likely use ast.walk instead here.

Issue running on project that is based on Poetry and pyenv

What happened?

I was expecting it to just work and it was complaining about not being able to find Could not find toml section project.dependencies.

Reproduction steps

1. pip install creosote
2. creosote (from inside the git folder as well as with the pyenv virtualenv active)

Relevant log output

❯_: creosote
Traceback (most recent call last):
  File "/home/ubuntu/.pyenv/versions/3.11.1/envs/<venv>/lib/python3.11/site-packages/creosote/parsers.py", line 44, in pyproject
    section_contents = dotty_contents[section]
                       ~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/ubuntu/.pyenv/versions/3.11.1/envs/<venv>/lib/python3.11/site-packages/dotty_dict/dotty_dict.py", line 162, in __getitem__
    return get_from(self._split(item), self._data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/.pyenv/versions/3.11.1/envs/<venv>/lib/python3.11/site-packages/dotty_dict/dotty_dict.py", line 154, in get_from
    data = data[it]
           ~~~~^^^^
KeyError: 'project'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/.pyenv/versions/<venv>/bin/creosote", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/ubuntu/.pyenv/versions/3.11.1/envs/<venv>/lib/python3.11/site-packages/creosote/cli.py", line 100, in main
    deps_reader.read(args.deps_file, args.sections)
  File "/home/ubuntu/.pyenv/versions/3.11.1/envs/<venv>/lib/python3.11/site-packages/creosote/parsers.py", line 124, in read
    self.pyproject(deps_file, sections)
  File "/home/ubuntu/.pyenv/versions/3.11.1/envs/<venv>/lib/python3.11/site-packages/creosote/parsers.py", line 46, in pyproject
    raise KeyError(f"Could not find toml section {section}.") from err
KeyError: 'Could not find toml section project.dependencies.'

Creosote pre-commit hook incorrectly flags python-dateutil as unused

What happened?

When running the Creosote pre-commit hook on my repository, it incorrectly identifies python-dateutil as an unused dependency, despite it being actively used in the code. The tool fails with an exit code of 1 and suggests that my virtual environment is bloated with unused dependencies.

Reproduction steps

1. Create a pyproject.toml with python-dateutil listed under [tool.poetry.dependencies].
2. Implement usage of python-dateutil in a Python file within the project.
3. Configure .pre-commit-config.yaml with Creosote to check for unused dependencies, specifically pointing to tool.poetry.dependencies.
4. Run poetry install to set up the environment.
5. Execute the pre-commit hook using poetry run pre-commit run creosote --all-files.

Files

───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: pyproject.toml
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ [tool.poetry]
   2   │ name = "alma"
   3   │ version = "0.1.0"
   4   │ description = ""
   5   │ authors = ["Balazs"]
   6   │ readme = "README.md"
   7   │
   8   │ [tool.poetry.dependencies]
   9   │ python = "^3.10"
  10   │ python-dateutil = "^2.8.2"
  11   │
  12   │
  13   │ [tool.poetry.group.dev.dependencies]
  14   │ pre-commit = "^3.5.0"
  15   │
  16   │ [build-system]
  17   │ requires = ["poetry-core"]
  18   │ build-backend = "poetry.core.masonry.api"
───────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: src/main.py
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ from dateutil.relativedelta import *
   2   │ from dateutil.easter import *
   3   │ from dateutil.rrule import *
   4   │ from dateutil.parser import *
   5   │ from datetime import *
   6   │ now = parse("Sat Oct 11 17:13:46 UTC 2003")
   7   │ today = now.date()
   8   │ year = rrule(YEARLY,dtstart=now,bymonth=8,bymonthday=13,byweekday=FR)[0].year
   9   │ rdelta = relativedelta(easter(year), today)
  10   │ print("Today is: %s" % today)
  11   │ print("Year with next Aug 13th on a Friday is: %s" % year)
  12   │ print("How far is the Easter of that year: %s" % rdelta)
  13   │ print("And the Easter of that year is: %s" % (today+rdelta))
───────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

[tool.poetry]
name = "alma"
version = "0.1.0"
description = ""
authors = ["Balazs Erdos"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.10"
python-dateutil = "^2.8.2"


[tool.poetry.group.dev.dependencies]
pre-commit = "^3.5.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
from dateutil.relativedelta import *
from dateutil.easter import *
from dateutil.rrule import *
from dateutil.parser import *
from datetime import *
now = parse("Sat Oct 11 17:13:46 UTC 2003")
today = now.date()
year = rrule(YEARLY,dtstart=now,bymonth=8,bymonthday=13,byweekday=FR)[0].year
rdelta = relativedelta(easter(year), today)
print("Today is: %s" % today)
print("Year with next Aug 13th on a Friday is: %s" % year)
print("How far is the Easter of that year: %s" % rdelta)
print("And the Easter of that year is: %s" % (today+rdelta))

Relevant log output

$ poetry run pre-commit run creosote --all-files --verbose

creosote.................................................................Failed
- hook id: creosote�[m
- duration: 0.1s�[m
- exit code: 1�[m

Creosote version: 3.0.0
Command: creosote --section=tool.poetry.dependencies --verbose
Default configuration (may have loaded pyproject.toml): Config(format='default', paths=['src'], sections=['project.dependencies'], exclude_deps=[], deps_file='pyproject.toml', venvs=['/home/foo/.cache/pre-commit/repojiy8wtb6/py_env-python3.10'], features=[])
Arguments: Namespace(verbose=True, format='default', paths=['src'], sections=['tool.poetry.dependencies'], exclude_deps=[], deps_file='pyproject.toml', venvs=['/home/foo/.cache/pre-commit/repojiy8wtb6/py_env-python3.10'], features=[])
Parsing src/main.py
Imports found in code:
- ImportInfo(module=['dateutil', 'relativedelta'], name=['*'], alias=None)
- ImportInfo(module=['dateutil', 'easter'], name=['*'], alias=None)
- ImportInfo(module=['dateutil', 'rrule'], name=['*'], alias=None)
- ImportInfo(module=['dateutil', 'parser'], name=['*'], alias=None)
- ImportInfo(module=['datetime'], name=['*'], alias=None)
Parsing pyproject.toml for dependencies...
['tool.poetry.dependencies']: {'python': '^3.10', 'python-dateutil': '^2.8.2'}
Detected Poetry toml section in pyproject.toml
Found dependencies in pyproject.toml: python-dateutil�[0m
Gathering all top_level.txt files in venv /home/foo/.cache/pre-commit/repojiy8wtb6/py_env-python3.10...
Found /home/foo/.cache/pre-commit/repojiy8wtb6/py_env-python3.10/lib/python3.10/site-packages/pip_requirements_parser-32.0.1.dist-info/top_level.txt
Found /home/foo/.cache/pre-commit/repojiy8wtb6/py_env-python3.10/lib/python3.10/site-packages/setuptools-68.2.2.dist-info/top_level.txt
Found /home/foo/.cache/pre-commit/repojiy8wtb6/py_env-python3.10/lib/python3.10/site-packages/pip-23.3.1.dist-info/top_level.txt
Found /home/foo/.cache/pre-commit/repojiy8wtb6/py_env-python3.10/lib/python3.10/site-packages/loguru-0.7.2.dist-info/top_level.txt
Found /home/foo/.cache/pre-commit/repojiy8wtb6/py_env-python3.10/lib/python3.10/site-packages/toml-0.10.2.dist-info/top_level.txt
Gathering all top_level.txt files in venv /home/foo/.cache/pre-commit/repojiy8wtb6/py_env-python3.10...
Found /home/foo/.cache/pre-commit/repojiy8wtb6/py_env-python3.10/lib/python3.10/site-packages/creosote-3.0.0.dist-info/RECORD
Found /home/foo/.cache/pre-commit/repojiy8wtb6/py_env-python3.10/lib/python3.10/site-packages/pip_requirements_parser-32.0.1.dist-info/RECORD
Found /home/foo/.cache/pre-commit/repojiy8wtb6/py_env-python3.10/lib/python3.10/site-packages/setuptools-68.2.2.dist-info/RECORD
Found /home/foo/.cache/pre-commit/repojiy8wtb6/py_env-python3.10/lib/python3.10/site-packages/pip-23.3.1.dist-info/RECORD
Found /home/foo/.cache/pre-commit/repojiy8wtb6/py_env-python3.10/lib/python3.10/site-packages/dotty_dict-1.3.1.dist-info/RECORD
Found /home/foo/.cache/pre-commit/repojiy8wtb6/py_env-python3.10/lib/python3.10/site-packages/packaging-23.2.dist-info/RECORD
Found /home/foo/.cache/pre-commit/repojiy8wtb6/py_env-python3.10/lib/python3.10/site-packages/loguru-0.7.2.dist-info/RECORD
Found /home/foo/.cache/pre-commit/repojiy8wtb6/py_env-python3.10/lib/python3.10/site-packages/toml-0.10.2.dist-info/RECORD
Found /home/foo/.cache/pre-commit/repojiy8wtb6/py_env-python3.10/lib/python3.10/site-packages/wheel-0.41.3.dist-info/RECORD
Found /home/foo/.cache/pre-commit/repojiy8wtb6/py_env-python3.10/lib/python3.10/site-packages/pyparsing-3.1.1.dist-info/RECORD
Attempting to find import names...
[python-dateutil] did not find top_level.txt in venv
[python-dateutil] did not find RECORD in venv
[python-dateutil] relying on canonicalization fallback: python_dateutil 🤞
Dependencies with populated 'associated_import' attribute are used in code. End result of resolve:
- DependencyInfo(name='python-dateutil', top_level_import_names=None, record_import_names=None, canonicalized_dep_name='python_dateutil', associated_imports=[])
�[31mOh no, bloated venv! 🤢 🪣
Unused dependencies found: python-dateutil

Note: The python-dateutil module is imported and used as follows:

from dateutil.relativedelta import *
from dateutil.easter import *
from dateutil.rrule import *
import dateutil.parser as date_parser

These calls are active in the codebase, and the error stating that python-dateutil is an unused dependency is incorrect.

License not showing on pypi.org page

For some reason, the MIT license is not shown on https://pypi.org/project/creosote/

Doesn't seem to work with Pipenv

What happened?

creosote --deps-file Pipfile --sections packages causes NotImplementedError: Dependency specs file Pipfile is not supported.

I'm confused because the FAQ says Pipenv is supported.

Reproduction steps

1. Have a project that uses Pipfile to define dependencies
2. run creosote --deps-file Pipfile --sections packages

Relevant log output

Creosote version: 2.6.2
Command: creosote --deps-file Pipfile --sections packages --verbose
Arguments: Namespace(verbose=True, format='default', paths=[], sections=['packages'], exclude_deps=[], deps_file='Pipfile', venvs=['.venv'], features=[])
Imports found in code:
Parsing Pipfile for dependencies...

Incorrectly identifies python-consul as unused

What happened?

I'm using python-consul. The package is usable with import consul. I expected Creosote to identify python-consul is used, but it did not.

I guess this is down to the package machinery in python-consul? It looks like py_modules is used in the setup & not packages - not sure if this could be the issue. This package is not actively maintained.

Reproduction steps

python -m venv .venv
. .venv/bin/activate
echo python-consul > requirements.txt
echo 'import consul; print consul.Consul()' > do_stuff.py
pip install -r requirements.txt
python do_stuff.py
# works
creosote --deps-file requirements.txt --paths do_stuff.py --verbose
# result below

Relevant log output

Creosote version: 2.4.0
Command: creosote --deps-file requirements.txt --paths do_stuff.py --verbose
Arguments: Namespace(version=False, verbose=True, format='default', paths=['do_stuff.py'], venv='.venv', deps_file='requirements.txt', sections=['project.dependencies'], exclude_deps=[])
Parsing /home/jack/scratch/try_out_creosote/do_stuff.py
Imports found in code:
- ImportInfo(module=[], name=['consul'], alias=None)
Parsing requirements.txt for dependencies...
Found dependencies in requirements.txt: python-consul
Dependencies found in requirements.txt:
- python-consul
Gathering all top_level.txt files in venv...
Found .venv/lib/python3.10/site-packages/certifi-2022.12.7.dist-info/top_level.txt
Found .venv/lib/python3.10/site-packages/charset_normalizer-3.1.0.dist-info/top_level.txt
Found .venv/lib/python3.10/site-packages/pip-22.3.1.dist-info/top_level.txt
Found .venv/lib/python3.10/site-packages/python_consul-1.1.0.dist-info/top_level.txt
Found .venv/lib/python3.10/site-packages/requests-2.28.2.dist-info/top_level.txt
Found .venv/lib/python3.10/site-packages/setuptools-65.5.0.dist-info/top_level.txt
Found .venv/lib/python3.10/site-packages/six-1.16.0.dist-info/top_level.txt
Found .venv/lib/python3.10/site-packages/urllib3-1.26.15.dist-info/top_level.txt
Attempting to find import names...
[python-consul] found import name via top_level.txt: consul/__init__,consul/aio,consul/base,consul/std,consul/tornado,consul/twisted ⭐️
Dependencies with populated 'associated_import' attribute are used in code. End result of resolve:
- DependencyInfo(name='python-consul', top_level_import_names=['consul/__init__', 'consul/aio', 'consul/base', 'consul/std', 'consul/tornado', 'consul/twisted'], distlib_db_import_name=None, canonicalized_dep_name='python_consul', associated_imports=[])

An --ignores argument?

It's possible there is a use case for an --ignore-packages argument. The user can ignore certain packages and creosote would simply ignore reporting them as unused. The ignores already exists internally here so all that is needed to enable this is to pass an argument to the CLI and pass it onto that function.

The use case could be e.g. using SQLAlchemy with a database driver. You wouldn't import the database driver in your own source code. Instead you would tell SQLAlchemy to use it via e.g. connection or engine creation string. This would be too tricky to support for Creosote, I think.

To avoid forgetting about the ignores (if e.g. discontinuing the use of said database driver above), Creosote could complain if an ignored package is no longer found in the dependency file (pyproject.toml, requirements.txt, ...), prompting the user to also remove the ignore.

🤔 Would it also be useful to ignore certain files/folders from the source code scan?
Anyhow, this would in that case be a different CLI argument, like --ignore-paths.

I'm just leaving this thought here for now to simmer. Please upvote if you think this would be a good addition to Creosote. 😄

Potentially replace top_level.txt logic

What feature request would you like to suggest?

The current logic/code for grabbing the top_level.txt file contents (in map_dep_to_import_via_top_level_txt_file) could potentially be replaced with:

from importlib_metadata import _top_level_declared, _top_level_inferred, distributions
import collections

site_packages_path = "/absolute/path/to/.venv/lib/python3.7/site-packages"
pkg_to_dist = collections.defaultdict(list)
for dist in package_distributions:
    for pkg in _top_level_declared(dist) or _top_level_inferred(dist):
        pkg_to_dist[pkg].append(dist.metadata['Name'])
print(dict(pkg_to_dist))

Output:

{'pathspec-0.11.1.dist-info': ['pathspec'], 'pathspec': ['pathspec'], '__pycache__': ['typing_extensions', 'black'], 'typing_extensions-4.5.0.dist-info': ['typing_extensions'], 'typing_extensions': ['typing_extensions'], 'certifi': ['certifi'], '_distutils_hack': ['setuptools'], 'pkg_resources': ['setuptools'], 'setuptools': ['setuptools'], '..': ['creosote', 'black', 'ruff'], 'creosote-2.4.0.dist-info': ['creosote'], '_ast27': ['typed-ast'], '_ast3': ['typed-ast'], 'typed_ast': ['typed-ast'], 'toml': ['toml'], 'urllib3': ['urllib3'], 'black': ['black'], '_black_version': ['black'], 'blackd': ['black'], 'black-23.1.0.dist-info': ['black'], 'blib2to3': ['black'], 'requests': ['requests'], 'consul/__init__': ['python-consul'], 'consul/aio': ['python-consul'], 'consul/base': ['python-consul'], 'consul/std': ['python-consul'], 'consul/tornado': ['python-consul'], 'consul/twisted': ['python-consul'], 'six': ['six'], '_pytest': ['pytest'], 'py': ['pytest'], 'pytest': ['pytest'], 'mypy_extensions': ['mypy-extensions'], 'loguru_mypy': ['loguru-mypy'], 'click': ['click'], 'iniconfig-2.0.0.dist-info': ['iniconfig'], 'iniconfig': ['iniconfig'], 'pip': ['pip'], 'dotty_dict': ['dotty-dict'], 'dotty_dict-1.3.1.dist-info': ['dotty-dict'], 'zipp': ['zipp'], 'pytest_mock': ['pytest-mock'], 'ruff-0.0.257.dist-info': ['ruff'], 'ruff': ['ruff'], 'platformdirs': ['platformdirs'], 'platformdirs-3.1.1.dist-info': ['platformdirs'], 'idna': ['idna'], 'idna-3.4.dist-info': ['idna'], 'distlib': ['distlib'], 'pluggy': ['pluggy'], 'toml-stubs': ['types-toml'], 'attr': ['attrs'], 'attrs': ['attrs'], 'tomli': ['tomli'], 'tomli-2.0.1.dist-info': ['tomli'], 'wheel': ['wheel'], 'exceptiongroup': ['exceptiongroup'], 'exceptiongroup-1.1.1.dist-info': ['exceptiongroup'], 'mypy': ['mypy'], 'mypyc': ['mypy'], 'packaging-23.0.dist-info': ['packaging'], 'packaging': ['packaging'], 'packaging_legacy_version': ['pip-requirements-parser'], 'pip_requirements_parser': ['pip-requirements-parser'], 'importlib_metadata': ['importlib-metadata'], 'charset_normalizer': ['charset-normalizer'], 'pyparsing-3.0.9.dist-info': ['pyparsing'], 'pyparsing': ['pyparsing'], 'loguru': ['loguru']}

What concerns me here is the docs mention that packages_distributions() was introduced in Python 3.10.

Just leaving this here for now.

Will the feature request break backwards compatibility?

No.

Dependencies of packages flagged as unused

Hi,

My requirements.txt contains packages installed as dependencies of other modules (ex. Pandas requires pytz, pdfplumber requires Pillow and loguru requires colorama) that are not used as imports in code. They are flagged as unused.

Can this be changed? The requirements are often available in METADATA file in dist-info folders.

Speed up test suite

What feature request would you like to suggest?

Instead of actually calling pip or poetry, I'd like to get rid of these slow tests and instead mock and use fast unit testing. Some refactoring is being made in #142 which will make it easier to mock the dependency file reader. Pytest fixtures should be made for adding dependencies, excluding dependencies, returning a deps/venv/distlib/canonicalization resolve etc.

• Add enough fixtures and tests around defining packages/venv etc so that the slow integration tests can be replaced with faster and equally reliable tests.
• Would also be nice to add a coverage report, so to easily spot uncovered logic.
• Would like to remove poetry as a required dependency for running the test suite.
• Use built in caching from actions/setup-python:

- uses: actions/setup-python@v4
  with:
    cache: 'pip'
    cache-dependency-path: 'pyproject.toml'

Will the feature request break backwards compatibility?

No.

Tests: top_level.txt is not found on GHA Windows runner

What happened?

The Windows GHA runner consistently fails to find the top_level.txt file when running the pytest workflow. This, however, works fine locally on a Windows machine of mine.

Judging from the log output, the glob in gather_top_level_filepaths works, and finds the top_level.txt files, but then the self.top_level_txt_pattern.findall fails in map_dep_to_import_via_top_level_txt_file.

The reason is that the found paths from the glob contains backslashes (of course), and seems to be absolute (contains C:\. And the compiled regex pattern uses forward slashes and does not take the C:\ into consideration.

At first glance, I don't quite understand how this is not also failing the test on my local Windows machine (I haven't fully debugged it there)...

This ugly fix looks like it fixes it, so it's about the backslashes:

        for top_level_filepath in self.top_level_filepaths:
            normalized_top_level_filepath = str(top_level_filepath).replace("\\", "/")
            matches = self.top_level_txt_pattern.findall(normalized_top_level_filepath)

Reproduction steps

1. Enable windows-latest in the GHA os matrix for pytest.
2. Observe how test_no_unused_and_found_in_top_level_txt fails where mocked_map_via_distlib.assert_not_called() fails as it is being called (which it shouldn't be).

Relevant log output

Creosote version: 2.4.0
Command: creosote -s -vv -k test_no_unused_and_found_in_top_level_txt
Arguments: Namespace(deps_file='pyproject.toml', exclude_deps=[], format='default', paths=['src'], sections=['project.dependencies'], venv='C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\test_no_unused_and_found_in_to0\\venv', verbose=True, version=False)
Parsing pyproject.toml for dependencies...
Found dependencies in pyproject.toml: yolo
Gathering all top_level.txt files in venv...
Found C:\Users\runneradmin\AppData\Local\Temp\pytest-of-runneradmin\pytest-0\test_no_unused_and_found_in_to0\venv\lib\python3.7\site-packages\yolo-1.2.3.dist-info\top_level.txt
Attempting to find import names...
FOR DEP IN SELF.DEPS
IN map_dep_to_import_via_top_level_txt_file
CHECKING C:\Users\runneradmin\AppData\Local\Temp\pytest-of-runneradmin\pytest-0\test_no_unused_and_found_in_to0\venv\lib\python3.7\site-packages\yolo-1.2.3.dist-info\top_level.txt
MATCHES: []   <------------- this is the problem
[yolo] did not find top_level.txt in venv
[yolo] relying on canonicalization fallback:  \U0001f91e
Dependencies with populated 'associated_import' attribute are used in code. End result of resolve:
- DependencyInfo(name='yolo', top_level_import_names=None, distlib_db_import_name=None, canonicalized_dep_name='', associated_imports=[])
Oh no, bloated venv! \U0001f922 \U0001faa3

Creosote 3.0.0. fails to detect usage of `ruamel-yaml` 0.18.6.

What happened?

Oh no, bloated venv! 🤢 🪣
Unused dependencies found: ruamel-yaml

Top top-level init.py:

from ruamel.yaml import YAML

Reproduction steps

Add a package(s) to pyproject.toml main dependencies using PDM 2.15.3:

│ ruamel.yaml             │ 0.18.6                        │                                                                        │
│ ruamel.yaml.clib        │ 0.2.8                         │    

Run Creosote.

Relevant log output

Creosote 3.0.0

Support PDM through `pyproject.toml`

Problem

When using PDM and not Poetry, creosote 1.0.3 fails:

Parsing pyproject.toml for packages
Traceback (most recent call last):
  File "/package/.venv/lib/python3.10/site-packages/creosote/parsers.py", line 24, in _pyproject
    deps = contents["tool"]["poetry"]["dependencies"]
KeyError: 'poetry'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/package/.venv/bin/creosote", line 8, in <module>
    sys.exit(main())
  File "/package/.venv/lib/python3.10/site-packages/creosote/cli.py", line 84, in main
    deps_reader.read(args.deps_file, args.dev)
  File "/package/.venv/lib/python3.10/site-packages/creosote/parsers.py", line 58, in read
    self.packages = self.packages_sans_ignored(self._pyproject(deps_file, dev))
  File "/package/.venv/lib/python3.10/site-packages/creosote/parsers.py", line 26, in _pyproject
    raise Exception("Could not find expected toml property.") from e
Exception: Could not find expected toml property.

Sketch of solution

Iterate over project.dependencies and the keys under tool.pdm.dev-dependencies in pyproject.toml.

Exception in package_to_module

Tried it on my project: https://github.com/staticf0x/mrmonitor

Invocation: creosote --venv (poetry env info --path) --deps-file pyproject.toml --paths . --sections tool.poetry.dependencies

Parsing mrmonitor.py
Parsing reviews.py
Parsing pyproject.toml for packages
Found packages in pyproject.toml: arrow, click, python-dotenv, python-gitlab, rich
Resolving...
Traceback (most recent call last):
  File "/home/username/.local/bin/creosote", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/username/.local/lib/python3.11/site-packages/creosote/cli.py", line 90, in main
    deps_resolver.resolve()
  File "/home/username/.local/lib/python3.11/site-packages/creosote/resolvers.py", line 109, in resolve
    self.populate_packages()
  File "/home/username/.local/lib/python3.11/site-packages/creosote/resolvers.py", line 87, in populate_packages
    self.package_to_module(package)
  File "/home/username/.local/lib/python3.11/site-packages/creosote/resolvers.py", line 53, in package_to_module
    for filename, _, _ in dist.list_installed_files():
  File "/home/username/.local/lib/python3.11/site-packages/distlib/database.py", line 677, in list_installed_files
    for result in self._get_records():
                  ^^^^^^^^^^^^^^^^^^^
  File "/home/username/.local/lib/python3.11/site-packages/distlib/database.py", line 596, in _get_records
    with contextlib.closing(r.as_stream()) as stream:
                            ^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'as_stream'

Same thing in another project with requirements.txt

Does not support PEP 621

TODO:

• Add support for dependencies.
• Add support for optional-dependencies.

UnicodeDecodeError while parsing

Hi,

I'm using creosote 2.2.0 and while parsing I get a following error:

Traceback (most recent call last):
File "C:\Users\8008701\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\8008701\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users\8008701\PycharmProjects\onesource_idt-intl-tools\venv_test\Scripts\creosote.exe_main.py", line 7, in
File "C:\Users\8008701\PycharmProjects\onesource_idt-intl-tools\venv_test\lib\site-packages\creosote\cli.py", line 77, in main
imports = parsers.get_modules_from_code(args.paths)
File "C:\Users\8008701\PycharmProjects\onesource_idt-intl-tools\venv_test\lib\site-packages\creosote\parsers.py", line 163, in get_modules_from_code
for imp in get_module_info_from_code(resolved_path):
File "C:\Users\8008701\PycharmProjects\onesource_idt-intl-tools\venv_test\lib\site-packages\creosote\parsers.py", line 137, in get_module_info_from_code
root = ast.parse(fh.read(), path)
File "C:\Users\8008701\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 3560: character maps to

All files are in UTF-8

Fix errors in PyPi README

What happened?

Instead of showing the noice emojis in the PyPi README, it e.g. says :zap:... yuck.

Reproduction steps

N/A

Relevant log output

N/A

Support sourceless packages

What feature request would you like to suggest?

Sometimes, packages are just bundles of external packages. For example, a pyproject.toml with just dev dependencies created to manage CI tooling not even targeting Python. These don't need a source code directory. Currently, Creosote requires a source directory, has a default of src/ and crashes if it doesn't exist.

I propose that Creosote warns if src/ doesn't exist, but doesn't crash and provides a zero exit status, possibly behind a configuration option if the fault is desired in the common case.

Will the feature request break backwards compatibility?

No.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• chore(deps): update python docker tag to v3.11

Detected dependencies

github-actions

.github/workflows/check.yml

• actions/checkout v3
• actions/setup-python v4
• actions/checkout v3
• actions/setup-python v4

.github/workflows/publish.yml

• actions/checkout v3
• actions/setup-python v4
• actions/upload-artifact v3
• actions/download-artifact v3

.github/workflows/stale.yml

• actions/stale v8

.github/workflows/test.yml

• actions/checkout v3
• actions/setup-python v4
• actions/upload-artifact v3
• actions/checkout v3
• actions/setup-python v4

pep621

pyproject.toml

• dotty-dict >=1.3.1,<1.4
• loguru >=0.6.0,<0.8
• pip-requirements-parser >=32.0.1,<33.1
• toml >=0.10.2,<0.11

pyenv

.python-version

• python 3.7

---

• Check this box to trigger a request for Renovate to run again on this repository

Use the appropriate default values depending on the environment

What feature request would you like to suggest?

Upon running creosote --deps-file Pipfile, a KeyError is raised with the message 'Could not find toml section project.dependencies.' Even though this information is documented in the readme, I suggest that the default value for sections should be changed to "packages" to align with Pipenv's default behavior. While I'm not well-versed in other environments, it seems logical to extend the same thought process to them. In essence, creosote should dynamically determine the appropriate default section based on the specified dependency file, ensuring a better user experience and consistency across different environments.

Will the feature request break backwards compatibility?

I don't think so

Automatically detect virtualenv

What feature request would you like to suggest?

It would be nice if creosote could simply detect the virtualenv it is installed in instead of having to pass it explicitly. A script can easily detect the current virtualenv by checking sys.base_prefix. If sys.base_prefix is different from sys.prefix, you're in a virtualenv.

Will the feature request break backwards compatibility?

I don't think this would break any backward compatibility. For example, you could:

1. Check for .venv (current default)
2. Check for sys.base_prefix
3. Give error if no virtualenv is found

False positives for extra_requires subdependencies, and for packages who's pypi name differs from import name

I just tried running creosote v2.2.0 on my project, Belay:

creosote --venv .venv --paths belay --deps-file pyproject.toml --sections project.dependencies

The output:

Unused packages found: aiohttp, gitpython, typing-extensions

Without understanding creosote's internals, here's my speculations:

1. aiohttp is a subdependency of fsspec[http]. Perhaps creosote doesn't handle extra_requires properly.
2. gitpython is used as import git; i.e. it's pypi package name and module name are different. This stackoverflow answer provides some idea of how one might draw the relation (note; it uses deprecated pkg_resources):

>>> import pkg_resources as pkg
>>> metadata_dir = pkg.get_distribution('gitpython').egg_info
>>> open('%s/%s' % (metadata_dir, 'top_level.txt')).read().rstrip()
"git"

3. typing-extensions, possibly same story as (2), it's used as import typing_extensions.

I imagine other popular python packages may run into issues, a few off the top of my head:

• pillow - import PIL
• opencv-python - import cv2

Unrelated side small feature request; allow the version to be printed out from the CLI via creosote --version. Thanks!

Default --venv to $VIRTUAL_ENV

What feature request would you like to suggest?

Idea: instead of requiring the --venv .venv, it could simply pick up the value of $VIRTUAL_ENV. If this environment variable is not set, Creosote will exit and prompt the user to provide the --venv argument.

🤔 also, should perhaps --venv be deprecated in favor for --venvs. The use case could be that -p . is used or that user employs pip install --user ... (like here). No, probably not. Seems like a weird use case.

Will the feature request break backwards compatibility?

Yep.

Creosote does not exit with expected status when unused deps are found

What happened?

It is customary for tools like Creosote to exit with a failure-signalling exit status (value other than zero) when an issue is found. Creosote doesn't. The effect is that CI/CD pipelines in principle cannot detect unused dependencies with Creosote.

Reproduction steps

$ creosote --paths azure/ core/ datapipeline_ff/
Found packages in pyproject.toml: nsone, pip, pulumi, pulumi-azure-native, pulumi-command, pulumi-kubernetes, pulumi-ns1, setuptools
Oh no, bloated venv! 🤢 🪣
Unused packages found: nsone, pip, setuptools
$ echo $status
0

Relevant log output

2.3.7 (current main branch revision)

Allow configuration via pyproject.toml

What feature request would you like to suggest?

It would be nice if we could configure creosote via a [tool.creosote] section in pyproject.toml. Similar to other tools like pytest, CLI arguments could override values provided in pyproject.toml.

E.g.

[tool.creosote]
venv=".venv"
paths="my_project/"
deps-file="pyproject.toml"
sections=tool.poetry.dependencies
exclude-deps=[
    "importlib_resources",
    "pydantic",
]

Implementation details:

• Care must be taken to support lists for arguments that can be supplied multiple times.

Will the feature request break backwards compatibility?

Yes, since all projects would currently have a non-existant [tool.creosote] section, so then the default behavior is current behavior.

Issue running creosote in a large workspace

Picking up on the discussion here #114 (comment) in this issue.

ping @sanmai-NL

Support Git packages

In requirements.txt and modern package managers in pyproject.toml alike: e.g.,Pattern @ git+https://github.com/uob-vil/pattern.git.

When such a dependency is specified, it's incorrectly reported as unused by creosote 1.0.3.