asottile-archive/seed-isort-config

Statically populate the `known_third_party` `isort` setting.

DEPRECATED

this is no longer needed as of isort>=5

---

seed-isort-config

Statically populate the known_third_party isort setting.

isort when run in isolation is not the best at determining what dependencies are third party.

aspy.refactor_imports is fortunately much better at this static analysis.

Why not just use reorder-python-imports? Well, it lacks a few features provided by isort (intentionally).

What this script does is seeds the known_third_party isort configuration automatically.

install

pip install seed-isort-config

usage

seed-isort-config provides a single executable by the same name. Run it inside a git repository.

To specify custom application roots (such as with the src pattern) pass a colon-separated --application-directories parameter.

Files may be excluded from the process using the --exclude flag. This argument takes a python regular expression.

For a full list of arguments, see seed-isort-config --help.

getting started

seed-isort-config looks for an existing known_third_party setting in an isort configuration file. It will modify that if it exists, otherwise it'll create a brand new .isort.cfg file.

The easiest way to get started is to just add a blank known_third_party = section to your isort configuration (or known_third_party = [] if you are using pyproject.toml).

usage with pre-commit

This works especially well when integrated with pre-commit.

-   repo: https://github.com/asottile/seed-isort-config
    rev: v2.2.0
    hooks:
    -   id: seed-isort-config
-   repo: https://github.com/timothycrosley/isort
    rev: 4.3.21  # pick the isort version you'd like to use from https://github.com/timothycrosley/isort/releases
    hooks:
    -   id: isort

In this configuration, seed-isort-config will adjust the known_third_party section of the isort configuration before isort runs!

Note that seed-isort-config doesn't act like a normal pre-commit linter so file exclusion must be configured through args: [--exclude=...] instead. For example: args: [--exclude=tests/.*\.py].

seed-isort-config's Issues

Option to not remove libraries already in `known_third_party`

I'm using pre-commit with a repo that has a bunch of git submodules. I have a .pre-commit-config.yml in the top level repo, and I do ln -s ../.pre-commit-config.yml .pre-commit-config.yml from every submodule. If I configure seed-isort-config like this

-   repo: https://github.com/asottile/seed-isort-config
    rev: v1.9.4
    hooks:
    -   id: seed-isort-config
        args: [--settings-path=/path/to/my/root/repo]

then running pre-commit run --all-files from each submodule will overwrite known_third_party in the top level isort.cfg with values for the repo I'm running it from.

If I do with isort.cfg the same thing I do with .pre-commit-config.yml (as in the actual file in /path/to/my/root/repo/.isort.cfg and then symlinking /path/to/my/roo/repo/sumbodule1/.isort.cfg to that file), the same thing happens, known_third_party is overwritten to only work for the last submodule I ran it from. Running pre-commit from sumbodule1/ will overwrite known_third_party with just the libraries that are used in the submodule.

I could add an isort.cfg file to every single submodule, but because I'm using black, I need to put 5 lines of black-compatible boilerplate config in each file, while it shouldn't change, it still doesn't seem good.

compatible with isort 5.0+

Hi,

For isort 5.0+, there is a new option extra_standard_library. Currently the seed-isort-config only filtered out modules at field starts with known.

Do we have a plan to support isort 5.0.0?

Pre-commit hook passing locally, failing in CI

Possibly related to #7

You should be able to see the failed build here.

Here it is passing in my local environment:

$ git clone [email protected]:rpdelaney/downforeveryone.git
Cloning into 'downforeveryone'...
remote: Enumerating objects: 239, done.
remote: Counting objects: 100% (239/239), done.
remote: Compressing objects: 100% (138/138), done.
remote: Total 239 (delta 131), reused 185 (delta 78), pack-reused 0
Receiving objects: 100% (239/239), 48.37 KiB | 1.21 MiB/s, done.
Resolving deltas: 100% (131/131), done.
$ cd downforeveryone/
direnv: loading .envrc
direnv: export +POETRY_ACTIVE +VIRTUAL_ENV -PS2 ~PATH
$ pre-commit run --all
Fix End of Files.........................................................Passed
Mixed line ending........................................................Passed
Trim Trailing Whitespace.................................................Passed
Check for added large files..............................................Passed
Check that executables have shebangs.....................................Passed
Check for broken symlinks............................(no files to check)Skipped
Check JSON...........................................(no files to check)Skipped
Check for merge conflicts................................................Passed
Detect Private Key.......................................................Passed
Debug Statements (Python)................................................Passed
Check docstring is first.................................................Passed
Check Toml...............................................................Passed
Check Yaml...............................................................Passed
flake8...................................................................Passed
Shell Syntax Check.......................................................Passed
mypy.....................................................................Passed
seed isort known_third_party.............................................Passed
isort....................................................................Passed
black....................................................................Passed
$

Also, if I remove known_third_party = [] from pyproject.toml then the hook creates an .isort.cfg like this:

[settings]
known_third_party = 

I haven't tried but I suspect that would also fail in CI?

Prevent library from being added by hook (google-cloud-bigquery)

Thanks for the package, I'm finding it really useful.

I'm currently using seed-isort-config as a pre-commit hook. It prevents library, google-cloud-bigquery, which defines the google namespace. Because of this, It will modify the [setting]known_third_party from google-cloud-bigquery to google.

Is there any way to keep google-cloud-bigquery in [setting]known_third_party ?
Regards,

Split config across multiple lines to reduce git churn

For larger projects .isort.cfg is a constant source of merge conflicts

Is there a way to populate the known third party list with 1 package per line to reduce git churn & make it easier to see which packages were added/removed?

does it identify absolute imports?

hey,

one quick question if i have code like this

import os
import sys

from rest_framework import status
from faker import Faker

from django.contrib.auth import authenticate
from django.http.response import HttpResponse

from . import utils
from . import constants
from data_controller import constants as dc_constants
from data_aggregator import utils as da_utils

will it consider data_controller and data_aggregator as third-party packages ?? because I cannot use relative imports like ..data_controller import constants as dc_constants in my django project as it is throwing ValueError: attempted relative import beyond top-level package for some reason, also it's not a good practice according to two scoops of django.

print changes on stdout

I run this now part of the tox steps (seed --check, black --check, isort --check). Currently when this command fails leaves you wondering what would actually change, can we print to the standard output the changes to the list (maybe via some verbose mode or by default)?

Prevent library from being added by hook

Thanks for the package, I'm finding it really useful.

I'm currently trying to use this when building a library, django-verified-user, which defines the verified_user namespace.

When I run seed-isort-config from the command line, it detects that the library has been installed using ./setup develop and prevents verified_user from being added to the list of known_third_party.

However, using seed-isort-config as a pre-commit hook runs the tool in a separate virtual environment, meaning it doesn't detect the local installation and adds verified_user to known_third_party.

I'd like some way to prevent seed-isort-config from adding verified_user to known_third_party when it's run as a hook.

One possible solution is similar to #10: detect that verified_user is specified in known_first_party to ensure it's never added to known_third_party.

Another would be to add an exclude-pkg (excludepkg? exclude-dep?) flag to the command, allowing me to simply specify args: [--exclude-pkg=verified_user,not_included_either] in .pre-commit-config.yaml

Happy to put together a PR if one of these sounds more desirable than the other. I'm leaning towards the second option due to implementation simplicity, but defer to you, @asottile.

FileNotFoundError thrown after tweaking directory structure

I am getting the following error suddenly after making a few tweaks to my directory structure:

hookid: seed-isort-config

Traceback (most recent call last):
  File "/Users/jadchaar/.cache/pre-commit/repojmc2zze7/py_env-python3/bin/seed-isort-config", line 10, in <module>
    sys.exit(main())
  File "/Users/jadchaar/.cache/pre-commit/repojmc2zze7/py_env-python3/lib/python3.7/site-packages/seed_isort_config.py", line 101, in main
    third_party = third_party_imports(filenames, appdirs)
  File "/Users/jadchaar/.cache/pre-commit/repojmc2zze7/py_env-python3/lib/python3.7/site-packages/seed_isort_config.py", line 51, in third_party_imports
    with open(filename, 'rb') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'tests/helpers/utils.py'

Here is the source code that is causing the issue: https://github.com/jadchaar/sec-edgar-downloader/tree/ab45fad4a6aaddb9df33d02d5a0e4a03469b4a2b.

Here are the steps I take before getting the error:

1. Move utils.py out of the helpers folder of my testing directory
2. Delete the helpers directory
3. Remove the line sys.path.append(str(Path(__file__).parent.joinpath("helpers"))) from the conftest.py
4. Change the import to from .utils import strip_cik, verify_directory_structure in test_filing_downloads.py

It definitely has something to do with removing sys.path.append(str(Path(__file__).parent.joinpath("helpers"))), but I have no idea how to fix this.

I tried deleting the pre-commit cache, but that did not help :(.

document correct usage of --exclude

in https://github.com/ManageIQ/integration_tests/blob/46a345b67970a04951616bdda1a755211a7b6eef/.pre-commit-config.yaml#L7-L12 i tried to use --exclude just to note that i'm hitting the same error as in #8

its not clear to me how to correctly use the flag

issues with generated pyproject.toml third party list

Hi guys!

The generated pyproject.toml doesn't work for me with v4.3.16 of isort. I think this is because the known_third_party entry in pyproject.toml is expected to be a comma-delimited string, rather than a list.

i.e.

[tool.isort]
known_third_party = "numpy,setuptools"

works

[tool.isort]
known_third_party = ["numpy", "setuptools"]

Doesn't. seed-isort-config creates the latter.

endless add/remove cycle for `known_third_party` under certain conditions

seed-isort-config seems to undergo an endless cycle of adding/removing modules to known_third_party under certain conditions:

1. There is more than one third-party module used in the Python source files visited by seed-isort-config.
2. There is exactly one of those third-party modules listed in the value for the known_third_party key in .isort.cfg.
3. A known_first_party key exists in .isort.cfg with no value.

This appears to be due to a greedy whitespace pattern in KNOWN_OTHER_RE which matches newlines and gloms known_third_party onto the value for known_first_party.

A full explanation and steps to reproduce are in https://github.com/jmknoble/testcase-seed-isort-config .

An apparent quick fix is in https://github.com/jmknoble/seed-isort-config . Caveats: I'm unsure whether this was intended to match patterns like:

known_first_party = a_module,
        another_module,
        yet_another_module

which is valid, but much harder to match with regexes.

I'm guessing the reason seed-isort-config is using regexes is because of the lack of fidelity in reading/writing CFG/INI files in the builtin ConfigParser. Ideally, seed-isort-config should probably avoid the pitfalls of regexes and use something like configupdater or ConfigObj for CFG/INI files and tomlkit for TOML files, but I understand reasons like "should support Python 2.7 while it's still alive" and "part of the reason for TOML is that CFG/INI file formats are all so subtly different".

So I'd rather see the whitespace be less greedy and expect known_* values to be all on one line rather than create other, possibly more subtle and less solvable, problems.

check mode

Can we create a check mode, to fail in CI if this makes changes (when using it outside of pre-commit)?

Not compatible with monorepos

Because seed-isort-config uses git ls-files -- *.py to find the eligible candidates for walking for third-party dependencies, this lumps all found python files into one category and does not allow for scoping by sub-project in monorepos (those repositories containing multiple sub-projects per git repository).

The support offered by the arguments settings-path and application-directory help here, but don't fix the problem entirely as we're still walking the all of the files to pick out their third-party deps. This means that for a monorepo with two sub-projects, the output for seed-isort-config will be the same for both.

For my usecase (and I think this generalizes decently), if I set application-directories I would only like seed-isort-config to walk the python files in those directories. Trimming the list of file paths returned by git ls-file -- *.py down to those that start with the path I have provided would be a great way to determine a set of files for seed-isort-config to walk.

I suppose that there's a use case where application-directories may not be local (if you're implementing a sub-project outside of the main repository - say for a plugin infrastructure or something). If that's a huge concern, I would think that an additional knob to tweak the path to the code to walk would be appropriate.

I am happy to put in the legwork to make this work right and open a pull request for the changes in question. Just let me know how you'd like me to proceed. Thanks for you time!

failure on ignored python file - should respect excludes?

with a pre-commit config of

exclude: ^(data|sprout)/
repos:
- repo: https://github.com/asottile/seed-isort-config
  rev: v1.0.1
  hooks:
  - id: seed-isort-config

i observe

seed isort known_third_party.............................................Failed
hookid: seed-isort-config

Traceback (most recent call last):
  File "/home/rpfannsc/.cache/pre-commit/repomfmabho9/py_env-python3/bin/seed-isort-config", line 11, in <module>
    sys.exit(main())
  File "/home/rpfannsc/.cache/pre-commit/repomfmabho9/py_env-python3/lib/python3.6/site-packages/seed_isort_config.py", line 67, in main
    third_party = ','.join(sorted(third_party_imports(filenames, appdirs)))
  File "/home/rpfannsc/.cache/pre-commit/repomfmabho9/py_env-python3/lib/python3.6/site-packages/seed_isort_config.py", line 45, in third_party_imports
    visitor.visit(ast.parse(f.read(), filename=filename))
  File "/usr/lib64/python3.6/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "data/bundles/merkyl/__init__.py", line 13
    print "allowed.files not found, exiting"
                                           ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("allowed.files not found, exiting")?

unfortunately its not clear how to fix since the tool has to scan the folders

Support pyproject.toml

Since isort now supports reading configuration from pyproject.toml, should support for it be added here?

observing unexpected regular changes for some users

hi, i'm reporting this issue on behalf of a coworker after doing some initial investigation

the git repo in question is https://github.com/ManageIQ/integration_tests

the user even after reinstalling pre-commit faced the following detail issues

diff --git a/.isort.cfg b/.isort.cfg
index f00acd605..be837f5e0 100644
--- a/.isort.cfg
+++ b/.isort.cfg
@@ -1,4 +1,4 @@
 [settings]
 line_length=100
 # NOTE: never change known_third_party by hand its managed by a hook
-known_third_party = Crypto,IPython,OpenSSL,PIL,_pytest,attr,bottle,bugzilla,cStringIO,cached_property,click,coverage,cycler,dateparser,dateutil,debtcollector,deepdiff,diaper,docker,dockerbot,dump2polarion,fauxfactory,freeze,gevent,git,github,glanceclient,humanfriendly,importscan,iso8601,jenkins,jinja2,jira,jsmin,keystoneauth1,kwargify,lxml,lya,manageiq_client,matplotlib,miq_version,navmazing,novaclient,numpy,paramiko,pathlib2,pdfminer,pika,pkg_resources,py,pygal,pytesseract,pytest,pytz,requests,riggerlib,rope,scp,selenium,sentaku,setuptools,six,slumber,sphinx,sphinx_rtd_theme,sqlalchemy,tabulate,tzlocal,urllib2,wait_for,werkzeug,widgetastic,widgetastic_patternfly,wrapanapi,xmltodict,yaml,yaycl,yaycl_crypt,zmq
+known_third_party = Crypto,IPython,OpenSSL,PIL,_pytest,attr,bottle,bugzilla,cached_property,click,concurrent,coverage,cycler,dateparser,dateutil,debtcollector,deepdiff,diaper,docker,dockerbot,dump2polarion,fauxfactory,freeze,gevent,git,github,glanceclient,humanfriendly,importscan,iso8601,jenkins,jinja2,jira,jsmin,keystoneauth1,kwargify,lxml,lya,manageiq_client,matplotlib,miq_version,navmazing,novaclient,numpy,paramiko,pathlib,pathlib2,pdfminer,pika,pkg_resources,py,pygal,pytesseract,pytest,pytz,requests,riggerlib,rope,scp,selenium,sentaku,setuptools,six,slumber,sphinx,sphinx_rtd_theme,sqlalchemy,tabulate,tzlocal,wait_for,werkzeug,widgetastic,widgetastic_patternfly,wrapanapi,xmltodict,yaml,yaycl,yaycl_crypt,zmq

the particular exchange in packages leads me to believe that pre-commit isn't quite sure which python version to use - and the seed tool adds certain packages based solely on whether its invoked on python3 or python2

Blanks out `known_third_party` when run from `emacs` `magit`

Disclaimer: I am not an emacs user, before today I knew how to quit emacs and that's about it -- if the things I am running are non-conventional cut me some slack :)

First I started with a dockerfile so I could reproduce this:

FROM ubuntu:bionic

ENV \
    DEBIAN_FRONTEND=noninteractive \
    LANG=C.UTF-8 \
    PIP_DISABLE_PIP_VERSION_CHECK=1 \
    PIP_NO_CACHE_DIR=off \
    VIRTUALENV_NO_DOWNLOAD=1

RUN : \
    && apt-get update \
    && apt-get install -y --no-install-recommends \
        emacs-nox \
        git \
        nano \
        python3 \
        virtualenv \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /tox
RUN : \
    && git clone https://github.com/tox-dev/tox --no-checkout . \
    && git -C /tox checkout 23a53529aa27c0010a891ccddb400b10b24bd4fb

ENV PATH=/venv/bin:$PATH
RUN : \
    && virtualenv /venv -ppython3.6 \
    && pip install pre-commit \
    && pre-commit install --install-hooks

RUN echo '\
(require '"'"'package)\
(add-to-list '"'"'package-archives\
             '"'"'("melpa" . "http://melpa.org/packages/") t)\
' > /root/.emacs

RUN : \
    && git config --global user.name 'Anthony Sottile' \
    && git config --global user.email '[email protected]'

Built and ran this with:

docker build -t magit .
docker run --rm -ti magit bash

Next I started emacs setup.py and installed magit by doing the following (following these instructions):

M-x  package-initialize
M-x package-refresh-contents
M-x package-install
magit

Edit the file in some way, I added HELLO WORLD to a docstring for instance

Then

M-x magit-status
s setup.py
c c

View the failed pre-commit by pressing $:

black....................................................................^[[42m$
blacken-docs.........................................(no files to check)^[[46;3$
seed isort known_third_party.............................................^[[41m$
hookid: seed-isort-config                                                       
                                                                                
Files were modified by this hook.                                               
                                                                                
isort....................................................................^[[42m$
Trim Trailing Whitespace.................................................^[[42m$
Fix End of Files.........................................................^[[42m$
Check Yaml...........................................(no files to check)^[[46;3$
Debug Statements (Python)................................................^[[42m$
Flake8...................................................................^[[42m$
pyupgrade................................................................^[[42m$

Then looking at the contents of the modified file:

diff --git a/tox.ini b/tox.ini
index 46a892b..081abbc 100644
--- a/tox.ini
+++ b/tox.ini
@@ -116,4 +116,4 @@ include_trailing_comma = True
 force_grid_wrap = 0
 line_length = 99
 known_first_party = tox
-known_third_party = pkg_resources,pluggy,py,pytest,setuptools,six
+known_third_party = 

Different results when run from pre-commit or cli

I am running into an issue where the pre-commit hook is generating a different result when it is run by a pre-commit hook or when it is run from the cli after installing the package.

When it is run from the CLI it is generating:

[settings]
known_third_party=cx_Freeze,py2exe,setuptools,versioneer,werkzeug

and the output from the pre-commit is from:

[settings]
known_third_party=cx_Freeze,py2exe,setuptools,werkzeug

I am unsure where to start with debugging this issue, what additional details or steps do I need to take to provide you with the results that you need?

This causes some issues with isort and it is breaking up third party library versioneer from setuptools in my setup.py file.

Skip files that fail to parse their AST

Currently this hook fails if it attempts to parse a python file that has invalid syntax. This means that a side-effect of this hook is requiring all files in the repository to be valid python. This seems like a large limitation of the hook - imports don't need to be sorted in files that don't parse, and we don't necessarily want code being checked as a side effect.

Would it be possible to just ignore files that fail to parse?

Incorrectly classifying `dataclasses` as a BUILTIN when it's being installed as a third party

Here's a tricky one. I'm trying to support Python 3.6, so I've included dataclasses in my setup.py as a dependency. So on my dev machine which is Python 3.6, I run seed-isort-config, and it correctly adds dataclasses as a third party lib. However on the CI server, under Python 3.7+ dataclasses is a built-in library, so precommit fails when seed-isort-config tries to move dataclasses to the built-in section. The correct behaviour here, I believe, is to count it as a third party because it's in the setup.py and also installed in the virtualenv under site_packages.

Although my setup is unusual, I think this will happen anytime you try to use a backport library that's already supported on certain Python versions as a built-in.

Ignore entries from other `known_` sections?

For a Django project, I like to have all the django imports in its own section. I do this using isort's custom sections:

known_django=django

As it is, seed-isort-config ignores this setting, adding django to known_third_party, which takes precedence in isort.

I think it would make sense to ignore modules from any other known_ section when assembling the known_third_party list.

module-per-line format

Isort supports a one known module per line format:

known_third_party =
    debug_toolbar
    django

seed-isort-config fails to parse this, and also it would be handy if it output like this, to avoid merge conflicts.

Different values in known_third_party if there's a folder/file with the same name

Hi, I think this issue is similar to this - #7

Basically the problem happens if we have a package in Python, e.g. flask_session which automatically also creates flask_session directory in the root of the git repository (used as a cache folder, note that this directory is in .gitignore). Than seed-isort-config cli produces different result than pre-commit run -a.

This is a problem, which I believe should be solved somehow. We also use alembic which creates alembic folder for migrations by default. If we run pre-commit run -a it will not add alembic to known_third_party because it uses the same nae as the Python package itself. Se our only option is to rename alembic folder to migrations or something like that.

This conflict in package name vs directory/filename is not good at all. And it also took some time to figure out why this is happening, so I believe it should be resolved somehow.

I'm using the latest version of seed-isort-config (v2.2.0), python 3.8 (I believe this is issue in all versions).

Support [tool:isort] section name in setup.cfg

It would be great if we could seed into setup.cfg instead of .isort.cfg if the latter doesn't exist.

Python 3.8 support

Hi! Thanks for this great tool! Is there any way to use this with Python 3.8? The pre commit hook (v2.2.0) currently fails on code containing the walrus operator. If there's anything I can do to help with it, I'd be happy to.

Walrus support

Any chance you will add support for python 3.8 / walrus operator?

QST: multiple application-directories

Hi Anthony,

I'm trying to use this tool to populate .isort.cfg for a repository in which I have multiple projects. I've made a small repo which reproduces the issue here.

The structure of the repo is as follows:

$ tree -I '.git|__pycache__' -a
.
├── .gitignore
├── .isort.cfg
├── .pre-commit-config.yaml
├── application_1
│   ├── __main__.py
│   └── utils.py
└── my_subfolder
    └── application_2
        ├── __main__.py
        └── utils.py

3 directories, 7 files

Inside application_1/utils.py, I import pandas. Inside my_subfolder/application_2/utils.py, I import numpy.

Here is my .pre-commit-config.yaml file:

repos:
-   repo: https://github.com/asottile/seed-isort-config
    rev: v2.2.0
    hooks:
    -   id: seed-isort-config
        args: ['--application-directories=my_subfolder,.']
-   repo: https://github.com/timothycrosley/isort
    rev: 4.3.21
    hooks:
    -   id: isort

If I run pre-commit run --all, then .isort.cfg is modified as follows:

[settings]
known_third_party =application_1,pandas

I was expecting to see

[settings]
known_third_party =numpy,pandas

Have I misunderstood how to use seed-isort-config, or am I using it wrong?