Unrelated testcases fail when total number of tests is increased beyond 367

[sumezulike]

Unrelated testcases fail when total number of tests is increased beyond 367

[asottile-sentry]

because I was bored I decided to throw detect-test-pollution at this:

$ # get the tests from that xdist worker -- exercise left to the reader but I put them in `testlist`
$ detect-test-pollution --testids testlist --failing $(tail -1 testlist)
discovering all tests...
-> pre-discovered 28 tests!
ensuring test passes by itself...
-> OK!
ensuring test fails with test group...
-> OK!
running step 1:
- 27 tests remaining (about 5 steps)
running step 2:
- 14 tests remaining (about 4 steps)
running step 3:
- 7 tests remaining (about 3 steps)
running step 4:
- 4 tests remaining (about 2 steps)
running step 5:
- 2 tests remaining (about 1 steps)
double checking we found it...
-> the polluting test is: tests/test_black.py::TestFileCollection::test_get_sources_with_stdin_symlink_outside_root

and sure enough:

$ pytest tests/test_black.py::TestFileCollection::test_get_sources_with_stdin_symlink_outside_root $(tail -1 testlist)
============================= test session starts ==============================
platform darwin -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
rootdir: /private/tmp/black
configfile: pyproject.toml
plugins: cov-4.1.0, xdist-3.5.0
collected 2 items                                                              

tests/test_black.py .F                                                   [100%]

=================================== FAILURES ===================================
__ TestFileCollection.test_get_sources_with_stdin_filename_and_extend_exclude __

self = <tests.test_black.TestFileCollection object at 0x102f98fb0>

    def test_get_sources_with_stdin_filename_and_extend_exclude(self) -> None:
        # Extend exclude shouldn't exclude stdin_filename since it is mimicking the
        # file being passed directly. This is the same as
        # test_exclude_for_issue_1572
        src = ["-"]
        path = THIS_DIR / "data" / "include_exclude_tests"
        stdin_filename = str(path / "b/exclude/a.py")
        expected = [f"__BLACK_STDIN_FILENAME__{stdin_filename}"]
>       assert_collected_sources(
            src,
            root=THIS_DIR.resolve(),
            expected=expected,
            extend_exclude=r"/exclude/|a\.py",
            stdin_filename=stdin_filename,
        )

tests/test_black.py:2742: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

src = ['-']
expected = ['__BLACK_STDIN_FILENAME__/private/tmp/black/tests/data/include_exclude_tests/b/exclude/a.py']

    def assert_collected_sources(
        src: Sequence[Union[str, Path]],
        expected: Sequence[Union[str, Path]],
        *,
        root: Optional[Path] = None,
        exclude: Optional[str] = None,
        include: Optional[str] = None,
        extend_exclude: Optional[str] = None,
        force_exclude: Optional[str] = None,
        stdin_filename: Optional[str] = None,
    ) -> None:
        gs_src = tuple(str(Path(s)) for s in src)
        gs_expected = [Path(s) for s in expected]
        gs_exclude = None if exclude is None else compile_pattern(exclude)
        gs_include = DEFAULT_INCLUDE if include is None else compile_pattern(include)
        gs_extend_exclude = (
            None if extend_exclude is None else compile_pattern(extend_exclude)
        )
        gs_force_exclude = None if force_exclude is None else compile_pattern(force_exclude)
        collected = black.get_sources(
            root=root or THIS_DIR,
            src=gs_src,
            quiet=False,
            verbose=False,
            include=gs_include,
            exclude=gs_exclude,
            extend_exclude=gs_extend_exclude,
            force_exclude=gs_force_exclude,
            report=black.Report(),
            stdin_filename=stdin_filename,
        )
>       assert sorted(collected) == sorted(gs_expected)
E       AssertionError: assert [] == [PosixPath('_...xclude/a.py')]
E         
E         Right contains one more item: PosixPath('__BLACK_STDIN_FILENAME__/private/tmp/black/tests/data/include_exclude_tests/b/exclude/a.py')
E         Use -v to get more diff

tests/test_black.py:2341: AssertionError
=========================== short test summary info ============================
FAILED tests/test_black.py::TestFileCollection::test_get_sources_with_stdin_filename_and_extend_exclude - AssertionError: assert [] == [PosixPath('_...xclude/a.py')]
========================= 1 failed, 1 passed in 0.10s ==========================

smells to me like a rogue lru_cache -- let's try commenting out the one most related to Path.resolve:

diff --git a/src/black/files.py b/src/black/files.py
index c0cadbf..a92d011 100644
--- a/src/black/files.py
+++ b/src/black/files.py
@@ -48,7 +48,7 @@ def _load_toml(path: Union[Path, str]) -> Dict[str, Any]:
         return tomllib.load(f)
 
 
-@lru_cache
+#@lru_cache
 def _cached_resolve(path: Path) -> Path:
     return path.resolve()
  

bingo!

$ pytest tests/test_black.py::TestFileCollection::test_get_sources_with_stdin_symlink_outside_root $(tail -1 testlist)
============================= test session starts ==============================
platform darwin -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
rootdir: /private/tmp/black
configfile: pyproject.toml
plugins: cov-4.1.0, xdist-3.5.0
collected 2 items                                                              

tests/test_black.py ..                                                   [100%]

============================== 2 passed in 0.10s ===============================

classic test pollution :)

[asottile-sentry]

introduced in 23dfc5b cc @hauntsaninja

[hauntsaninja]

Oh amazing, thank you asottile for figuring it out and sumezulike for the issue report! #4287 should fix