Skip to content

Fix quadratic collection when passing many file arguments from one directory - #15230

Open
suryakant-source wants to merge 7 commits into
pytest-dev:mainfrom
suryakant-source:patch-1
Open

suryakant-source wants to merge 7 commits into
pytest-dev:mainfrom
suryakant-source:patch-1

Conversation

@suryakant-source

Copy link
Copy Markdown

closes #15187

Problem

When several test files from the same directory are passed as explicit command-line arguments (e.g. pytest a/test_1.py a/test_2.py ...), Session.collect() re-collects the parent directory once per file argument. Cost grows with (number of file arguments) x (number of entries in the directory), so a CI job passing ~100 explicit files from a ~940-entry directory spends its collection time re-scanning the same directory ~100 times. Reported in #15187 and again in #15221.

Root cause

In Session.collect(), the parent of the last path part of a file argument is collected with handle_dupes=False so that files given directly multiple times on the command line are not deduplicated. That forces a full re-collection of the parent directory for every file argument, even though only the one requested file is needed.

Fix

In the default mode, normalize_collection_arguments() has already removed duplicate and overlapping arguments, so the cached collection of the parent directory can always be reused. The fresh re-collection is only required with --keep-duplicates, so handle_dupes=False is now gated on that option.

Measured with 200 test files in one directory and a counting pytest_ignore_collect hook:

file args directory-entry scans before scans after
10 1902 193
20 3622 183
40 6442 163

Verification

  • New regression test test_multiple_file_args_do_not_rescan_parent_directory fails without the fix (~400 pytest_collect_file hook calls for 20 files) and passes with it.
  • Duplicate semantics unchanged, re-verified after the patch: --keep-duplicates with overlapping file+dir args still runs the file twice; pytest a.py a.py still runs the file once; pytest a.py::test_x a.py::test_y still selects exactly the requested items.
  • testing/test_collection.py, test_session.py, test_pathlib.py, test_conftest.py, test_config.py all pass locally (621 passed, plus the new test).

Checklist:

  • Include new tests or update existing tests when applicable.
  • Allow maintainers to push and squash when merging my commits.
  • Added closes #15187 above.
  • AI agents were used; they are credited in Co-authored-by commit trailers, per the AI/LLM-Assisted Contributions Policy.
  • New changelog file changelog/15187.bugfix.rst.
  • Added myself to AUTHORS.

suryakant-source and others added 4 commits October 21, 2026 00:09
Session.collect() collected the parent directory of a file argument with handle_dupes=False so that directly repeated files are not deduplicated. That re-collected the whole parent directory once per file argument: O(file args x directory entries). In the default mode, normalize_collection_arguments() has already removed duplicate and overlapping paths, so the cached collection can always be reused. Only --keep-duplicates still needs the fresh re-collection.

Fixes pytest-dev#15187

Co-authored-by: Instinct AI Assistant <ai-assistant@users.noreply.github.com>
)

Co-authored-by: Instinct AI Assistant <ai-assistant@users.noreply.github.com>
@pyba-chronographer pyba-chronographer Bot added the bot:chronographer:provided (automation) changelog entry is part of PR label Oct 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:chronographer:provided (automation) changelog entry is part of PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Collection is quadratic in the number of file arguments from one directory: each file argument re-collects its parent directory

1 participant