Repository navigation
Fix quadratic collection when passing many file arguments from one directory - #15230
Open
suryakant-source wants to merge 7 commits into
Open
suryakant-source wants to merge 7 commits into
suryakant-source wants to merge 7 commits into
Conversation
Session.collect() collected the parent directory of a file argument with handle_dupes=False so that directly repeated files are not deduplicated. That re-collected the whole parent directory once per file argument: O(file args x directory entries). In the default mode, normalize_collection_arguments() has already removed duplicate and overlapping paths, so the cached collection can always be reused. Only --keep-duplicates still needs the fresh re-collection. Fixes pytest-dev#15187 Co-authored-by: Instinct AI Assistant <ai-assistant@users.noreply.github.com>
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #15187
Problem
When several test files from the same directory are passed as explicit command-line arguments (e.g.
pytest a/test_1.py a/test_2.py ...),Session.collect()re-collects the parent directory once per file argument. Cost grows with (number of file arguments) x (number of entries in the directory), so a CI job passing ~100 explicit files from a ~940-entry directory spends its collection time re-scanning the same directory ~100 times. Reported in #15187 and again in #15221.Root cause
In
Session.collect(), the parent of the last path part of a file argument is collected withhandle_dupes=Falseso that files given directly multiple times on the command line are not deduplicated. That forces a full re-collection of the parent directory for every file argument, even though only the one requested file is needed.Fix
In the default mode,
normalize_collection_arguments()has already removed duplicate and overlapping arguments, so the cached collection of the parent directory can always be reused. The fresh re-collection is only required with--keep-duplicates, sohandle_dupes=Falseis now gated on that option.Measured with 200 test files in one directory and a counting
pytest_ignore_collecthook:Verification
test_multiple_file_args_do_not_rescan_parent_directoryfails without the fix (~400pytest_collect_filehook calls for 20 files) and passes with it.--keep-duplicateswith overlapping file+dir args still runs the file twice;pytest a.py a.pystill runs the file once;pytest a.py::test_x a.py::test_ystill selects exactly the requested items.testing/test_collection.py,test_session.py,test_pathlib.py,test_conftest.py,test_config.pyall pass locally (621 passed, plus the new test).Checklist:
closes #15187above.Co-authored-bycommit trailers, per the AI/LLM-Assisted Contributions Policy.changelog/15187.bugfix.rst.AUTHORS.