Avoid scanning the filesystem to filter files per check#1262
Open
Wigny wants to merge 2 commits intorrrene:masterfrom
Open
Avoid scanning the filesystem to filter files per check#1262Wigny wants to merge 2 commits intorrrene:masterfrom
Wigny wants to merge 2 commits intorrrene:masterfrom
Conversation
Owner
|
First, thx for the PR! I don't have access to my dev machine at the moment, but I am reasonably sure that this change will cause problems when piping files via STDIN (via I will have to verify if these scenarios are sufficiently covered by tests right now (and improve the test suite if not) 👍 |
Wigny
force-pushed
the
fix/avoid-filesystem-scan-in-check-runner
branch
from
de901fe to
326a676
Compare
Contributor
Author
When a check defines custom `files` patterns (e.g. `files: %{excluded: ...}`),
the runner used `Sources.find_in_dir` with `Path.wildcard` to list matching
file paths from the filesystem. This was slow in some cases as the glob
patterns could cause a recursive scan of the entire project tree.
Now the runner filters the already known file paths in memory using
`Sources.filename_matches?` instead of listing them from the filesystem again.
Wigny
force-pushed
the
fix/avoid-filesystem-scan-in-check-runner
branch
from
326a676 to
45cc6c4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We started noticing
credobecame much slow latelly when trying to execute checks that setfiles.includedglob patterns like the Credo.Check.Refactor.PassAsyncInTestCases.This slowdown seems to be caused, according to my test, by
Credo.Sources.find_in_dir/3, which usesPath.wildcardto list matching file paths from the filesystem, which might be running a recursive scan of the entire project tree.This PR changes the code to filter the already known file paths in memory using
Credo.Sources.filename_matches?instead of listing them from the filesystem.Tested it in our project, and the slowdown was reduced from several minutes to seconds.