chunked file finder

[staabm]

Feature Request

when changing rules in big codebases, I can see in blackfire profiles, that a huge portion of the time rector processes is spent in the file finder (not the actual code modifications).

I wonder whether we can/could chunk the file finding and e.g. start worker processes for the e.g. the first chunk of files, while we are still analyzing the file-system to find more files..?

see

e.g. we could try to turn the FileCacheStorage->load() method into a generator and return several times in chunks (or file by file?)

[TomasVotruba]

Thanks for investigation 👍

Without much thinking, I think PHPStan resource loader needs all files before hand, and would be a blocker for this.

How does PHPStan handle file loading?

What is the time spent on these ~700 files?

If you find any way to improve this part, I'm all ears

[staabm]

What is the time spent on these ~700 files?

on my macbook m4 pro it takes ~1minute (87% of the overall time).

I will play a bit with the idea

[TomasVotruba]

That's really huge number.
We have a project with 5 000 files and Rector stars working under 5 seconds. I'm on Ubuntu