What is the best way to walk a file tree while skipping certain directories?

Long story short, I am building a search tool that searches for text inside of files, and returns all files that it matches lines in. Not returning line numbers for now, maybe later.

• The search tool recursively searches a single directory.
• There are many levels of directories in the one folder.
• Total number of files across all levels is 100-200 thousand.
• Each file ranges from 1-50 kilobytes.

My first attempt was to use the Files.walk(Path) method, and it worked out very nicely! It was especially fun to turn on .parallel(), and get a free performance boost for so little effort lol. However, once I tested it on the larger dataset, the performance wasn't quite where I wanted it to be, even with parallel turned on.

But then I actually looked at the work I was doing, and I saw that I was searching several folders that I absolutely do not need to search. For example, the .git folder, amongst several others.

My next attempt was to simply add a .filter() that simply looped through the Path (who knew that Path was iterable?!), and checked for a .git folder, and the performance of that was ok, still not ideal. But that made sense -- it still is marching through the .git folder, and just not searching any files in it. What I really want is to just not enter the .git folder at all. Skip over that folder entirely as soon as I find it.

I then saw that there is another method on Files -- the Files.walkFileTree(Path, FileVisitor). Unlike Files.walk, this one short-circuits, which was exactly what I want! However, this strategy completely abandons the streaming strategy, instead going for a Visitor Pattern instead. That's not bad, but now I have all sorts of concerns, like how to search in parallel, watching out for race conditions, etc. Each surmountable, but it feels like I am solving the problem at the wrong level, if that makes sense.

Putting together a solution using Files.walkFileTree doesn't sound too hard at all, especially thanks to the SimpleFileVisitor class. But I am not clear on how to apply parallelism to it, as parallelism was such a good idea for the stream solution.

But I also want to know if I am missing something, or if there is a better way than what I found. Obviously, I could use a library for this, but I am treating this like a design exercise, to see how best to tackle this.

And to be clear, I originally started off by using grep, but it was horrifically slow. As in, 30+ seconds to find a non-regex search. Here is the command I was using.

grep -r "import java" ./

And to skip the .git folder, I also tried the following. Still very slow.

for codebase in $(ls)
do
    git grep -r "import java" ./$codebase
done

Now, this only skips the .git folder. There are other folders I want to skip as well, but I don't see an easy way to do so. Some are a few levels deep, others are more. They all have the same path pattern though, so it is easy enough to do it in code, but I don't know how to do it here in bash.

As an aside, I also asked the Unix SE for help on improving the grep command, in case I am just using it wrong -- How can I improve the performance of my grep search?

Anyways, I am just looking for ideas on the overall design here.

• Files.walk was fast, but it did so much unnecessary work, and I am certain that that was a contributing factor for it being slower than I'd like.
  • I don't see a way to short-circuit Files.walk -- I am pretty sure I am forced to step all the way through the .git folder, and just reject every member of it. Am I wrong?
• Files.walkFileTree is flexible enough to short-circuit, but it is not clear to me how to make it parallel.
  • I can certainly just take my FileVisitor and run that independently from the Files class, but now I am wondering if I am just missing something fundamental here.

Anyways, ideas are welcome. Example code is appreciated, but not necessary.