During COVID I was bored and wondered why nobody (I was ignorant at that time) used the MFT to read the NTFS in one swoop and not do this item by item sys-call flow.
Then I started looking around and found a few tools actually which did this. One of them was a CPP implemented “Swiftsearch”(https://sourceforge.net/projects/swiftsearch/) tool. I liked the tool, but it was clumsy and not very user friendly. It did have a CLI, but that was a bare minimum. So I sat out to fork it and add the missing pieces, which I did (https://github.com/githubrobbi/Ultra-Fast-File-Search-legacy...) … and I learned a lot.
Then I stumbled upon RUST doing another project (derivative trading engine) and with this I thought, I can rebuild a RUST based NTFS reader. (The CPP code was a convoluted monolithic code base and to re-write the CPP code was the same as to start from scratch and doing that I had the “freedom of choice” to use a language I thought might even be better suited.) I chose RUST.
Everything and a few other tools do it, but their CLI / scripting support is thin and that is where I thought I can contribute.
So my toolset UFFS reads the MFT and builds a local index, which can serve all metadata for every file of the underlying drive. It is Rust, MPL-2.0, local-only, and has no accounts or telemetry.
For freshness, UFFS follows NTFS changes through the USN Journal rather than requiring a full rescan for every change.
I created a benchmark harness to see how well my RUST based approach would do against Everything. So far I saw the UFFS was ~2.8 times faster than Everything ?! I am sure I missed a thing or two and that I might have a fundamental flaw in the way I measured. Anyhow, maybe there is a specialist out there, who can shed some light on what I could do better for a truly meaningful comparison.
I was surprised at how fast RUST is able to push data out to the terminal / file etc. In my measures UFFS streamed 23.3M rows across seven volumes to CSV in 12.0s. The es.exe path I tested hit a limit there. This is a bulk-export workload I think can be useful for automation / scripting.
This is not a “fastest Windows search” claim. Everything is excellent, and for many interactive desktop lookups it is still the tool I would recommend. UFFS is aimed more at large multi-drive machines, scripted searches, full-result exports, and treating file metadata as a queryable dataset.
Here is what I am looking for help with: feedback, for things I missed to look at, and even scenarios, where Everything should be the clear winner.
- This is a WINDOWS-only tool … and yes, you could analyze digest an OFFLINE MFT on linux or MAC too, but the real value is for WINDOWS users.
- MFT access needs ADMIN rights each time you use it … or use the UFFS-BROKER which will just need a one-time elevation and stays running as a service.
- The binaries on GitHub are NOT signed yet they are built from the source code by GitHub Actions and carry a SHA256 with them … just not signed yet. Here is how to verify where the binaries came from: “gh attestation verify <file> --owner skyllc-ai”
During COVID I was bored and wondered why nobody (I was ignorant at that time) used the MFT to read the NTFS in one swoop and not do this item by item sys-call flow.
Then I started looking around and found a few tools actually which did this. One of them was a CPP implemented “Swiftsearch”(https://sourceforge.net/projects/swiftsearch/) tool. I liked the tool, but it was clumsy and not very user friendly. It did have a CLI, but that was a bare minimum. So I sat out to fork it and add the missing pieces, which I did (https://github.com/githubrobbi/Ultra-Fast-File-Search-legacy...) … and I learned a lot.
Then I stumbled upon RUST doing another project (derivative trading engine) and with this I thought, I can rebuild a RUST based NTFS reader. (The CPP code was a convoluted monolithic code base and to re-write the CPP code was the same as to start from scratch and doing that I had the “freedom of choice” to use a language I thought might even be better suited.) I chose RUST.
And this is what I build:
On windows systems, the traditional file system tools are VERY slow, because of the way they process the file system, the tree traversal. (look here for what I am talking about: https://dev.to/robertnio/dont-walk-the-tree-how-ntfs-file-se...)
Everything and a few other tools do it, but their CLI / scripting support is thin and that is where I thought I can contribute.
So my toolset UFFS reads the MFT and builds a local index, which can serve all metadata for every file of the underlying drive. It is Rust, MPL-2.0, local-only, and has no accounts or telemetry.
For freshness, UFFS follows NTFS changes through the USN Journal rather than requiring a full rescan for every change.
I created a benchmark harness to see how well my RUST based approach would do against Everything. So far I saw the UFFS was ~2.8 times faster than Everything ?! I am sure I missed a thing or two and that I might have a fundamental flaw in the way I measured. Anyhow, maybe there is a specialist out there, who can shed some light on what I could do better for a truly meaningful comparison.
I was surprised at how fast RUST is able to push data out to the terminal / file etc. In my measures UFFS streamed 23.3M rows across seven volumes to CSV in 12.0s. The es.exe path I tested hit a limit there. This is a bulk-export workload I think can be useful for automation / scripting.
This is not a “fastest Windows search” claim. Everything is excellent, and for many interactive desktop lookups it is still the tool I would recommend. UFFS is aimed more at large multi-drive machines, scripted searches, full-result exports, and treating file metadata as a queryable dataset.
Here is what I am looking for help with: feedback, for things I missed to look at, and even scenarios, where Everything should be the clear winner.
Repo: https://github.com/skyllc-ai/UltraFastFileSearch Benchmark writeup: https://uffs.io/blog/benchmarking-against-everything/ Demos: https://www.youtube.com/playlist?list=PLTTMn_SUmnqqTVXWHYPSy...
Just to let you know:
- This is a WINDOWS-only tool … and yes, you could analyze digest an OFFLINE MFT on linux or MAC too, but the real value is for WINDOWS users. - MFT access needs ADMIN rights each time you use it … or use the UFFS-BROKER which will just need a one-time elevation and stays running as a service. - The binaries on GitHub are NOT signed yet they are built from the source code by GitHub Actions and carry a SHA256 with them … just not signed yet. Here is how to verify where the binaries came from: “gh attestation verify <file> --owner skyllc-ai”
[dead]