Problem
SKiM identifies which microbes are in a DNA sample from an Oxford Nanopore sequencer, and was published in Bioinformatics (2025). Its reference database didn’t fit in the memory of the small devices it was built for, like NVIDIA Jetson boards.
What I built
I built the caching layer. Every step was benchmarked before the next one:
- Moved the database to SSD behind mmap, so it could run at all.
- Switched to Direct I/O with a dedicated I/O thread pool, so rayon workers stop blocking on disk.
- Deduplicated keys, so the same key is never fetched twice at once.
- Replaced the hash map with minimal perfect hashing, removing per-key overhead across 63M keys.
- Swapped in a non-blocking cache library. The old one’s bounded channel left hundreds of threads waiting.
I also added a tunable cache page size for different storage devices, and a watch mode that classifies files the moment the sequencer writes them.
Stack
Rust · Tokio · rayon · Direct I/O · Minimal perfect hashing