Thing that burned me was forgetting about page cache and THP, two runs of the same binary ten minutes apart differed by 15% until I dropped caches and set khugepaged off. Also worth reporting variance across reboots, not just across runs.
Biggest thing for me was pinning CPU governor to performance and disabling turbo entirely. Slower numbers but they stop drifting as the box heats up. Also isolcpus for the load generator, otherwise you're benchmarking your own noise.
Thing that burned me was forgetting about page cache and THP, two runs of the same binary ten minutes apart differed by 15% until I dropped caches and set khugepaged off. Also worth reporting variance across reboots, not just across runs.
What do you mean about reboots? Never heard of rebooting tbh
Biggest thing for me was pinning CPU governor to performance and disabling turbo entirely. Slower numbers but they stop drifting as the box heats up. Also isolcpus for the load generator, otherwise you're benchmarking your own noise.
Nice one! Yeah, it might be a bit slower, but for benchmarking reproducibility would be more important
Turbo only possible to have enabled while benchmarking if you have hardware good enough to cool down the server