Finding a needle in a 4 GB haystack: from 0.75 GB/s to 49 GB/s in Go

Fri, 29 May 2026 00:40:00 +0200

I had a 4 GiB file that’s almost entirely zeros, exactly one non-zero int64 is hiding at offset Size - 8 (the last aligned slot). The task: find that offset, as fast as possible, in Go on Linux.

It’s a deliberately silly problem. There’s no parsing, no indexing, no cleverness on the algorithm side. The only thing it measures is how much data we can pull through a CPU per second. Exactly the kind of micro-task that exposes every layer of the stack: the Go runtime, the standard library, the kernel, the page cache, the memory hierarchy, and SIMD, including Go 1.26’s brand-new simd/archsimd package that lets you write AVX-512 in pure Go.

Starting from the most obvious os.ReadFile + for range we get 0.75 GB/s. Thirteen variants later we’re at 49 GB/s, a 66× speedup, and we’ll know exactly which wall we hit and why.

Mmap on Segflow

Finding a needle in a 4 GB haystack: from 0.75 GB/s to 49 GB/s in Go