Stackness
All moves

Loop coding agents against benchmarks to iteratively speed up code

by Stackness

Languages & Frameworks

Instead of asking an agent to write fast code once, developers set up a benchmark harness and let the agent iterate repeatedly, measuring, profiling and rewriting until measurable speedups appear. Guardrails or constraints keep the agent from changing semantics while it explores optimizations like kernel rewrites or algorithmic changes. Reports show 2x to 20x gains on things like CUDA kernels and Rust algorithms when the loop is properly instrumented.

What it is

This move treats speed as something an agent discovers through repeated measurement, not something it gets right on the first try. Instead of a single prompt asking for optimized code, you build a benchmark harness that can run automatically, then let a coding agent propose a change, compile it, measure it, and compare it against the previous best, looping through many attempts.

How to do it

  • Set up a harness that runs a reference implementation and any candidate implementation under the same inputs, so outputs can be diffed for correctness before speed is even considered.
  • Wire in a benchmarking tool that can track timing across iterations and flag regressions automatically, such as Rust's criterion crate, which the agent can invoke on its own.
  • For GPU or systems code, give the agent access to lower-level signals like GPU properties, profiler counters, or documentation lookups so it can reason about why a kernel is slow, not just that it is slow.
  • Define guardrails up front: forbid unsafe constructs where relevant, require every candidate to pass a correctness check against the reference before it is considered, and cap the number of repair attempts per iteration.
  • After each run, feed the timing and correctness results back into the next iteration, keep the fastest validated candidate, and repeat.
  • Log every experiment, including a record of what changed and its measured result, so you can audit the search afterward.

When it helps

This works well when you have a well-defined, measurable performance target such as kernel latency or algorithm runtime, and a reference implementation to check correctness against. It has been used to rewrite CUDA kernels and to reimplement algorithms like dimensionality reduction in Rust, in both cases producing implementations reported as 2x to 20x faster than the starting point after many iterations.

Pitfalls

Without a strict correctness check on every candidate, the agent can drift into changes that are faster but semantically different from the original. Loops also need a repair budget or iteration cap, since not every proposed change will compile or run correctly, and unconstrained exploration can burn time on invalid candidates. Guardrails matter: leaving optimization goals underspecified can let the agent chase speed by cutting corners rather than genuinely improving the implementation.

Sources

Tools

Rust