Case study
Ferrox
An LMAX-inspired order matching engine in Rust, built one phase at a time with a benchmark proving each optimization, then ported in part to C++20 to measure what memory ordering really costs.
Results
- simulated orders/sec
- 4.7M63% filled / 37% resting, on a Ryzen 7 7735HS laptop.
- worst P99 per order
- 1.06µsP50 130–210 ns, P99 832–1,062 ns, P99.9 1.4–1.7 µs over 990K orders, TSC clock. The target was 50 µs.
- worst-case cancel latency
- -85%cancel_best_level, −82% to −85% from the HashMap-to-BTreeMap swap.
- allocations per 990K orders
- 51,936 bytes in all, every one a BTreeMap node for a new price level.
Problem
A matching engine sits at the core of every exchange, and its microseconds translate directly into money. The common ways to make one slow are well known: floating-point prices, a heap allocation on every order, lock-based concurrency, and naive structures like a HashMap of queues.
The goal was to fix each one in its own phase, with a benchmark before and after, against a stated target: sub-50µs P99 at more than 1M orders a second, with zero allocation on the hot path.
What I built
- 01DecodeTCP · fixed-size binary messages
- 02Ringlock-free SPSC, Disruptor-style
- 03WALmmap append + CRC32, before matching
- 04Matchprice-time FIFO order book
- 05PublishUDP multicast, sequenced reports
Two threads. One decodes fixed-size little-endian messages (a new order is 40 bytes, a cancel 16) and pushes them through a lock-free single-producer ring buffer. The other appends each command to a memory-mapped write-ahead log, then matches, then publishes a 48-byte execution report with a sequence number so subscribers can detect gaps. A snapshot every 10,000 commands bounds replay time.
Prices are i64 ticks. Orders live in a pre-allocated arena of one million 64-byte, cache-line-aligned nodes, linked into an intrusive list per price level, so a cancel is an O(1) unlink. Price levels sit in a BTreeMap with the best bid and ask cached, so finding the next best price after a cancel is O(log n) instead of a scan.
The ring buffer was then ported line for line to C++20 with the same compile-time switches, to measure each design choice in both languages: the memory ordering, whether the two cursors share a cache line, whether each side caches the other's cursor, and the bounds check.
Considered and set aside
- f64 prices—rounding in a system that must be exact; i64 ticks instead
- HashMap + VecDeque book—O(n) cancels and an O(n) best-price recompute
- SeqCst ordering—5.4–6.0x slower, for a guarantee SPSC does not need
- Removing the bounds check—costs nothing measurable across threads, so it stays
Reading the numbers
Memory ordering: Acquire/Release compiles to a plain mov on x86 in both languages; SeqCst becomes an xchg and is 5.4x slower in Rust and 6.0x in C++. Two cursors on one cache line cost 2x. The bounds check costs 10–25% in a single-threaded loop and nothing measurable across threads.
Rust against C++: Rust is 10–17% faster on 48-byte cross-thread transfers (p ≤ 0.003) and ties on u64. C++ was 1.35–1.7x faster single-threaded. The cause was Arc's drop taking &mut self out of line, which stopped LLVM keeping the cursors in registers. Holding the Arc in a ManuallyDrop took push+pop from 2.752 ns to 1.610 ns (p = 1.4e-14, n = 40).
Durability: a WAL append costs 56 ns; recovery takes 1.4 ms. Crash-fault injection (3,362 injected faults, 12 process kills) found two real defects, both fixed.
Lessons
- A gap between languages is a question, not an answer. The single-threaded gap looked like Rust's bounds checks; it was one out-of-line drop.
- Say where it stops. One instrument only, no fsync (so no durability against power loss), gaps detected but never retransmitted, and no benchmark yet on a quiet, pinned Linux machine.
NextKernel-bypass networking (AF_XDP or DPDK), multiple instruments, a FIX gateway, and hot-hot failover.