Backtesting & Mirror Trading: How We De-Risk a Prediction-Market Engine Before Going Live

We build custom trading and automation systems for clients, and this is the companion piece to two others: the Polymarket trading system case study and the write-up on how we moved the hot path from Python to Rust. Both of those are about making the engine fast and correct. This one is about the part that comes before any of that speed is allowed near a live balance: proving, on evidence, that a strategy is worth funding at all. On a fast market a wrong-but-confident bot can bleed money quietly for a long time, so the interesting engineering is in refusing to trust it.

Fills modelled against real depth, not the midpoint

The most common way to lie to yourself in a backtest is to assume every order fills instantly at the mid. Real markets do not work like that: a taker who wants size sweeps the book, paying progressively worse prices as each level is consumed. So our fill simulator models exactly that. It walks the ask book price-by-price, taking the size available at each level, capping the buy at the target notional, and stopping when it reaches a price limit or when the liquidity simply runs out. What it returns is not a yes/no fill — it is the achieved average price and a fill ratio. A strategy that looks profitable only because it assumed full fills at the touch will show its true, partially-filled, slippage-eroded economics here.

Maker fills are modelled separately, and deliberately more conservatively. A resting bid is only considered filled when the opposite side of the market actually reprices down to touch it — we never assume a passive order gets hit just because the price wandered near it. The replay even offers two variants of that assumption: an optimistic one, where a resting order is filled the moment a trade prints at its price, and a queue-aware one, where it is only filled after the volume standing ahead of it in the book has cleared first. Reporting both is a way of bracketing reality rather than picking the flattering number.

Three replay sources, increasing in realism

A fill model is only as honest as the price data you feed it. We use three sources, each more realistic — and more expensive to work with — than the last:

• A six-year minute-level price history — millions of rows, fully offline. This is the cheap, broad net: it is fast enough to sweep across a huge space of strategy ideas and throw out the ones that never had a chance, but it is coarse and cannot see intrabar dynamics.
• Real recorded venue prices, captured from production logs. This is the accurate source, because it replays the actual book the engine saw in production rather than a synthetic reconstruction. Parameter search on it is governed by explicit risk constraints — a floor on how many trades per week a strategy must make, a floor on win rate, and a cap on how bad the worst loss is allowed to be relative to the average win — rather than by a single return number that is trivially gamed.
• Full captured order-book sessions — hundreds of megabytes — for the market-making work. These are reconstructed onto a common, monotonic clock so every event lines up, then swept across a latency ladder: the same session is replayed assuming a range of realistic delays to see which opportunities actually survive the time it takes to react. An edge that only exists at zero latency is not an edge.

The progression matters. Cheap synthetic history is for search; recorded venue prices are for judgement; full order-book replay under a latency ladder is for the market-making strategies where a millisecond of assumed delay changes the answer. Optimising on the cheapest source and declaring victory is exactly the trap the ladder is built to avoid.

The promotion path: inert → virtual → mirror → real

A strategy that survives backtesting has still never touched real money, and we keep it that way for as long as possible. Promotion happens along a ladder, and each rung is gated by evidence from the one below it.

The mirror gate is the sharp edge of the whole system. Real money is only allowed to move when every one of a set of conditions holds simultaneously: execution is enabled, mirror mode is enabled, the price feed is fresh, exposure caps are satisfied, the day's loss is under its limit, no circuit breaker has tripped — and, critically, the trailing paper expected value over the recent window is still positive. That last condition is an auto-disable: a virtual twin of the strategy runs alongside the real one, and the instant that paper twin stops being profitable over the recent window, real trading switches itself off. The engine does not wait for a human to notice.

Staged rollout: how a strategy actually earns real size

For the highest-stakes strategies, the bring-up to real money is a fixed sequence, and each step can kill the rollout:

• Fix the telemetry first, so every later claim is measured rather than guessed at.
• Run an honest backtest with realistic single-leg fills — and if it is not above breakeven, stop here.
• Paper-trade at least ~30 lots and pass roughly five quantitative gates (trade frequency, win rate, loss-to-win ratio and the like) before anything real is armed.
• Do a small real bring-up on a single, calm position — not a portfolio, one market — to confirm the live path behaves like the paper path.
• Scale by diversification: add more markets and assets, not bigger bets. Growth comes from breadth, so a single bad market can never sink the book.

One subtlety in judging whether the real path matches the paper path: we compare them on normalized return — profit per unit of cost — rather than absolute profit and loss. That is deliberate, because the real path intentionally runs the same strategy shape at a fraction of the size while it is being trusted. Comparing raw dollars would make the smaller real book look like a failure even when it is a faithful, correctly-scaled copy of the virtual one; normalizing by cost is what makes the parity check meaningful.

Certification before the switch is armed

For the highest-stakes execution path, passing the gates still is not enough on its own. Before real execution is armed, the change goes through a formal live certification — for example, a small three-lot run that has to show zero accounting drift, one hundred percent of fills going through the canonical code path, zero leaked secrets and zero unexpected network calls — alongside multiple rounds of independent AI code review of the change itself. The merge is blocked until all of that passes.

The point of the ladder is not to prove the strategy works. It is to make sure that the moment it stops working, the money stops moving on its own.

That is the whole philosophy: fast execution is worth nothing if you cannot trust what it is executing. Backtesting against real depth, a mirror mode that auto-disables the instant the paper twin turns unprofitable, and a certified bring-up are what let a client run an aggressive strategy without betting the account on it being right. If you are building an automated trading or execution system and want that kind of discipline baked in, see the full Polymarket trading system case study or read about our AI agents and bots work.