Temper
A reinforcement-learning execution agent, graded against Almgren–Chriss closed forms. The agent works a parent order in a simulated market; an independently derived oracle defines what optimal is, and every threshold is written down before training starts. Under linear impact it rediscovers the analytic schedule to four decimal places. Under a 0.6-power impact law — where the closed form is no longer solving the right problem — it captures 99.4% of the available difference against a certified optimum. Given an invented price signal explaining one part in ten thousand of the next bin’s return variance, it captures 93% of the available advantage while paying 1.05× the optimum’s own execution premium. The same agent then worked a real order on a live matching engine over a wire, as that venue’s third independent client, with every fill predicted in closed form before anything was sent. Python, PyTorch, single-file PPO, CPU only.
// Overview
Temper trains a reinforcement-learning agent to work a parent order: sell a large position over a trading day without moving the price against itself more than necessary. It grades the agent against the Almgren–Chriss closed forms already implemented by FrontierView, another project in this portfolio — the differential-oracle pattern from Anvil applied to RL. An independently derived reference engine defines the right answer, so the agent’s score is never a matter of opinion.
Under the exact assumptions Almgren–Chriss makes, its closed form is provably optimal — so if the agent appears to beat it, something in the harness is broken. Phase 1 therefore claims only that the agent independently converges to the analytic schedule, within a tolerance written down before training started. Every milestone is specified in a brief that fixes ε, the seed count and the confidence level in advance; every reported number regenerates from a committed config and seed; every chart carries TWAP and the analytic schedule as baselines and shows the spread across seeds, not the best run.
This caught a real bug. The vendored κ turned out to be a continuum-limit expression that drops a factor at realistic order sizes: the two rates differ by about 2.2×, both produce sinh-shaped schedules, and the difference is invisible by eye. Grading against it would have shown a correctly trained agent scoring up to 18% better than "optimal" — exactly the red flag above. The oracle now carries both conventions. The finding itself is narrow: the risk-aversion dial is mislabelled by about 5× at usable settings, not that Almgren–Chriss is wrong. The vendored schedule is exactly optimal for a different risk aversion, and sits on the efficient frontier, not inside it.
The second finding was about noise, not finance. At the reference case the quantity being optimised is ~2.4 basis points, while a single episode’s cost varies by ~95. At that signal-to-noise ratio, training outcomes are close to random: five seeds scored 0.066, 0.009, 0.819, 0.098 and 0.147 of the available gap — one essentially rediscovered the analytic schedule, another barely learned at all. The same agent with the same hyperparameters, trained on a reward whose noise is cancelled by an exact identity the simulator publishes, clears the bar with a spread orders of magnitude tighter. Both runs are committed and differ in one config field, so "the estimator closed the gap" is a measurement, not a story. A related discovery, now a portfolio-wide rule: the same seed scored 0.165 on four threads and 0.066 on eight, because multithreaded floating-point reductions sum in a thread-dependent order. Thread count is a committed input, not a property of the host.
Phase 2 is where the agent is allowed to win, because the closed form is now solving the wrong problem. FrontierView’s temporary impact is a 0.6-power law; Almgren–Chriss has no closed form for that, so the library linearises at a tangent and solves the linear problem instead. With the power law as the simulator’s actual dynamics, that costs the closed form 1.54% of the objective at the reference case — 16,878 shares of a 100,000-share order in trajectory space. The power-law world’s own optimum has no formula, so it is solved numerically: Newton iteration on the KKT system, certified by a positive-definite Hessian, a relative residual of 1.2e-15, 3,600 perturbations that all increase the cost, and an independent bisection solver agreeing to within 3.1e-15 of the parent order. Ten seeds captured a median 99.4% of the available difference, worst seed 99.0% — landing 727 shares from the certified optimum, where the closed form sits 16,878 away.
The second part of Phase 2 introduces the first advantage no fixed schedule can capture at all. Per-bin liquidity becomes a second, independent noise source, and the agent observes it. The benchmark is therefore no longer a mis-specified formula but the best possible fixed schedule that knows the liquidity law: 2.49661 basis points, against 2.43449 for the optimum over reacting policies — an adaptive advantage of 0.06212 bps, or 2.55% of the objective. (The liquidity process is Temper’s own invention: a per-bin multiplier with one parameter, σ_L = 0.5.) That 0.06212 bps is the denominator, not the distance from M4a’s schedule — 0.00234 bps of the latter is a level shift any static solver picks up for free. Ten seeds captured a median 99.0% of it, worst seed 96.3%, landing 0.00064 bps above the reference. That reference is converged and bracketed, not certified: a stochastic dynamic program has no certificate to offer, so the word is not used. The control carries the claim: re-graded with observed liquidity drawn independently of the liquidity actually charged, the same policies score −1.01 against a bar of 0.15 fixed before the run — about one advantage-unit worse than the best static schedule. Reacting to a decorrelated signal costs almost exactly what reacting to the true signal gains, and the gap between +0.99 and −1.01 is the evidence that the agent is using the signal rather than finding a better fixed schedule.
The third part, M5, puts a weak one-step-ahead price signal in the observation and asks the agent to tilt the schedule for it. The signal is Temper’s own invention with one parameter, ρ = 0.01: it explains one part in ten thousand of the next bin’s return variance. Monetising a signal costs execution quality — the converged optimum itself gives back 45.5% of the gross effect to do so — which means no single capture fraction can separate a policy that traded the signal well from one that traded it badly and executed well. The result is therefore three numbers, always reported together, each against a bar fixed before the run: alpha capture 0.9767, of the 0.14808 basis points of gross alpha available; execution premium 1.0480×, of the 0.06732 basis points the optimum itself pays; net capture 0.9341, which is 0.00532 basis points over that optimum on an advantage of 0.08076. Ten seeds, worst seed 0.8925. The denominator is the net advantage rather than the gross alpha, which would have read 1.83× larger in the flattering direction. As with liquidity, the control carries the claim: re-graded with the observed signal shuffled, the same policies score a median −0.87 against a bar of −0.50 stated in advance. The reference is again a dynamic program, converged rather than certified — but with a certified floor under the part of the objective the signal cannot touch. Impact and risk are convex and carry no signal, so M4a’s certified optimum bounds them for any policy whatever, which makes the red-flag test a proof rather than a threshold.
The two ends of the project make one point, because it is the same agent at the same budget both times. At a 1:70 per-episode signal-to-noise ratio on the realised reward, rediscovering a closed form was a lottery. A signal seven hundred times weaker was recovered almost completely once the reward was the conditional grade, whose conditional variance is exactly zero. The agent fails on a large signal under noise and succeeds on a tiny one without it: across both phases, the binding constraint was the estimator’s variance, not the signal’s strength.
M4a’s methodological finding is worth more than its result, and the two milestones after it repeated the lesson. The tolerance every earlier milestone used — 5% of the distance TWAP covers — is 1.8× the entire effect M4a measures, so an agent capturing none of the mis-specification would have passed comfortably. A tolerance is only portable within the question it was written for. M5 extends the point: when the advantage is a difference of two larger numbers that move together, no single fraction is portable either, and the headline has to be the whole decomposition.
None of this establishes real-market performance. Every number is a statement about a simulator whose dynamics are analytic and whose parameters are a calibrated synthetic set — and the liquidity process and the alpha signal are Temper’s own inventions, calibrated against nothing, so the 2.55% that seeing liquidity is worth and the 0.081 bps the signal is worth are statements about those models, not about a market. There are no real fills and no historical order-book data anywhere in the project. The differential tests make it a good simulator, and the honest reading of the results is "the machinery is correct", not "the strategy would make money".
The per-commit suite runs from a clean clone in about two minutes ten seconds, inside a three-minute ceiling every brief states in advance; the deep Monte-Carlo tiers and multi-hour acceptance runs sit behind markers and drivers, so the commit loop never waits on them. Everything is CPU-only and every milestone is sized to train in an evening on one desktop, so progress never depends on access to a GPU.
The trained policy also ran against a live venue. M4a’s median seed — exported as a committed .npz and run as a NumPy forward pass, with no training stack on the client’s import path — worked a 1,000-share parent order over 13 bins on a live Anvil book, another project in this portfolio. It is that protocol’s third independent client after the browser UI and DepthCharge, required zero changes to Anvil, and reports 11.21 basis points of arrival slippage.
That number was predicted before anything was sent. A committed counterparty ladder, a deterministic policy and deterministic matching make every fill price and quantity computable in closed form in advance — and three ladder shapes came back matching level for level: 11.21 bps on the reference book, 22.85 on a deliberately thin one, 33.63 on a wide-spread one, each to the digit and each reproduced across two revisions. A dozen lines of Python on one side, a C++ matching engine over a wire on the other. Arrival slippage against a book the client built itself says nothing about execution quality; predicting it first makes it a test of client correctness, which is the one thing it can certify.
The thin book produced a state the policy had never seen: the simulator always filled exactly what was asked — its step function clips the trade to available inventory — so partial fills never occurred in training. Bin one asked for 421 shares against 375 of depth, swept all eight price levels, was filled short, had its resting remainder cancelled, and carried the shortfall into the next observation. Anvil has no market orders: an aggressive order is a limit priced to cross, and a limit that doesn’t cross rests silently. Accepted is not filled.
The same client then worked the same order against the public deployment over TLS, on a shared, unauthenticated book. It filled 1,000 of 1,000, attributed every share, and reported the measurement void: 236 fills by other participants landed while the order was being worked. That is the milestone’s own pre-stated condition firing exactly as written — a third-party fill makes the run a successful demonstration and a void measurement, and both halves get reported. It is the only void run of the five, and void for the healthiest possible reason: the venue was real and other people were trading on it. A clean number would have meant nobody else was.
This is plumbing evidence, not execution-quality evidence: the flow is a synthetic non-adversarial feeder, the sample is one order, and there is no baseline to compare against fairly. What it establishes is that the policy speaks a versioned venue protocol end to end — session, order entry, verdict channel, event stream — and copes with a book that does not give it what it asked for.
// Key decisions
Why is the agent never allowed to beat Almgren–Chriss?
Under the exact assumptions the closed form makes, it is provably optimal — so an agent that scores better has found a bug in the harness, the metric or the environment, not an improvement. Phase 1 claims only convergence within a pre-stated tolerance, and a hard test fails the run if any seed beats the certified optimum by more than arithmetic noise. Advantage is claimed only in Phase 2, where the assumptions are deliberately broken: the closed form solves the wrong problem in the first of those worlds and has no answer at all in the other two. The test survives even there, and by the last milestone it becomes a proof rather than a threshold — the graded cost splits into a part the signal can move and a part it cannot, the second part is convex and contains no signal, so the earlier world’s certified optimum is a floor under it for any policy whatever. An agent below that floor is provably a defect, not an unlucky seed, and telling those two situations apart is the whole project.
Why a separate repo from FrontierView, rather than importing its model?
The boundary is versioned artefacts, not shared code. FrontierView’s outputs are exported by a read-only script and vendored here with the source commit stamped; nothing upstream changed for Temper’s benefit. Independently re-deriving the closed forms cost about 55 lines of parallel code, and it is what surfaced the κ convention problem — a shared import would have inherited it silently. A reference engine that shares code with the thing it checks is not a reference engine.
Why is the agent graded analytically instead of by Monte Carlo?
Through Phase 1 the observation carried no price, so a deterministic policy produced the same schedule under every price draw — and a fixed schedule’s exact cost is a closed form. One rollout gives the answer with no sampling error. That is not just faster: resolving the milestone’s tolerance by sampling would need on the order of 10⁷ episodes per seed, and would still report an interval where this reports a number. An assertion keeps the shortcut sound rather than merely convenient: the policy is rolled out on two unrelated random streams and the schedules must match bit-for-bit, so it fails loudly the moment price reaches the observation. Stochastic liquidity retires half of that rule and shows which half was load-bearing. The observation now carries liquidity, so the schedule reacts and there is no single trajectory to hand to a closed form — but price still never enters the decision, so conditioning on the liquidity path removes all the price randomness analytically and the conditional expectation is a closed form again, averaged over 20,000 held-out paths with no price sampling anywhere. What made analytic grading legitimate was never that the schedule was open-loop; it was that price never entered the decision. The alpha signal then crosses that line on purpose: the observation now carries something that predicts the next bin’s shock. The old ban is replaced by a guard in two clauses — pin the signal and vary the price, and the observations must come back bit-for-bit identical; and no observation coordinate may correlate with any shock whose cost the current decision can no longer change. Realised price in any form is still refused, as is any realised cost, as is any signal about a shock that is already committed, however far ahead it lands. Grading keeps pace with the observation: conditioning on the signal path makes the training reward the conditional grade itself, with zero conditional variance and no price sampling anywhere in the milestone, and a check rejects outright any grade whose conditioning set is not exactly what the policy saw.
Why write the success criteria down before training?
Because post-hoc thresholds are how reinforcement-learning projects mislead their own authors. Each milestone’s brief fixes the tolerance, the seed count and the confidence level before any training code runs, and loosening one means amending the brief before the work, in the commit history. This has caught a real problem: the tolerance inherited from three earlier milestones turned out to be 1.8× the size of the effect the next milestone was measuring — visible only because the bar had been written down and could be compared against the oracle in advance.
Why does the oracle carry two decay rates instead of one?
Because the vendored library’s κ is a continuum-limit expression and the discrete problem’s stationarity condition gives a different rate — about 2.2× apart at realistic order sizes, with both producing sinh-shaped schedules, so the difference is invisible on a chart. Keeping both means the goldens still match the vendored library exactly while the agent is graded against the optimum of the objective it actually trains on. A test pins the mapping between the two, and a guard fails if anyone collapses them back together.
Why Python, when the rest of this portfolio is C++?
The reference model’s compute core is Python, the RL ecosystem is Python, and the C++ story is already told by Anvil and Crucible. The interesting engineering here is the grading discipline, not the language. A C++ inference leg — exporting the trained policy and measuring it under load — is on the backlog, and would tie the two halves of the portfolio together.
Your live run reports a number and then says it's void. Which is it?
Both — they answer different questions. The run is a successful demonstration: the client held a session, submitted orders, consumed the event stream, reconciled every fill to its own order ids, and finished the parent order. The measurement is void because 236 fills by other participants landed on the book while the order was being worked — and the voiding condition was written into the milestone brief before the run, not after seeing the number. It is reported this way because a clean number from a shared public book would have been the weaker result: it would have meant nobody else was trading, which is the opposite of what the run exists to demonstrate. The measured runs are the ones against a book the client built itself, where every fill was predicted in closed form beforehand and matched to the digit. There is deliberately no reconciliation path in the code — the moment one exists, every number the client produces becomes an estimate.