How you measure a town

Part eight of a series. The part before this one, the car brakes too hard, ends on a promise: that the only thing which reliably works is to measure. This article is about what "measure" means when the thing being measured is a town full of cars.

Start with the awkward fact. A traffic simulation has no score.

A compiler either produces the right program or it does not. A physics solver conserves energy or it does not. A town of forty cars driving around for seven minutes is not right or wrong. It is better or worse, along a dozen axes at once, and most of those axes pull against each other. Fewer crashes means more waiting. More throughput means tighter gaps. Smoother steering means slower cornering.

So the first job is not to write tests. It is to decide what a test even is.

Rules written as sentences

Every rule in the suite is a plain English sentence. Not a metric name, not an enum, a sentence that reads like something a person would say about traffic:

• two cars never overlap
• two cars never take the same crossing at once
• a car never stands still inside a junction
• a target speed is one the car can still reach
• braking stays inside the comfortable deceleration
• a wait at a junction is bounded
• a ring of waiting cars breaks inside the deadlock timeout
• a car only expects a yield the other driver still gives
• oncoming traffic is always on the same side
• the car never corners harder than its profile allows

There are about twenty of them. The sentence is the actual identifier in the code, the thing that gets printed, and the key in the stored budget file. That is not cosmetic. Three things come out of it.

The name survives redesign. "A wait at a junction is bounded" stayed true and stayed meaningful through four complete rewrites of how right of way works. A name like "junction priority check, version two" would have needed renaming three times, and would have started meaning something else along the way.

A violation is readable without the code. Every one carries the step, the time, the car, and a sentence of detail. You read "at 143.2 seconds, car 31, braking stays inside the comfortable deceleration" and you know where to look, without opening the file that produced it.

The list is arguable. Anyone can read twenty sentences and say "that one is wrong" or "you are missing one". Nobody reviews a list of assertion names.

Beside the rules there are metrics, which are descriptions rather than judgements: trips finished, junction throughput, arrivals a minute, the ratio of travel time to free flow time, the share of time cars spend standing still, how far the worst car drifted from its lane line, how much the steering weaved, and the share of cars that got lost. None of those can pass or fail. They are the shape of the traffic, and they are what you read when a rule count moves and you want to know what it cost.

One detail in how those metrics combine across runs matters more than it looks. Most are averaged. Two are not: the worst lane drift and the worst distance outside the lane keep the largest value seen in any run, because the average of a worst case is not a worst case. Averaging a maximum is one of the easiest ways to make a bad number disappear.

The budget, and the rule that protects it

The rules do not have hand written thresholds. There is a stored file per scenario, written by the harness itself, holding every metric and every rule count from the last run that was accepted.

A run compares itself against that file and prints one of three verdicts per rule: held, worse, or "better, write the budget again". A single "worse" fails the whole run.

Two things make this work.

A rule with no entry in the file is allowed nothing. When a rule count reaches zero and the budget is rewritten, that rule disappears from the file entirely, and from that moment one single occurrence fails the harness. There is no separate list of "rules that are now strict". Reaching zero is what makes a rule strict, automatically and permanently. Four rules crossed that line in one sitting, including both collision rules, which had previously been allowed counts in the thousands.

A count may only fall. This is the rule that actually protects the whole system, and it is a social rule rather than a technical one, because the file is right there and anybody can regenerate it. Never raise a count to make a run pass. If a change trades one rule for another, that is a decision somebody makes out loud, and the budget is rewritten deliberately with the trade written down beside it. That happened twice in this project, and both times the note attached to the new budget turned out to be worth more than the numbers.

There is a failure mode on the other side, which is a budget that is too loose. At one point every scenario passed with five times the headroom, because the stored numbers described driving from before a large piece of work. A budget that loose cannot catch a regression. Rewriting it after every accepted change is not bureaucracy, it is the only thing that keeps the gate tight enough to trip.

Why one run proves nothing

Now the part that took a full day to learn and saves weeks afterwards.

Cars start at random spots. Change the starting seed and you get a different run: different queues, different near misses, different everything. The question is how different, and the answer turned out to be "it depends what you are counting". That is why the naive approach fails so badly.

There are two separate kinds of noise, and they need different treatment.

Throughput noise is a density problem. Drive the town at rising car counts and the spread between seeds stays small until a threshold, then explodes. Sixteen seeds at each density, on the town the driving tests use:

The knee is between thirty six and forty two cars, and it is sharp. Throughput stops rising, standing time doubles, and the spread jumps by a factor of ten. So the scenario the harness gates on drives thirty six, just under it. Nothing else moves that noise. You cannot average it away, because past the knee a run is not a noisy sample of one behaviour. It is sometimes one behaviour and sometimes another.

Collision noise is not a density problem. Crash counts swing by about a third between seeds at every density, because they count rare events, and rare events stay noisy no matter how busy the road is. The only thing that helps there is more seeds. Sixteen seeds put the error on the mean at roughly eight percent, where four seeds leave sixteen percent, which is larger than most changes worth keeping.

So the gate runs sixteen seeds of one scenario, and the saturated scenario became a printed trend rather than a gate. I will say that one plainly, because it feels like giving up: a single seed of a saturated town blocks good work and waves bad work through in equal measure. It is not a weak test, it is an actively misleading one.

Across the sweep, rule counts are summed and metrics are averaged, and the report also prints the slowest single seed. A mean throughput with one gridlocked seed buried inside it is a number that hides the only interesting thing in the run.

A count is not an incident

One more trap, and this one can misdirect months of work.

The harness counts samples. A sample is one pair of cars on one tick. So a rule count of five thousand does not mean five thousand crashes. It means five thousand ticks on which some pair was in contact.

Grouped into episodes, the two largest collision counts, about five thousand four hundred samples and about two thousand two hundred, turn out to be two hundred and thirty one incidents and one hundred and seventy, with a median length of about half a second.

That changes what the numbers mean twice over. It makes them smaller, which is a relief. It also makes them real: almost every one of those episodes starts with both cars moving well above walking pace, and the faster car is typically doing seven metres a second. These are crashes, not grazes.

The practical rule that came out of it: do not rank work by the largest count alone. A count tells you how long something lasted, not how often it happened. Ask how many incidents it is before deciding it is the biggest problem you have.

Determinism is a test, not an assumption

Two of the smallest tests in the suite carry more weight than their size suggests. The same scenario driven twice must produce an identical trace, car by car, tick by tick. A different seed must produce a different one.

The first catches the day something non deterministic reaches the driver, which is the day every measurement above stops meaning anything. The second catches the day the seed stops being wired through, which would turn a sixteen seed sweep into sixteen copies of one run while reporting a reassuring spread of zero.

Both have caught exactly the thing they exist for.

The town cannot tell you whether the road model is right

Everything so far measures the town against itself. It says whether today is worse than yesterday. It does not say whether any of it resembles traffic.

For that there is a second, completely different rig: one closed loop of a single lane, two hundred metres across, no junctions, no lane changes, nothing to decide. Cars are placed at a chosen density and left to settle, and the run reports speed and flow. Raise the density step by step and you get the fundamental diagram, which is the oldest and most standardised measurement in traffic engineering.

What it produces:

Six checks run on that curve. Free flow holds the speed limit. Capacity lands inside the band the Highway Capacity Manual gives one real lane, fifteen hundred to twenty four hundred vehicles an hour, and the measured peak is a little over eighteen hundred. Flow falls once past capacity and never rises again. The jam branch reaches a genuine standstill, at a density that works out to exactly one car length plus one gap. The measured curve matches the analytic prediction of the following rule it was built from, to one percent. And the answer does not depend on the seed.

That last check is the odd one, and it is the only one guarding the rig rather than the road model. A closed loop conserves its own mean speed, so the starting layout cannot move the answer, and the measured spread across sixteen seeds is under a hundredth of a percent. If that spread ever grows, the loop stopped being a loop, or something non deterministic got in. This is the exact opposite of the busy town, where seed spread is the dominant source of error. Same simulation, two rigs, completely different noise behaviour.

The number that must never become a gate

The loop reports one more thing, under the pass lines, and it is deliberately not a check.

Real human traffic on a ring road breaks into stop and go waves at around thirty vehicles per kilometre. This is the Sugiyama experiment, where drivers on a circular track are told to hold a steady speed and a phantom jam appears out of nothing within a couple of minutes. The loop here shows none of that. The spread of speed between cars inside a single tick is two hundredths of a kilometre per hour at every density. The following rule is unconditionally stable, so this simulation will never produce a phantom jam on its own.

That is a real difference from human traffic, and it is not a defect today. It is printed with the human number beside it, and there is a note in the code saying it must never become a pass or a fail, in either direction. A gate that demands waves stays red forever. A gate that demands no waves writes the stable follower down as correct, and blocks the day somebody actually wants phantom jams.

The report keeps this number visible but does not pass or fail on it. There is no agreed target yet, and turning it into a gate would pretend that there is.

The busy town tells me whether a change is worse than the previous accepted version. The closed loop checks the model against an outside reference. I need both. I also need the raw reports, because a large sample count may describe a much smaller number of incidents, and an interesting measurement may not yet have a useful pass condition.

---

The next article goes back a step, to where the town itself comes from: one number, a noise field, and a graph that turns into streets.