Accuracy methodology

What this covers#

This is how CapitolMath measures and publishes the accuracy of its predictions, set out before the first results are published. It covers:

• Whip counts: projections of how each member of a chamber will vote on a bill's next floor vote. They're the feature measured in full, and the only one that can be withdrawn.
• Bill outcome predictions: none are made. If we ever make them, their baseline and test will be published here before the first one is offered.
• Model legislation: which bills share near-identical text. It's matching, not forecasting. We publish how stable its groupings are from one build to the next. Its precision and recall will be measured against a labeled set once one exists.

Every rule below is applied by software, on a published schedule, whatever the numbers say. Nobody at CapitolMath can change a figure, a verdict or a withdrawal by hand.

The record#

• Written before it's shown. Every whip count is written to a prediction log at the moment it's made, before anyone sees it: through the API, the MCP server, the app or a widget. If it can't be written, it isn't shown.

• Never edited. The log can only be added to. Each entry carries a hash of the one before it, so changing or removing an entry breaks the chain. The chain is checked every night. Each publication states the log's latest hash, and that hash is also included in our daily public audit anchor.

• Not only what people asked for. Some projections are made and logged without being shown to anyone:

  • every week, a fixed 1% sample of pending bills, chosen by a hash of each bill's identifier;
  • each bill once, when it enters a modeled chamber's floor path (second or third reading; in Congress, placement on a calendar or a committee's report);
  • projections computed for an organization whose plan doesn't include them.

  This keeps the record from being drawn only from the bills that drew attention.

• Placeholder bills get no projection. A placeholder only states an intent to legislate and changes no law, so it has nothing to project from.

How a prediction is scored#

Each member's projected vote is compared with the official roll call for the chamber's first floor vote on the bill's passage.

Only resolved predictions count toward accuracy. Every publication reports all four counts and the resolution rate: resolved predictions as a share of those whose outcome is due.

What is measured#

Panels. Results are reported per chamber and session (for example, the California State Assembly, 2025–26). They're never pooled across states into a headline figure. A special or extraordinary session is its own session.

Lead times. How far ahead a prediction was made changes how hard it was, so each panel is reported at four lead times. For each member, bill and model version:

• At introduction: the earliest prediction made within 14 days of the bill's first recorded action.
• 90, 30 and 7 days out: the latest prediction made at least that many days before the vote.

A prediction served again and again counts once.

Figures, each with its count and a 95% interval:

• accuracy, overall and on the uncertain band: the members whose calls have been right less than 75% of the time at that confidence;
• reliability: in each range of stated confidence, how often the call was right;
• Brier score and log loss, on the stated probability of a yes vote.

Baselines, computed on exactly the same predictions:

• Party line, the primary baseline: every member votes the way the projection itself calls most of their party. A projection that only called each party's position would score exactly level, so it can beat this only by correctly calling members who break from their party. A member with no party, or whose party the projection splits evenly, has no baseline and is left out of both figures. We don't use how each party actually voted on the roll call: that's known only after the vote, which no forecast can know.
• Member habit: every member votes as they did on the most similar earlier bill in the same chamber and session, by shared topics.

Intervals come from resampling whole bills (1,000 resamples, from a fixed seed), because one roll call's members don't vote independently of each other.

Minimum data. A panel with fewer than 20 resolved chamber-votes shows its counts and "insufficient data", never a figure.

Also reported:

• the refusal rate: of the chamber-sessions with floor votes, the share where no model was published because it couldn't pass its checks;
• results per model version and combined, with the change of version marked;
• accuracy split by whether a prediction was shown to anyone.

The verdict#

Each chamber gets a verdict at each publication. It's judged on its latest regular session (a special session's figures are published on their own, but don't decide the verdict), on the predictions made at least seven days before the vote, and on the margin: the model's accuracy minus the party-line baseline's on the uncertain band.

If fewer than 20 resolved chamber-votes include a member in the uncertain band, but at least 20 resolved chamber-votes exist, the verdict is judged on every member instead, and the scoreboard says so. Without that, a chamber whose model confidently places everyone could never be judged.

Publications#

• Schedule: every Monday while any modeled legislature is in session, and the first Monday of the month while none is. A chamber whose legislature isn't sitting is judged only at the monthly publication; in between, its result carries over unchanged.
• The first public publication is on March 1, 2027. Publications before it are internal dry runs.
• Never skipped. A publication runs unattended on its date, whatever the numbers say. If a run is missed, it publishes late and says so. When several dates are published at once, only the latest is judged, so the same evidence never counts twice.
• What each includes: every published chamber's figures and verdict, whether its whip counts are offered or withdrawn and since when, the five worst chambers, the refusal rate, the methodology version, and the prediction log's latest hash.
• Kept as published. A publication is never recomputed. A retrained model is a new model version, and earlier predictions stay with the version that made them.
• The data behind it. Each publication's predictions, joined to their outcomes, are published with it under a permissive license, including the predictions that weren't shown to anyone, marked as such.

Which panels are published#

Panels are published in stages, starting with the one that performs best:

• The first public publication publishes one chamber's panel, chosen by this rule. Among the chambers with at least 20 resolved chamber-votes and a verdict of "above baseline", it's the one whose margin has the highest lower bound on its interval. If none qualifies, nothing is published that day, and the launch waits for a publication where one does.
• From then on, each chamber's panel is published at the first publication where it qualifies by the same test, and stays published.
• The five worst are named from the published panels. While fewer than five are published, all of them are named, with the number not yet published.
• Every chamber with enough data will later be published whatever its verdict. That change will be a new version of this methodology, dated in advance.

Other chambers show "not yet published", with the threshold.

Withdrawal#

The unit is a chamber, for example the California State Assembly. A verdict on one chamber never affects another, including the other chamber of the same legislature. A withdrawal applies to a chamber from the publication that first publishes its panel. It stays with the chamber across sessions until the chamber earns its return.

Withdrawn at the publication whose verdict is "below baseline". From then on:

• the API and the MCP server return status: "withdrawn" with the date and a link to the scoreboard, in place of a projection, and bill detail reports has_whip_count: false;
• the app and widgets say the whip count is withdrawn, and why;
• the scoreboard keeps showing the chamber's figures, marked "below baseline", with the date;
• projections are still made and logged, not shown, so the chamber keeps being scored.

Nothing else changes: other chambers, and other analysis of the same bills. Withdrawal doesn't change what any customer pays. If withdrawing cost us revenue, that would be a reason not to withdraw, and this rule exists so that no such reason can operate.

Returning. A withdrawn chamber is judged only on predictions made after the publication that withdrew it. It returns after two publications whose verdict on those new votes is "above baseline", with no "below baseline" between them. "Inconclusive" and "insufficient data" don't count toward a return, and they don't reset one. After a return, its next verdicts are judged on predictions made since the return.

New sessions. A new session's model is offered only if it passes its build check on roll calls held back from fitting:

• it must do better than guessing that every vote goes the way most of that chamber's votes go;
• its margin on the uncertain band must not be "below baseline".

A model that fails isn't published, and that counts toward the refusal rate. A chamber that was withdrawn when a session ended starts the next session withdrawn. Its new model must pass its build check and then return on new votes from the new session. A withdrawal late in a session that has no votes left to resolve carries into the next session.

What this can't capture#

• The selection effect. A projection resolves only if its bill reaches a floor vote, and leadership schedules bills it expects to pass. So accuracy is measured on an easier set of bills than the one whip counts are used on. Read it beside the resolution rate.
• A prediction can change the outcome. A whip count can shape lobbying and so the vote itself. That can't be removed, so we disclose it, and report accuracy split by whether a prediction was shown to anyone.
• Abstention isn't predicted. A member who doesn't vote is unresolvable, not wrong. Abstention becomes a predicted outcome only once roll calls record why members didn't vote.

Changing this methodology#

Every change gets a new version, dated, and the version it replaces stays published at its own address. A change that would make any reported figure or verdict look better is announced before the publication it would first affect. No change applies retroactively: past publications, verdicts and withdrawals stand as published.