this-that-model-1.0: A typed decision model that decides in 30 ms, for a millionth of a cent
Zehua Cheng, Wei Dai, Jiahao Sun
cs.CL
Quick verdict
A 2B typed decision head claiming 30.9 ms answers at 0.941 on a 68-item cohort; candid about limits, but small samples and in-distribution training weaken it.
Summary
The paper releases this-that-model-1.0, a 1.88B-parameter typed decision model that answers caller-declared multiple-choice questions by reading hidden states at designated slots and normalizing over option label embeddings, so no text is generated and no output can be malformed. Adapted from the open decider-2b checkpoint on a delta-rule linear-attention hybrid backbone, it is trained with a convex combination of cross-entropy and Brier score, including an unbiased single-outcome estimator with a paired control variate for outcome supervision and an abstention-insertion scheme; the fine-tune ships as one interpolation scalar for exact rollback. Reported results: 30.9 ms per decision and 32 decisions per second on a consumer GPU; 0.941 accuracy and 0.042 Brier on a third party's recorded 68-question cohort versus 0.765 and 0.133 for the hosted Jev service (the 18-point gap rests on 12 questions); near-ceiling calibration (0.745 against 0.746) on a stochastic-actuator family; and a released 7,305-question benchmark on which one round of targeted training lifted the model from 0.409 to 0.839 while hosted systems scored 0.425-0.897 zero-shot. The authors are candid: they lose on branch-shaped families (0.818 versus 0.837-0.968), manage only 0.560 on multi-step arithmetic, and answer shortest-path parity at chance (0.513 versus 0.500). Main limitations: no confidence intervals or significance tests, small cohorts, an in-distribution-versus-zero-shot headline comparison, and an unreleased internal 42-family suite.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A typed answer head that reads hidden states at designated slots and softmaxes over declared options makes malformed outputs unrepresentable and answers N questions about one state in a single forward pass (Section 3.1, Eq. 2-3).
- this-that-model-1.0 decides in 30.9 ms with zero generated tokens, sustaining 32 decisions per second on one consumer GPU, where a frontier API call costs 8758 ms (abstract, Section 1).
- On the third-party recorded 68-question cohort it scores 0.941 accuracy and 0.042 Brier versus 0.765 and 0.133 for the hosted Jev service on identical inputs (Table 1).
- Trained against a proper scoring rule, its probabilities are usable: 0.745-0.750 accuracy against a computed 0.746 ceiling on stochastic-actuator distributions, while hosted endpoints exposing probabilities score worse than a constant (Table 2).
- One round of targeted training lifted its benchmark score from 0.409 to 0.839, but nothing transferred to the 13 untrained families and search-shaped questions remain unsolved (0.574 average; parity at 0.513 vs coin 0.500) (Section 4.3).
Strengths
Malformed output is unrepresentable by construction
Eq. (2) softmaxes only over the label-token rows of the declared option set at fixed answer slots; no answer is written back, so N questions factorize as conditionally independent given the state (Eq. 3) and share one forward pass — removing the decoding loop, parser and retry path that Section 1 dissects, with the schema-first layout amortizing the shared prefix (Eq. 5).
Statistically literate training and evaluation machinery
The U-statistic estimator (Eq. 9) is genuinely unbiased for 2p_y - ||p||^2 with a zero-mean paired control variate (Eq. 10); Eq. (13)'s Brier decomposition is used to derive a computed-distribution ceiling (0.746) rather than trusting sampled labels, and the abstention-insertion design makes the option's presence uninformative (Section 3.3).
Unusually candid negative results
Auditable competitor comparison
Strong release artifacts
Weaknesses
No uncertainty quantification
In-distribution vs zero-shot headline
Unattributed latency and cost anchors
Benchmark self-dealing
Asymmetric measurement
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This cs.CL preprint (arXiv 2609.23886) introduces and releases this-that-model-1.0, a 1.88B-parameter typed decision model adapted from the open decider-2b checkpoint. The thesis: software branches need 'one of n declared options and a number it can threshold', not prose, and today that costs a frontier round trip the authors repeatedly anchor at 8758 ms. Five claims are made: a typed answer head that makes malformed output unrepresentable and answers N questions in one forward pass (Section 3.1); 30.9 ms per decision with zero generated tokens and 32 decisions/s on a consumer GPU (abstract, Section 1); 0.941 accuracy and 0.042 Brier versus 0.765 and 0.133 for the hosted Jev service on a third-party recorded 68-question cohort (Table 1); usable probabilities, 0.745-0.750 against a computed 0.746 ceiling, where hosted endpoints that expose probabilities score worse than a constant (Table 2); and a released 7,305-question benchmark on which one training round lifted the model 0.409 to 0.839 (Section 4.3). The authors explicitly disclaim frontier-level capability: 'a frontier model is a better judge and a worse component'.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a typed decision model?
A model that, given a state and a caller-declared set of options, returns a probability distribution over exactly those options by reading its hidden state at a fixed position — no text is generated, so no output can be malformed or need parsing. this-that-model-1.0 is a 1.88B-parameter open-source example that answers multiple declared questions in a single forward pass.
How fast and cheap is this-that-model-1.0?
The paper measures 30.9 ms per decision and 32 decisions per second on one consumer GPU, generating zero output tokens; one pass over the 68-question recorded cohort costs about $0.000014 of electricity (80 W card, $0.30/kWh), versus per-token API pricing for hosted rivals.
Does this-that-model-1.0 beat GPT or Claude?
Not overall. It outscores the hosted Jev service on a third-party 68-question cohort (0.941 vs 0.765) and is competitive on its own benchmark after training on it (0.844 vs gpt-5.6's 0.897 zero-shot), but it loses on branch-shaped tasks (0.818 vs 0.837-0.968), multi-step arithmetic (0.560), and cannot do graph search — shortest-path parity is answered at chance level (0.513 where a coin gives 0.500).
Are the model and data publicly available?
Yes: MIT-licensed weights on Hugging Face (flock-io/this-that-model-1.0), inference and measurement scripts on GitHub (FLock-io/this-that-model), and the 7,305-question benchmark with generators and contamination fingerprints (this-that-spatial-bench). The internal 42-family evaluation suite is not released.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as caption text; plot-level numbers (e.g. Figure 1's '91,510x' multiplier, per-size ladder curves) could not be independently verified against tables.
- The abstract's branch-shaped-family (0.818 vs 0.837-0.968) and multi-step-arithmetic (0.560) results have no per-family breakdown in the extracted body text.
- References to 2026-era systems (gpt-5.6, claude-opus-5, Jev, NanoJev, decider-2b) could not be cross-checked offline; identities and versions are taken as stated.
- Some extraction layouts are garbled (e.g. the surface-rendering table in Section 4.3); affected numbers were cross-checked against the surrounding prose where duplicated.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.