NumericJev: Jev-like LLM Numerical Decoding with Multiway Decision Trees
Weiwei Ye, Hangchen Liu, Renhe Jiang
stat.ML · cs.AI · cs.LG
Quick verdict
Training-free interval-tree decoding gives choice-only LLMs numeric output; honest stats, but one hosted model, tiny synthetic tasks, mixed significance.
Summary
NumericJev is a training-free decoding algorithm that extracts numerical values from language models exposing only a structured-choice interface (called Jev-like). Instead of regressing from hidden states or requesting free-form text, it discretizes a user-supplied range at a requested resolution, asks the model which of K intervals contains the target in the context of the original question, and recursively refines the chosen interval — a multiway decision tree needing two rounds at K=10 on a 100-value grid. On 256 synthetic arithmetic expressions (64 seeded families × four affine transforms × two option orders) using hosted model jev-1.13-20260917, ten-way refinement places 83.40% of outputs within 5% relative error versus 80.47% for direct choice among all 100 grid values; range-normalized MAE improves from 5.18% to 1.84%, with significant paired NMAE differences but MAPE confidence intervals crossing zero. Index-digit and bit encodings fail badly (84–95% MAPE), and a supplied-answer control shows readout can still fail even when the value is given. A three-date S&P 500 recall study yields 4.58% mean error. Main limitations: a single proprietary model version, synthetic endpoint-valued arithmetic with no free-form generation baseline, a three-date historical sample, and a post-hoc 5% tolerance metric anchoring the headline claim.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Numerical range selection is itself a decision problem that Jev-like structured-choice LLMs can solve, enabling numerical output without parameter updates, hidden-state access, or tools.
- Ten-way NumericJev places 83.40% of outputs within 5% relative error on 256 arithmetic expressions versus 80.47% for direct choice, and reduces range-normalized MAE from 5.18% to 1.84% (the NMAE difference statistically significant, the MAPE difference not).
- The representation requested at the interface is a major error source: decimal-digit and binary-bit index encodings fail at 84–95% MAPE even when the correct value is supplied in the state.
- A three-date S&P 500 recall study yields 4.58% mean relative error, dropping to 0% readout error when the closing level is supplied.
Strengths
Candid significance reporting
Section 5.1 explicitly states that paired MAPE differences vs direct choice cross zero for all trees (e.g., K=10: [−14.05, 12.77]) and Section 5.3 admits a 16.41-point gap to the answer-supplied control — the text repeatedly reins in its own headline numbers.
Pre-registered, hashed evaluation protocol
Section 4: cases, prompts, model version, and job order were fixed and hashed before execution, and prompts 'were not revised after inspecting these results'; Appendix B adds a deterministic exact-label oracle verifying decoding mechanics for all 256 cases before any API call.
Informative controls
Negative results retained in the artifact
Formal error analysis matched to intuition
Weaknesses
Single proprietary model, unverifiable interface
Missing free-form generation baseline
Small effective samples with strong domain reversals
Post-hoc headline metric
Confounded continuous-head comparison
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
NumericJev (arXiv 2609.28587, stat.ML) is a method paper with a controlled empirical evaluation. Its claim: numerical values can be decoded from LLMs exposing only a structured-choice interface ('Jev-like', per the cited Jev Choice/Score API documentation) through recursive multiway interval refinement — no fine-tuning, no hidden states, no tools. Three headline results: (1) 83.40% of outputs within 5% relative error on 256 arithmetic expressions versus 80.47% for direct choice among 100 candidate values (+2.93 points); (2) range-normalized MAE 1.84% versus 5.18%, with statistically significant paired differences; (3) a three-date S&P 500 recall study at 4.58% mean error, 0% when the close is supplied. The reduction-to-classification lineage (Torgo & Gama 1997; Langford et al. 2006) is correctly credited; the novelty is the constraint setting — a fixed hosted choice endpoint, inference-time-changeable alternatives, and no calibration assumption.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is NumericJev?
NumericJev is a training-free decoding algorithm that obtains numerical values from language models which only expose a structured multiple-choice interface. It repeatedly asks the model which sub-interval of a user-specified range contains the answer, narrowing the range like a decision tree until reaching the requested precision.
How much better is the interval tree than just listing all candidate values?
On 256 synthetic arithmetic expressions, ten-way refinement placed 83.40% of outputs within 5% relative error versus 80.47% for direct choice among all 100 grid values; range-normalized mean absolute error improved from 5.18% to 1.84%, though the MAPE improvement was not statistically significant in this sample.
Does the method work on real-world numbers like stock prices?
Only weak evidence exists: a small study asked for S&P 500 closing levels on three dates and got 4.58% mean relative error by recall, and 0% error when the value was supplied in the input. Three dates are far too few to establish real-world forecasting or generalization.
Is code or data available for NumericJev?
The paper points to github.com/Bring-AI/jev-numeric. The authors state the fixed evaluation is reproducible offline from saved responses, decision traces, and a fixed bootstrap seed, but the underlying hosted model (jev-1.13-20260917) is proprietary and re-querying it may fail.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 are represented only by captions in the extracted text; visual details (error bar magnitudes, the Figure 4 CDF shape) could not be verified.
- The 'Jev' hosted model and its Choice/Score API documentation are cited but not independently verifiable from the text.
- Appendix A proofs are summarized in the extraction; full derivations were not checked line by line.
- Whether the released repository actually contains the saved response traces required for offline reproduction could not be confirmed from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.