FirmCORe: A Benchmark for Structured Reasoning about Inter-Firm Collaboration Opportunities
Tian Du, Tiantong Wu, Yafei Wang, Mengyu Liu, Xingyan Chen, Mu Wang
cs.AI
Quick verdict
Solid 2,805-pair benchmark; LLMs detect collaboration opportunities far better than they recover its type/direction, but skewed sub-labels temper results.
Summary
FirmCORe is a human-annotated benchmark testing whether LLMs can reason about inter-firm collaboration opportunities from two weakly structured firm profiles. Each of 2,805 Chinese firm pairs, mined from four strata (strong-signal, borderline, same-industry hard, random non-edge) and labeled by two source-blind annotators with third-annotator tuple adjudication, carries a four-field gold label: opportunity existence, strength, primary collaboration type, and role direction. A parallel English set was built by controlled, partially human-reviewed field-level translation. Seventeen LLMs were evaluated zero-shot at temperature 0 with a unified prompt, parser, and metrics. The best detector (Qwen3.6-27B) reaches 74.51 macro-F1 but only 61.57% four-field exact match; Kimi-K2.7 leads positive exact match (61.98%). Error analyses show systematic conflation of relatedness with complementarity (78.5% of false positives self-rated High confidence), severe confusion of Technology Transfer with Supply (only 6.5% correct on Technology instances), majority-class direction bias, and model-, instance-, and category-dependent language effects, with high cross-lingual agreement often masking shared errors. An A/B order-permutation probe shows 23.53% of predictions fail exact equivariance. Limitations: zero-shot single-run evaluation only, extreme sub-label imbalance (35 Weak, 15 Capital), subjective gold standards on very short profiles (30.55% adjudication rate), and release restricted to the gold data.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 2
● low 4
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- FirmCORe is the first benchmark to jointly evaluate whether two firm profiles provide evidence-supported collaboration opportunity, and if so its strength, primary type, and role direction.
- The strongest LLM reaches 74.51 macro-F1 for opportunity detection but only 61.57% four-field exact match, showing detection is far easier than recovering the full relational structure.
- LLMs systematically conflate business relatedness with capability complementarity, with 78.5% of false-positive decisions assigned High self-reported confidence.
- Input-language effects are model-, instance-, and category-dependent, and high Chinese-English agreement can reflect identical errors repeated across languages rather than robust reasoning.
- Predictions are sensitive to A/B presentation order: 23.53% of predictions fail exact equivariance under a direction-balanced permutation of the benchmark.
Strengths
Rigorous, transparent annotation protocol
Dual source-blind annotation with tuple-level aggregation (avoiding internally inconsistent field-wise votes), third-annotator adjudication for 30.55% of pairs, 195 unresolvable cases excluded, and agreement honestly reported (84.26% field, 69.45% tuple) — Section 3.2C/D and Table 1.
Stratified hard-negative design counters trivial benchmarks
Same-industry hard and borderline strata prevent easy negatives; mining scores hidden from annotators; stratum-level analysis shows shortcut reliance (Qwen3.6-27B drops from 66.42 to 58.77 macro-F1 on same-industry hard vs random pairs) — Section 3.2B and Figure 2.
Robustness probes beyond headline metrics
Honest baseline context and interpretation guardrails
Controlled parallel-language construction
Weaknesses
Extreme sub-label imbalance undermines the structured fields
Construct validity of gold labels is not stress-tested
Narrow evaluation scope
Reproducibility gaps around mining and baselines
Model-roster bookkeeping inconsistency
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the FirmCORe benchmark?
FirmCORe is a human-annotated dataset of 2,805 pairs of company profiles (with parallel Chinese and English versions) used to test whether large language models can judge whether two firms have a plausible collaboration opportunity and, if so, predict its strength, primary type (supply, distribution, technology transfer, R&D, or capital), and direction of resource flow.
What are the main findings of the FirmCORe paper?
The best model detects collaboration opportunities reasonably well (74.51 macro-F1) but gets all four output fields simultaneously correct only about 61.57% of the time. Models overpredict opportunities based on superficial similarity, confuse technology transfer with supply cooperation, are biased toward the majority role direction, and often repeat the same mistake in both Chinese and English.
Is the FirmCORe dataset publicly available?
The paper states that only the final gold-labeled benchmark will be released, because the underlying commercial platform data is restricted by licensing and privacy constraints. No code repository or URL is stated in the reviewed text, so full reproduction of the data-mining pipeline and baselines is not currently possible from the paper alone.
Why is predicting collaboration role direction so hard for LLMs on FirmCORe?
The direction labels are highly imbalanced (700 of 968 positive pairs are B2A), so simply always guessing B2A already achieves 72.31% accuracy. The best model exceeds this by only about 5 points, and swapping the display order of the two firms breaks the expected direction reversal in a substantial share of predictions, indicating the models are not reasoning about resource flow in an order-invariant way.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures were extracted as raw number streams; values were cross-checked against tables where possible, but figure-only content (e.g., Figure 8 and the confusion-matrix layout) could not be visually verified.
- No explicit code-release statement or repository URL appears in the extracted text; codeAvailable reflects code, whereas gold-data release is asserted subject to licensing constraints.
- The abstract is truncated mid-sentence in the source metadata, but the full text covers the same claims.
- Model names (e.g., Qwen3.6-Max, GLM-5.1, GPT5.6-Thinking, DeepSeek-V4-Flash) are taken as given from the text and were not independently verifiable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.