Guiding Worker Self-Selection in Crowdsourcing Contests: An LLM-Augmented Algorithmic Approach
Nguyen Thach, Hau Chan, David Parkes, Karim Lakhani
cs.LG · cs.GT
Quick verdict
Extends SSTC contest theory with LLM-evolved worker-ordering heuristics; strong synthetic results and special-case proofs, but validation is synthetic-only.
Summary
The paper formalizes worker self-selection in crowdsourcing contests as SSTC, a two-stage Tullock-contest game in which workers first choose which contests to enter and then compete through effort. It contributes GRAF, a greedy polynomial-time framework that orders workers by a score vector and sequentially assigns each to a best-response contest; for restricted cases (homogeneous values, single-contest selection, linear production and cost) it provably returns a pure-strategy Nash equilibrium with zero worker regret (Theorem 2), and in uniform cases with n ≥ 2m every PSNE is platform-optimal (Theorem 3). Because designing the ordering is hard with heterogeneous workers, LLMScore, an EoH-style LLM evolutionary framework, evolves human-readable scoring code using a regret-penalized fitness that handles both the bi-objective (platform effort plus worker regret) and intractable regret evaluation; training uses only small instances of one setting yet transfers to larger, structurally different ones. Across 1,000 synthetic instances in four settings, GRAF[σLLM] typically outperforms iterated best response (IBR) and seven hand-crafted scores on overall effort and regret (e.g., 35.749 vs 31.265 median OE in SSTC 4 at n=10, m=3) while running at least 10× faster; IBR could not finish the largest configuration within a week. Main limitations: validation is entirely synthetic with no real platform data, equilibrium and regret in the harder settings are computed with approximate best-response dynamics lacking convergence guarantees, and the theory covers only restricted special cases.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- GRAF computes a pure-strategy Nash equilibrium with zero worker regret in polynomial time for special SSTC, and in uniform SSTC with n ≥ 2m every PSNE is platform-optimal (Theorems 2-3).
- LLMScore, an EoH-style LLM evolutionary framework with a regret-penalized fitness, automatically designs GRAF's scoring algorithm, jointly optimizing platform effort and worker regret despite intractable regret evaluation.
- Trained only on small instances of one setting, GRAF[σLLM] transfers to larger and structurally different settings, achieving high or near-optimal overall effort with low regret across 1,000 synthetic instances in four settings.
- GRAF[σLLM] runs at least 10× faster per instance than iterated best response (Figure 2), with training cost amortized offline, and general SSTC with heterogeneous workers and nonlinear functions is uncharted prior to this work (Table 3).
Strengths
Model generalization with honest theory placement
Table 3 contrasts prior SSTC work ([5,10]: identical valuations, linear/identity functions, single-contest selection) with the general heterogeneous model; Theorem 2 gives polynomial-time PSNE construction with a full inductive proof (Claims 3-4 in the appendix), and Theorem 3 adds platform optimality for uniform games with n ≥ 2m.
Fitness design that addresses the stated challenges head-on
The regret-penalizing scheme (threshold τ=0.3) handles the bi-objective fitness problem (Challenge I), and offline training on small instances sidesteps intractable per-instance regret evaluation (Challenge II); all hyperparameters are disclosed (population η=20, ≤20 generations, 180 s/instance, prompt-refinement λ=t=δ=3, ≤5 strategies).
Decisive wins in the regimes without equilibrium characterization
Scalability and transfer demonstrated, not just asserted
Interpretability as a first-class outcome
Weaknesses
Entirely synthetic evaluation
Gap between guarantees and headline results
Approximate evaluation stack contaminating metrics and fitness
Median-only statistical reporting
Narrow baseline family
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets contest recommendation on crowdsourcing platforms, modeled as self-selection in Tullock contests (SSTC): workers first pick contests (up to k_i each), then compete via effort in a Tullock contest with production f_i(e)=q_i·e^{r_i} and cost c_i(e) (Section 2). Two claims anchor the work. First, GRAF — a greedy, score-ordered sequential assignment — provably reaches a pure-strategy Nash equilibrium (zero worker regret) in polynomial time for special SSTC (Theorem 2) and platform-optimal overall effort in uniform games with n ≥ 2m (Theorem 3). Second, LLMScore, an EoH-style LLM evolutionary framework, evolves the scoring function σ so that across 1,000 synthetic instances in four settings, the augmented GRAF achieves high or near-optimal effort with low regret, transferring from small single-setting training. An AAMAS 2020 extended abstract partially overlaps this work ([9]); relative to prior SSTC characterizations ([5,10], Table 3), the heterogeneous general model and the LLM-augmented design loop appear new.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SSTC (self-selection in Tullock contests)?
It is a two-stage game model of crowdsourcing contests: in stage one each worker chooses which contest(s) to enter (up to a personal limit k_i); in stage two, workers inside each contest exert costly effort, and a worker's probability of winning the prize rises with their effort relative to others (the Tullock rule). The paper studies how a platform can recommend contests so that the resulting self-selection yields high total effort for the platform and low regret for workers.
What guarantees does the GRAF algorithm provide?
GRAF provably returns a pure-strategy Nash equilibrium — meaning zero worker regret — in polynomial time, but only for restricted special SSTC games (same value per contestant, single-contest selection, linear production and cost). In uniform games with at least twice as many workers as contests, every such equilibrium is also platform-optimal. For general heterogeneous settings there are no formal guarantees; performance there is established empirically.
Was the method validated on real crowdsourcing data?
The reported experiments use 1,000 synthetic instances across four model settings; real platforms such as Topcoder and Freelancer motivate the work, but in the text reviewed here no experiments on real platform data are reported.
How much faster is GRAF[σLLM] than the IBR baseline?
Figure 2 reports that a single iterated-best-response run takes at least 10 times the time of GRAF on the hardest setting (SSTC 4), with the gap widening as the number of workers grows; IBR also failed to finish all n=60, m=5 test instances within a week, so that configuration is omitted for IBR in Table 2(b).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The excerpt contains five [...sampling gap...] regions (parts of Sections 2-3, the LLMScore evolutionary loop in Section 4, Section 5 setup, and Table 1(a) rows); judgments about skipped content rest on head/tail fragments only.
- The underlying LLM's identity and any code/data release statement were not visible in the provided text; they may appear in skipped sections.
- Figures 1 and S1-S2 and the result figures (S8-S11) are images not represented in the extracted text; only captions and in-text references were available.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.