Pairit: A Platform for Live Experiments on Human-AI Collaboration
cs.HC · cs.AI
Quick verdict
Well-documented YAML platform for live human-AI experiments, field-deployed at scale, but validation is self-referential and hosting is commercial.
Summary
Pairit is a platform for running live experiments on human-AI collaboration in which an entire protocol—pages, conditional routing, randomization, matchmaking, chat, shared workspaces, surveys, timers, and sandboxed custom HTML components—is declared in a single YAML file that compiles to an executable directed graph. Server-hosted LLM agents are first-class participants: triggers (every message, on room join, every N messages) and conditional prompt blocks govern when they speak, and JSON-Schema tools (assign_state, end_chat, write_workspace) let them update session state and co-edit documents. A Manager CLI lints configurations against a strict schema; a Lab Server coordinates sessions, matchmaking, and agent calls, persisting events to MongoDB and exporting six file types in CSV/JSON/JSONL with full model metadata (prompts, parameters, raw outputs). Feasibility is argued through the authors' own deployments: 2,234 Prolific participants in human-human versus human-AI advertising teams, personality-prompt randomization published in PNAS, and 2,500 participants in dyadic negotiations with embedded AI assistance. Main limitations: no platform performance benchmarks (latency, concurrency, reliability), all validation is self-referential, the authors are co-founders of the commercial host Pairium, reproducibility covers protocol structure rather than participant data, and replicating agent behavior is bound to specific model versions.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A single YAML configuration can fully declare an executable live experiment — pages, routing, randomization, matchmaking, chat, shared workspaces, surveys, timers, custom HTML — combining any number of human and AI participants.
- Server-hosted LLM agents are first-class participants that both converse and take protocol-defined actions (assign_state, end_chat, write_workspace) within the declared graph.
- The platform's feasibility is validated by the authors' live deployments, including 2,234 Prolific participants in team-composition experiments and 2,500 participants in AI-assisted negotiation, including peer-reviewed published work.
- Representing protocols as standardized, auditable configuration files makes live human-AI experiments shareable, inspectable research objects that other laboratories can adapt and run.
Strengths
Declarative single-file protocol with a real toolchain
§2 and Appendix A/G describe a YAML experiment graph (pages as nodes, actions/routing as edges) that a Manager CLI lints against a strict JSON schema, checks for dangling page targets and duplicate IDs, and compiles before upload — a genuinely different model from the imperative codebases of oTree, Empirica, jsPsych, and nodeGame.
Agents as first-class actors, not just chatbots
Appendix D specifies trigger rules (every_message, on_join, {every: N}), replyCondition filters, conditional prompt blocks interpolated from session state, and JSON-Schema tools (assign_state, end_chat, write_workspace) so agents can update state, end discussions, and co-edit documents — beyond the chat-only support the paper attributes to Deliberate Lab.
Deployment precedent at meaningful scale
Complete audit trail for agent behavior
Candid boundary-drawing
Weaknesses
No quantitative platform evaluation
Self-referential validation
Commercial conflict of interest
Protocol-level, not results-level, reproducibility
Ethics enforced only as configuration toggles
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Pairit — Structured Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Pairit?
Pairit is an online platform for running live experiments on human-AI collaboration. Researchers describe an entire experiment — pages, conditional routing, randomization, team matchmaking, chat, shared documents, surveys, timers, and LLM-powered AI agents — in a single YAML configuration file, which the platform validates, compiles, and runs as live sessions mixing real human participants with server-hosted AI agents.
How is Pairit different from oTree or Empirica?
oTree, Empirica, jsPsych, and nodeGame embed study logic in imperative Python or JavaScript code, so adding AI agents or shared document editing requires custom engineering. Pairit declares the whole protocol, including server-hosted LLM agents that both chat and take actions (update session state, end discussions, edit shared documents), in one auditable configuration file, with a lint/compile pipeline and standardized CSV/JSON/JSONL data exports.
Is Pairit open source and free to use?
The paper states the source code, CLI tools, and core server packages are publicly inspectable at github.com/pairium/pairit, with documentation and browser demos at docs.pairium.ai/pairit. However, studies run through the authors' hosted platform (Pairium), researchers must supply their own encrypted LLM API keys, and the authors are co-founders of the company; external labs were in beta rollout as of publication.
Has Pairit been used in real published research?
According to the paper, the architecture powered a study with 2,234 Prolific participants comparing human-human and human-AI advertising teams, a PNAS-published study randomizing AI personality in human-AI teams, and a 2,500-participant negotiation study on AI assistance — all conducted by the authors' own research groups; the platform itself had not yet been independently deployed by outside laboratories.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.