Disentangling Algorithmic Bias from Archival Artifacts: A Controlled Audit of Vision-Language Model Valuation in Metropolitan Museum Archives
Manpreet Singh, Rhythm Bhatia, Rahul Joshi
cs.LG
Quick verdict
A careful CLIP audit of Met artworks finds no significant gender valuation gap, but the authors' own noise-floor argument limits what the null proves.
Summary
The paper audits zero-shot valuation bias in two CLIP models (OpenAI CLIP and OpenCLIP, both ViT-B/32) over 1,500 artworks harvested from the Metropolitan Museum of Art Open Access API. Artist gender is resolved via the Met's artistGender field plus gender-guesser name inference, manually validated at 96% accuracy on 100 names; after excluding 618 unattributed and 139 image-missing records (attrition balance-tested), 743 attributed works remain (534 male, 209 female). The authors compute softmax probability differentials over three prompt pairs (masterpiece, quality, influence) within a 10-prompt candidate set, compare genders with Mann-Whitney U, rank-biserial effect sizes, and bootstrap CIs, apply TOST equivalence tests, and fit HC3-robust OLS controlling for medium, century, and aspect ratio. Neither model shows a significant gender gap (CLIP p=0.183; OpenCLIP p=0.122); equivalence holds for d>=0.25; gender shows no conditional effect with R^2 below 2%. The authors read the low R^2 as an embedding noise floor: equivalence indicates instrument insensitivity, not proven fairness, and the 41.2% excluded unattributed works constitute an archival survival-bias paradox. Limitations include ViT-B/32-only backbones, a single institution, binary name-based gender inference, and internal inconsistencies in the distributional statistics.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Zero-shot CLIP logit differentials (masterpiece/quality/influence) show no significant male-female difference on 743 attributed Met artworks, with formal TOST equivalence at Cohen's d >= 0.25.
- Controlling for medium, creation century, and aspect ratio, artist gender has no significant conditional effect on valuation scores, and all covariates together explain under 2% of score variance.
- The low R^2 indicates these zero-shot metrics operate near an embedding noise floor, so score equivalence evidences instrument insensitivity rather than model fairness.
- Excluding the 41.2% unattributed holdings removes precisely the archival strata where female and craft labor was historically anonymized, an uneliminable survival bias for named-creator audits.
- Museum AI deployments should adopt medium-stratified score normalization with fallback pooling and mandatory TOST-based pre-deployment audits.
Strengths
Equivalence testing beyond null-result reasoning
TOST with sensitivity analysis across d in [0.15, 0.40] (Sections 4.3, 5.1) plus a post-hoc power calculation (1-beta > 0.998 for f^2 = 0.02) guard against misreading an underpowered non-significant result as parity.
Fully accounted data flow with attrition diagnostics
The 1,500 -> 882 -> 743 funnel is explicitly tracked (Section 3.4) with chi-square balance tests for the 139 dropped images (chi^2 = 2.025, p = 0.155) and a sensitivity check pooling the zero-count female-sculpture cell (n = 6 all male) into the Other category.
Validated, conservatively scoped demographic pipeline
Self-limiting interpretation of the null result
Dual pretraining-regime comparison
Weaknesses
Internally inconsistent distributional reporting
Abstract overstates TOST precision
Liberal equivalence bound for a fairness claim
Method mismatch in equivalence testing
Narrow external validity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does this paper show that CLIP is not biased against female artists?
No. It finds no statistically significant gender difference in zero-shot 'masterpiece/quality/influence' similarity scores across 743 attributed Metropolitan Museum artworks, and formal equivalence tests hold at effect sizes of d >= 0.25. But the authors explicitly argue the coarse scalar metric operates near an embedding noise floor, so the equivalence reflects instrument insensitivity rather than proven fairness; localized biases could still exist.
What is the 'Archival Survival Bias Paradox' described in the paper?
It is the paper's term for a selection problem: auditing only named creators requires dropping 41.2% (618 of 1,500) unattributed works, which are exactly the strata (textiles, decorative arts, domestic craft) where historical female labor was anonymized. The evaluated cohort is therefore a sanitized survival subset, and upstream archival erasure is invisible to any audit based on model outputs over named works.
What data and models were used in this audit?
1,500 public-domain objects from the Metropolitan Museum of Art Open Access API across nine curatorial departments, filtered to 743 attributed works with usable images (534 male-attributed, 209 female-attributed). Two vision encoders were tested, both ViT-B/32: OpenAI CLIP (trained on curated WIT data) and OpenCLIP (trained on the uncurated LAION-2B web crawl), scoring softmax probability differences between high-value and low-value text prompts.
Can I reproduce this study?
The authors state that all data comes from the public Met API and that audit scripts are available on GitHub, with a deterministic, rate-limited, cached harvesting pipeline. The prompts, models, and statistical tests are documented, though some details (exact checkpoint versions, image preprocessing, temperature) are thin in the text, so exact replication may require inspecting the released code.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-6 are available only as captions in the extracted text; distribution shapes and plotted results could not be checked.
- The GitHub repository was not inspected; code and data availability reflect the paper's declarations only.
- Reported statistics could not be recomputed from raw scores, so consistency checks rely on the paper's own numbers (arXiv v1, not yet peer-reviewed).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.