Multimodal Conditioning of Fine-Tuned Stable Diffusion XL for Controllable and Culturally Faithful Ulos Motif Generation
Humasak Simanjuntak, Tamara Yunika Sianipar, Bronson T. M Siallagan, Difya Laurensya Ambarita, Arlinta Barus
cs.AI
Quick verdict
Applies LoRA-tuned SDXL plus LLaVA and ControlNet conditioning to Ulos motif generation; its ablation finding that more conditioning is not better is interesting, but evidence rests on a 7-image test set with inconsistent metric reporting and no text-only baseline.
Summary
The paper fine-tunes Stable Diffusion XL v1.0 with LoRA to generate Batak Ulos weaving motifs, adding LLaVA-1.5-7B (itself LoRA-tuned for Ulos captioning) as a 'representation' conditioning channel and ControlNet driven by a six-class semantic map extracted via Otsu thresholding, connected-component analysis, Hough lines, and contour analysis. Training used 125 curated Ulos images (from DiTenun and field collection) augmented to 431; testing used seven reference images across three scenarios (shape/motif transformation, recolouring, high-complexity input). A five-level ablation over four conditioning combinations reportedly shows non-additive effects: Text+Image+Semantic Map attained the best FID (~270) and CLIP Score (0.65-0.70) but the weakest SSIM (~0.65); Text+Image+Representation gave the best balance (SSIM ~0.84, FID ~280); all four combined degraded to FID ~330, attributed to conflicting optimization signals. Thirty inference steps outperformed 40/50 (average FID 253.67). Questionnaires from nine weavers and thirty public participants showed above-neutral acceptance (Wilcoxon p=0.007 and p<0.001). Key limitations: a seven-image test set, inconsistently defined and reported metrics, no text-only baseline despite motivating the work by prompt limitations, and an empirically untested disentanglement claim borrowed from EncDiff.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Four complementary conditioning mechanisms (text, image, LLaVA-derived representation, ControlNet semantic map) enable controllable, culturally faithful generation of Batak Ulos motifs on a LoRA-fine-tuned SDXL v1.0 backbone.
- Conditioning effectiveness is not proportional to the number of mechanisms combined: Text+Image+Semantic Map gives the best FID (~270) but weakest SSIM (~0.65), Text+Image+Representation offers the best overall balance (SSIM ~0.84, FID ~280), and all four combined degrade FID to ~330 due to conflicting optimization signals.
- Thirty inference steps are optimal for this domain; 40 and 50 steps progressively worsen FID (279.00, 289.38) and structural fidelity.
- Generated motifs achieve statistically significant above-neutral acceptance by nine traditional weavers (Wilcoxon p = 0.007) and thirty public participants (p < 0.001), including on cultural authenticity.
- Scenario 3 (high-complexity input) received the highest and most stable human ratings from both respondent groups.
Strengths
Genuinely interesting and honestly reported headline finding: conditioning is non-additive
Table 2 and Fig. 4 show the full four-mechanism configuration (E4) achieves the worst FID (~330) with unstable validation loss, while single additions help; the authors openly report this against their own design intuition rather than hiding it, and Section 3.1 traces it to conflicting optimization signals - a useful caution for multimodal-conditioning designs beyond Ulos.
Thorough multi-axis experimental design
Five ablation levels x three task scenarios (shape transformation, recolouring, high-complexity input) x strength values 0.65/0.75/0.85 x 30/40/50 inference steps, with guidance-scale sweeps (Fig. 5) - unusually systematic for a domain-application paper, and the step ablation (30 steps best, avg FID 253.67) is a practical, actionable result.
Well-specified semantic-map construction pipeline
Culturally grounded evaluation with domain stakeholders
Clear positioning against prior Ulos/batik generative work
Weaknesses
Quantitative evaluation is severely underpowered and internally inconsistent
Missing key baseline: fine-tuned SDXL with text prompts alone
Mechanism claims asserted, not tested
Potential circularity in the caption/representation pipeline
Qualitative evidence shows direction of preference, not authenticity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the paper do?
It fine-tunes Stable Diffusion XL (SDXL) with LoRA to generate new motifs for Ulos, a traditional woven textile of the Batak people in North Sumatra, Indonesia. Generation is guided by up to four conditioning inputs at once: a text prompt, a reference image, a machine-generated visual description of the motif produced by a fine-tuned LLaVA multimodal language model, and a six-class spatial 'semantic map' processed through ControlNet. The authors also built a web prototype supporting text-to-image and image-to-image motif generation.
What are the main results?
In a five-level ablation across three tasks (shape transformation, recolouring, and high-complexity input), adding more conditioning was not always better: text + image + semantic map achieved the best FID (about 270) and CLIP Score (0.65-0.70) but the weakest structural similarity (SSIM about 0.65); text + image + representation gave the best overall balance (SSIM about 0.84, FID about 280); and combining all four mechanisms produced the worst FID (about 330). Thirty inference steps worked better than 40 or 50. Questionnaires with nine traditional weavers and thirty public participants showed ratings significantly above the neutral midpoint.
Are the generated Ulos motifs culturally authentic?
The paper argues they are culturally faithful: the model was fine-tuned on 125 curated authentic Ulos images, and evaluations by nine experienced weavers from Desa Pintu Bantu plus thirty culturally familiar public participants scored the outputs significantly above a neutral rating, including on pattern authenticity and colour alignment. However, the statistics only show ratings leaned positive; the review cautions that this does not by itself prove cultural authenticity, and independent expert validation beyond the two questionnaire groups would strengthen the claim.
Why does combining all four conditioning mechanisms hurt performance?
The authors observe that the full four-mechanism configuration had the weakest and least stable FID and CLIP scores and attributed this to 'conflicting optimization signals': jointly enforcing LLaVA-based representation constraints and semantic-map spatial constraints appears to pull the diffusion model in different directions during training rather than combining additively. The review notes this interpretation rests on limited quantitative evidence (a seven-image test set without variance estimates) and that no controlled experiment isolates the conflict.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (architecture Fig. 1, training curves Fig. 3-4, sweeps Fig. 5) and the visual comparison Table 3 are not analyzable from extracted text, so qualitative image-quality claims could not be independently checked.
- Table 1 is internally inconsistent in the extracted text (caption says six classes, five rows listed, class-ID column duplicates '3' with no ID 5); this may partly reflect extraction artefacts.
- The web prototype is asserted in the abstract and conclusion but never described in the body, and its availability cannot be verified from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.