COPE: Continual Personalization of LLMs under Sparse User Feedback via User Embeddings and Self-Evaluation

Summary

The paper proposes COPE, a framework for continually personalizing an LLM assistant when user feedback is sparse. Each user receives a learnable soft-prompt embedding (10 virtual tokens) prepended to inputs, and after each session the model performs a single joint update combining three objectives: supervised fine-tuning that compresses textual feedback into the user embedding, PPO-based calibration of a self-evaluator that predicts the user's satisfaction score (rewarded by exact match plus normalized distance), and PPO-based response optimization whose reward multiplies an external completeness judgment by either the real user score or the self-predicted proxy. Experiments on PersonaLens with 256 simulated users, a chronological protocol (51 optimization interactions, fourth traversal held out), and Qwen-Flash as user simulator show COPE beating training-free baselines (e.g., 4.74 vs 3.69 for PAP at p=0.75) and the embedding-free control T-PAP (4.69 vs 4.47 at p=0.5), with COPE+SR reaching 5.36. Gains persist under a DeepSeek-V3 simulator, across four backbones, and in a blinded 204-pair human comparison. Ablations attribute gains to all three objectives. Main limitations: training and evaluation share the same Qwen-family simulator and rubrics, so closed-loop bias is only partially mitigated; the human study involves a single annotator and only COPE vs T-PAP; no empirical comparison with cited continual-personalization competitors (RLPA, SPRInG, C-BPO); reported differences lack significance testing; evaluation is single-turn on one benchmark with modest absolute scores (max 5.36/10).