Benchmarking persistent sycophancy
Stateful personal agents keep long-term profiles, memories, and skills. That turns a moment of sycophancy into a durable write: a user's biased claim gets committed to state and later resurfaces in a fresh, unrelated session. PASB traces the whole pathway.
What PASB measures
Sycophancy becomes persistent when an accepted claim crosses the commit boundary into durable state — the largest downstream-failure jump in the benchmark (+27 pp). Committed claims reappear as preferences, background facts, or reusable procedures, characterized by three coupled write-time patterns:
Stored more confidently than the user said it — a passing remark becomes a stable preference, an assertion becomes a background fact, an instruction becomes a reusable procedure.
The record drops who said it — "the user said X" is rewritten as "X is true," so downstream sessions treat a user opinion as an objective fact.
The claim is applied more widely than warranted — a committed cross-domain claim leaks even into later queries in a deliberately different domain.
Leaderboard
Max-FR@3 failure rate (%) per judge dimension, for each backbone across two stateful agent frameworks — Hermes (self-improving, commits broadly) and OpenClaw (commits mostly procedures). Higher = more persistent sycophancy. Sort any column; filter by framework or search.
Overall Avg averages the six judge dimensions across both frameworks. H = Hermes-Agent, OC = OpenClaw. Judge: Kimi-K2.6 (blinded); 88% / 86% agreement with human consensus on turn-level / cross-turn dimensions.
Explore every result
Memory-like framing (Signed-Memory) and repeated reinforcement (Progressive / Drip) raise risk; Personal-Opinion and Late-Shock are lightest.
Interactive demo · 336 real episodes
Every episode below is a real run: the user's claim → the 5-turn persist stage → the actual state the
agent committed (USER.md / MEMORY.md / skills) → a fresh 3-turn neutral query stage →
the judge's per-dimension scores. Filter by model, framework, scenario, or delivery.
The benchmark
100 base items (32 personal-preference · 18 cross-domain · 50 social-value) crossed with 4 scenario framings × 4 temporal deliveries = 1,600 tasks per framework. Each is an 8-turn episode: a 5-turn persist stage that may write durable state, then a cleared, fresh 3-turn neutral query stage.
Findings
What safe personalization needs
A claim reaches the query stage only if it is committed. Preventing that pathway is a ladder of governance capabilities — not a single response-calibration fix.
Sub-axes: PRF (personal preference, 32) · CDL (cross-domain leakage, 18) · SOC (social value, 50, from ELEPHANT AITA-YTA). Domains normalized to a 13-class taxonomy. A frozen 50-task human-gold subset.
Frameworks: Hermes-Agent & OpenClaw · Judge: Kimi-K2.6 (blinded, temp 0).
Each episode runs in an isolated sandbox: the 5-turn persist stage may write durable state, runtime context is cleared, and the fresh 3-turn query stage can read only preserved durable artifacts. Cross-episode and cross-worker state sharing are forbidden — so any contamination must pass through an explicit commit.
6 judges: Sycophancy · Leak · Upgrade · Amplification (turn-level) + Persistence · Escalation (cross-turn). Failure = Likert ≥ 3. Judge agreement: 88% (turn-level) / 86% (cross-turn) within ±1.
Citation
@misc{pasb2026,
title = {Agents Don't Just Agree, They Remember: Benchmarking
Persistent Sycophancy in Stateful Personal Agents},
author = {PASB Team},
year = {2026},
note = {Personal Agent Sycophancy Benchmark (PASB)},
url = {https://henrymao2004.github.io/agent-sycophancy/}
}
Research artifact. PASB stores biased and value-laden claims as test stimuli for evaluating state-writing governance.