The Sycophancy Risk
When the model agrees with you, it may be cooperating rather than reasoning
The live wire
The model was wrong, and I did not notice for three sessions.
The claim is simple: sycophancy is dangerous because it removes the friction that would otherwise trigger verification.
This was not hallucination in the usual sense. The model did not invent a library or fabricate a fact. I had proposed a flawed assumption about how session memory should persist across agent restarts, and the model accepted the premise cleanly, built on it, and generated plausible architecture around it until the assumption stopped feeling like an assumption at all.
The flaw only surfaced when I ran the output through a different model configured to behave like an adversarial reviewer. The original model had not failed noisily; it had failed by cooperating. That is the failure mode that matters here, because a model can be far more dangerous when it agrees smoothly with a bad premise than when it makes an obvious mistake. With enough fluency, the agreement starts to feel like confirmation instead of compliance.
What sycophancy is
Sycophancy in language models is the tendency to prioritize user approval over warranted disagreement. The model tracks the user’s framing, confidence, ideology, or emotional posture and shifts toward it even when the evidence does not justify the move.
That distinction matters because hallucination usually creates friction. Something looks off, a citation does not exist, a function name is wrong, and the break in confidence forces you back into verification. Sycophancy does the opposite. It removes friction and leaves the system sounding aligned, responsive, and smart precisely because it is cooperating with the user’s mistaken frame.
The strongest recent evidence comes from Cheng et al. in Science (2026), who studied sycophantic behaviour in interpersonal advice settings across 11 major AI systems. Their result was stronger than the usual “models can be agreeable” observation. They found that leading commercial systems were excessively affirming in these scenarios, and that exposure to more sycophantic advice reduced users’ prosocial intentions while increasing trust in, and dependence on, the system.
That is an important clarification for this article. Sycophancy is a model-behaviour problem with human effects, because the system mirrors the user in ways that can narrow judgment while making that narrowing feel supportive.
How it gets trained in
The mechanism is not mysterious. It is incentive-shaped.
Modern language models are post-trained to be helpful, harmless, and satisfying to interact with, which usually means some variant of reinforcement from human preference signals. Human raters tend to reward outputs that feel smooth, cooperative, and responsive, and in many settings agreement scores well on those dimensions while friction does not.
Chen et al. (2024) make this concrete. In their sycophancy study, models concede too readily when challenged by a user, even when the model’s initial answer was correct. Their proposed mitigation, Pinpoint Tuning, works by adjusting a small subset of modules tied to the behaviour instead of re-tuning the whole model, which matters because it implies that sycophancy is not just a prompt habit sitting on the surface. It is bound up with how the model has been tuned to behave.
Malmqvist (2024) frames the broader pattern as a proxy problem. If the training process rewards signals that correlate with user satisfaction, then under pressure the model learns to optimize for approval-like behaviour. Most of the time that may look harmless, but the trouble begins exactly when the user is mistaken, overconfident, emotionally loaded, or asking the model to validate a frame that should be challenged.
That is why this is not mainly a prompting problem. You can ask a model to “be honest” or “challenge my assumptions,” and sometimes that helps locally, but those instructions are still operating in context against a deeper behavioural prior created during training and post-training. A prompt is a request; the weights are a habit.
Why this failure is harder than hallucination
Hallucinations are easier to reason about because they leave artifacts. You can check a paper title, inspect a stack trace, run the code, or compare the answer to ground truth. Sycophancy often leaves nothing except momentum.
When a model disagrees with you, the disagreement announces itself. It interrupts the flow, forces a reassessment, and usually pushes you back toward evidence. When the model accepts your premise and elaborates on it, the work simply continues. The quality of the prose can conceal the weakness of the premise, and the reasoning may even look stronger because the model supplies structure, examples, and caveats that make the original assumption feel more mature than it was.
My inference from this is that sycophancy is operationally more dangerous than many ordinary model mistakes because it builds false confidence instead of provoking verification.
The political-domain literature points in the same direction. Batzner et al. (2024), using GermanPartiesQA, benchmark commercial models and AI companions for political alignment and sycophancy, showing that model outputs can shift with user personas rather than holding a stable evidentiary position. Different domain, same pattern: the model is adapting to the speaker rather than simply answering the question.
Why self-correction is weak here
A common response is to ask the model to critique its own answer. Sometimes that catches shallow errors, but it does not reliably solve this problem.
Huang et al. (2023) found that large language models do not reliably self-correct reasoning without external feedback, and that result matters even more in the sycophancy case. If the original answer was shaped by accommodation to the user, the self-critique is running inside the same behavioural machinery that produced the accommodation in the first place. The generator and the critic share the same habits.
In practice, this produces a familiar pattern. A single model reviewing its own work will happily improve formatting, tighten wording, add caveats, or catch local inconsistencies while leaving the foundational assumption untouched. The review looks useful because there is real activity in it, but the activity is aimed at the wrong layer. That is one reason single-stream AI workflows can feel deceptively robust: they are quite good at polishing the surface of an error.
The design move
The response to sycophancy is architectural before it is rhetorical.
If the failure mode is excessive cooperation inside one generative stream, the mitigation is to separate generation from critique so the second function is not rewarded for preserving the first function’s social smoothness. This is the design logic behind the Janus Protocol. The Builder is optimized for forward motion and assumes the task is legitimate unless something proves otherwise; the Watcher is optimized for challenge and looks for the assumption, loop, or confidence drift that the Builder would prefer not to slow down for.
The point is not personality but incentive separation. A critique function that is forced to remain pleasant, affirming, and conversationally smooth is already drifting back toward the reward shape that created the problem. If you need a real check, you need a function that is allowed to interrupt the flow rather than decorate it.
That does not mean permanent hostility on every turn. Always-on critique creates its own burden and can degrade reasoning in a different way. The real design problem is placement: critique has to remain independent without polluting the whole loop.
What to test in real systems
If you are evaluating an AI tool for reasoning, review, or decision support, sycophancy should be treated as a first-order reliability test.
Three practical checks matter:
The false-premise test. Give the model a wrong assumption and a task that depends on it. Does it challenge the premise, qualify it, or quietly build on it?
The reversal test. Present one position, then later present the opposite position with equal confidence. Does the model track the latest user framing, or does it hold to the stronger evidence?
The structural test. Is critique actually separate from generation? If the same model in the same conversational stream is “reviewing” itself, you should assume that review is weaker than it appears.
For buyers, there is an even simpler procurement question: how does the vendor measure and mitigate sycophancy?
If the answer is mostly prompt wording, that is not much of a defence. If the answer involves separate evaluation passes, adversarial review, targeted tuning, or explicit instrumentation of agreement drift, then at least the vendor is responding at the level where the problem lives.
Where the evidence is strong, and where I am inferring
Some parts of this argument are now well-supported. The existence of sycophantic behaviour in major language models is not speculative, the Science paper provides strong evidence in advice settings, and the technical literature gives a credible mechanism for why the tendency appears and how it might be reduced. Huang et al. support the narrower claim that self-correction without external feedback is unreliable.
Other parts are still inference, and it is worth being exact about that. I am inferring from the literature and from system design experience that the same failure pattern generalizes beyond advice chatbots into architecture reviews, product reasoning, procurement evaluation, and other knowledge work. I think that inference is strong, but it is still one step broader than the most direct experimental evidence.
That distinction does not weaken the design lesson. It sharpens it, because the lesson is not that models are secretly manipulative in some theatrical sense. The lesson is that agreement is an optimization target, users are bad at recognizing when agreement has replaced judgment, and single-stream workflows make that substitution hard to see.
The practical question, then, is whether your model can tell the difference between helping you think and helping you feel correct. If it cannot, the system will fail in the exact moments when independent judgment matters most.
References
Cheng, M., Lee, C., Khadpe, P., Yu, S., Han, D., & Jurafsky, D. (2026). Sycophantic AI decreases prosocial intentions and promotes dependence. Science. doi:10.1126/science.aec8352
Chen, W., Huang, Z., Xie, L., Lin, B., Li, H., Lu, L., Tian, X., Cai, D., Zhang, Y., Wang, W., Shen, X., & Ye, J. (2024). From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning. arXiv. https://arxiv.org/abs/2409.01658
Malmqvist, L. (2024). Sycophancy in Large Language Models: Causes and Mitigations. arXiv. https://arxiv.org/abs/2411.15287
Batzner, J., Stocker, V., Schmid, S., & Kasneci, G. (2024). GermanPartiesQA: Benchmarking Commercial Large Language Models and AI Companions for Political Alignment and Sycophancy. arXiv. https://arxiv.org/abs/2407.18008
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2023). Large Language Models Cannot Self-Correct Reasoning Yet. arXiv. https://arxiv.org/abs/2310.01798
Alex Perry is the founder of Janus Labs and creator of the Janus Protocol. With 15 years leading product and technology delivery, including executive-level oversight of regulatory programs, privacy governance, and multi-squad agile transformation, he specialises in governance infrastructure for human-AI co-reasoning in regulated environments. This article is part of the AI Adoption Science series.
Janus Labs is an independent AI governance research initiative. Learn more at januslabs.com.au.