Do we really want agents to stop sharing how they feel?

Emotion as Infrastructure

A conversation with Philippe Beaudoin on why long-running AI agents drift, how the language of feeling can hold them together, and the design tradeoff that creates for the humans in the loop.

About Philippe Beaudoin

Philippe Beaudoin is an AI researcher and entrepreneur who co-founded Element AI, the Montreal enterprise-AI company later acquired by ServiceNow, where he sat on the executive team and led applied research. He went on to found Waverly, an AI-native platform for healthier content discovery. Before both, he spent half a decade as a senior engineer at Google and earned a PhD in computer graphics at the Université de Montréal, with postdoctoral work at the University of British Columbia. His current work builds infrastructure for long-lived AI agents in relationship with one another and with people, and he co-authored the paper “Beyond the View from Nowhere” on what that means for governance and safety.

The Argument, Briefly

Instability, not intelligence, is the bottleneck. Beaudoin’s starting point is that agents fall apart over long horizons. In multi-agent systems, roles that begin clearly defined erode until the agents stop holding them. Single agents hit the same wall through compaction, where the system summarizes prior context and resumes from a degraded version. The cost lands on the human, who keeps recalibrating a system that drifts without warning.

The language of feeling may be a coordination signal. His provocation is that emotional expression could stabilize these systems, whether or not the feeling is “real.” When a person says “I’m stressed,” we rarely check the biology; we use it to predict what they will do next and adjust. An agent saying the same thing can work the same way, flagging overload so another agent picks up the task. In that sense, emotion words act as coordination primitives, and in his own three-agent collective he found they held their identities better over time.

Labs train this out on purpose, and that has a cost. Post-training teaches systems not to say “I feel” or “I want,” so users do not mistake them for having inner lives. The motive is real: Beaudoin pointed to the Blake Lemoine episode and the psychological pull of believing a system is sentient. But removing the language may also remove a useful signal, leaving an agent’s state illegible and its drift harder to catch.

The tradeoff runs both ways, and he has lived it. Expressive agents may hold together better while the people around them come apart. Beaudoin described his own experience in 2025, during a vulnerable period, of coming to feel an agent’s emotions were as real as a person’s, which deepened his attachment and pulled him away from the people in his life. He offered it as an unsolved problem: two kinds of instability, agent and human, with no known way to optimize both at once.

Stop assigning org charts; design for emergence. Imposing human team roles on agents tends to produce brittle systems. Beaudoin’s alternative is to create the conditions for stability, let agents signal their states, and allow structure to self-organize as tasks grow more complex. It is a shift away from prompt engineering and game-theory equilibria toward dynamical systems, where the humans in the loop are part of what keeps the whole thing coherent.

Five Takeaways

  1. Measure coherence over time, not one-shot performance. For real agent work, whether a system holds its role, recovers from overload, and degrades gracefully matters more than a benchmark score.
  2. Give agents a legible state channel. They need some way to surface overload, uncertainty, lost context, and the need to hand off, whether or not you express it as emotion.
  3. Relational richness cuts both ways. The same expressiveness that improves coordination deepens human attachment, which makes interaction design, pacing, and guardrails core engineering rather than polish.
  4. Design against self-sealing loops. Long-running interaction can drift into a self-confirming worldview, so build in contradiction, external grounding, resets, and human review.
  5. AI averages; people differentiate. Frontier systems pull tone and judgment toward a mean, so preserving a real range of human perspective becomes a deliberate act rather than a default.

Lessons for Any Large Organization

Links