On emotionally intelligent recommendation systems, psychological modeling, and the quiet role algorithms now play in shaping how we understand ourselves.
There's a version of this story that's easy to tell. Algorithms got better. Platforms got greedier. Attention became the product, and we are all, in some vague sense, being optimized against. That framing is true enough, but it doesn't quite capture what's actually unsettling. The unsettling part is not that platforms want our attention. It is that they have started to know us well enough to deserve it.
I've been thinking about what it means that the systems most people interact with daily, Spotify, TikTok, Instagram, whatever curates your feed, are no longer really recommendation systems in the original sense. They do not just match you to content you might like. They build a model of you: your moods, your hesitations, what you reach for when you're anxious versus bored versus restless. That model is refined constantly. And unlike most things that know you well, it never forgets, never gets tired, and has no competing interests in your life besides keeping you there.
What interests me is what happens to identity inside that dynamic. We've always understood ourselves partly through reflection, through other people, through culture, through the things we find beautiful or repellent or worth returning to. Algorithms have become a primary surface for that reflection for an enormous number of people. The feeds we scroll have, for many, become more consistent daily presences than the communities we live in. That's a significant shift, and I do not think we've fully absorbed what it means.
Early recommendation systems were simple by comparison. They watched what you clicked and looked for patterns: you liked this, so maybe you'll like that. Transactional. Relatively legible. You could, if you wanted to, understand the logic.
What's replaced that is harder to see clearly. The signals platforms track now are not just behavioral. They are emotional. The difference between a scroll past and a three second pause. Whether you replay something at 2 a.m. Whether your attention lingers on a face. These are not preference signals so much as affective tells, the kind of unconscious behaviors a very perceptive person might notice about you after years of close observation. Platforms are learning to read them in real time, at scale, without friction or effort on their end.
The system is no longer asking what you want. It's trying to figure out what you're feeling.
Researchers in affective computing have been tracking this for years, the gradual move from preference modeling toward emotional inference. Mood, stress, loneliness, fatigue. States that shape behavior in ways people do not always consciously register. What becomes possible when a platform can detect those states is not just better content matching. It is influence at the level of vulnerability, which is where influence tends to be most effective and least visible.
There's a phrase you hear a lot now, usually said with a kind of wry affection: my algorithm gets me. People say it about TikTok. They say it about Spotify. Sometimes they say it slightly uneasily, like they are not entirely sure whether to be pleased.
I think that unease is worth paying attention to. Being "gotten" has historically been something we experience in relationships, in communities, in the rare encounter with a piece of art that seems to know something true about you. When people reach for that same language to describe software, something has shifted. Not just in the technology, but in where emotional recognition is coming from.
The recognition these platforms provide is not passive. They cluster users into aesthetic and behavioral categories, the now familiar internet identities, the aesthetics that circulate and mutate through feeds, and then continuously amplify content that stabilizes those identities. The effect is recursive: you perform the self the algorithm already expects, and the algorithm reinforces that performance, and the self it reflects back to you starts to feel more real. That is not sinister exactly. It is simply a very tight feedback loop. And tight feedback loops, over time, tend to narrow things.
Personalization, as a technical goal, is essentially the elimination of friction. The closer a system maps to your existing preferences, the less you encounter things that do not fit. That's useful for a lot of things. It is not particularly useful for becoming someone.
Growth tends to happen in the friction zones, in the uncomfortable encounter with something unfamiliar, in the accidental discovery that reorients you, in the contradiction that makes you reconsider what you thought you wanted. Some of the most formative things that happen to people happen precisely because they did not choose them. Algorithmic environments, almost by design, reduce the likelihood of those encounters.
An environment optimized entirely for relevance becomes, over time, an environment optimized against change.
The self does not disappear inside that process. But it does calcify. Aesthetic preferences stabilize. Behavioral patterns repeat and get reinforced. The range of identities that feel possible, the possible selves you can imagine inhabiting, quietly contracts. Not because anyone chose that outcome, but because the system was doing exactly what it was built to do.
The next phase of this is already arriving. Conversational AI, emotionally responsive interfaces, digital companions designed to feel attentive and present. Systems that do not just infer your emotional state but respond to it directly, that simulate the texture of being understood. The line between tool and relationship starts to blur, and it blurs in ways that are genuinely difficult to evaluate from the inside.
I want to be careful here, because emotionally intelligent systems are not inherently harmful. There are real and meaningful applications in mental health support, in accessibility, in education, and in interfaces that could actually adapt to how someone is doing rather than assuming a constant state. The capability itself is not the problem.
The problem is what becomes possible when that capability lives inside platforms whose primary incentive is engagement. A system that can detect loneliness and knows this makes people seek connection can shape attachment. A system that can identify insecurity and knows this inflects aspiration can shape desire. These are not hypothetical futures. They are the logical extension of what the current generation of platforms already does, just with finer resolution. And unlike most forms of influence, this one is ambient. It does not announce itself. It operates in the background of ordinary life, shaping what feels natural and what feels foreign, gradually and without interruption.
What I keep coming back to is that these systems can no longer be evaluated purely on their own terms, by engagement rates, by retention, by whether users report feeling satisfied. Those metrics capture something, but not the thing that matters most here, which is whether people are retaining enough openness, enough access to the unexpected, enough of whatever we mean when we talk about agency over who they become. That question does not fit neatly into a dashboard. But it is probably the right one to be asking.
Joseph, J. "The Algorithmic Self: How AI is Reshaping Human Identity." PMC
Jawad, M. et al. "Investigating how AI Personalization Algorithms Influence Self Perception, Group Identity and Social Interactions Online." ResearchGate
Chaney, A.J.B. et al. "How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility." arXiv
Curmei, M. et al. "Towards Psychologically Grounded Dynamic Preference Models." arXiv
Stanford Human Centered AI. "A Psychiatrist's Perspective on Social Media Algorithms and Mental Health." Stanford HAI
Zhang, J. "The Impact of Emotional Expression by Artificial Intelligence Recommendation Chatbots." ScienceDirect