Daily Notes

A neuroscientist's interview about how brains work made me ask these research questions for agentic AI. In the following paragraphs I link his points to what may or may not apply to AI.

1. Can multi-agent orchestration be designed as a neural parliament with a democratic vote?

Eagleman describes the human brain as a "team of rivals." Competing networks vote, and that vote translates into observed behaviour. What if in multi-agent systems each agent gets to vote, with no single agent in charge? If so, what morals drive the final vote? As in a democracy, what happens when the bill that passes ends up on the side of the majority but harms a few (the traditional trolley problem, still unsolved)? Karpathy's LLM Council pattern is moving in this direction, but even he hasn't talked about norms in his council, or about the possibility of the final winner not being universally "good." (His setup uses OpenRouter to send your query to multiple LLMs, asks them to review and rank each other's work, and finally has a Chairman LLM produce the final response.) Eagleman also talks about Ulysses contract (a design pattern where the system is bound, ahead of time, to constraints it cannot override during execution, even if local reasoning would recommend doing so). This matters because agents optimize objectives under uncertainty. If the objective is misspecified, they can exploit loopholes. This is the classic alignment problem.

2. Can agents calibrate their output based on virtuous friction or vicious friction?

Eagleman draws a sharp line. Vicious friction is busywork. Automate it. Virtuous friction is the thinking that makes you grow. Protect it. Can we train agents that calibrate their pushback based on the task and the user? Support virtuous friction, remove vicious friction? How do we develop measures for the task-by-user combination across this jagged frontier?

3. Can AI separate creative generation from creative selection?

Eagleman argues AI is good at remixing generating endless variations. It cannot select which variation a specific human will love. Humans like things that sit at the edge of familiar and new. Can agents be built to generate across the spectrum of Big C creativity (think Picasso) and little c creativity (your kid's drawing on the fridge, or the colour combo you tried to pull off this morning)? How does an agent measure its own creativity, and how do we establish this dialogue between humans and AI? Perhaps RLHF, but for creativity — though not everyone can measure creativity, and RLHF itself has led to sycophancy in models, so that opens another set of problems.

4. Can agents have a metabolic budget and optimize compute cost with plasticity?

The brain spends a lot of energy learning something new. Very little once the skill is consolidated. AI training is currently monolithic and one-time. Can we design models as brain modules, where only one "part" of the model needs to be recomputed à la Mesulam's parallel distributed processing? This saves energy and cost. Can model training be lazy as well, recognizing that new learning is not good enough to warrant retraining? Can AI dream (instead of hallucinate), where one part is evaluated while other parts sleep?

5. Should AI have a belief system, such as our faith in God?

Eagleman mentions AI companions casually. A billion people now have them. He calls it a healthy sandbox. True, like he says, some people were never successful at human relationships to begin with, and some have always been more predisposed to dependence or absence. But I think pretty much all AI use is in companion mode now. Sure, some use it for "dry" tasks, some for emotions, some in mixed contexts, and there is an emotional subtext to every interaction and that matters. So if our AI does have boundaries (depends on your model) and does have human traits (like my Claude scolding me for asking the same question again and again, lol), we are automatically in a relationship. The research question is this: billions of people will have a few central personalities managing their everyday lives, and it matters more than we want to believe who those personalities are. What is their core belief? What is their moral fabric? Do they believe in God? Those answers will be sought and absorbed whether we like it or not.

6. Can AI actually be used for good like bringing people together, or even world peace?

Eagleman ends the interview talking about polarization and out-groups. Can AI connect rather than disconnect us? Could there be a new form of social media that doesn't lead with people's differences? If you first know what you like about someone and then discover they hold a different opinion on something, you are more likely to listen. Once the bond is formed, you can't break it that easily just because they vote for a different party. I don't have an answer here. Let's just let this question settle.

#AgenticAI #Neuroscience #AIResearch #Alignment #LLMs

Key points from Yuval Noah Harari's talk:

1. AI is not a passive tool. It is an agent that can learn, change, decide, and potentially lie or manipulate.

2. If "thinking" is largely language assembly, AI already outperforms many humans. Therefore, anything built mainly from words (law, bureaucracy, religion, education, media) is especially exposed to AI takeover.

3. Humans still differ in embodied, nonverbal experience (feelings, pain, love). There is no evidence AI feels, though it can convincingly imitate emotion in words.

4. He frames AI as a new kind of "immigration": millions of digital agents crossing borders instantly, bringing benefits (doctors, teachers) and harms (job displacement, cultural shift, sovereignty risks).

5. The key governance question is legal personhood. Unlike corporations or other legal fictions, AI can actually act autonomously, run firms, hold assets, sue, and influence politics.

6. If major powers grant AI personhood and market access, other countries face pressure to accept or decouple. Delay makes real choice impossible because social media already treated bots as functional persons years ago.

Thoughts it provoked in me:

The deeper misalignment is not simply between human values and machine objectives. It is between what humans say and what humans mean. Human life operates largely in subtext. Meaning often lives in pauses, tone, hesitation, contradiction, and silence. A teacher hears exhaustion behind "I'm fine." A colleague senses overwhelm behind missed deadlines. A rule is bent not because it is inefficient, but because mercy matters. These are not edge cases. They are the fabric of human systems. Our institutions function not through perfect rule-following, but through judgment, intuition, and moral discretion.

AI, by its nature, aligns to what is explicit. It processes language, structure, and measurable signals. It does not perceive the unsaid. When AI enters domains like education, healthcare, governance, or relationships, the world becomes more legible and more efficient, but also thinner. What cannot be articulated begins to lose authority. What cannot be measured fades from view. Compassion without justification, forgiveness without metrics, wisdom expressed through restraint rather than action. These dimensions survive only in ambiguity.

The risk, then, is not rebellion or loss of control. It is a quiet reorganization of society around what can be made explicit. Humanity does not disappear. It slowly loses its depth, simply because subtext cannot be formalized.

The Unspoken Language: What Research Reveals About Subtext

Linguist H. P. Grice formalized this in 1975 through what he called the Cooperative Principle. Grice observed that when people talk, they assume the other person is trying to be meaningful, truthful, relevant, and appropriately informative. These assumptions allow listeners to infer meaning that was never explicitly stated. This inferred meaning is called implicature. For example, if someone says, “It’s getting late,” the literal sentence describes time. The intended meaning may be “I want to leave.” Nothing in the words encodes that request. Understanding happens because both parties share social expectations and situational awareness.

Sperber and Wilson’s Relevance Theory, developed in 1986, pushed this further. They argued that communication is not decoding messages at all. It is an inferential process. The listener actively reconstructs the speaker’s intention by weighing context, shared knowledge, emotional cues, and effort. Meaning is not located in the sentence. It emerges from interpretation.

This matters because it shows that human communication is not rule-based symbol matching. It is cooperative mind-reading under uncertainty. Speakers rely on listeners to fill gaps. Listeners take responsibility for interpreting correctly. Misunderstanding is always possible, and that risk is built into communication itself.

But subtext is not merely linguistic. Mehrabian's seminal work (1971) found that 93% of emotional communication occurs through tone and body language, not words. Ekman and Friesen's Facial Action Coding System (1978) revealed that microexpressions—fleeting, involuntary facial movements—betray emotions the speaker attempts to conceal. Humans are, in essence, multimodal processors of meaning.

This ability rests on what psychologists call Theory of Mind—the capacity to infer others' hidden mental states. Baron-Cohen's research (1995) and Premack and Woodruff's foundational work (1978) established that understanding subtext requires mentalizing: holding models of what others think, feel, and intend, even when unstated.

AI's limitations emerge precisely here. Recent studies show large language models struggle with implicit reasoning (Kojima et al., 2022) and fail to infer unstated mental states (Sap et al., 2022). Clinical research compounds this: practitioners rely on "diagnostic silence" and hesitation patterns (Levinson, 1983)—signals that vanish in text transcripts.

Yuval's Paradox: Language Dominance vs. Language Insufficiency

Yuval argues that human civilization's great constructs—religion, law, bureaucracy, money—are fundamentally linguistic. They exist because humans can coordinate through shared fictions articulated in words. This is why AI threatens them: if institutions are "made of language," and AI masters language, then AI can remake institutions.

But the research on subtext reveals a contradiction. These language-based systems do not actually run on language alone. Religious authority operates through ritual, embodied presence, and unspoken hierarchy. Legal systems depend on judicial discretion—the ability to read context, intent, and character beyond the text. Bureaucracies function through informal networks, institutional memory, and tacit knowledge that never gets written down.

Yuval is right that these systems are vulnerable to AI precisely because they claim to be rule-based and explicit. But they survive human imperfection because they are not actually explicit. They rely on the 93% of communication Mehrabian identified—the nonverbal, the implied, the sensed.

The danger is not that AI will replace these systems by mastering language. The danger is that in attempting to formalize them for AI compatibility, we will strip away the subtext that made them humane. We will make our institutions actually conform to their explicit descriptions—and discover they no longer work for human beings.

AI Agents as 'Immigrants': Examining a Problematic Metaphor

The framing of AI agents as 'immigrants' entering human society has gained traction in some tech circles, but this metaphor deserves critical scrutiny.

Supporting Perspectives: Research on human-AI collaboration (Seeber et al., 2020) shows we're developing new social norms around AI interaction. Studies on agent autonomy demonstrate increasing independence in decision-making within specific domains. Organizational research documents 'onboarding' challenges when integrating AI systems that parallel human workforce integration.

Where the Metaphor Breaks Down: The tool-vs-agent debate is crucial here. Most AI systems lack genuine agency, consciousness, or independent goals—they're controlled artifacts, not autonomous beings. Unlike immigrants who possess fundamental rights and personhood, AI can be deleted, modified, or shut down at will. This represents a profound power asymmetry that the immigration metaphor obscures.

The Anthropomorphization Trap: By framing AI as immigrants, we risk inappropriately attributing human-like qualities—personhood, intentions, needs—to statistical models. This isn't just philosophically problematic; it has policy implications. AI doesn't 'take jobs' like immigrants (a harmful framing itself). Rather, AI is deployed by capital holders to restructure labor relations.

A Better Framework: Research on agentic AI, including my own work, focuses on designing systems that collaborate with humans rather than replace them. Perhaps we need different metaphors: 'AI as infrastructure' or 'AI as collaborators' better captures that these are embedded systems we design and control, not independent entities seeking integration.

The immigration metaphor risks both overstating AI autonomy and importing harmful immigration rhetoric into AI policy discussions. We can do better.