The Missing Piece

taking AI from useful to delightful

2025 was a crazy year for AI. So, so much progress. I can’t help but (begrudgingly) agree with those cringey “progress is accelerating” posts. It is. The models are getting really smart really quickly. The better these models get, the more I use them. And the more I use them, the more I see something is missing. Yes, these systems are incredibly useful. But they’re not delightful. To get there, we need targeted investment in some neglected research areas.

Useful & Delightful

AI is becoming increasingly more autonomous. These systems can go off and do more and more on their own without any human supervision.

For complex, multi-step tasks, some level of autonomy is necessary. But that’s only 1 piece of the puzzle. Working with AI should feel collaborative, not transactional. Collaboration, in turn, depends on (at least) three core capabilities:

1. Autonomy. The system should be able to carry out meaningful tasks on its own

2. Personalization. The system should recognize that different people want different things and adapt their behavior and style to these preferences

3. Humility. The system should know what it doesn’t know and react fluidly to new information, including mid-task changes, corrections, or interruptions

The big players seem pretty focused on autonomy. This makes sense; scaling RL shows no signs of slowing down, and I’m trying to mentally prepare myself for the crazy long-horizon tasks Claude 5 is going to be able to do.

We should build systems that boost both productivity and fulfillment

That said, if someone were to invest serious research effort and compute into personalization and humility, they could build a highly collaborative AI system. Instead of replacing people, this system would empower them to solve harder and harder problems.

Personalization

Personalization is an evergreen problem in tech. From tweaking app icon colors to asking ChatGPT to be cheeky without being cringey, people want technology to reflect their own tastes. That desire is fundamentally human, and we should build systems that can accommodate it.

Personalization options in ChatGPT

Claude Code and ChatGPT¹ take a simple approach to personalization: let the user describe what they want in natural language. Similarly, allow the model to choose when to “remember” something, where memories are stored as natural language.

I like this approach. It simplifies deployment: serving one set of weights is much easier than figuring out e.g. multi-tenant LoRA’s at scale². It’s interpretable: users can directly see and understand how the system is being personalized for them and make changes if needed.

But there’s a pitfall here. This system assumes:

1. The user knows what they want

2. The user can fully articulate what they want

3. The user can fully articulate what they want in a way the model understands

That last bit is especially important: just saying “don’t be cringey” might not curb ChatGPT’s emoji use, even though I feel like it should; not everyone finds emojis in every response cringey.

Natural language is useful because it’s fuzzy and flexible. But this flexibility introduces challenges for user-elicited natural language feedback. The interpretability of natural language can be deceiving: the system knows I want X, but what does X mean to the system?

To that end, a human-centric system should be curious about its users. It might sacrifice getting something done ASAP to learn more about their preferences. Classical RL literature calls this the explore vs. exploit tradeoff. While RL for LMs has completely taken off in the last year, very little work treats an entire lifespan of interactions between a person and an AI system as a single trajectory³. I don’t know what the right solution is here. But the current paradigm of user-defined natural language preferences leaves a lot on the table⁴

Humility

A collaborative system needs to be humble; it should know what it doesn’t know⁵. The system needs to fluidly respond to interruptions and new information. Contrast this with a system which thinks on its own for a very long time, makes a grand plan, and tries to complete a task in one shot.

A humble system isn’t scared of making mistakes. It should happily try things out and accept feedback from its user. There’s something deeply frustrating and somewhat unsettling about how badly Claude wants to avoid messing up.

Current training regimes optimize for single conversations. If a model fails to solve your problem, it gets zero reward. But a system optimizing over a lifespan of interactions has different incentives. Failing on a hard problem today might surface information that makes tomorrow’s conversation more productive. A willingness to try, fail, and learn across conversations is how really hard problems get solved.

“Tries things out and isn’t afraid to make mistakes” is a weird way to describe an AI assistant, but a perfectly normal way to describe your favorite coworker. They’re not afraid of being wrong; they know it’s part of getting things right, together. Training a model to be ok with being wrong is a bit counterintuitive. But it’s a necessary piece of a delightful, collaborative partner, AI or human.

You & Me & AI

Let me be clear: modern AI systems are incredibly useful. It’s a funny thing to say about such a new technology, but I can’t imagine doing my job without the AI tools I use today. It’ll be cool to see how useful fully autonomous systems will get.

But if this technology is going to be as pervasive as it seems, we shouldn’t settle for useful. We should be building systems that are collaborative, human-centric, and genuinely delightful to work with—not just impressive in isolation.

Our hardest problems aren’t solved by lone geniuses. They’re solved by teams of people working together. I think the same will be true for AI. Cancer and fusion won’t be solved by a model thinking on its own, but by teams of people working together with AI. I’m excited to build systems that enable this. Systems at the intersection of humans& AI.

If that excites you too, I’d love to chat.

1. I’m talking about claude.md, agents.md, skills, ChatGPT personalization, and just about every other memory mechanism in production today

2. Although there’s a good amount of nice systems work here, like S-LoRA and Punica. It also seems like some of this research has made its way into modern inference engines like vllm and sglang

3. The relevant buzzword here is “continual learning”. I should mention that RL’ing a big model at super long context lengths is pretty hard: consider just the size of the KV cache during rollouts, for example. Popular tasks like coding agents have been an excellent test bed to get the training fundamentals + systems-level work down.

4. “But wait!” you, the astute reader, interjects: “you mentioned that the model can edit the memory/personalization files. Can’t you solve the issues you described by making the models really good at this? The user wouldn’t need to understand how to phrase things to the model!” That’s a good point, astute reader. While I’m not fully convinced we need something weight-based here, I do think really nuanced preferences could benefit from weight-based updates rather than natural language. Over long horizons, it’s not clear to me if natural language or weight-based preferences will win out. The former suffers from bloat and context limits (or bottlenecked by retrieval). The latter from forgetting. I really liked this tweet arguing for the usefulness of weight-based personalization/continual learning: “maybe it's better to absorb [preference information] in you rather than looking up how to behave every time you need to interact with someone”. Ok this footnote is getting outta hand sorry

5. This isn’t a new idea. ML literature calls this calibration. A calibrated model knows how confident it is on any given inference. There are formal definitions of calibration that come with an epsilon and a delta. I’m not arguing that AI systems need calibration in the formal sense. But they should be able to tell when a problem is readily solvable and when it’s more worth it to explore and gather new information. This is a related but different idea to what Nathan Lambert calls “thinking calibration” in this blog