AI Session Review

This is a story about designing for a product that keeps changing. 

To improve AI Agents, you have to read conversations. When an Agent does something unexpected, AI Session Review is where teams go to read conversations. 

AI Session Review brings the customer conversation and Agent trace together in one place. Helping teams understand what happened and why.

Inspect, the trace component of the AI Session reviewer, has evolved alongside the Agent itself.

V1 simplified the complexity.
↓
V2 exposed more of it.
↓
Just when we thought we had Inspect in a good spot, the underlying agent architecture changed. So now we're on to V3.

If there's anything I've learned from designing AI products, it's that the technology, our understanding of it, and what users need change fast.

Gladly is built on the belief that customers should be able to manage their own AI Agents. 

For that to work, Inspect needed to do more than expose information. It needed to help customers understand how their Agent thinks and makes decisions.

And if it looks so technical their eyes glaze over, we've failed.

V1 — Hide the complexity.

Our first approach was to use narrative to make AI behavior understandable without exposing the technical system underneath.

The conversation stayed primary. Expanding a turn revealed a simple, human-readable explanation of what happened.

But as our internal AI teams used Session Reviewer to investigate Agent behavior, we learned we'd abstracted away too much. They were leaving our product for LangSmith to access the data they needed.

They didn't need us to explain what happened. They needed to see what actually happened.

Example of a technical trace.

Try it! Example of the same technical trace in Gladly.

Stay in the conversation. Inspect the trace without losing the customer interaction that prompted the investigation.

V2 — Bring the trace into Gladly.

People managing AI Agents needed the actual trace: what ran, what went in, what came out, and how those components led to the response.

They could get that information from developer tools like LangSmith, but that meant leaving the customer conversation and interpreting the raw trace somewhere else.

We brought that depth into Inspect.

The goal wasn't to simplify the system anymore. It was to make the complexity navigable. 

See the algorithms that ran and how they contributed to the response. Get into the technical detail when you need it.

Just when we had Inspect in a good spot, the agent harness changed.

The next ask sounded straightforward. Make Inspect easier to discover and sessions faster to review.

I prototyped a few options to share ahead of a conversation with our AI research lead. That conversation surfaced something much bigger: we were trying to make the V2 Inspector fit a fundamentally different Agent architecture.

Inspect still needed to answer the question, "Why did my Agent do that?" But the information someone needed to answer that question had changed.

The difference between how the algorithm works in v2 compared to v3.

Same question. Different model.

V2 showed the sequence of algorithms that ran: what went in, what came out, and how each step contributed to the response.

But the new agent harness worked differently. Rather than following a sequence of steps, it accumulated context throughout the conversation: instructions, tools, results, and memory.

Our design needed to reflect that shift. It was no longer just about showing what the Agent did, but what it knew when it responded.

Try it! The trace can get more sophisticated without asking the customer to become more technical.

Represent what the Agent knows.

The new model changed what Inspect needed to show.

Instead of a sequence of algorithms, we needed to represent the context the Agent had available at each turn — and how that context accumulated over the conversation.

That led to a simple distinction: New this turn / Carried over

Working on AI products requires understanding the system deeply to represent it clearly for someone else.

But the system you're designing for today may not be the system you're designing for next month.

V1 wasn't wrong. Neither was V2.
Each reflected how the Agent worked, and what customers needed to understand about it, at that moment.

Get deep enough into the system to represent it clearly, while embracing the reality that today's solution is going to change.
Maybe sooner than you think.