Signals is an LLM-as-judge eval system.

A customer describes what they want to measure in plain language, and conversations are labeled and scored against it.

No eval experience required.

Before Signals, customers had to leave Gladly to understand how their AI Agents were performing.

Now they stay in Gladly, and use what they find to decide what to automate next. 

Gladly's own teams use Signals too. The first internal Signal separated sales conversations from support, replacing an estimate with data the company could use to inform a major product bet.

Dark solid lines indicate decisions I made. Dashed lines represent features that we cut.

Zooming out. This is Signals end to end.
A dashboard, two wizards, a criteria library, a detail page and a run page.

I led design on all of it, across two product versions and a mid-project pricing change.

Try it! Might be better in a new tab though. PS Desktop only.

The product.
Open the prototype in a new tab to play →

Cost estimates changed based on number of sessions.

Customer facing evals were a critical product gap.

The first hypothesis. Customers would pay to run evals, so they needed visibility and control over their spend.

Everything followed from that assumption.

• Scope controls let customers narrow an eval to a subset of conversations instead of running it across everything.
• An estimate show the expected cost before a run.

Manual vs LLM generated criteria.

It was assumed CX leads would write the evaluation criteria themselves.

The first version gave them maximum control: they defined the criteria, label set, and boundaries.

When we talked to customers, they told us the hard part wasn't creating an eval. It was trusting that they created a good one. 

Without confidence in the results, we had more than a UX problem. We had a product and business problem.

CX managers describe what they want to measure. Gladly generates the criteria and labels. They edit anything that looks wrong.

I pushed the team to make LLM generation the default path for both criteria and Signals. 

Rather than starting with a blank form, CX leads would start with AI-generated criteria they could review, refine, and approve before running at scale.

The design decision was to shift the burden of creating a good eval from the customer to Gladly, while keeping customers in control of the final result.

Designing with AI shaped the product, too.

I used AI to prototype directly in the codebase, letting me work through real interactions quickly instead of stopping at static screens.

Working this way reinforced something I'd experienced myself: when you're navigating something unfamiliar, AI is often more useful when it gives you a strong starting point to react to rather than a blank slate.

That helped shape my push to make generation the default. CX leads didn't need to become eval experts. They needed something credible to start from, then the control to make it theirs.

I used VS Code to prototype in the code base.

The pivot.
Partway through, the revenue model changed.

Signals would be free, and we wanted customers running evals across all of their conversations.

That fundamentally changed how we thought about the design. Cost didn’t disappear. It moved to Gladly, where the person spending it had no reason to think about it.

We wanted customers to use Signals broadly, but we also didn’t want them generating thousands of low-value evaluations at our expense.

The challenge became: How do we make Signals easy to use at scale while minimizing waste?

This measures the LLM judge against human ground truth, ensuring customers were measuring what they intended.

Before a Signal could run at scale, it had to be accurate.

I proposed an agreement gate before publishing: customers review 25 AI-labeled conversations and mark whether they agree or disagree with each result.

At 85%+ agreement, the Signal clears the bar for publishing.

Below that, we show what's driving the disagreements and what to change. The test isn't just a pass/fail check. It helps customers improve the Signal before letting it run across thousands of conversations.

Customers called Signals one of the most innovative products Gladly was building.
The product also turned one of our biggest skeptics into a referenceable customer.

It closed a key competitive gap, strengthening Gladly’s position against other AI companies.

Internally, Signals also gave teams new ways to answer questions from conversation data — including replacing an estimated sales-vs-support split with actual data to inform a major product bet.

Signals home

Signals details

Signals run

Where I'd take Signals next

Ground the judge in Guides and knowledge.
Without the procedure or source of truth, a Signal can measure how a customer felt, but not whether the agent followed the right process.

Connect Signals to performance reporting.
Tag Signals and criteria so results can roll into the dashboards CX leads already use. Today, Signal results live on individual run pages, disconnected from the rest of their reporting.

Decide whether Signals should measure human agents, too.
Every customer asked for this. The opportunity is clear, but expanding Signals beyond AI agents would require weighing the value against other priorities for the team platform.