Skip to content

Case study · Custom AI Systems

A product audit that surfaced six hidden issues

The company used their own product every day. A first-time user hits things the team has long since stopped seeing.

Verified

6issues surfaced

Client
a SaaS product company
Problem
A team too close to its own product to see what a first-time user hits.
Approach
Use the company's product like a new customer for a day, then rank what surfaces.
Verified result
6 issues surfaced

The problem

The company knew their product worked. They used it every day, which is exactly why some of its rough edges had gone invisible: the team had built those paths and learned to walk them without noticing.

The gap is not effort, it is proximity. A first-time user meets the product with none of that context, and hits friction the team has stopped registering.

The reality behind the problem

The obvious fix is to ask the team to look harder at their own product. It does not work, because the thing hiding the issues is not a lack of attention, it is the attention itself: the team has walked these paths so many times that the friction has stopped registering as friction.

A fresh pair of eyes inside the team runs into the same wall within days. Proximity accumulates fast, and once someone knows where a path bends, they stop feeling the bend. The only reliable way to meet the product the way a first-time user does is to come to it with none of that accumulated context.

What was assumed

The product is fine, we use it every day.

What Stallwart asked

What does a first-time user hit that the team has stopped seeing?

The desired outcome

The goal was not a list of opinions about the product. It was a set of findings the team could act on, in order, without having to argue first about whether each one was real.

  • Every finding met the way a first-time user meets it, not the way the team remembers building it.
  • Each issue named specifically, so it points at what to fix rather than that something feels off.
  • The findings ranked, so the team knows where to start rather than guessing.
  • Each item actionable on its own, without a follow-up to work out what it means.
  • Nothing the team already knew and had chosen to live with dressed up as a discovery.

The system we designed

Stallwart spent a day using the company's product the way a first-time customer would, with none of the team's built-in knowledge. What surfaced became a ranked, specific report: six usability issues, each one actionable rather than a vague note that something could be better.

How it works

  1. 01

    Use it like a first-time customer

    Approach the product with no inherited context, following the paths a new user would, so the friction a new user meets shows up instead of being walked past.

  2. 02

    Record what the team has stopped seeing

    Note each point of friction as it happens, the ones built past and no longer registered, before familiarity explains them away.

  3. 03

    Rank it and make it actionable

    Turn the findings into a ranked, specific list. Each item names what to fix, not that something could be better, so the team can act in order.

The findings

Six issues the team had stopped seeing.

6issues surfaced

A day spent using the company's product like a first-time customer surfaced six usability issues the team had built past and stopped noticing. The report was ranked and specific, and every item was actionable.

Architecture

What sits underneath.

The audit was structured as a pass from the outside in, each layer feeding the next, so that what a new user hits becomes something the team can order and act on.

01

The first-time pass

The product is used the way a new customer would use it, following the paths a new user takes with none of the team's inherited context. This is the layer that sees what proximity hides.

02

The friction log

Each point of friction is recorded as it happens, before familiarity has a chance to explain it away. The log captures what surfaced, not a tidied-up recollection of it.

03

Ranking and judgment

The logged findings are ordered by what matters most, so the report tells the team where to start. The ranking is argued rather than asserted, since order is a judgment call.

04

The actionable report

Each finding is written to name the exact issue and what it affects, so an item can be picked up and fixed without a round of interpretation first.

The hard parts

Where the engineering judgment was.

01

Audit from the outside, not the team's seat

The team cannot un-see what it built. A first-time pass is the only way to meet the product as a user does.

The trade-off

An outside pass misses internal context, which is the point: that context is what hid the issues.

02

Rank the findings, do not just list them

An unranked list leaves the team guessing what matters. Order tells them where to start.

The trade-off

Ranking is a judgment call, so it is argued, not just asserted.

03

Be specific, never 'this could be better'

A vague note is not actionable. Naming the exact issue is what lets the team fix it.

The trade-off

Specific findings take longer to write than general impressions, and are worth more.

Reliability & guardrails

How it avoids the wrong call.

An audit is only worth acting on if the findings hold up. The approach is built to keep them honest.

Findings grounded in what surfaced

Every issue traces back to a specific moment in the first-time pass, not a general impression. A finding that cannot be pointed at is not reported.

Specific, not vague

Each item names the exact issue rather than noting that something could be better. A vague note cannot be acted on, and cannot be checked against the product either.

Ranking argued, not asserted

Order is a judgment call, so the reasoning behind the ranking is shown. The team can disagree with a position and still see why it was placed there.

No padding

Things the team already knew and had chosen to live with are not re-reported as discoveries. The report stays short enough to act on and honest about what is new.

Productionization

What turns it from a demo into software.

What turns a day of using the product into something the team can work from.

A ranked, written report

The findings are handed over as a specific, ordered document rather than a conversation, so the team can work through them in sequence and track what is done.

Each item actionable

Every finding names what to change, so it can go straight into the team's own backlog without a translation step.

Severity made visible

The ranking makes the relative weight of each issue explicit, so the team spends its effort on what matters first instead of the easiest thing to fix.

Reproducible from the outside

Because each finding is tied to a path a first-time user takes, the team can walk the same path and see the issue for themselves before and after a fix.

What changed

The team got back a view of their own product that they could no longer produce from the inside: the friction a first-time customer meets, named and ordered. The issues had not been hidden by a lack of care, they had been worn smooth by daily use, and seeing them again was the point.

What changed was not just the list but the footing. Instead of a vague sense that the product could be smoother somewhere, the team had specific, ranked items they could take in order, each one pointing at what to fix rather than that something felt off.

  • From a product the team knew worked to one they could see the way a new customer does.
  • From a general sense that something could be better to specific, named issues.
  • From guessing where to start to a ranked order to work through.
  • From impressions that fade to a written record the team can act on and check against.

In practice

Founder, a SaaS product company

The company's founder described Stallwart spending a day using the product like a first-time customer and surfacing six issues the team had stopped seeing because they had built them. The report was ranked and specific, with every item actionable rather than a vague note that something could be better.

Ownership & handover

What you receive, and keep.

The audit is yours to act on, with nothing held back.

  • The full ranked report of the usability issues that surfaced.
  • The specific path to reproduce each finding as a first-time user would.
  • The reasoning behind the ranking, so the order can be revisited as priorities change.
  • A clear statement of what each finding affects, ready to drop into your own backlog.
  • The findings as your own record, with no dependency on Stallwart to interpret or act on them.

What we learned

The insight that generalizes: the hardest problems to see in your own product are the ones you have learned to walk past, and no amount of looking harder from the inside recovers that first-time view. The value of an audit is not the hours spent but the context deliberately left behind. Meet the product the way a new user does, write down what surfaces before it explains itself away, and the issues that had gone invisible become a list you can act on in order.

This work connects to

  • Custom AI Systems
  • Product and UX audit
  • Production AI systems

Frequently asked

What is an AI product audit?

It is a structured review of a product carried out as a first-time user would experience it, surfacing usability issues the team has stopped seeing. The output is a ranked, specific, actionable list rather than general impressions.

Why can't the team find these issues themselves?

Because they built the product and use it daily. That proximity is what hides the issues: the team learned the rough paths and stopped registering them. A first-time pass meets the product without that context.

What does the report actually contain?

A ranked list of specific usability issues, each one actionable. For the company that was six issues, ordered by what to address first, with each item naming the problem rather than noting that something could be better.

How long does an audit like this take?

The audit was a day spent using the product like a new customer, followed by ranking and writing up what surfaced. The value is in the fresh perspective, not the hours.

What does a first-time user hit that your team has stopped seeing?

We use your product like a new customer and hand you a ranked, specific, actionable report.

Last updated: October 7, 2026