From Idea to Production in 72 Hours: What Shipping a Support AI Agent Taught Us About Our Own Product

Every support queue tells the same story if you read enough of it. Ours did. We run SpeedyIndex, a link indexing service for SEO teams, and for months our human support was answering variations of the same fifteen questions: how token refunds work, why a report shows "crawled, not indexed", what the day-7 check means. Real people were spending real hours on answers that already existed in our documentation.

So we built a support AI agent. The part that surprised us was not that it worked — it was how little time the build actually required. From the first planning message to production, three days passed. One week after launch, 57% of incoming questions were resolved by the agent before a ticket ever reached a human, with an average response time of 2.3 seconds (internal data, first week after launch).

This article is not a product announcement. It is a build log, written so you can repeat the process on your own queue — including the parts that did not work.

Day One: Your Support Archive Is Already a Dataset

The biggest misconception about support agents is that you need to write a knowledge base from scratch. You do not. You already have one — it is sitting in your ticket history.

We started by pulling the last few months of resolved conversations and sorting them by intent, not by topic. The distinction matters. "Billing" is a topic. "I was charged and the link is not indexed — where is my refund?" is an intent, and an intent maps directly to one answer the agent can learn.

The sorting produced three piles:

1. Answerable from documentation. Questions whose correct answer already existed in our FAQ, docs or past replies. These became the agent's core knowledge.

2. Answerable from account data. Questions like "what is the status of my task?" that need a lookup, not a paragraph. We left these out of version one — resist the urge to build everything at once.

3. Genuinely human. Disputes, edge cases, angry customers, anything involving judgment. These were explicitly excluded and routed to people.

The lesson from day one: the work is editorial, not technical. We spent most of those hours rewriting answers so each one was self-contained — no "as mentioned above", no links standing in for explanations. An answer the agent serves must survive being read in isolation, because that is exactly how it will be read.

Day Two: Teaching the Agent to Refuse

Day two went almost entirely into rules about what the agent must not do. In hindsight, this was the highest-leverage day of the three.

An agent that answers everything is a liability. Ours has hard refusal rules: it does not promise indexing outcomes (no tool can guarantee what a search engine decides), it does not discuss refunds beyond stating the documented policy, and it does not guess. When confidence is low or the question matches a "genuinely human" pattern from day one, it says so and hands the conversation to support — with the context attached, so the customer never repeats themselves.

We also wrote the escalation phrasing by hand. The default text most frameworks ship — some variant of "I'll connect you with a specialist" — reads as a brush-off when a customer is already frustrated. Ours states plainly: a person will answer, here is what I have passed along, here is what to expect.

The lesson from day two: define failure before you define success. A support agent's reputation is set by its worst answer, not its average one. Every hour spent on refusal rules bought us more trust than any hour spent expanding the knowledge base.

Day Three: Launch — Quietly

We did not announce the agent. We turned it on as the first responder in the existing chat and watched.

The first hours produced a short list of fixes: two answers that were technically correct but confusing without a screenshot, one intent we had missed entirely, one phrasing that sounded fine in documentation and cold in a chat window. All four fixes were edits to text, not code. That pattern held: since launch, maintaining the agent has meant maintaining the writing.

The lesson from day three: launch is the beginning of the editorial loop, not the end of the project. Read the transcripts. The agent's mistakes are a map of what your documentation never explained well in the first place.

The Metric Trap: Why We Do Not Count "Hours Saved"

The obvious way to sell a project like this internally is "hours of support work saved". We deliberately do not use that number, and I would argue you should not either.

"Hours saved" assumes every deflected question would have taken a fixed amount of human time, that support would have handled the same volume at the same pace and that none of the deflected questions would simply have gone unasked. None of those assumptions survives contact with a real queue. The number is unfalsifiable — which is exactly why it looks so good in slide decks.

Two metrics hold up better, and both come straight from logs:

• Deflection rate. The share of incoming questions resolved without a ticket reaching a human. Ours settled at 57% in the first week (internal data). This is a countable event, not an estimate: either a ticket was created or it was not.

• Time to first meaningful answer. Not "first response" — autoresponders made that metric meaningless years ago — but the moment a customer gets an answer that addresses their actual question. For the agent it averages 2.3 seconds (internal data). For human queues, first meaningful answers are typically measured in hours, and industry benchmarks commonly put them at several hours or more (industry estimate — measure your own baseline before quoting anyone's).

There is a third number worth watching: the escalation share — the 43% that does reach people. Ours did not just shrink; it changed composition. Humans now spend their time on disputes, unusual configurations and questions that require judgment. That shift is hard to put in a single number, and it is the most valuable outcome of the whole project.

What Did Not Work

Honesty section, because every build log needs one. The agent still cannot handle questions that mix two intents in one message — those escalate more often than they should. Screenshots from customers are ignored entirely in version one. And our early attempt to let the agent paraphrase documentation freely produced answers that drifted from policy; we pulled that back to tightly edited source texts within the first day.

A 20-Minute Exercise for Your Own Queue

If you want to test whether this is worth three days of your time, you do not need any tools yet. Open your last 50 resolved tickets and sort them into the three piles from day one. If the "answerable from documentation" pile holds more than a third of the total, you have a case. If it holds more than half — as ours did — you are paying people to retype your own FAQ, and 72 hours of focused work can change that.