5 AI features I owned in the onboarding platform

The throughline

The time lost searching transcripts, the hours writing tickets, the context that died in meeting recordings - these shaped what I built and why. I owned all five features end-to-end. On Ask FDE, Jira ticket creation, and weekly update drafting, I owned prompt design and the user-side behavior (when AI fires, how outputs surface, accept/edit/dismiss controls), built on data-ingestion rails engineering put in place. On meeting matching and spec generation, I owned requirements, testing, and acceptance while engineering owned the build. This case study covers all five. Every feature is instrumented in PostHog so adoption is visible, not assumed; Langfuse handles LLM-trace telemetry for production monitoring.

1 · Ask FDE - RAG-powered workspace Q&A

Onboarding analysts run 5–6 back-to-back calls daily across 10–17 concurrent accounts. Between calls, answering "what's blocking this account right now?" used to mean switching between Circleback transcripts, the spec document, and Jira. Ask FDE is a conversational assistant embedded in each workspace, grounded in three sources - the specification, uploaded meeting transcripts, and synced Jira tickets - using a per-workspace semantic search store. Every answer carries source citations so users see exactly which meeting, spec section, or ticket it came from, and it degrades gracefully to spec-only context before meetings are uploaded.

2 · Jira ticket generation

After spec sign-off, a three-phase AI pipeline parses the spec into categories, generates tickets per category using domain-specific prompts, and persists them as reviewable drafts. Managers approve, edit, or reject individual tickets before a batch export to Jira with parallel execution and relevant data files attached - replacing the manual creation of 30–80 tickets per account.

3 · Weekly update drafting

Writing weekly updates across 10–17 accounts was a top pain point - one senior manager did it on weekends because there was no time during the week. Now an AI draft is generated automatically when a meeting is approved, answering the team's standard five-question format and grounded in the transcript. Three affordances: post as-is, edit, or dismiss. An earlier version over-dramatized issues ("AI will pick up one thing and call it a critical blocker"), so I redesigned the prompts around evidence grounding, not inference.

Usage in production

The AI PM work isn't shipping AI. It's shipping AI that real users actually adopt. Every feature is instrumented in PostHog so I can see whether analysts are reaching for it, and which ones stuck.

Combined usage across the three features I shipped grew ~51x in 6 months: from 7 events in January 2026 to 357 by 06/26/26, with 4 days still to come in the month. Monthly active analysts on these features went from 1 to 32 over the same window.

Feature	First month → latest month	Peak unique analysts
Ask FDE (RAG Q&A)	2 events (January) → 308 (April)	31
Jira tickets	Per-distributor cadence; 33 ticket-batch runs across 7 managers in last 90 days	7
Weekly updates	24 events (May, launched 05/18/26) → 245 (June)	9

Ask FDE and weekly updates are now part of the analyst team's daily workflow. Jira ticket generation fires per new distributor, which is the right cadence for what it does.

Evals: calibrating the eval before trusting the numbers

I wrote an LLM-as-Judge eval script for Ask FDE using Claude Code, informed by eval methodology I'd studied. The judge was Claude Haiku 4.5, scoring production traces on two dimensions - Grounding (are factual claims traceable to retrieved tool outputs?) and Relevance (does the response answer the question?) - with targets of 88% and 92% respectively.

The first run returned suspicious numbers: 46.7% grounding, 99% relevance. High relevance with low grounding is a red flag - it means the model is answering confidently from sources it doesn't have. I investigated and found two methodology bugs that made the numbers uninterpretable.

Bug 1: Context truncation. The eval was capping total tool call output at 12,000 chars before passing it to the judge. Ask FDE routinely makes 20+ tool calls per trace; actual retrieved content ran 60,000+ chars. The judge was seeing 3 of 23 tool calls and scoring everything else as unsupported. Traces with rich retrieval were being unfairly penalized.

Bug 2: Temperature not set. The judge was running at default temperature (~1.0), making scores non-reproducible. The same input produced scores swinging 90 points across runs. The 46.7% grounding and 99% relevance numbers from the first run were both artifacts of the methodology, not the feature.

I fixed both bugs, re-ran the April traces to get a valid baseline, then ran June traces to check for drift. The calibrated results told a different story:

Period	Grounding	Relevance	Traces
April 2026	75.0%	90.0%	30
June 2026	60.0%	81.0%	30
Change	-15.0 pts	-9.0 pts	-

A 15-point grounding regression, surfaced cold - before it appeared in any user feedback. I traced it to three root causes: a new usage pattern (analysts asking Ask FDE to draft emails and Slack messages - responses that sound right but invent action items not in any meeting); a retrieval strategy issue (the model calling search instead of read, then authoring detailed summaries beyond what the excerpts contained); and persistent product knowledge hallucination when retrieved context is thin. All three are fixable in the system prompt and tool call behavior without a model change.

I ran this once. The priority then shifted to shipping the remaining features and maintaining PostHog instrumentation. But the eval is calibrated and the script exists - running it again is a single command.

Time returned per analyst

Cadence	Feature	Saved
Daily	Ask FDE	~20–30 min/day
Weekly	Weekly update draft	~50–55 min/week
Per distributor	Jira ticket generation	~2 hrs each

Recurring: ~2.5–3.5 hours/week per analyst from the three features I shipped. Per new distributor: ~2 hours on ticket generation. Time that was previously absorbed into evenings and weekends.

Stack