Building a Production Voice AI Pipeline: WhatsApp, Whisper, and Prompt Caching

The Problem

A recovery-support SaaS client needed their AI sponsor chatbot reachable by voice over WhatsApp, not just text — a real constraint for users who wanted to talk through something difficult without typing it out. That meant a full round trip: receive a voice note, understand it, generate a thoughtful reply, and send that reply back as synthesized speech, all inside a latency budget that still felt like a conversation.

The Pipeline

Twilio's WhatsApp Business API receives the incoming voice note and hands it off. Whisper transcribes it to text. That text goes to Claude along with the conversation's system prompt and history. The reply comes back as text, gets synthesized to speech, and goes out over WhatsApp as a voice reply.

Every one of those hops adds latency, so the part of the system I spent the most time on wasn't any single component — it was making sure the LLM call itself, the most expensive and slowest hop, wasn't also the most wasteful one.

Where Prompt Caching Actually Pays Off

The system prompt for this chatbot is long: personality, safety guardrails, crisis-detection instructions, conversation guidelines. Without caching, that entire block gets re-sent and re-processed as input tokens on every single turn of every conversation — for a product meant to be available 24/7, that adds up fast.

Prompt caching fixes this by letting the model provider cache the processing of a stable prefix and reuse it across calls, so you only pay full input-token price on the first turn. On this project it brought input-token costs down by roughly 90%.

The catch is that caching only helps if the cached prefix is genuinely stable and comes first in the prompt. I structure every request as: system prompt and static instructions first, then conversation history, then the new user turn last. The moment something volatile gets inserted before the stable block, the cache stops matching and you're back to paying full price without realizing it — so I added logging on cache-hit rate specifically, not just on latency or token count, because it's the one metric that silently degrades without showing up anywhere else.

Testing a Pipeline You Can't Easily Eyeball

Voice is hard to spot-check the way you can read a chat transcript. I built a 12-category automated test suite covering the chat logic, session handling, webhook behavior, and security boundaries separately from the audio hops, so a regression in conversation quality wasn't hiding behind "did the audio play correctly."

Lessons Learned

Latency budget forces different tradeoffs than a text chatbot. A response that reads as "fine" when typed out can feel sluggish when someone's waiting for a voice reply, so I ended up caring more about time-to-first-token than I would have for a purely text-based product.

And caching is a design constraint, not a config flag you turn on afterward — the prompt structure has to be built around it from the start, or you end up rewriting the request-assembly logic once you realize the savings aren't materializing.