AI Voice Agent · Personal Project
A real phone call. Answered by AI.
Built a production AI voice receptionist that answers live phone calls, books appointments through natural conversation, and streams everything to a real-time doctor dashboard — in one session.
What I built
Someone calls a real Twilio number. An AI answers, understands natural speech, books appointments by asking for name, pet, and preferred time — then confirms out loud.
A doctor's dashboard updates in real time: live call status, full transcript as it happens, appointments logged the moment the AI confirms them. No simulation. No mock data.
Three decisions that defined the build
The OpenAI Agents SDK didn't export the Twilio transport at the package level. I reverse-engineered the dist bundle, found OpenAIRealtimeWebSocket in the exports, and wired bidirectional audio manually using the raw WebSocket protocol. Zero documentation. Shipped in 20 minutes.
The doctor's dashboard needed live call state — not polling, not refresh. I built a WebSocket broadcast layer inside the same Fastify server: every call event fans out to all connected dashboard clients instantly. One process. Multiple live clients.
No NLP library. When the call ends, the system reads the full transcript for patterns — name, pet name, time — then falls back to reading the AI's own confirmation to extract what it just scheduled. If the AI confirmed it, the dashboard shows it.
The dashboard
When a call comes in, the topbar flips to Live with a timer. Every sentence the AI or caller speaks appears in the transcript panel in real time. When the AI confirms an appointment, it appears in the list with name, pet, date, and time — before they even hang up.
Vue 3 + Pinia. PrimeVue components. Click any appointment — drawer opens with the full booking conversation that led to it.
Full stack
Backend
Frontend
Future upgrades
Call the number. The AI picks up.
Twilio trial account — you'll hear a 3-second notice, then the AI takes over. Real PSTN. Real AI. Real appointments.
"I don't just consume tutorials.
I break them, fix them,
and ship something real."