A voice agent that books appointments is a system where someone talks, a model understands what they are asking for, calls real tools, and hangs up with an appointment on a calendar. I built one in a lab repo: ElevenLabs Agents handles the voice—transcription, turn-taking, synthesis—Cal.com handles the calendar, and in between sits a Fastify backend whose only job is to make sure the model never has to guess. Not what day it is, not which time the chosen option corresponds to, not how that time is written in ISO with an offset. This post is the whole walkthrough: the architecture, the three tools, the prompt, and how I tested it without burning the 15 voice minutes the free plan gives you each month. The code is all on GitHub. TL;DR The backend hands the agent at most three options already phrased for speech, and booking happens with an optionId (opt_1), never with a date the model writes itself. Booking is the only irreversible action: it requires explicit confirmation, blocks interruptions while it runs, and is idempotent per conversation. Tool errors come back as HTTP 200 with a sentence the agent can read out loud. A 5xx would leave the model improvising in the middle of a call. What a voice agent that books appointments actually solves The task is concrete and narrow: someone opens a page, talks to an assistant in Spanish, and hangs up with a confirmed 30-minute appointment on the business calendar. It is not a chatbot that answers questions, and not a general assistant. It does one thing, and that is exactly what makes it buildable. The system has four pieces and one direction: Browser (ElevenLabs SDK) │ WebRTC ▼ ElevenLabs Agents transcription · turn-taking · LLM · speech synthesis │ webhook tools (HTTPS + bearer) ▼ Backend (Fastify + TypeScript) │ ▼ Cal.com API v2 │ ▼ Google Calendar ElevenLabs never sees the Cal.com API key and never assembles a booking payload. The backend is the only component that talks to the calendar, and that boundary is what makes everything else reasonable. The conversation follows a fixed order, written into the prompt as ten numbered steps: greet, pin down a concrete date, check availability, read the options, wait for a choice, ask for name and email separately, put the details on screen and spell the email back, wait for an explicit yes, book, say goodbye. Every step exists because the previous one can go wrong. Why there is a backend and not a proxy The first temptation is to point the agent's tools straight at Cal.com. It works in the demo and breaks on the second call. The reason a backend sits in between is that every responsibility you take away from the model is an entire class of errors that stops existing. Responsibility Where it lives What it prevents Date and timezone arithmetic src/lib/time.ts, with frozen-clock tests The model computing offsets or formatting ISO Choosing which times to offer The backend, after querying Cal.com The agent reading thirty slots out loud Phrasing how each time sounds The backend, in Spanish The model improvising "a las 14:00" Booking idempotency One key per conversation Two appointments from one repeated confirmation Payload validation Zod, before anything reaches Cal.com A malformed field landing on the calendar Traceability One structured log line per call Not being able to reconstruct a call that went wrong Dates are the clearest case. The agent sends 2026-09-08 and afternoon. The backend resolves timezone, offset and format. No other part of the project converts dates, and that rule is written into the repo's CLAUDE.md so it is still true six months from now. One detail only shows up if you read the Cal.com API instead of assuming it: GET /v2/slots interprets start and end as UTC, but groups the response keys by the timeZone you requested. And each endpoint requires a different cal-api-version—2024-09-04 for slots, 2026-02-25 for bookings. Sending the wrong one does not produce a clear error. Keep reading That is the first half. The full walkthrough — with the rest of the implementation, the trade-offs and the things that only show up in production — is on my blog: Read the full post on ramonchancay.me → Originally published at www.ramonchancay.me/blog/voice-agent-books-appointments-elevenlabs-calcom.