WhatsApp voice notes: automatic transcription + AI replying in text

In Brazil, 40-55% of all messages customers send to a business are voice notes (US and EU sit at 5-15%, but the trend is rising). Your rep is in a meeting and can't sit through eight 2-minute voice notes back to back. The customer waits, then leaves. This post shows how to set up automatic transcription (Whisper / Gemini) plus an AI that understands and replies, with real costs (~$0.0002 per minute of audio), response times under 5 seconds, and the 3 mistakes that destroy this feature in practice.

The problem nobody measures correctly

Open your WhatsApp dashboard. Count how many of the last 100 inbound messages are voice notes. Likely 40-55% if your audience is Latin American (US/EU is 5-15%). Then ask: does the team listen to all of them? Honest answer: no. Voice piles up, the rep procrastinates, the customer gives up.

What the bot needs to do with audio

1. Download the audio from Meta's server (link expires in <5 min — has to be immediate).
2. Transcribe to text in about 3-5 seconds.
3. Understand the intent (same AI that handles text).
4. Reply in short text, quoting what it heard if the case is ambiguous.
5. Save the transcript to history so a human rep can read it later.

Production-ready tech stack

Option 1: OpenAI Whisper API (most popular)

• Cost: $0.006 per minute (~$0.009 for a typical 90s voice note).
• Latency: 1-3s for <30s audio, 3-6s for 1-2 min.
• Quality: excellent across English, Spanish, Portuguese — handles slang, accents, moderate noise.
• Limits: 25MB per file (covers up to ~25 min — enough for 99% of cases).

Option 2: Gemini Audio (Google)

• Cost: $0.0003 per minute (10× cheaper than Whisper).
• Latency: 2-4s.
• Quality: good, slightly weaker on regional slang.
• Edge: understands audio + replies directly (transcribe + process in a single call).

Option 3: Self-host Whisper (open source)

• Cost: server only (CPU/GPU). About $0.0002 per minute on CPU, $0.0001 on GPU.
• Latency: 5-15s on CPU, 1-3s on GPU.
• Edge: data never leaves your infra (HIPAA / GDPR-sensitive).
• Downside: keeping the model updated and the infra healthy is real work.

Implementation on Cloudflare Worker + Whisper

// Cloudflare Worker — receives WhatsApp webhook
export default {
  async fetch(req, env) {
    const body = await req.json();
    const msg = body.entry[0].changes[0].value.messages?.[0];
    if (msg?.type !== 'audio') return new Response('ok');

    // 1. Get the audio URL (link expires in 5 min!)
    const mediaId = msg.audio.id;
    const mediaInfo = await fetch(
      `https://graph.facebook.com/v18.0/${mediaId}`,
      {headers: {Authorization: `Bearer ${env.META_TOKEN}`}}
    ).then(r => r.json());

    // 2. Download the audio (ogg/opus)
    const audioBlob = await fetch(mediaInfo.url, {
      headers: {Authorization: `Bearer ${env.META_TOKEN}`}
    }).then(r => r.blob());

    // 3. Transcribe via Whisper
    const formData = new FormData();
    formData.append('file', audioBlob, 'audio.ogg');
    formData.append('model', 'whisper-1');
    formData.append('language', 'en');

    const transcript = await fetch('https://api.openai.com/v1/audio/transcriptions', {
      method: 'POST',
      headers: {Authorization: `Bearer ${env.OPENAI_KEY}`},
      body: formData
    }).then(r => r.json());

    // 4. Save to history + process as a text message
    await saveTranscript(msg.from, transcript.text);
    await processMessage({from: msg.from, text: transcript.text, source: 'audio'});

    return new Response('ok');
  }
}

Total time: customer sends audio → bot replies

How the bot should reply to audio (good UX)

✅ Good — confirms understanding implicitly

Customer: [audio: "I wanted to know the price for a keratin
treatment on long hair"]
Bot: "Keratin treatment on long hair: $44 (single session).
Want to book? Any preferred day?"

❌ Bad — asks them to retype it

Customer: [audio]
Bot: "I don't listen to audio. Please type it out."

The customer loses patience. Real metric: 35% don't reply after that "I don't listen" line.

✅ Great — quotes back only when ambiguous

Customer: [noisy audio + 2 questions blended together]
Bot: "Just to confirm: you want the keratin treatment price
AND availability on Saturday, right? If so: keratin is $44,
Saturday has 2 PM and 4 PM open."

Real-world costs (5,000 voice notes/month)

• Whisper API: 5,000 × 90s = 7,500 min × $0.006 = $45/month.
• Gemini Audio: 5,000 × 90s = 7,500 min × $0.0003 = $2.25/month.
• Self-hosted: $20-60/month server (fixed cost, only worth it past 10k voice notes/month).

Real recommendation: start with Whisper (quality > cost at this scale). Migrate to Gemini Audio once you cross 20k voice notes/month.

3 mistakes that destroy this feature

1. Not downloading immediately. Meta's link expires in 5 min. If the webhook lags, the audio is gone. You have to process synchronously.
2. Not handling transcription errors. Whisper fails ~2% of the time (clipped audio, extreme noise). The bot needs a fallback: "I couldn't hear that clearly. Mind trying again or typing it out?"
3. Bot processing its own audio. If you also send voice notes in your replies, make sure the webhook ignores messages from the bot itself (infinite loop).

Privacy / GDPR

Audio is personal data. A few considerations:

• Disclosure on first interaction: "Your conversation may be processed by AI for support purposes."
• Controllable retention: delete the raw audio after transcription if you don't need the original (keep text only).
• OpenAI Whisper: does not train on API data (current policy). Same for Gemini.
• Regulated industries (healthcare, finance): self-hosting is safer.