Real-time bidirectional voice streaming with Google Gemini Live API for React.

Why This Exists

Building real-time voice with Gemini Live is harder than it looks:

Audio format juggling: Gemini wants 16kHz PCM input, sends 24kHz output, but browsers use 44.1kHz/48kHz
Endianness matters: Gemini sends little-endian PCM16. Use Int16Array directly and you'll get garbage on some devices
Buffer management: Play audio too early = choppy. Buffer too much = laggy
Playback chaining: New audio chunks arrive while playing. Chain them wrong = gaps and clicks

I spent some hours figuring this out. Hope it saves you some time!

Quick Start

1. Install

npm install gemini-live-react

2. Deploy the proxy

Copy packages/proxy-deno/index.ts to your Supabase project:

mkdir -p supabase/functions/gemini-live-proxy
cp node_modules/gemini-live-react/packages/proxy-deno/index.ts supabase/functions/gemini-live-proxy/
# Set your API key
supabase secrets set GOOGLE_AI_API_KEY=your-key
# Deploy
supabase functions deploy gemini-live-proxy

3. Use the hook

import { useGeminiLive } from 'gemini-live-react';
function App() {
  const { connect, disconnect, transcripts, isConnected, isSpeaking } = useGeminiLive({
    proxyUrl: 'wss://your-project.supabase.co/functions/v1/gemini-live-proxy',
  });
  return (
    <div>
      <button onClick={() => isConnected ? disconnect() : connect()}>
        {isConnected ? 'End Call' : 'Start Call'}
      </button>
      {isSpeaking && <p>🔊 AI is speaking...</p>}
      {transcripts.map(t => (
        <p key={t.id}><b>{t.role}:</b> {t.text}</p>
      ))}
    </div>
  );
}

See It In Production

deflectionrate.com - AI-powered customer support deflection built with this library.

Features

Voice in/out - Full duplex audio streaming
Screen sharing - Gemini can see what you share
Tool calling - Let AI execute functions and get results back
Voice Activity Detection - Only send audio when speaking (saves bandwidth)
Transcription - Real-time speech-to-text for both sides
Streaming transcripts - Show partial transcripts as users speak
Welcome messages - Auto-trigger AI greeting on connect
Connection state machine - Unified state management (idle → connecting → connected)
Debug mode - Built-in logging for diagnosing issues
Auto-reconnect - Configurable exponential backoff
Session resumption - Pick up where you left off
Session Recording - Record everything, export as JSON, replay for debugging
Workflow Builder - Define multi-step automations AI can execute
Smart Element Detection - AI identifies clickable elements without selectors
TypeScript - Full type definitions

Packages

Package	Description
`gemini-live-react`	React hook
`proxy-deno`	Supabase Edge Function proxy

API

const {
  // State
  isConnected,       // Connected to proxy
  isConnecting,      // Attempting connection
  connectionState,   // 'idle' | 'connecting' | 'connected' | 'reconnecting' | 'error' | 'disconnected'
  isSpeaking,        // AI audio playing
  isMuted,           // Mic muted
  isUserSpeaking,    // User speaking (VAD)
  error,             // Error message
  transcripts,       // Conversation history
  streamingText,     // AI's current partial transcript (real-time)
  streamingUserText, // User's current partial transcript (real-time)
  // Actions
  connect,          // Start session (optional: pass video element)
  disconnect,       // End session
  sendText,         // Send text message
  sendToolResult,   // Send tool result back to AI
  setMuted,         // Mute/unmute mic
  clearTranscripts, // Clear history
} = useGeminiLive({
  proxyUrl: string,              // Required
  sessionId?: string,            // Optional session identifier
  welcomeMessage?: string,       // Sent to AI on connect to trigger greeting
  debug?: boolean | DebugCallback, // Enable logging (true or custom callback)
  // Tool calling
  tools?: ToolDefinition[],      // Function definitions for AI
  onToolCall?: (name, args) => result, // Handle tool calls
  // Voice Activity Detection
  vad?: boolean,                 // Only send audio when speaking
  vadOptions?: { threshold?, minSpeechDuration?, silenceDuration? },
  // Reconnection
  reconnection?: {
    maxAttempts?: number,        // Default: 5
    initialDelay?: number,       // Default: 1000ms
    maxDelay?: number,           // Default: 10000ms
    backoffFactor?: number,      // Default: 2
  },
  // Callbacks
  onTranscript?: (t) => void,
  onError?: (e) => void,
  onConnectionChange?: (c) => void,
  // Audio tuning
  minBufferMs?: number,          // Default: 200
  transcriptDebounceMs?: number, // Default: 1500
  // Session Recording
  recording?: RecordingConfig,   // Enable recording
  onRecordingEvent?: (event) => void,
  // Smart Detection
  smartDetection?: SmartDetectionConfig,
});

Guides

Voices

Voice	Style
Zephyr	Bright, clear (default)
Puck	Warm, friendly
Charon	Deep, authoritative
Kore	Soft, gentle
Fenrir	Strong, confident
Aoede	Melodic, expressive

Change voice via proxy query param: ?voice=Kore

Screen Sharing

const videoRef = useRef<HTMLVideoElement>(null);
const startWithScreen = async () => {
  const stream = await navigator.mediaDevices.getDisplayMedia({ video: true });
  videoRef.current!.srcObject = stream;
  await videoRef.current!.play();
  await connect(videoRef.current!); // Pass video element
};

Frames are sent at 1 FPS, scaled to max 1024px width.

Mobile Support

iOS Safari and some mobile browsers don't support screen recording. The library exports utilities for detection and fallback:

import {
  shouldUseCameraMode,
  canScreenRecord,
  isIOS,
  isMobile,
} from 'gemini-live-react';
// Decide between screen share and camera
const startWithVideo = async () => {
  let stream: MediaStream;
  if (canScreenRecord()) {
    stream = await navigator.mediaDevices.getDisplayMedia({ video: true });
  } else {
    // Camera fallback for mobile
    stream = await navigator.mediaDevices.getUserMedia({
      video: { facingMode: 'environment' }
    });
  }
  videoRef.current!.srcObject = stream;
  await videoRef.current!.play();
  await connect(videoRef.current!);
};

Important for iOS: Always add playsInline to video elements:

Table of Contents