WebSocket Connection

WebSocket connections are used to establish connection and send user audio data to the NavTalk API for processing. This is the primary channel for transmitting your audio input to the digital human system. The complete connection process involves one unified WebSocket connection that handles:

Real-time API communication - for sending audio input and receiving text/audio responses
WebRTC signaling - for establishing video stream (WebRTC signaling messages are sent through the same WebSocket connection)

Step 1: Establish WebSocket Connection

First, establish a unified WebSocket connection to the NavTalk API. This single connection will be used for all communication including audio data, text/audio responses, and WebRTC signaling.NavTalk supports two connection entry points:

REST token flow: Call POST /api/open/v1/realtime-chat/connection, then connect to the returned wsUrl. Use this when your backend should validate the license, avatar, domain/IP restrictions, or maximum call duration before opening the WebSocket. The request body must include exactly one of avatarId or name, and can optionally include durationSeconds.
Direct WebSocket flow: Connect to wss://transfer.navtalk.ai/wss/v2/realtime-chat with license and either name or avatarId in the URL. Use this for quick starts or client flows that already pass the license directly.

In the REST token flow, the request body only uses avatarId/name and optional durationSeconds; do not include model or voice in that token request.After the WebSocket is connected, both methods use the same event sequence and message types.

// Define event type constants
const NavTalkMessageType = Object.freeze({
    CONNECTED_SUCCESS: "conversation.connected.success",
    CONNECTED_FAIL: "conversation.connected.fail",
    CONNECTED_CLOSE: "conversation.connected.close",
    INSUFFICIENT_BALANCE: "conversation.connected.insufficient_balance",
    CONNECTED_WARNING: "conversation.connected.warning",
    WEB_RTC_OFFER: "webrtc.signaling.offer",
    WEB_RTC_ANSWER: "webrtc.signaling.answer",
    WEB_RTC_ICE_CANDIDATE: "webrtc.signaling.iceCandidate",
    REALTIME_SESSION_CREATED: "realtime.session.created",
    REALTIME_SESSION_UPDATED: "realtime.session.updated",
    REALTIME_SPEECH_STARTED: "realtime.input_audio_buffer.speech_started",
    REALTIME_SPEECH_STOPPED: "realtime.input_audio_buffer.speech_stopped",
    REALTIME_CONVERSATION_ITEM_COMPLETED: "realtime.conversation.item.input_audio_transcription.completed",
    REALTIME_RESPONSE_AUDIO_TRANSCRIPT_DELTA: "realtime.response.audio_transcript.delta",
    REALTIME_RESPONSE_AUDIO_DELTA: "realtime.response.audio.delta",
    REALTIME_RESPONSE_AUDIO_TRANSCRIPT_DONE: "realtime.response.audio_transcript.done",
    REALTIME_RESPONSE_AUDIO_DONE: "realtime.response.audio.done",
    REALTIME_RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE: "realtime.response.function_call_arguments.done",
    REALTIME_INPUT_AUDIO_BUFFER_APPEND: "realtime.input_audio_buffer.append",
    REALTIME_INPUT_TEXT: "realtime.input_text",
    REALTIME_INPUT_IMAGE: "realtime.input_image"
});

// Configuration
const LICENSE = 'your-api-key-here';
const CHARACTER_NAME = 'navtalk.Leo';
// Optional: Use avatarId for precise avatar lookup
const AVATAR_ID = 'your-avatar-id'; // Get from avatar list API

// Build WebSocket URL with query parameters
const websocketUrl = 'wss://transfer.navtalk.ai/wss/v2/realtime-chat';

// Method 1: Direct WebSocket flow using character name
const websocketUrlWithParams = `${websocketUrl}?license=${encodeURIComponent(LICENSE)}&name=${encodeURIComponent(CHARACTER_NAME)}`;

// Method 1 alternative: Direct WebSocket flow using avatarId (recommended, higher priority)
// const websocketUrlWithParams = `${websocketUrl}?license=${encodeURIComponent(LICENSE)}&avatarId=${encodeURIComponent(AVATAR_ID)}`;

// Method 2: REST token flow
// const connectionResponse = await fetch('https://api.navtalk.ai/api/open/v1/realtime-chat/connection', {
//   method: 'POST',
//   headers: {
//     'Content-Type': 'application/json',
//     license: LICENSE
//   },
//   body: JSON.stringify({
//     avatarId: AVATAR_ID,
//     durationSeconds: 600
//   })
// });
// const connectionResult = await connectionResponse.json();
// const websocketUrlWithParams = connectionResult.data.wsUrl;

// Create WebSocket connection
const socket = new WebSocket(websocketUrlWithParams);
// Important: set binary type to 'arraybuffer' for handling binary audio data
socket.binaryType = 'arraybuffer';

// Connection event handlers
socket.onopen = () => {
  console.log('WebSocket connection established');
};

socket.onerror = (error) => {
  console.error('WebSocket error:', error);
  // Handle connection errors
};

socket.onclose = (event) => {
  console.log('WebSocket connection closed', event.code, event.reason);
  // Handle connection closure
  // Common reasons: 'Insufficient points', normal closure, etc.
};

import asyncio
import websockets
from urllib.parse import quote
import json

# Configuration
LICENSE = 'your-api-key-here'
CHARACTER_NAME = 'navtalk.Leo'

# Build WebSocket URL with query parameters
websocket_url = 'wss://transfer.navtalk.ai/wss/v2/realtime-chat'

# Method 1: Direct WebSocket flow using character name
websocket_url_with_params = f'{websocket_url}?license={quote(LICENSE)}&name={quote(CHARACTER_NAME)}'

# Method 2: REST token flow
# import requests
# response = requests.post(
#     'https://api.navtalk.ai/api/open/v1/realtime-chat/connection',
#     headers={'license': LICENSE},
#     json={'name': CHARACTER_NAME, 'durationSeconds': 600}
# )
# websocket_url_with_params = response.json()['data']['wsUrl']

async def connect():
    async with websockets.connect(websocket_url_with_params) as websocket:
        print('WebSocket connection established')
        # Handle messages here
        await handle_messages(websocket)

asyncio.run(connect())

Both connection entry points are supported:

REST token flow: POST /api/open/v1/realtime-chat/connection returns a short-lived wsUrl.
Direct WebSocket flow: the URL includes license and either name or avatarId.

Direct WebSocket Query Priority: For the direct WebSocket flow, if both avatarId and name are provided, avatarId takes precedence. For the REST token flow, pass exactly one of avatarId or name.Multiple Avatars Warning: If using name query and multiple avatars share the same name, the system will:

Automatically select the most recently updated avatar
Send a conversation.connected.warning event with the selected avatarId immediately after the connection success event

This unified connection handles both real-time API communication and WebRTC signaling, eliminating the need for a separate WebRTC WebSocket connection.

Step 2: Configure Session and Handle Session Events

After the WebSocket connection is established, the server will automatically send realtime.session.created and realtime.session.updated events. After receiving realtime.session.created, send conversation history (if any). Once you receive realtime.session.updated, you can start sending audio data.

// Global variables for session management
let socket; // WebSocket connection
let conversationHistory = []; // Optional: store conversation history

socket.onmessage = (event) => {
  // Handle both string (JSON) and binary messages
  if (typeof event.data === 'string') {
    try {
      const data = JSON.parse(event.data);
      handleReceivedMessage(data);
    } catch (e) {
      console.error('Failed to parse JSON message:', e);
    }
  } else if (event.data instanceof ArrayBuffer) {
    // Handle binary audio data if needed
    handleReceivedBinaryMessage(event.data);
  }
};

async function handleReceivedMessage(data) {
  // Extract data from nested structure
  const nav_data = data.data;
  
  switch (data.type) {
    // Connection events
    case NavTalkMessageType.CONNECTED_SUCCESS:
      console.log('WebSocket connection successful');
      // Extract sessionId and iceServers from data
      if (nav_data && nav_data.sessionId) {
        const sessionId = nav_data.sessionId;
        console.log('Received session ID:', sessionId);
        // Store sessionId for logging or session tracking
        // Store iceServers for WebRTC configuration
        if (nav_data.iceServers) {
          configuration.iceServers = nav_data.iceServers;
          console.log('ICE servers configured:', configuration.iceServers);
        }
      }
      break;
    
    case NavTalkMessageType.CONNECTED_FAIL:
      const errorMessage = data.message || 'Unknown error';
      console.error('Connection failed:', errorMessage);
      // Handle connection failure
      break;
    
    case NavTalkMessageType.CONNECTED_CLOSE:
      console.log('Connection closed:', data.message || 'Normal closure');
      // Handle connection closure
      break;
    
    case NavTalkMessageType.INSUFFICIENT_BALANCE:
      console.error('Insufficient balance, service has stopped, please recharge!');
      // Handle insufficient balance
      break;
    
    case NavTalkMessageType.CONNECTED_WARNING:
      const warningMsg = data.message || 'Connection warning';
      console.warn('Connection warning:', warningMsg);
      // Display warning to user (e.g., toast notification)
      break;
    
    // Step 2.1: Session created - send conversation history
    case NavTalkMessageType.REALTIME_SESSION_CREATED:
      console.log('Session created, sending conversation history.');
      await sendSessionUpdate();
      break;
    
    // Step 2.2: Session updated - ready to send audio
    case NavTalkMessageType.REALTIME_SESSION_UPDATED:
      console.log('Session updated. Ready to receive audio.');
      // Now you can start recording and sending audio
      startRecording();
      break;

// Send conversation history after session is created
async function sendSessionUpdate() {
  // Get conversation history from storage
  const history = localStorage.getItem('realtimeChatHistory');
  const conversationHistory = history ? JSON.parse(history) : [];
  
  // Send each item in history
  // Note: Only send user messages, assistant messages are handled by the server
  conversationHistory.forEach((msg) => {
    if (msg.role === 'user') {
      const messageConfig = {
        type: 'conversation.item.create',
        item: {
          type: 'message',
          role: msg.role,
          content: [
            {
              type: 'input_text',
              text: msg.content
            }
          ]
        }
      };
      
      try {
        socket.send(JSON.stringify(messageConfig));
        console.log('Sent history message:', msg.role);
      } catch (e) {
        console.error('Error sending history message:', e);
      }
    }
  });
}

import json

async def handle_messages(websocket):
    async for message in websocket:
        if isinstance(message, str):
            data = json.loads(message)
            nav_data = data.get('data')
            
            message_type = data.get('type')
            
            # Connection events
            if message_type == 'conversation.connected.success':
                print('WebSocket connection successful')
                if nav_data and nav_data.get('sessionId'):
                    session_id = nav_data['sessionId']
                    print('Received session ID:', session_id)
                    # Store sessionId for logging or session tracking
                    if nav_data.get('iceServers'):
                        configuration['iceServers'] = nav_data['iceServers']
                        print('ICE servers configured')
            
            elif message_type == 'conversation.connected.fail':
                error_message = data.get('message', 'Unknown error')
                print(f'Connection failed: {error_message}')
            
            elif message_type == 'conversation.connected.close':
                print('Connection closed:', data.get('message', 'Normal closure'))
            
            elif message_type == 'conversation.connected.insufficient_balance':
                print('Insufficient balance, service has stopped, please recharge!')
            
            # Session events
            elif message_type == 'realtime.session.created':
                print('Session created, sending conversation history.')
                await send_session_update(websocket)
            
            elif message_type == 'realtime.session.updated':
                print('Session updated. Ready to receive audio.')
                # Start audio recording here
                await start_recording(websocket)
            
            else:
                # Handle other message types
                await handle_other_messages(data)

async def send_session_update(websocket):
    # Get conversation history from storage
    # In Python, you might use a database or file storage
    conversation_history = []  # Load from your storage
    
    # Send each user message in history
    for msg in conversation_history:
        if msg['role'] == 'user':
            message_config = {
                'type': 'conversation.item.create',
                'item': {
                    'type': 'message',
                    'role': msg['role'],
                    'content': [
                        {
                            'type': 'input_text',
                            'text': msg['content']
                        }
                    ]
                }
            }
            
            try:
                await websocket.send(json.dumps(message_config))
                print('Sent history message:', msg['role'])
            except Exception as e:
                print(f'Error sending history message: {e}')

Session configuration (voice, prompt, tools) can be set via the Console interface or API configuration. The sendSessionUpdate() function is used to send conversation history after receiving the realtime.session.created event.The conversation history allows the AI to maintain context across sessions. Only user messages need to be sent; assistant messages are handled by the server.

Step 3: Capture and Send Audio

Once you receive the realtime.session.updated event, you can start capturing audio from the user’s microphone and sending it through the WebSocket connection. The audio must be in PCM16 format at 24kHz sample rate, mono channel.

// Global variables for audio processing
let audioContext;
let audioProcessor;
let audioStream;

function startRecording() {
  // Request microphone access
  navigator.mediaDevices.getUserMedia({ audio: true })
    .then(stream => {
      // Create AudioContext with 24kHz sample rate (required by API)
      audioContext = new (window.AudioContext || window.webkitAudioContext)({ 
        sampleRate: 24000 
      });
      audioStream = stream;
      
      // Create audio source from microphone stream
      const source = audioContext.createMediaStreamSource(stream);
      
      // Create ScriptProcessorNode to process audio chunks
      // Parameters: bufferSize (8192), inputChannels (1), outputChannels (1)
      audioProcessor = audioContext.createScriptProcessor(8192, 1, 1);
      
      // Process audio data in real-time
      audioProcessor.onaudioprocess = (event) => {
        // Only send if WebSocket is open
        if (socket && socket.readyState === WebSocket.OPEN) {
          // Get audio data from input buffer (Float32Array, range -1.0 to 1.0)
          const inputBuffer = event.inputBuffer.getChannelData(0);
          
          // Step 3.1: Convert Float32Array to PCM16 (16-bit signed integers)
          const pcmData = floatTo16BitPCM(inputBuffer);
          
          // Step 3.2: Encode PCM16 data to base64 for JSON transmission
          const base64PCM = base64EncodeAudio(new Uint8Array(pcmData));
          
          // Step 3.3: Send audio chunks (split large base64 strings into smaller chunks)
          // Chunk size: 4096 characters to avoid message size limits
          const chunkSize = 4096;
          for (let i = 0; i < base64PCM.length; i += chunkSize) {
            const chunk = base64PCM.slice(i, i + chunkSize);
            
            // Send audio chunk as JSON message
            socket.send(JSON.stringify({
              type: NavTalkMessageType.REALTIME_INPUT_AUDIO_BUFFER_APPEND,
              data: { audio: chunk }
            }));
          }
        }
      };
      
      // Connect audio processing chain
      source.connect(audioProcessor);
      audioProcessor.connect(audioContext.destination);
      
      console.log('Recording started');
    })
    .catch(error => {
      console.error('Unable to access microphone:', error);
      // Handle microphone access errors
    });
}

// Helper function: Convert Float32Array to 16-bit PCM
// Input: Float32Array with values in range [-1.0, 1.0]
// Output: ArrayBuffer containing 16-bit PCM data
function floatTo16BitPCM(float32Array) {
  const buffer = new ArrayBuffer(float32Array.length * 2); // 2 bytes per sample
  const view = new DataView(buffer);
  let offset = 0;
  
  for (let i = 0; i < float32Array.length; i++, offset += 2) {
    // Clamp value to [-1, 1] range
    let s = Math.max(-1, Math.min(1, float32Array[i]));
    
    // Convert to 16-bit signed integer
    // Negative values: s * 0x8000 (range: -32768 to 0)
    // Positive values: s * 0x7FFF (range: 0 to 32767)
    view.setInt16(offset, s < 0 ? s * 0x8000 : s * 0x7FFF, true);
  }
  
  return buffer;
}

// Helper function: Encode audio to base64
// Input: Uint8Array of PCM16 data
// Output: Base64-encoded string
function base64EncodeAudio(uint8Array) {
  let binary = '';
  const chunkSize = 0x8000; // 32KB chunks to avoid stack overflow
  
  // Convert binary data to string (chunk by chunk)
  for (let i = 0; i < uint8Array.length; i += chunkSize) {
    const chunk = uint8Array.subarray(i, i + chunkSize);
    binary += String.fromCharCode.apply(null, chunk);
  }
  
  // Encode to base64
  return btoa(binary);
}

// Stop recording and cleanup
function stopRecording() {
  // Disconnect audio processor
  if (audioProcessor) {
    audioProcessor.disconnect();
    audioProcessor = null;
  }
  
  // Stop all audio tracks
  if (audioStream) {
    audioStream.getTracks().forEach(track => track.stop());
    audioStream = null;
  }
  
  // Close WebSocket connection
  if (socket) {
    socket.close();
  }
  
  console.log('Recording stopped');
}

import pyaudio
import base64
import asyncio
import json

# Audio configuration (must match API requirements)
CHUNK = 8192  # Buffer size
FORMAT = pyaudio.paInt16  # 16-bit PCM
CHANNELS = 1  # Mono
RATE = 24000  # 24kHz sample rate

audio = pyaudio.PyAudio()

def audio_callback(in_data, frame_count, time_info, status):
    # Convert PCM16 bytes to base64
    audio_base64 = base64.b64encode(in_data).decode('utf-8')
    
    # Send audio chunks (split into smaller chunks)
    chunk_size = 4096
    for i in range(0, len(audio_base64), chunk_size):
        chunk = audio_base64[i:i + chunk_size]
        message = {
            'type': 'realtime.input_audio_buffer.append',
            'data': { 'audio': chunk }
        }
        # Send asynchronously
        asyncio.create_task(websocket.send(json.dumps(message)))
    
    return (None, pyaudio.paContinue)

async def start_recording(websocket):
    stream = audio.open(
        format=FORMAT,
        channels=CHANNELS,
        rate=RATE,
        input=True,
        frames_per_buffer=CHUNK,
        stream_callback=audio_callback
    )
    
    stream.start_stream()
    print('Recording started')

Critical Audio Requirements:

Format: PCM16 (16-bit signed integers)
Sample Rate: 24kHz (24000 Hz)
Channels: Mono (1 channel)
Encoding: Base64 for JSON transmission

Make sure your audio processing matches these specifications exactly, otherwise the API may reject the audio data.

Step 4: Handle Response Messages

Process incoming messages from the API. The WebSocket connection will send various event types including transcriptions, AI responses, and status updates.

When the WebSocket connection is established, you will receive a conversation.connected.success event. This event contains:

data.sessionId: The session ID for logging, billing, and session tracking
data.iceServers: ICE server configuration used when creating the WebRTC peer connection

Capture iceServers as soon as it arrives and pass it into your RTCPeerConnection configuration. The sessionId is useful for logs and backend correlation, but the unified v2 WebRTC signaling flow does not require sending it back as a userId parameter.

// Enhanced message handler with all event types
async function handleReceivedMessage(data) {
  // Extract data from nested structure
  const nav_data = data.data;
  
  switch (data.type) {
    // Connection events
    case NavTalkMessageType.CONNECTED_SUCCESS:
      console.log('WebSocket connection successful');
      // Extract sessionId and iceServers from data
      if (nav_data && nav_data.sessionId) {
        const sessionId = nav_data.sessionId;
        console.log('Received session ID:', sessionId);
        // Store sessionId for logging or session tracking
        // Store iceServers for WebRTC configuration
        if (nav_data.iceServers) {
          configuration.iceServers = nav_data.iceServers;
          console.log('ICE servers configured:', configuration.iceServers);
        }
      }
      break;
    
    case NavTalkMessageType.CONNECTED_FAIL:
      const errorMessage = data.message || 'Unknown error';
      console.error('Connection failed:', errorMessage);
      break;
    
    case NavTalkMessageType.CONNECTED_CLOSE:
      console.log('Connection closed:', data.message || 'Normal closure');
      break;
    
    case NavTalkMessageType.INSUFFICIENT_BALANCE:
      console.error('Insufficient balance, service has stopped, please recharge!');
      break;
    
    case NavTalkMessageType.CONNECTED_WARNING:
      const warningMsg = data.message || 'Connection warning';
      console.warn('Connection warning:', warningMsg);
      // Display warning to user (e.g., toast notification)
      break;
    
    // Session events (handled in Step 2)
    case NavTalkMessageType.REALTIME_SESSION_CREATED:
      await sendSessionUpdate();
      break;
    
    case NavTalkMessageType.REALTIME_SESSION_UPDATED:
      console.log('Session updated. Ready to receive audio.');
      startRecording();
      break;
    
    // User speech detection events
    case NavTalkMessageType.REALTIME_SPEECH_STARTED:
      console.log('Speech started detected by server.');
      // User started speaking - you might want to stop any current audio playback
      stopCurrentAudioPlayback();
      break;
    
    case NavTalkMessageType.REALTIME_SPEECH_STOPPED:
      console.log('Speech stopped detected by server.');
      // User stopped speaking - server will process the audio
      break;
    
    // User transcription events
    case NavTalkMessageType.REALTIME_CONVERSATION_ITEM_COMPLETED:
      const transcript = nav_data.content;
      console.log('Received transcription:', transcript);
      // Display user message in UI
      displayUserMessage(transcript);
      // Save to conversation history
      await saveToHistory('user', transcript);
      break;
    
    // AI response text stream (streaming)
    case NavTalkMessageType.REALTIME_RESPONSE_AUDIO_TRANSCRIPT_DELTA:
      // This event fires multiple times as the AI generates text
      const deltaTranscript = nav_data.content; // Incremental text chunk
      const responseId = nav_data.id; // Unique ID for this response
      
      // Accumulate text chunks (you may need to buffer incomplete content)
      accumulateResponseText(responseId, deltaTranscript);
      
      // Display streaming text in UI
      displayAIResponseStream(responseId, deltaTranscript);
      break;
    
    // AI response text completed
    case NavTalkMessageType.REALTIME_RESPONSE_AUDIO_TRANSCRIPT_DONE:
      const finalTranscript = nav_data.content;
      console.log('AI response complete:', finalTranscript);
      // Display final text
      displayAIResponseComplete(finalTranscript);
      // Save to conversation history
      await saveToHistory('assistant', finalTranscript);
      break;
    
    // AI response audio stream (if you're receiving audio via WebSocket)
    case NavTalkMessageType.REALTIME_RESPONSE_AUDIO_DELTA:
      // Handle audio delta if needed
      if (nav_data.delta) {
        // Process audio chunk
        handleAudioChunk(nav_data.delta);
      }
      break;
    
    // AI response audio completed
    case NavTalkMessageType.REALTIME_RESPONSE_AUDIO_DONE:
      console.log('Audio response complete.');
      // Handle audio completion
      break;
    
    // Function call events
    case NavTalkMessageType.REALTIME_RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE:
      console.log('Function call received:', nav_data);
      await handleFunctionCall(nav_data);
      break;
    
    // WebRTC signaling events (handled in Step 5)
    case NavTalkMessageType.WEB_RTC_OFFER:
      handleOffer(nav_data);
      break;
    
    case NavTalkMessageType.WEB_RTC_ANSWER:
      handleAnswer(nav_data);
      break;
    
    case NavTalkMessageType.WEB_RTC_ICE_CANDIDATE:
      handleIceCandidate(nav_data);
      break;
    
    // Error events
    case NavTalkMessageType.INSUFFICIENT_BALANCE:
      console.error('Insufficient balance, service has stopped, please recharge!');
      break;
    
    default:
      console.warn('Unhandled event type:', data.type);
  }
}

// Helper function to accumulate streaming text
let responseBuffers = new Map();

function accumulateResponseText(responseId, delta) {
  if (!responseBuffers.has(responseId)) {
    responseBuffers.set(responseId, '');
  }
  const current = responseBuffers.get(responseId);
  responseBuffers.set(responseId, current + delta);
}

// Handle binary messages (if any)
function handleReceivedBinaryMessage(arrayBuffer) {
  // Process binary audio data if needed
  console.log('Received binary message:', arrayBuffer.byteLength, 'bytes');
}

async def handle_messages(websocket):
    response_buffers = {}  # Store streaming text buffers
    
    async for message in websocket:
        if isinstance(message, str):
            data = json.loads(message)
            message_type = data.get('type')
            
            # Connection events
            if message_type == 'conversation.connected.success':
                nav_data = data.get('data')
                if nav_data and nav_data.get('sessionId'):
                    session_id = nav_data['sessionId']
                    print('Received session ID:', session_id)
                    # Store sessionId for logging or session tracking
                    if nav_data.get('iceServers'):
                        configuration['iceServers'] = nav_data['iceServers']
                        print('ICE servers configured')
            
            elif message_type == 'realtime.session.created':
                await send_session_update(websocket)
            
            elif message_type == 'realtime.session.updated':
                print('Session updated. Ready to receive audio.')
                await start_recording(websocket)
            
            elif message_type == 'input_audio_buffer.speech_started':
                print('Speech started detected by server.')
            
            elif message_type == 'input_audio_buffer.speech_stopped':
                print('Speech stopped detected by server.')
            
            elif message_type == 'conversation.item.input_audio_transcription.completed':
                transcript = data.get('transcript')
                print('Received transcription:', transcript)
                # Display and save user message
            
            elif message_type == 'response.audio_transcript.delta':
                response_id = data.get('response_id')
                delta = data.get('delta')
                # Accumulate streaming text
                if response_id not in response_buffers:
                    response_buffers[response_id] = ''
                response_buffers[response_id] += delta
                # Display streaming text
            
            elif message_type == 'response.audio_transcript.done':
                transcript = data.get('transcript')
                print('AI response complete:', transcript)
                # Display final text
            
            elif message_type == 'response.audio.done':
                print('Audio response complete.')
            
            elif message_type == 'session.insufficient_balance':
                print('Insufficient balance, service has stopped, please recharge!')

The API sends events in a specific sequence:

conversation.connected.success → Connection established, contains sessionId for tracking and iceServers for WebRTC
realtime.session.created → Send conversation history
realtime.session.updated → Start sending audio
realtime.input_audio_buffer.speech_started → User starts speaking
realtime.input_audio_buffer.speech_stopped → User stops speaking
realtime.conversation.item.input_audio_transcription.completed → User speech transcribed
realtime.response.audio_transcript.delta → AI response text (streaming, multiple events)
realtime.response.audio_transcript.done → AI response text complete
realtime.response.audio.done → AI response audio complete

Note: Event data may be nested in a data field. Always check both data.data and data when accessing event properties.

Step 5: Establish WebRTC Connection (for Video)

To receive the digital human’s video stream, WebRTC signaling messages are sent through the same unified WebSocket connection. This is covered in detail in the WebRTC Connection guide.

In the new unified API, WebRTC signaling (offer, answer, ICE candidates) is handled through the same WebSocket connection using event types:

webrtc.signaling.offer - Receive WebRTC offer
webrtc.signaling.answer - Send WebRTC answer
webrtc.signaling.iceCandidate - Exchange ICE candidates

No separate WebRTC WebSocket connection is needed.

Getting Started

Real-time Digital Human API

Video Synthesis API

Custom Avatar Training

Resources

WebSocket Connection

WebSocket Connection

​WebSocket Connection

WebSocket Connection