> ## Documentation Index
> Fetch the complete documentation index at: https://docs.navtalk.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# WebSocket Connection

> Learn how to establish WebSocket connections to send user audio input for processing

## WebSocket Connection

WebSocket connections are used to **establish connection and send user audio data** to the NavTalk API for processing. This is the primary channel for transmitting your audio input to the digital human system.

The complete connection process involves **one unified WebSocket connection** that handles:

1. **Real-time API communication** - for sending audio input and receiving text/audio responses
2. **WebRTC signaling** - for establishing video stream (WebRTC signaling messages are sent through the same WebSocket connection)

<Steps>
  <Step title="Step 1: Establish WebSocket Connection">
    First, establish a unified WebSocket connection to the NavTalk API. This single connection will be used for all communication including audio data, text/audio responses, and WebRTC signaling.

    NavTalk supports two connection entry points:

    * **REST token flow**: Call `POST /api/open/v1/realtime-chat/connection`, then connect to the returned `wsUrl`. Use this when your backend should validate the license, avatar, domain/IP restrictions, or maximum call duration before opening the WebSocket. The request body must include exactly one of `avatarId` or `name`, and can optionally include `durationSeconds`.
    * **Direct WebSocket flow**: Connect to `wss://transfer.navtalk.ai/wss/v2/realtime-chat` with `license` and either `name` or `avatarId` in the URL. Use this for quick starts or client flows that already pass the license directly.

    In the REST token flow, the request body only uses `avatarId`/`name` and optional `durationSeconds`; do not include `model` or `voice` in that token request.

    After the WebSocket is connected, both methods use the same event sequence and message types.

    <CodeGroup>
      ```javascript JavaScript theme={null}
      // Define event type constants
      const NavTalkMessageType = Object.freeze({
          CONNECTED_SUCCESS: "conversation.connected.success",
          CONNECTED_FAIL: "conversation.connected.fail",
          CONNECTED_CLOSE: "conversation.connected.close",
          INSUFFICIENT_BALANCE: "conversation.connected.insufficient_balance",
          CONNECTED_WARNING: "conversation.connected.warning",
          WEB_RTC_OFFER: "webrtc.signaling.offer",
          WEB_RTC_ANSWER: "webrtc.signaling.answer",
          WEB_RTC_ICE_CANDIDATE: "webrtc.signaling.iceCandidate",
          REALTIME_SESSION_CREATED: "realtime.session.created",
          REALTIME_SESSION_UPDATED: "realtime.session.updated",
          REALTIME_SPEECH_STARTED: "realtime.input_audio_buffer.speech_started",
          REALTIME_SPEECH_STOPPED: "realtime.input_audio_buffer.speech_stopped",
          REALTIME_CONVERSATION_ITEM_COMPLETED: "realtime.conversation.item.input_audio_transcription.completed",
          REALTIME_RESPONSE_AUDIO_TRANSCRIPT_DELTA: "realtime.response.audio_transcript.delta",
          REALTIME_RESPONSE_AUDIO_DELTA: "realtime.response.audio.delta",
          REALTIME_RESPONSE_AUDIO_TRANSCRIPT_DONE: "realtime.response.audio_transcript.done",
          REALTIME_RESPONSE_AUDIO_DONE: "realtime.response.audio.done",
          REALTIME_RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE: "realtime.response.function_call_arguments.done",
          REALTIME_INPUT_AUDIO_BUFFER_APPEND: "realtime.input_audio_buffer.append",
          REALTIME_INPUT_TEXT: "realtime.input_text",
          REALTIME_INPUT_IMAGE: "realtime.input_image"
      });

      // Configuration
      const LICENSE = 'your-api-key-here';
      const CHARACTER_NAME = 'navtalk.Leo';
      // Optional: Use avatarId for precise avatar lookup
      const AVATAR_ID = 'your-avatar-id'; // Get from avatar list API

      // Build WebSocket URL with query parameters
      const websocketUrl = 'wss://transfer.navtalk.ai/wss/v2/realtime-chat';

      // Method 1: Direct WebSocket flow using character name
      const websocketUrlWithParams = `${websocketUrl}?license=${encodeURIComponent(LICENSE)}&name=${encodeURIComponent(CHARACTER_NAME)}`;

      // Method 1 alternative: Direct WebSocket flow using avatarId (recommended, higher priority)
      // const websocketUrlWithParams = `${websocketUrl}?license=${encodeURIComponent(LICENSE)}&avatarId=${encodeURIComponent(AVATAR_ID)}`;

      // Method 2: REST token flow
      // const connectionResponse = await fetch('https://api.navtalk.ai/api/open/v1/realtime-chat/connection', {
      //   method: 'POST',
      //   headers: {
      //     'Content-Type': 'application/json',
      //     license: LICENSE
      //   },
      //   body: JSON.stringify({
      //     avatarId: AVATAR_ID,
      //     durationSeconds: 600
      //   })
      // });
      // const connectionResult = await connectionResponse.json();
      // const websocketUrlWithParams = connectionResult.data.wsUrl;

      // Create WebSocket connection
      const socket = new WebSocket(websocketUrlWithParams);
      // Important: set binary type to 'arraybuffer' for handling binary audio data
      socket.binaryType = 'arraybuffer';

      // Connection event handlers
      socket.onopen = () => {
        console.log('WebSocket connection established');
      };

      socket.onerror = (error) => {
        console.error('WebSocket error:', error);
        // Handle connection errors
      };

      socket.onclose = (event) => {
        console.log('WebSocket connection closed', event.code, event.reason);
        // Handle connection closure
        // Common reasons: 'Insufficient points', normal closure, etc.
      };
      ```

      ```python Python theme={null}
      import asyncio
      import websockets
      from urllib.parse import quote
      import json

      # Configuration
      LICENSE = 'your-api-key-here'
      CHARACTER_NAME = 'navtalk.Leo'

      # Build WebSocket URL with query parameters
      websocket_url = 'wss://transfer.navtalk.ai/wss/v2/realtime-chat'

      # Method 1: Direct WebSocket flow using character name
      websocket_url_with_params = f'{websocket_url}?license={quote(LICENSE)}&name={quote(CHARACTER_NAME)}'

      # Method 2: REST token flow
      # import requests
      # response = requests.post(
      #     'https://api.navtalk.ai/api/open/v1/realtime-chat/connection',
      #     headers={'license': LICENSE},
      #     json={'name': CHARACTER_NAME, 'durationSeconds': 600}
      # )
      # websocket_url_with_params = response.json()['data']['wsUrl']

      async def connect():
          async with websockets.connect(websocket_url_with_params) as websocket:
              print('WebSocket connection established')
              # Handle messages here
              await handle_messages(websocket)

      asyncio.run(connect())
      ```
    </CodeGroup>

    <Note>
      Both connection entry points are supported:

      * REST token flow: `POST /api/open/v1/realtime-chat/connection` returns a short-lived `wsUrl`.
      * Direct WebSocket flow: the URL includes `license` and either `name` or `avatarId`.

      **Direct WebSocket Query Priority**: For the direct WebSocket flow, if both `avatarId` and `name` are provided, `avatarId` takes precedence. For the REST token flow, pass exactly one of `avatarId` or `name`.

      **Multiple Avatars Warning**: If using `name` query and multiple avatars share the same name, the system will:

      * Automatically select the most recently updated avatar
      * Send a `conversation.connected.warning` event with the selected avatarId immediately after the connection success event

      This unified connection handles both real-time API communication and WebRTC signaling, eliminating the need for a separate WebRTC WebSocket connection.
    </Note>
  </Step>

  <Step title="Step 2: Configure Session and Handle Session Events">
    After the WebSocket connection is established, the server will automatically send `realtime.session.created` and `realtime.session.updated` events. After receiving `realtime.session.created`, send conversation history (if any). Once you receive `realtime.session.updated`, you can start sending audio data.

    <CodeGroup>
      ```javascript JavaScript theme={null}
      // Global variables for session management
      let socket; // WebSocket connection
      let conversationHistory = []; // Optional: store conversation history

      socket.onmessage = (event) => {
        // Handle both string (JSON) and binary messages
        if (typeof event.data === 'string') {
          try {
            const data = JSON.parse(event.data);
            handleReceivedMessage(data);
          } catch (e) {
            console.error('Failed to parse JSON message:', e);
          }
        } else if (event.data instanceof ArrayBuffer) {
          // Handle binary audio data if needed
          handleReceivedBinaryMessage(event.data);
        }
      };

      async function handleReceivedMessage(data) {
        // Extract data from nested structure
        const nav_data = data.data;
        
        switch (data.type) {
          // Connection events
          case NavTalkMessageType.CONNECTED_SUCCESS:
            console.log('WebSocket connection successful');
            // Extract sessionId and iceServers from data
            if (nav_data && nav_data.sessionId) {
              const sessionId = nav_data.sessionId;
              console.log('Received session ID:', sessionId);
              // Store sessionId for logging or session tracking
              // Store iceServers for WebRTC configuration
              if (nav_data.iceServers) {
                configuration.iceServers = nav_data.iceServers;
                console.log('ICE servers configured:', configuration.iceServers);
              }
            }
            break;
          
          case NavTalkMessageType.CONNECTED_FAIL:
            const errorMessage = data.message || 'Unknown error';
            console.error('Connection failed:', errorMessage);
            // Handle connection failure
            break;
          
          case NavTalkMessageType.CONNECTED_CLOSE:
            console.log('Connection closed:', data.message || 'Normal closure');
            // Handle connection closure
            break;
          
          case NavTalkMessageType.INSUFFICIENT_BALANCE:
            console.error('Insufficient balance, service has stopped, please recharge!');
            // Handle insufficient balance
            break;
          
          case NavTalkMessageType.CONNECTED_WARNING:
            const warningMsg = data.message || 'Connection warning';
            console.warn('Connection warning:', warningMsg);
            // Display warning to user (e.g., toast notification)
            break;
          
          // Step 2.1: Session created - send conversation history
          case NavTalkMessageType.REALTIME_SESSION_CREATED:
            console.log('Session created, sending conversation history.');
            await sendSessionUpdate();
            break;
          
          // Step 2.2: Session updated - ready to send audio
          case NavTalkMessageType.REALTIME_SESSION_UPDATED:
            console.log('Session updated. Ready to receive audio.');
            // Now you can start recording and sending audio
            startRecording();
            break;

      // Send conversation history after session is created
      async function sendSessionUpdate() {
        // Get conversation history from storage
        const history = localStorage.getItem('realtimeChatHistory');
        const conversationHistory = history ? JSON.parse(history) : [];
        
        // Send each item in history
        // Note: Only send user messages, assistant messages are handled by the server
        conversationHistory.forEach((msg) => {
          if (msg.role === 'user') {
            const messageConfig = {
              type: 'conversation.item.create',
              item: {
                type: 'message',
                role: msg.role,
                content: [
                  {
                    type: 'input_text',
                    text: msg.content
                  }
                ]
              }
            };
            
            try {
              socket.send(JSON.stringify(messageConfig));
              console.log('Sent history message:', msg.role);
            } catch (e) {
              console.error('Error sending history message:', e);
            }
          }
        });
      }
      ```

      ```python Python theme={null}
      import json

      async def handle_messages(websocket):
          async for message in websocket:
              if isinstance(message, str):
                  data = json.loads(message)
                  nav_data = data.get('data')
                  
                  message_type = data.get('type')
                  
                  # Connection events
                  if message_type == 'conversation.connected.success':
                      print('WebSocket connection successful')
                      if nav_data and nav_data.get('sessionId'):
                          session_id = nav_data['sessionId']
                          print('Received session ID:', session_id)
                          # Store sessionId for logging or session tracking
                          if nav_data.get('iceServers'):
                              configuration['iceServers'] = nav_data['iceServers']
                              print('ICE servers configured')
                  
                  elif message_type == 'conversation.connected.fail':
                      error_message = data.get('message', 'Unknown error')
                      print(f'Connection failed: {error_message}')
                  
                  elif message_type == 'conversation.connected.close':
                      print('Connection closed:', data.get('message', 'Normal closure'))
                  
                  elif message_type == 'conversation.connected.insufficient_balance':
                      print('Insufficient balance, service has stopped, please recharge!')
                  
                  # Session events
                  elif message_type == 'realtime.session.created':
                      print('Session created, sending conversation history.')
                      await send_session_update(websocket)
                  
                  elif message_type == 'realtime.session.updated':
                      print('Session updated. Ready to receive audio.')
                      # Start audio recording here
                      await start_recording(websocket)
                  
                  else:
                      # Handle other message types
                      await handle_other_messages(data)

      async def send_session_update(websocket):
          # Get conversation history from storage
          # In Python, you might use a database or file storage
          conversation_history = []  # Load from your storage
          
          # Send each user message in history
          for msg in conversation_history:
              if msg['role'] == 'user':
                  message_config = {
                      'type': 'conversation.item.create',
                      'item': {
                          'type': 'message',
                          'role': msg['role'],
                          'content': [
                              {
                                  'type': 'input_text',
                                  'text': msg['content']
                              }
                          ]
                      }
                  }
                  
                  try:
                      await websocket.send(json.dumps(message_config))
                      print('Sent history message:', msg['role'])
                  except Exception as e:
                      print(f'Error sending history message: {e}')
      ```
    </CodeGroup>

    <Note>
      Session configuration (voice, prompt, tools) can be set via the Console interface or API configuration. The `sendSessionUpdate()` function is used to send conversation history after receiving the `realtime.session.created` event.

      The conversation history allows the AI to maintain context across sessions. Only user messages need to be sent; assistant messages are handled by the server.
    </Note>
  </Step>

  <Step title="Step 3: Capture and Send Audio">
    Once you receive the `realtime.session.updated` event, you can start capturing audio from the user's microphone and sending it through the WebSocket connection. The audio must be in **PCM16 format at 24kHz sample rate, mono channel**.

    <CodeGroup>
      ```javascript JavaScript theme={null}
      // Global variables for audio processing
      let audioContext;
      let audioProcessor;
      let audioStream;

      function startRecording() {
        // Request microphone access
        navigator.mediaDevices.getUserMedia({ audio: true })
          .then(stream => {
            // Create AudioContext with 24kHz sample rate (required by API)
            audioContext = new (window.AudioContext || window.webkitAudioContext)({ 
              sampleRate: 24000 
            });
            audioStream = stream;
            
            // Create audio source from microphone stream
            const source = audioContext.createMediaStreamSource(stream);
            
            // Create ScriptProcessorNode to process audio chunks
            // Parameters: bufferSize (8192), inputChannels (1), outputChannels (1)
            audioProcessor = audioContext.createScriptProcessor(8192, 1, 1);
            
            // Process audio data in real-time
            audioProcessor.onaudioprocess = (event) => {
              // Only send if WebSocket is open
              if (socket && socket.readyState === WebSocket.OPEN) {
                // Get audio data from input buffer (Float32Array, range -1.0 to 1.0)
                const inputBuffer = event.inputBuffer.getChannelData(0);
                
                // Step 3.1: Convert Float32Array to PCM16 (16-bit signed integers)
                const pcmData = floatTo16BitPCM(inputBuffer);
                
                // Step 3.2: Encode PCM16 data to base64 for JSON transmission
                const base64PCM = base64EncodeAudio(new Uint8Array(pcmData));
                
                // Step 3.3: Send audio chunks (split large base64 strings into smaller chunks)
                // Chunk size: 4096 characters to avoid message size limits
                const chunkSize = 4096;
                for (let i = 0; i < base64PCM.length; i += chunkSize) {
                  const chunk = base64PCM.slice(i, i + chunkSize);
                  
                  // Send audio chunk as JSON message
                  socket.send(JSON.stringify({
                    type: NavTalkMessageType.REALTIME_INPUT_AUDIO_BUFFER_APPEND,
                    data: { audio: chunk }
                  }));
                }
              }
            };
            
            // Connect audio processing chain
            source.connect(audioProcessor);
            audioProcessor.connect(audioContext.destination);
            
            console.log('Recording started');
          })
          .catch(error => {
            console.error('Unable to access microphone:', error);
            // Handle microphone access errors
          });
      }

      // Helper function: Convert Float32Array to 16-bit PCM
      // Input: Float32Array with values in range [-1.0, 1.0]
      // Output: ArrayBuffer containing 16-bit PCM data
      function floatTo16BitPCM(float32Array) {
        const buffer = new ArrayBuffer(float32Array.length * 2); // 2 bytes per sample
        const view = new DataView(buffer);
        let offset = 0;
        
        for (let i = 0; i < float32Array.length; i++, offset += 2) {
          // Clamp value to [-1, 1] range
          let s = Math.max(-1, Math.min(1, float32Array[i]));
          
          // Convert to 16-bit signed integer
          // Negative values: s * 0x8000 (range: -32768 to 0)
          // Positive values: s * 0x7FFF (range: 0 to 32767)
          view.setInt16(offset, s < 0 ? s * 0x8000 : s * 0x7FFF, true);
        }
        
        return buffer;
      }

      // Helper function: Encode audio to base64
      // Input: Uint8Array of PCM16 data
      // Output: Base64-encoded string
      function base64EncodeAudio(uint8Array) {
        let binary = '';
        const chunkSize = 0x8000; // 32KB chunks to avoid stack overflow
        
        // Convert binary data to string (chunk by chunk)
        for (let i = 0; i < uint8Array.length; i += chunkSize) {
          const chunk = uint8Array.subarray(i, i + chunkSize);
          binary += String.fromCharCode.apply(null, chunk);
        }
        
        // Encode to base64
        return btoa(binary);
      }

      // Stop recording and cleanup
      function stopRecording() {
        // Disconnect audio processor
        if (audioProcessor) {
          audioProcessor.disconnect();
          audioProcessor = null;
        }
        
        // Stop all audio tracks
        if (audioStream) {
          audioStream.getTracks().forEach(track => track.stop());
          audioStream = null;
        }
        
        // Close WebSocket connection
        if (socket) {
          socket.close();
        }
        
        console.log('Recording stopped');
      }
      ```

      ```python Python theme={null}
      import pyaudio
      import base64
      import asyncio
      import json

      # Audio configuration (must match API requirements)
      CHUNK = 8192  # Buffer size
      FORMAT = pyaudio.paInt16  # 16-bit PCM
      CHANNELS = 1  # Mono
      RATE = 24000  # 24kHz sample rate

      audio = pyaudio.PyAudio()

      def audio_callback(in_data, frame_count, time_info, status):
          # Convert PCM16 bytes to base64
          audio_base64 = base64.b64encode(in_data).decode('utf-8')
          
          # Send audio chunks (split into smaller chunks)
          chunk_size = 4096
          for i in range(0, len(audio_base64), chunk_size):
              chunk = audio_base64[i:i + chunk_size]
              message = {
                  'type': 'realtime.input_audio_buffer.append',
                  'data': { 'audio': chunk }
              }
              # Send asynchronously
              asyncio.create_task(websocket.send(json.dumps(message)))
          
          return (None, pyaudio.paContinue)

      async def start_recording(websocket):
          stream = audio.open(
              format=FORMAT,
              channels=CHANNELS,
              rate=RATE,
              input=True,
              frames_per_buffer=CHUNK,
              stream_callback=audio_callback
          )
          
          stream.start_stream()
          print('Recording started')
      ```
    </CodeGroup>

    <Warning>
      **Critical Audio Requirements:**

      * **Format**: PCM16 (16-bit signed integers)
      * **Sample Rate**: 24kHz (24000 Hz)
      * **Channels**: Mono (1 channel)
      * **Encoding**: Base64 for JSON transmission

      Make sure your audio processing matches these specifications exactly, otherwise the API may reject the audio data.
    </Warning>
  </Step>

  <Step title="Step 4: Handle Response Messages">
    Process incoming messages from the API. The WebSocket connection will send various event types including transcriptions, AI responses, and status updates.

    <Note>
      When the WebSocket connection is established, you will receive a `conversation.connected.success` event. This event contains:

      * `data.sessionId`: The session ID for logging, billing, and session tracking
      * `data.iceServers`: ICE server configuration used when creating the WebRTC peer connection

      Capture `iceServers` as soon as it arrives and pass it into your `RTCPeerConnection` configuration. The `sessionId` is useful for logs and backend correlation, but the unified v2 WebRTC signaling flow does not require sending it back as a `userId` parameter.
    </Note>

    <CodeGroup>
      ```javascript JavaScript theme={null}
      // Enhanced message handler with all event types
      async function handleReceivedMessage(data) {
        // Extract data from nested structure
        const nav_data = data.data;
        
        switch (data.type) {
          // Connection events
          case NavTalkMessageType.CONNECTED_SUCCESS:
            console.log('WebSocket connection successful');
            // Extract sessionId and iceServers from data
            if (nav_data && nav_data.sessionId) {
              const sessionId = nav_data.sessionId;
              console.log('Received session ID:', sessionId);
              // Store sessionId for logging or session tracking
              // Store iceServers for WebRTC configuration
              if (nav_data.iceServers) {
                configuration.iceServers = nav_data.iceServers;
                console.log('ICE servers configured:', configuration.iceServers);
              }
            }
            break;
          
          case NavTalkMessageType.CONNECTED_FAIL:
            const errorMessage = data.message || 'Unknown error';
            console.error('Connection failed:', errorMessage);
            break;
          
          case NavTalkMessageType.CONNECTED_CLOSE:
            console.log('Connection closed:', data.message || 'Normal closure');
            break;
          
          case NavTalkMessageType.INSUFFICIENT_BALANCE:
            console.error('Insufficient balance, service has stopped, please recharge!');
            break;
          
          case NavTalkMessageType.CONNECTED_WARNING:
            const warningMsg = data.message || 'Connection warning';
            console.warn('Connection warning:', warningMsg);
            // Display warning to user (e.g., toast notification)
            break;
          
          // Session events (handled in Step 2)
          case NavTalkMessageType.REALTIME_SESSION_CREATED:
            await sendSessionUpdate();
            break;
          
          case NavTalkMessageType.REALTIME_SESSION_UPDATED:
            console.log('Session updated. Ready to receive audio.');
            startRecording();
            break;
          
          // User speech detection events
          case NavTalkMessageType.REALTIME_SPEECH_STARTED:
            console.log('Speech started detected by server.');
            // User started speaking - you might want to stop any current audio playback
            stopCurrentAudioPlayback();
            break;
          
          case NavTalkMessageType.REALTIME_SPEECH_STOPPED:
            console.log('Speech stopped detected by server.');
            // User stopped speaking - server will process the audio
            break;
          
          // User transcription events
          case NavTalkMessageType.REALTIME_CONVERSATION_ITEM_COMPLETED:
            const transcript = nav_data.content;
            console.log('Received transcription:', transcript);
            // Display user message in UI
            displayUserMessage(transcript);
            // Save to conversation history
            await saveToHistory('user', transcript);
            break;
          
          // AI response text stream (streaming)
          case NavTalkMessageType.REALTIME_RESPONSE_AUDIO_TRANSCRIPT_DELTA:
            // This event fires multiple times as the AI generates text
            const deltaTranscript = nav_data.content; // Incremental text chunk
            const responseId = nav_data.id; // Unique ID for this response
            
            // Accumulate text chunks (you may need to buffer incomplete content)
            accumulateResponseText(responseId, deltaTranscript);
            
            // Display streaming text in UI
            displayAIResponseStream(responseId, deltaTranscript);
            break;
          
          // AI response text completed
          case NavTalkMessageType.REALTIME_RESPONSE_AUDIO_TRANSCRIPT_DONE:
            const finalTranscript = nav_data.content;
            console.log('AI response complete:', finalTranscript);
            // Display final text
            displayAIResponseComplete(finalTranscript);
            // Save to conversation history
            await saveToHistory('assistant', finalTranscript);
            break;
          
          // AI response audio stream (if you're receiving audio via WebSocket)
          case NavTalkMessageType.REALTIME_RESPONSE_AUDIO_DELTA:
            // Handle audio delta if needed
            if (nav_data.delta) {
              // Process audio chunk
              handleAudioChunk(nav_data.delta);
            }
            break;
          
          // AI response audio completed
          case NavTalkMessageType.REALTIME_RESPONSE_AUDIO_DONE:
            console.log('Audio response complete.');
            // Handle audio completion
            break;
          
          // Function call events
          case NavTalkMessageType.REALTIME_RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE:
            console.log('Function call received:', nav_data);
            await handleFunctionCall(nav_data);
            break;
          
          // WebRTC signaling events (handled in Step 5)
          case NavTalkMessageType.WEB_RTC_OFFER:
            handleOffer(nav_data);
            break;
          
          case NavTalkMessageType.WEB_RTC_ANSWER:
            handleAnswer(nav_data);
            break;
          
          case NavTalkMessageType.WEB_RTC_ICE_CANDIDATE:
            handleIceCandidate(nav_data);
            break;
          
          // Error events
          case NavTalkMessageType.INSUFFICIENT_BALANCE:
            console.error('Insufficient balance, service has stopped, please recharge!');
            break;
          
          default:
            console.warn('Unhandled event type:', data.type);
        }
      }

      // Helper function to accumulate streaming text
      let responseBuffers = new Map();

      function accumulateResponseText(responseId, delta) {
        if (!responseBuffers.has(responseId)) {
          responseBuffers.set(responseId, '');
        }
        const current = responseBuffers.get(responseId);
        responseBuffers.set(responseId, current + delta);
      }

      // Handle binary messages (if any)
      function handleReceivedBinaryMessage(arrayBuffer) {
        // Process binary audio data if needed
        console.log('Received binary message:', arrayBuffer.byteLength, 'bytes');
      }
      ```

      ```python Python theme={null}
      async def handle_messages(websocket):
          response_buffers = {}  # Store streaming text buffers
          
          async for message in websocket:
              if isinstance(message, str):
                  data = json.loads(message)
                  message_type = data.get('type')
                  
                  # Connection events
                  if message_type == 'conversation.connected.success':
                      nav_data = data.get('data')
                      if nav_data and nav_data.get('sessionId'):
                          session_id = nav_data['sessionId']
                          print('Received session ID:', session_id)
                          # Store sessionId for logging or session tracking
                          if nav_data.get('iceServers'):
                              configuration['iceServers'] = nav_data['iceServers']
                              print('ICE servers configured')
                  
                  elif message_type == 'realtime.session.created':
                      await send_session_update(websocket)
                  
                  elif message_type == 'realtime.session.updated':
                      print('Session updated. Ready to receive audio.')
                      await start_recording(websocket)
                  
                  elif message_type == 'input_audio_buffer.speech_started':
                      print('Speech started detected by server.')
                  
                  elif message_type == 'input_audio_buffer.speech_stopped':
                      print('Speech stopped detected by server.')
                  
                  elif message_type == 'conversation.item.input_audio_transcription.completed':
                      transcript = data.get('transcript')
                      print('Received transcription:', transcript)
                      # Display and save user message
                  
                  elif message_type == 'response.audio_transcript.delta':
                      response_id = data.get('response_id')
                      delta = data.get('delta')
                      # Accumulate streaming text
                      if response_id not in response_buffers:
                          response_buffers[response_id] = ''
                      response_buffers[response_id] += delta
                      # Display streaming text
                  
                  elif message_type == 'response.audio_transcript.done':
                      transcript = data.get('transcript')
                      print('AI response complete:', transcript)
                      # Display final text
                  
                  elif message_type == 'response.audio.done':
                      print('Audio response complete.')
                  
                  elif message_type == 'session.insufficient_balance':
                      print('Insufficient balance, service has stopped, please recharge!')
      ```
    </CodeGroup>

    <Note>
      The API sends events in a specific sequence:

      1. `conversation.connected.success` → Connection established, contains `sessionId` for tracking and `iceServers` for WebRTC
      2. `realtime.session.created` → Send conversation history
      3. `realtime.session.updated` → Start sending audio
      4. `realtime.input_audio_buffer.speech_started` → User starts speaking
      5. `realtime.input_audio_buffer.speech_stopped` → User stops speaking
      6. `realtime.conversation.item.input_audio_transcription.completed` → User speech transcribed
      7. `realtime.response.audio_transcript.delta` → AI response text (streaming, multiple events)
      8. `realtime.response.audio_transcript.done` → AI response text complete
      9. `realtime.response.audio.done` → AI response audio complete

      Note: Event data may be nested in a `data` field. Always check both `data.data` and `data` when accessing event properties.
    </Note>
  </Step>

  <Step title="Step 5: Establish WebRTC Connection (for Video)">
    To receive the digital human's video stream, WebRTC signaling messages are sent through the same unified WebSocket connection. This is covered in detail in the [WebRTC Connection guide](/api/real-time-digital-human-api/quick-start/webrtc-connection).

    <Note>
      In the new unified API, WebRTC signaling (offer, answer, ICE candidates) is handled through the same WebSocket connection using event types:

      * `webrtc.signaling.offer` - Receive WebRTC offer
      * `webrtc.signaling.answer` - Send WebRTC answer
      * `webrtc.signaling.iceCandidate` - Exchange ICE candidates

      No separate WebRTC WebSocket connection is needed.
    </Note>
  </Step>
</Steps>