> ## Documentation Index
> Fetch the complete documentation index at: https://docs.navtalk.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# WebRTC Connection

> Learn how to establish WebRTC connections to receive audio and video responses from the NavTalk Real-time Digital Human API

## WebRTC Connection

WebRTC connections are used to **receive the returned audio and video data** from the NavTalk API. After processing your audio input via WebSocket, the digital human's audio and video responses are delivered through WebRTC for optimal streaming performance.

<Note>
  In the new unified API, WebRTC signaling messages (offer, answer, ICE candidates) are sent through the same WebSocket connection used for real-time API communication. You no longer need a separate WebRTC WebSocket connection.
</Note>

<Steps>
  <Step title="Handle WebRTC Signaling Through Unified WebSocket">
    In the new unified API, WebRTC signaling messages are sent and received through the same WebSocket connection used for real-time API communication. You need to handle WebRTC events in your message handler:

    <CodeGroup>
      ```javascript JavaScript theme={null}
      // Define event type constants (should be defined at the top of your file)
      const NavTalkMessageType = Object.freeze({
          WEB_RTC_OFFER: "webrtc.signaling.offer",
          WEB_RTC_ANSWER: "webrtc.signaling.answer",
          WEB_RTC_ICE_CANDIDATE: "webrtc.signaling.iceCandidate",
          // ... other event types
      });

      // In your message handler (same socket used for real-time API)
      async function handleReceivedMessage(data) {
        const nav_data = data.data;
        
        switch (data.type) {
          // ... other event handlers ...
          
          // WebRTC signaling events
          case NavTalkMessageType.WEB_RTC_OFFER:
            handleOffer(nav_data);
            break;
          
          case NavTalkMessageType.WEB_RTC_ANSWER:
            handleAnswer(nav_data);
            break;
          
          case NavTalkMessageType.WEB_RTC_ICE_CANDIDATE:
            handleIceCandidate(nav_data);
            break;
        }
      }

      // Helper functions to send WebRTC signaling messages

      // Note: sendOfferMessage is typically not needed in the normal flow,
      // as the server sends the offer first. However, it may be needed for
      // renegotiation scenarios (e.g., when onnegotiationneeded is triggered)
      function sendOfferMessage(sdp) {
        const message = {
          type: NavTalkMessageType.WEB_RTC_OFFER,
          data: { sdp: sdp }
        };
        socket.send(JSON.stringify(message));
      }

      // This function is called after receiving an offer and creating an answer
      function sendAnswerMessage(sdp) {
        const message = {
          type: NavTalkMessageType.WEB_RTC_ANSWER,
          data: { sdp: sdp }
        };
        socket.send(JSON.stringify(message));
      }

      // This function is called when ICE candidates are generated
      function sendIceMessage(candidate) {
        const message = {
          type: NavTalkMessageType.WEB_RTC_ICE_CANDIDATE,
          data: { candidate: candidate }
        };
        socket.send(JSON.stringify(message));
      }
      ```

      ```python Python theme={null}
      # In your message handler (same websocket used for real-time API)
      async def handle_messages(websocket):
          async for message in websocket:
              if isinstance(message, str):
                  data = json.loads(message)
                  nav_data = data.get('data')
                  
                  message_type = data.get('type')
                  
                  if message_type == 'webrtc.signaling.offer':
                      await handle_offer(nav_data, websocket)
                  elif message_type == 'webrtc.signaling.answer':
                      await handle_answer(nav_data)
                  elif message_type == 'webrtc.signaling.iceCandidate':
                      await handle_ice_candidate(nav_data)
                  # ... other event handlers ...

      # Helper functions to send WebRTC signaling messages

      # Note: send_offer_message is typically not needed in the normal flow,
      # as the server sends the offer first. However, it may be needed for
      # renegotiation scenarios (e.g., when onnegotiationneeded is triggered)
      async def send_offer_message(websocket, sdp):
          message = {
              'type': 'webrtc.signaling.offer',
              'data': { 'sdp': sdp }
          }
          await websocket.send(json.dumps(message))

      # This function is called after receiving an offer and creating an answer
      async def send_answer_message(websocket, sdp):
          message = {
              'type': 'webrtc.signaling.answer',
              'data': { 'sdp': sdp }
          }
          await websocket.send(json.dumps(message))

      # This function is called when ICE candidates are generated
      async def send_ice_message(websocket, candidate):
          message = {
              'type': 'webrtc.signaling.iceCandidate',
              'data': { 'candidate': candidate }
          }
          await websocket.send(json.dumps(message))
      ```
    </CodeGroup>

    <Note>
      **Important**: WebRTC signaling is now integrated into the unified WebSocket connection. You no longer need to:

      * Create a separate WebRTC WebSocket connection
      * Use `sessionId` as `userId` query parameter
      * Send `{ type: 'create', targetSessionId: ... }` message

      Instead, WebRTC offer/answer/ICE candidate messages are automatically handled through the same WebSocket connection using the event types shown above.
    </Note>
  </Step>

  <Step title="Handle WebRTC Offer">
    When you receive an offer from the server, create a peer connection, set the remote description, create an answer, and send it back:

    <CodeGroup>
      ```javascript JavaScript theme={null}
      let peerConnection;

      // ICE server configuration for NAT traversal
      // This default configuration will be used initially,
      // and will be updated by the server in the 'conversation.connected.success' message
      const configuration = {
        iceServers: [
          { urls: 'stun:stun.l.google.com:19302' }
        ]
      };

      // Update ICE servers when receiving connection success message
      function handleConnectionSuccess(data) {
        if (Array.isArray(data.iceServers) && data.iceServers.length > 0) {
          configuration.iceServers = data.iceServers;
          console.log('ICE servers updated from server:', configuration.iceServers);
        }
      }

      function handleOffer(message) {
        const offer = new RTCSessionDescription(message.sdp);
        console.log("Received offer SDP:", offer);
        
        // ICE servers are provided by the server in the 'conversation.connected.success' message
        // The configuration variable should already be updated when the connection is established
        // No need to fetch ICE servers separately
        
        // Create RTCPeerConnection with the configuration
        peerConnection = new RTCPeerConnection(configuration);
        
        // Set remote description
        peerConnection.setRemoteDescription(offer)
          .then(() => peerConnection.createAnswer())
          .then(answer => peerConnection.setLocalDescription(answer))
          .then(() => {
            // Send answer back to server through unified WebSocket
            sendAnswerMessage(peerConnection.localDescription);
          })
          .catch(err => console.error('Error handling offer:', err));
        
        // Handle incoming tracks (audio/video) - this is where video stream is displayed
        // Make sure you have: <video id="character-video" autoplay playsinline></video> in your HTML
        peerConnection.ontrack = (event) => {
          console.log('Received remote track:', event);
          const remoteVideo = document.getElementById('character-video');
          if (remoteVideo && event.streams && event.streams[0]) {
            remoteVideo.srcObject = event.streams[0];
            remoteVideo.play()
              .then(() => console.log('Video playback started'))
              .catch(e => console.error('Video play failed:', e));
          }
        };
        
        // Handle ICE candidates
        peerConnection.onicecandidate = (event) => {
          if (event.candidate) {
            // Send ICE candidate through unified WebSocket
            sendIceMessage(event.candidate);
          }
        };
        
        // Monitor connection state
        peerConnection.oniceconnectionstatechange = () => {
          console.log('ICE connection state:', peerConnection.iceConnectionState);
          if (peerConnection.iceConnectionState === 'connected') {
            console.log('WebRTC connection fully established!');
          } else if (peerConnection.iceConnectionState === 'failed') {
            console.log('ICE connection failed');
          }
        };
      }
      ```

      ```python Python theme={null}
      # Python WebRTC implementation requires additional libraries
      # such as aiortc or python-webrtc
      # This is a conceptual example

      async def handle_offer(message, websocket):
          from aiortc import RTCPeerConnection, RTCSessionDescription
          
          offer = RTCSessionDescription(sdp=message['sdp']['sdp'], type=message['sdp']['type'])
          
          pc = RTCPeerConnection()
          
          @pc.on('track')
          def on_track(track):
              print('Received track:', track.kind)
              # Handle incoming audio/video track
              # Play in appropriate media element
          
          await pc.setRemoteDescription(offer)
          answer = await pc.createAnswer()
          await pc.setLocalDescription(answer)
          
          # Send answer through unified WebSocket
          await send_answer_message(websocket, {
              'sdp': pc.localDescription.sdp,
              'type': pc.localDescription.type
          })
          
          # Handle ICE candidates
          @pc.on('icecandidate')
          def on_ice_candidate(candidate):
              if candidate:
                  await send_ice_message(websocket, {
                      'candidate': candidate.candidate,
                      'sdpMid': candidate.sdpMid,
                      'sdpMLineIndex': candidate.sdpMLineIndex
                  })
      ```
    </CodeGroup>
  </Step>

  <Step title="Handle ICE Candidates">
    Exchange ICE candidates to establish the optimal network path for media streaming:

    <CodeGroup>
      ```javascript JavaScript theme={null}
      function handleIceCandidate(message) {
        const candidate = new RTCIceCandidate(message.candidate);
        
        if (peerConnection) {
          peerConnection.addIceCandidate(candidate)
            .then(() => console.log('ICE candidate added successfully'))
            .catch(err => console.error('Error adding ICE candidate:', err));
        }
      }
      ```

      ```python Python theme={null}
      async def handle_ice_candidate(message):
          candidate_info = message['candidate']
          candidate = RTCIceCandidate(
              candidate=candidate_info['candidate'],
              sdpMid=candidate_info['sdpMid'],
              sdpMLineIndex=candidate_info['sdpMLineIndex']
          )
          if pc:
              await pc.addIceCandidate(candidate)
              print('ICE candidate added successfully')
      ```
    </CodeGroup>
  </Step>

  <Step title="Video Stream Display">
    The video stream is automatically handled by the `ontrack` event handler set up in **Step 2** (Handle WebRTC Offer). When the WebRTC connection is established and media tracks are received, the `ontrack` event fires and displays the video in your HTML video element.

    **Important points:**

    * The `ontrack` handler is set up **inside** the `handleOffer` function (Step 2), right after creating the RTCPeerConnection
    * This ensures the handler is ready before any tracks arrive
    * The video element ID should match what you use in your HTML: `<video id="character-video" autoplay playsinline></video>`

    **Complete flow:**

    1. Server sends WebRTC offer → `handleOffer` is called
    2. `handleOffer` creates RTCPeerConnection and sets up `ontrack` handler
    3. When media tracks arrive → `ontrack` event fires automatically
    4. Video is displayed in the HTML video element

    <Note>
      The `ontrack` event handler is set up once when creating the peer connection. You don't need to set it up again elsewhere. All incoming audio/video tracks will automatically trigger this handler.
    </Note>
  </Step>

  <Step title="Optional: Handle Renegotiation (Advanced)">
    In some advanced scenarios, you may need to handle WebRTC renegotiation. This happens when the connection needs to be re-established (e.g., after adding/removing media tracks, changing codecs, etc.). In such cases, you may need to send an offer:

    <CodeGroup>
      ```javascript JavaScript theme={null}
      // Handle renegotiation when needed
      peerConnection.onnegotiationneeded = async () => {
        console.log('Renegotiation needed');
        
        try {
          // Create a new offer
          const offer = await peerConnection.createOffer();
          await peerConnection.setLocalDescription(offer);
          
          // Send the offer through unified WebSocket
          sendOfferMessage(peerConnection.localDescription);
        } catch (err) {
          console.error('Error during renegotiation:', err);
        }
      };
      ```

      ```python Python theme={null}
      # Handle renegotiation when needed
      @pc.on('negotiationneeded')
      async def on_negotiation_needed():
          print('Renegotiation needed')
          
          try:
              # Create a new offer
              offer = await pc.createOffer()
              await pc.setLocalDescription(offer)
              
              # Send the offer through unified WebSocket
              await send_offer_message(websocket, {
                  'sdp': pc.localDescription.sdp,
                  'type': pc.localDescription.type
              })
          except Exception as err:
              print(f'Error during renegotiation: {err}')
      ```
    </CodeGroup>

    <Note>
      **When is renegotiation needed?**

      * Typically, the server sends the initial offer, and you respond with an answer
      * Renegotiation is only needed in advanced scenarios, such as:
        * Dynamically adding/removing media tracks
        * Changing codec preferences
        * Recovering from connection failures
      * In most cases, you won't need to handle `onnegotiationneeded` or call `sendOfferMessage`
    </Note>
  </Step>
</Steps>