WebRTC Connection

WebRTC connections are used to receive the returned audio and video data from the NavTalk API. After processing your audio input via WebSocket, the digital human’s audio and video responses are delivered through WebRTC for optimal streaming performance.

In the new unified API, WebRTC signaling messages (offer, answer, ICE candidates) are sent through the same WebSocket connection used for real-time API communication. You no longer need a separate WebRTC WebSocket connection.

Handle WebRTC Signaling Through Unified WebSocket

In the new unified API, WebRTC signaling messages are sent and received through the same WebSocket connection used for real-time API communication. You need to handle WebRTC events in your message handler:

// Define event type constants (should be defined at the top of your file)
const NavTalkMessageType = Object.freeze({
    WEB_RTC_OFFER: "webrtc.signaling.offer",
    WEB_RTC_ANSWER: "webrtc.signaling.answer",
    WEB_RTC_ICE_CANDIDATE: "webrtc.signaling.iceCandidate",
    // ... other event types
});

// In your message handler (same socket used for real-time API)
async function handleReceivedMessage(data) {
  const nav_data = data.data;
  
  switch (data.type) {
    // ... other event handlers ...
    
    // WebRTC signaling events
    case NavTalkMessageType.WEB_RTC_OFFER:
      handleOffer(nav_data);
      break;
    
    case NavTalkMessageType.WEB_RTC_ANSWER:
      handleAnswer(nav_data);
      break;
    
    case NavTalkMessageType.WEB_RTC_ICE_CANDIDATE:
      handleIceCandidate(nav_data);
      break;
  }
}

// Helper functions to send WebRTC signaling messages

// Note: sendOfferMessage is typically not needed in the normal flow,
// as the server sends the offer first. However, it may be needed for
// renegotiation scenarios (e.g., when onnegotiationneeded is triggered)
function sendOfferMessage(sdp) {
  const message = {
    type: NavTalkMessageType.WEB_RTC_OFFER,
    data: { sdp: sdp }
  };
  socket.send(JSON.stringify(message));
}

// This function is called after receiving an offer and creating an answer
function sendAnswerMessage(sdp) {
  const message = {
    type: NavTalkMessageType.WEB_RTC_ANSWER,
    data: { sdp: sdp }
  };
  socket.send(JSON.stringify(message));
}

// This function is called when ICE candidates are generated
function sendIceMessage(candidate) {
  const message = {
    type: NavTalkMessageType.WEB_RTC_ICE_CANDIDATE,
    data: { candidate: candidate }
  };
  socket.send(JSON.stringify(message));
}

Important: WebRTC signaling is now integrated into the unified WebSocket connection. You no longer need to:

Create a separate WebRTC WebSocket connection
Use sessionId as userId query parameter
Send { type: 'create', targetSessionId: ... } message

Instead, WebRTC offer/answer/ICE candidate messages are automatically handled through the same WebSocket connection using the event types shown above.

Handle WebRTC Offer

When you receive an offer from the server, create a peer connection, set the remote description, create an answer, and send it back:

let peerConnection;

// ICE server configuration for NAT traversal
// This default configuration will be used initially,
// and will be updated by the server in the 'conversation.connected.success' message
const configuration = {
  iceServers: [
    { urls: 'stun:stun.l.google.com:19302' }
  ]
};

// Update ICE servers when receiving connection success message
function handleConnectionSuccess(data) {
  if (Array.isArray(data.iceServers) && data.iceServers.length > 0) {
    configuration.iceServers = data.iceServers;
    console.log('ICE servers updated from server:', configuration.iceServers);
  }
}

function handleOffer(message) {
  const offer = new RTCSessionDescription(message.sdp);
  console.log("Received offer SDP:", offer);
  
  // ICE servers are provided by the server in the 'conversation.connected.success' message
  // The configuration variable should already be updated when the connection is established
  // No need to fetch ICE servers separately
  
  // Create RTCPeerConnection with the configuration
  peerConnection = new RTCPeerConnection(configuration);
  
  // Set remote description
  peerConnection.setRemoteDescription(offer)
    .then(() => peerConnection.createAnswer())
    .then(answer => peerConnection.setLocalDescription(answer))
    .then(() => {
      // Send answer back to server through unified WebSocket
      sendAnswerMessage(peerConnection.localDescription);
    })
    .catch(err => console.error('Error handling offer:', err));
  
  // Handle incoming tracks (audio/video) - this is where video stream is displayed
  // Make sure you have: <video id="character-video" autoplay playsinline></video> in your HTML
  peerConnection.ontrack = (event) => {
    console.log('Received remote track:', event);
    const remoteVideo = document.getElementById('character-video');
    if (remoteVideo && event.streams && event.streams[0]) {
      remoteVideo.srcObject = event.streams[0];
      remoteVideo.play()
        .then(() => console.log('Video playback started'))
        .catch(e => console.error('Video play failed:', e));
    }
  };
  
  // Handle ICE candidates
  peerConnection.onicecandidate = (event) => {
    if (event.candidate) {
      // Send ICE candidate through unified WebSocket
      sendIceMessage(event.candidate);
    }
  };
  
  // Monitor connection state
  peerConnection.oniceconnectionstatechange = () => {
    console.log('ICE connection state:', peerConnection.iceConnectionState);
    if (peerConnection.iceConnectionState === 'connected') {
      console.log('WebRTC connection fully established!');
    } else if (peerConnection.iceConnectionState === 'failed') {
      console.log('ICE connection failed');
    }
  };
}

Handle ICE Candidates

Exchange ICE candidates to establish the optimal network path for media streaming:

function handleIceCandidate(message) {
  const candidate = new RTCIceCandidate(message.candidate);
  
  if (peerConnection) {
    peerConnection.addIceCandidate(candidate)
      .then(() => console.log('ICE candidate added successfully'))
      .catch(err => console.error('Error adding ICE candidate:', err));
  }
}

Video Stream Display

The video stream is automatically handled by the ontrack event handler set up in Step 2 (Handle WebRTC Offer). When the WebRTC connection is established and media tracks are received, the ontrack event fires and displays the video in your HTML video element.Important points:

The ontrack handler is set up inside the handleOffer function (Step 2), right after creating the RTCPeerConnection
This ensures the handler is ready before any tracks arrive
The video element ID should match what you use in your HTML: <video id="character-video" autoplay playsinline></video>

Complete flow:

Server sends WebRTC offer → handleOffer is called
handleOffer creates RTCPeerConnection and sets up ontrack handler
When media tracks arrive → ontrack event fires automatically
Video is displayed in the HTML video element

The ontrack event handler is set up once when creating the peer connection. You don’t need to set it up again elsewhere. All incoming audio/video tracks will automatically trigger this handler.

Optional: Handle Renegotiation (Advanced)

In some advanced scenarios, you may need to handle WebRTC renegotiation. This happens when the connection needs to be re-established (e.g., after adding/removing media tracks, changing codecs, etc.). In such cases, you may need to send an offer:

// Handle renegotiation when needed
peerConnection.onnegotiationneeded = async () => {
  console.log('Renegotiation needed');
  
  try {
    // Create a new offer
    const offer = await peerConnection.createOffer();
    await peerConnection.setLocalDescription(offer);
    
    // Send the offer through unified WebSocket
    sendOfferMessage(peerConnection.localDescription);
  } catch (err) {
    console.error('Error during renegotiation:', err);
  }
};

When is renegotiation needed?

Typically, the server sends the initial offer, and you respond with an answer
Renegotiation is only needed in advanced scenarios, such as:
- Dynamically adding/removing media tracks
- Changing codec preferences
- Recovering from connection failures
In most cases, you won’t need to handle onnegotiationneeded or call sendOfferMessage

Getting Started

Real-time Digital Human API

Custom Avatar Training

Video Synthesis API

Resources

WebRTC Connection