realtime.input_image

Send camera image to AI for visual recognition and analysis

Send a camera snapshot or image to the AI for visual recognition and analysis. This event is used to transmit base64-encoded images captured from the user’s camera for scene analysis, object recognition, or visual Q&A.

Provider Compatibility: Image input is only supported with the OpenAIRealtime provider. This feature is not available when using other AI providers (e.g., ElevenLabs).

Message Format

This is a client-to-server event that you send to transmit image data.

type · string · required

Event type. Must be "realtime.input_image".

data · object · required

Event data object containing the image content and reply flag.

Show data properties

content · string · required

Base64-encoded image in Data URL format. Format: data:image/jpeg;base64,{base64_string}

Supported formats: JPEG, PNG, WebP

Recommended: JPEG with quality 0.7 (70%) for optimal balance between quality and payload size

reply · integer · default: "0"

Whether to request an immediate AI response for this image:

• 0: No immediate reply (default) - Image is added to conversation context
• 1: Request reply - AI will analyze and respond to the image

Use 0 for periodic camera frames where you don’t need a response for every frame. Use 1 when the user explicitly asks the AI to analyze the current camera view.

Request Example

JavaScript

const NavTalkMessageType = {
  REALTIME_INPUT_IMAGE: "realtime.input_image"
};

// Send image without requesting immediate reply
function sendImageMessage(imageUrl) {
  if (!imageUrl || !socket || socket.readyState !== WebSocket.OPEN) {
    console.warn("WebSocket not open or empty input");
    return;
  }
  
  const message = {
    type: NavTalkMessageType.REALTIME_INPUT_IMAGE,
    data: { 
      content: imageUrl,  // e.g., "data:image/jpeg;base64,/9j/4AAQ..."
      reply: 0            // Don't request immediate reply
    }
  };
  
  socket.send(JSON.stringify(message));
}

// Example: Capture camera frame and send
async function captureAndSend() {
  const video = document.getElementById('camera-preview');
  
  // Create canvas to capture frame
  const canvas = document.createElement('canvas');
  canvas.width = video.videoWidth || 640;
  canvas.height = video.videoHeight || 360;
  const ctx = canvas.getContext('2d');
  ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
  
  // Convert to base64
  canvas.toBlob(async (blob) => {
    const arrayBuffer = await blob.arrayBuffer();
    const uint8Array = new Uint8Array(arrayBuffer);
    const base64 = btoa(String.fromCharCode.apply(null, uint8Array));
    const imageUrl = `data:${blob.type};base64,${base64}`;
    
    sendImageMessage(imageUrl);
  }, "image/jpeg", 0.7);
}

Message Structure

{
  "type": "realtime.input_image",
  "data": {
    "content": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQ...",
    "reply": 0
  }
}

Usage Scenarios

Scenario 1: Periodic Scene Recognition

Send camera frames periodically (e.g., every 2 seconds) without requesting replies:

// Start periodic capture
setInterval(() => {
  captureAndSend();  // reply: 0 (no immediate response)
}, 2000);

The AI builds visual context over time and can reference what it “sees” when responding to user questions.

Scenario 2: On-demand Visual Q&A

User asks “What do you see?” - capture and send with reply: 1:

// User asks a visual question
async function handleVisualQuestion() {
  const video = document.getElementById('camera-preview');
  
  // Capture current frame
  const imageUrl = await captureFrame(video);
  
  // Send with reply flag to get AI's analysis
  const message = {
    type: "realtime.input_image",
    data: { 
      content: imageUrl,
      reply: 1  // Request AI to analyze and respond
    }
  };
  
  socket.send(JSON.stringify(message));
}

Scenario 3: Hybrid Approach (Recommended)

Combine periodic background capture (reply: 0) with on-demand analysis (reply: 1):

// Background: Periodic capture for context (every 2 seconds)
setInterval(() => {
  captureAndSend(false);  // reply: 0
}, 2000);

// Foreground: User-initiated visual analysis
async function analyzeCurrentView() {
  await captureAndSend(true);  // reply: 1
}

Best Practices

Image Format and Quality

• Format: Use JPEG for optimal compression and compatibility
• Quality: Set to 0.7 (70%) to balance quality and payload size
• Resolution: 640x360 is sufficient for most recognition tasks
• File Size: Target 20-50KB per frame after compression

Transmission Frequency

• Periodic capture: 2-3 seconds interval for most use cases
• Adjust based on use case:
  • Fast-moving scenes: 1 second
  • Static scenes: 3-5 seconds
• Don’t exceed 1 frame per second to avoid overwhelming the AI and consuming excessive bandwidth

Reply Flag Usage

• Use reply: 0 (default) for:
  • Background periodic capture
  • Building visual context
  • Continuous monitoring without responses
• Use reply: 1 when:
  • User explicitly asks about the camera view
  • You need immediate AI analysis
  • Triggering specific visual recognition tasks

Error Handling

function captureFrameAndSend() {
  try {
    // Check prerequisites
    if (!socket || socket.readyState !== WebSocket.OPEN) {
      console.warn("WebSocket not ready");
      return;
    }
    
    if (!cameraStream || !video || video.readyState < 2) {
      console.warn("Camera not ready");
      return;
    }
    
    // Capture and send
    // ... implementation ...
  } catch (error) {
    console.error("Error capturing frame:", error);
    // Don't stop the interval - just skip this frame
  }
}

Related Events

Camera Recognition Guide

Complete guide for implementing camera recognition with both WebRTC and snapshot methods

WebRTC Connection

Learn how to set up WebRTC connections for video transmission

For more details on camera recognition implementation, including WebRTC video stream transmission, see the Camera Recognition guide.