> ## Documentation Index
> Fetch the complete documentation index at: https://docs.navtalk.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# realtime.input_image

> Send camera image to AI for visual recognition and analysis

Send a camera snapshot or image to the AI for visual recognition and analysis. This event is used to transmit base64-encoded images captured from the user's camera for scene analysis, object recognition, or visual Q\&A.

<Warning>
  **Provider Compatibility**: Image input is only supported with the **OpenAIRealtime** provider. This feature is not available when using other AI providers (e.g., ElevenLabs).
</Warning>

## Message Format

This is a **client-to-server event** that you send to transmit image data.

<ParamField body="type" type="string" required>
  Event type. Must be `"realtime.input_image"`.
</ParamField>

<ParamField body="data" type="object" required>
  Event data object containing the image content and reply flag.

  <Expandable title="data properties">
    <ParamField body="content" type="string" required>
      Base64-encoded image in Data URL format. Format: `data:image/jpeg;base64,{base64_string}`

      **Supported formats**: JPEG, PNG, WebP

      **Recommended**: JPEG with quality 0.7 (70%) for optimal balance between quality and payload size
    </ParamField>

    <ParamField body="reply" type="integer" default="0">
      Whether to request an immediate AI response for this image:

      * `0`: No immediate reply (default) - Image is added to conversation context
      * `1`: Request reply - AI will analyze and respond to the image

      **Use `0`** for periodic camera frames where you don't need a response for every frame. **Use `1`** when the user explicitly asks the AI to analyze the current camera view.
    </ParamField>
  </Expandable>
</ParamField>

## Request Example

<RequestExample>
  ```javascript JavaScript theme={null}
  const NavTalkMessageType = {
    REALTIME_INPUT_IMAGE: "realtime.input_image"
  };

  // Send image without requesting immediate reply
  function sendImageMessage(imageUrl) {
    if (!imageUrl || !socket || socket.readyState !== WebSocket.OPEN) {
      console.warn("WebSocket not open or empty input");
      return;
    }
    
    const message = {
      type: NavTalkMessageType.REALTIME_INPUT_IMAGE,
      data: { 
        content: imageUrl,  // e.g., "data:image/jpeg;base64,/9j/4AAQ..."
        reply: 0            // Don't request immediate reply
      }
    };
    
    socket.send(JSON.stringify(message));
  }

  // Example: Capture camera frame and send
  async function captureAndSend() {
    const video = document.getElementById('camera-preview');
    
    // Create canvas to capture frame
    const canvas = document.createElement('canvas');
    canvas.width = video.videoWidth || 640;
    canvas.height = video.videoHeight || 360;
    const ctx = canvas.getContext('2d');
    ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
    
    // Convert to base64
    canvas.toBlob(async (blob) => {
      const arrayBuffer = await blob.arrayBuffer();
      const uint8Array = new Uint8Array(arrayBuffer);
      const base64 = btoa(String.fromCharCode.apply(null, uint8Array));
      const imageUrl = `data:${blob.type};base64,${base64}`;
      
      sendImageMessage(imageUrl);
    }, "image/jpeg", 0.7);
  }
  ```

  ```python Python theme={null}
  import json
  import base64
  import cv2
  from io import BytesIO
  from PIL import Image

  async def send_image_message(websocket, image_url, request_reply=False):
      """Send image to AI"""
      if not image_url or websocket.closed:
          print("WebSocket not open or empty input")
          return
      
      message = {
          "type": "realtime.input_image",
          "data": {
              "content": image_url,  # Data URL format
              "reply": 1 if request_reply else 0
          }
      }
      
      await websocket.send(json.dumps(message))

  # Example: Capture from OpenCV camera and send
  async def capture_and_send(camera, websocket):
      """Capture frame from camera and send to AI"""
      ret, frame = camera.read()
      if not ret:
          return
      
      # Convert to PIL Image
      frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
      pil_image = Image.fromarray(frame_rgb)
      
      # Convert to base64
      buffer = BytesIO()
      pil_image.save(buffer, format='JPEG', quality=70)
      image_bytes = buffer.getvalue()
      base64_image = base64.b64encode(image_bytes).decode('utf-8')
      image_url = f"data:image/jpeg;base64,{base64_image}"
      
      # Send without requesting reply
      await send_image_message(websocket, image_url, request_reply=False)
  ```
</RequestExample>

## Message Structure

```json theme={null}
{
  "type": "realtime.input_image",
  "data": {
    "content": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQ...",
    "reply": 0
  }
}
```

## Usage Scenarios

### Scenario 1: Periodic Scene Recognition

Send camera frames periodically (e.g., every 2 seconds) without requesting replies:

```javascript theme={null}
// Start periodic capture
setInterval(() => {
  captureAndSend();  // reply: 0 (no immediate response)
}, 2000);
```

The AI builds visual context over time and can reference what it "sees" when responding to user questions.

### Scenario 2: On-demand Visual Q\&A

User asks "What do you see?" - capture and send with `reply: 1`:

```javascript theme={null}
// User asks a visual question
async function handleVisualQuestion() {
  const video = document.getElementById('camera-preview');
  
  // Capture current frame
  const imageUrl = await captureFrame(video);
  
  // Send with reply flag to get AI's analysis
  const message = {
    type: "realtime.input_image",
    data: { 
      content: imageUrl,
      reply: 1  // Request AI to analyze and respond
    }
  };
  
  socket.send(JSON.stringify(message));
}
```

### Scenario 3: Hybrid Approach (Recommended)

Combine periodic background capture (`reply: 0`) with on-demand analysis (`reply: 1`):

```javascript theme={null}
// Background: Periodic capture for context (every 2 seconds)
setInterval(() => {
  captureAndSend(false);  // reply: 0
}, 2000);

// Foreground: User-initiated visual analysis
async function analyzeCurrentView() {
  await captureAndSend(true);  // reply: 1
}
```

## Best Practices

<AccordionGroup>
  <Accordion title="Image Format and Quality" icon="image">
    * **Format**: Use JPEG for optimal compression and compatibility
    * **Quality**: Set to 0.7 (70%) to balance quality and payload size
    * **Resolution**: 640x360 is sufficient for most recognition tasks
    * **File Size**: Target 20-50KB per frame after compression
  </Accordion>

  <Accordion title="Transmission Frequency" icon="clock">
    * **Periodic capture**: 2-3 seconds interval for most use cases
    * **Adjust based on use case**:
      * Fast-moving scenes: 1 second
      * Static scenes: 3-5 seconds
    * **Don't exceed 1 frame per second** to avoid overwhelming the AI and consuming excessive bandwidth
  </Accordion>

  <Accordion title="Reply Flag Usage" icon="flag">
    * **Use `reply: 0`** (default) for:
      * Background periodic capture
      * Building visual context
      * Continuous monitoring without responses
    * **Use `reply: 1`** when:
      * User explicitly asks about the camera view
      * You need immediate AI analysis
      * Triggering specific visual recognition tasks
  </Accordion>

  <Accordion title="Error Handling" icon="shield">
    ```javascript theme={null}
    function captureFrameAndSend() {
      try {
        // Check prerequisites
        if (!socket || socket.readyState !== WebSocket.OPEN) {
          console.warn("WebSocket not ready");
          return;
        }
        
        if (!cameraStream || !video || video.readyState < 2) {
          console.warn("Camera not ready");
          return;
        }
        
        // Capture and send
        // ... implementation ...
      } catch (error) {
        console.error("Error capturing frame:", error);
        // Don't stop the interval - just skip this frame
      }
    }
    ```
  </Accordion>
</AccordionGroup>

## Related Events

<CardGroup cols={2}>
  <Card title="Camera Recognition Guide" icon="camera" href="/api/real-time-digital-human-api/camera-recognition">
    Complete guide for implementing camera recognition with both WebRTC and snapshot methods
  </Card>

  <Card title="WebRTC Connection" icon="video" href="/api/real-time-digital-human-api/quick-start/webrtc-connection">
    Learn how to set up WebRTC connections for video transmission
  </Card>
</CardGroup>

<Note>
  For more details on camera recognition implementation, including WebRTC video stream transmission, see the [Camera Recognition](/api/real-time-digital-human-api/camera-recognition) guide.
</Note>