> ## Documentation Index
> Fetch the complete documentation index at: https://docs.navtalk.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Function Calling

> Integrate external APIs and execute custom functions during conversations

NavTalk supports function calling to integrate digital humans with external systems, enabling the execution of custom functions and API calls during conversations. This allows the digital human to interact with weather APIs, databases, CRM systems, IoT devices, and other external services.

<Note>
  **Provider Support**:

  * **OpenAIRealtime**: Function calling can be defined and configured directly via API parameters (as shown in this guide). You can pass function definitions programmatically.
  * **ElevenLabs**: Function calling must be configured through the [ElevenLabs dashboard](https://elevenlabs.io/). It cannot be defined via API parameters - you need to set up functions in your ElevenLabs account settings before using them.
</Note>

<Steps>
  <Step title="Define Function Tools">
    Define the functions that can be called by including them in the `tools` array. In v2, function tools configuration is sent immediately after the WebSocket connection is established (in the `onopen` event), not after receiving `realtime.session.created`:

    <CodeGroup>
      ```javascript JavaScript theme={null}
      const NavTalkMessageType = Object.freeze({
          REALTIME_INPUT_FUNCTION_CALL: "realtime.input_function_call",
          REALTIME_RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE: "realtime.response.function_call_arguments.done",
          REALTIME_FUNCTION_CALL_OUTPUT: "realtime.function_call_output",
          // ... other event types
      });

      // Send function tools configuration when WebSocket connection is established
      function sendFunctionCallConfig() {
        const functions = [
          {
            type: 'function',
            name: 'function_call_judge',
            description: 'Automatically triggers function extension calls when user requests exceed the current conversation capabilities.',
            parameters: {
              type: 'object',
              properties: {
                userInput: {
                  type: 'string',
                  description: 'Raw user request content to be processed'
                }
              },
              required: ['userInput']
            }
          }
        ];
        
        const functionJson = JSON.stringify(functions);
        const message = {
          type: NavTalkMessageType.REALTIME_INPUT_FUNCTION_CALL,
          data: {
            content: functionJson
          }
        };
        
        socket.send(JSON.stringify(message));
      }

      // Call this when WebSocket connection is established
      socket.onopen = () => {
        console.log('WebSocket connection established');
        sendFunctionCallConfig();
      };
      ```

      ```python Python theme={null}
      NAV_TALK_MESSAGE_TYPE = {
          'REALTIME_INPUT_FUNCTION_CALL': 'realtime.input_function_call',
          'REALTIME_RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE': 'realtime.response.function_call_arguments.done',
          'REALTIME_FUNCTION_CALL_OUTPUT': 'realtime.function_call_output',
          # ... other event types
      }

      # Send function tools configuration when WebSocket connection is established
      async def send_function_call_config(websocket):
          functions = [
              {
                  'type': 'function',
                  'name': 'function_call_judge',
                  'description': 'Automatically triggers function extension calls when user requests exceed the current conversation capabilities.',
                  'parameters': {
                      'type': 'object',
                      'properties': {
                          'userInput': {
                              'type': 'string',
                              'description': 'Raw user request content to be processed'
                          }
                      },
                      'required': ['userInput']
                  }
              }
          ]
          
          function_json = json.dumps(functions)
          message = {
              'type': NAV_TALK_MESSAGE_TYPE['REALTIME_INPUT_FUNCTION_CALL'],
              'data': {
                  'content': function_json
              }
          }
          
          await websocket.send(json.dumps(message))

      # Call this when WebSocket connection is established
      async def connect_websocket():
          websocket = await websockets.connect(websocket_url)
          await send_function_call_config(websocket)
          return websocket
      ```
    </CodeGroup>

    <Note>
      You can define multiple functions in the `tools` array. Each function should have a unique name, clear description, and well-defined parameters. Function tools configuration is sent immediately after the WebSocket connection is established, not after receiving `realtime.session.created`. Connection identity is passed through the WebSocket URL (`license` plus `name` or `avatarId`); per-session configuration such as voice, prompt, and tools is sent with `realtime.input_config`.
    </Note>
  </Step>

  <Step title="Listen for Function Call Events">
    Listen for function call events in your WebSocket message handler. When a function needs to be called, you'll receive a `realtime.response.function_call_arguments.done` event:

    <CodeGroup>
      ```javascript JavaScript theme={null}
      const NavTalkMessageType = Object.freeze({
          REALTIME_RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE: "realtime.response.function_call_arguments.done",
          // ... other event types
      });

      socket.onmessage = (event) => {
        if (typeof event.data === 'string') {
          const data = JSON.parse(event.data);
          const nav_data = data.data;
          
          switch (data.type) {
            case NavTalkMessageType.REALTIME_RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE:
              handleFunctionCall(data);
              break;
            
            // ... handle other message types
          }
        }
      };
      ```

      ```python Python theme={null}
      NAV_TALK_MESSAGE_TYPE = {
          'REALTIME_RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE': 'realtime.response.function_call_arguments.done',
          # ... other event types
      }

      async def handle_messages(websocket):
          async for message in websocket:
              if isinstance(message, str):
                  data = json.loads(message)
                  nav_data = data.get('data')
                  
                  if data.get('type') == NAV_TALK_MESSAGE_TYPE['REALTIME_RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE']:
                      await handle_function_call(data, websocket)
                  
                  # ... handle other message types
      ```
    </CodeGroup>

    The `realtime.response.function_call_arguments.done` event contains the following properties:

    <ResponseField name="data.function_name" type="string">
      Name of the function to be called.

      **Example**: `"function_call_judge"`
    </ResponseField>

    <ResponseField name="data.arguments" type="string | object">
      Function call parameters. May be a JSON string or already parsed object containing the parameter values.

      **Example**: `"{\"userInput\": \"What's the weather in Beijing?\"}"` or `{"userInput": "What's the weather in Beijing?"}`
    </ResponseField>

    <ResponseField name="data.call_id" type="string">
      Unique identifier for this function call. **Required** when returning results - use this `call_id` when sending the function call output back to the AI.

      **Example**: `"call-123456"`
    </ResponseField>
  </Step>

  <Step title="Handle Function Calls">
    Parse the function call arguments and execute your custom logic:

    <CodeGroup>
      ```javascript JavaScript theme={null}
      function handleFunctionCall(data) {
        try {
          const nav_data = data.data;
          if (!nav_data) {
            console.error('Invalid function call message:', data);
            return;
          }
          
          // Extract function call details
          const functionName = nav_data.function_name;
          const callId = nav_data.call_id;
          let functionCallArgs = {};
          
          // Parse function arguments
          if (nav_data.arguments) {
            try {
              if (typeof nav_data.arguments === 'string') {
                functionCallArgs = JSON.parse(nav_data.arguments);
              } else {
                // arguments is already an object, use it directly
                functionCallArgs = nav_data.arguments;
              }
            } catch (parseError) {
              console.error('Failed to parse function call arguments:', parseError);
            }
          }
          
          // Handle different function calls
          switch (functionName) {
            case 'function_call_judge':
              const userInput = functionCallArgs.userInput;
              if (userInput) {
                // Call your external API or business logic
                callExternalAPI(userInput)
                  .then(result => {
                    // Send the result back
                    sendFunctionCallResult(result, callId);
                  })
                  .catch(error => {
                    console.error('Function call failed:', error);
                    // Send error result
                    sendFunctionCallResult('Error: ' + error.message, callId);
                  });
              } else {
                console.error('Missing required parameter: userInput');
              }
              break;
            
            default:
              console.warn('Unhandled function call:', functionName, functionCallArgs);
          }
        } catch (error) {
          console.error('Error parsing function call arguments:', error);
        }
      }
      ```

      ```python Python theme={null}
      async def handle_function_call(data, websocket):
          try:
              nav_data = data.get('data')
              if not nav_data:
                  print(f'Invalid function call message: {data}')
                  return
              
              # Extract function call details
              function_name = nav_data.get('function_name')
              call_id = nav_data.get('call_id')
              function_call_args = {}
              
              # Parse function arguments
              arguments_param = nav_data.get('arguments')
              if arguments_param:
                  try:
                      if isinstance(arguments_param, str):
                          function_call_args = json.loads(arguments_param)
                      else:
                          # arguments is already a dictionary, use it directly
                          function_call_args = arguments_param
                  except json.JSONDecodeError as parse_error:
                      print(f'Failed to parse function call arguments: {parse_error}')
              
              # Handle different function calls
              if function_name == 'function_call_judge':
                  user_input = function_call_args.get('userInput')
                  if user_input:
                      # Call your external API or business logic
                      try:
                          result = await call_external_api(user_input)
                          # Send the result back
                          await send_function_call_result(result, call_id, websocket)
                      except Exception as error:
                          print(f'Function call failed: {error}')
                          # Send error result
                          await send_function_call_result(f'Error: {str(error)}', call_id, websocket)
                  else:
                      print('Missing required parameter: userInput')
              else:
                  print(f'Unhandled function call: {function_name}, {function_call_args}')
          except Exception as error:
              print(f'Error handling function call: {error}')
      ```
    </CodeGroup>
  </Step>

  <Step title="Integrate External APIs">
    Call your external APIs or backend services to handle the business logic:

    <CodeGroup>
      ```javascript JavaScript theme={null}
      async function callExternalAPI(userInput) {
        return new Promise(async (resolve, reject) => {
          // Example: Call your backend API
          fetch('https://your-api.com/api/function-call', {
            method: 'POST',
            headers: {
              'Content-Type': 'application/json',
            },
            body: JSON.stringify({
              userInput: userInput,
              license: 'your-license-key',
              // ... other parameters
            }),
          })
            .then(response => {
              if (!response.ok) {
                throw new Error(`HTTP error! status: ${response.status}`);
              }
              return response.text(); // or response.json() for JSON responses
            })
            .then(result => {
              console.log('API result:', result);
              resolve(result);
            })
            .catch(error => {
              console.error('API call error:', error);
              reject(error);
            });
        });
      }

      // Example: Weather API integration
      async function getWeather(location) {
        const response = await fetch(`https://api.weather.com/forecast?location=${location}`);
        const data = await response.json();
        return `Weather in ${location}: ${data.condition}, ${data.temperature}°C`;
      }

      // Example: Knowledge base query
      async function queryKnowledgeBase(query) {
        const response = await fetch('https://your-kb-api.com/search', {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify({ query: query })
        });
        const results = await response.json();
        return results.summary || 'No results found';
      }
      ```

      ```python Python theme={null}
      import aiohttp

      async def call_external_api(user_input):
          """Call your external API or backend service"""
          async with aiohttp.ClientSession() as session:
              async with session.post(
                  'https://your-api.com/api/function-call',
                  json={
                      'userInput': user_input,
                      'license': 'your-license-key',
                      # ... other parameters
                  },
                  headers={'Content-Type': 'application/json'}
              ) as response:
                  if response.status == 200:
                      result = await response.text()
                      print(f'API result: {result}')
                      return result
                  else:
                      raise Exception(f'HTTP error! status: {response.status}')

      # Example: Weather API integration
      async def get_weather(location):
          async with aiohttp.ClientSession() as session:
              async with session.get(f'https://api.weather.com/forecast?location={location}') as response:
                  data = await response.json()
                  return f"Weather in {location}: {data['condition']}, {data['temperature']}°C"

      # Example: Knowledge base query
      async def query_knowledge_base(query):
          async with aiohttp.ClientSession() as session:
              async with session.post(
                  'https://your-kb-api.com/search',
                  json={'query': query}
              ) as response:
                  results = await response.json()
                  return results.get('summary', 'No results found')
      ```
    </CodeGroup>
  </Step>

  <Step title="Optionally Return Results to AI">
    After processing the function call, send the result back only when you want the AI to continue the conversation using the function result. If the function call only needs to trigger external business logic and no follow-up AI response is needed, you can omit this step.

    <CodeGroup>
      ```javascript JavaScript theme={null}
      function sendFunctionCallResult(result, callId) {
        // Send function call output back to NavTalk.
        // reply: "1" asks the AI to continue with this result.
        const resultJson = {
          type: NavTalkMessageType.REALTIME_FUNCTION_CALL_OUTPUT,
          data: {
            content: result,
            call_id: callId,
            reply: "1"
          }
        };
        
        socket.send(JSON.stringify(resultJson));
        console.log('Sent function call result:', resultJson);
      }
      ```

      ```python Python theme={null}
      async def send_function_call_result(result, call_id, websocket):
          # Send function call output back to NavTalk.
          # reply: "1" asks the AI to continue with this result.
          result_json = {
              'type': NAV_TALK_MESSAGE_TYPE['REALTIME_FUNCTION_CALL_OUTPUT'],
              'data': {
                  'content': result,
                  'call_id': call_id,
                  'reply': '1'
              }
          }
          
          await websocket.send(json.dumps(result_json))
          print(f'Sent function call result: {result_json}')
      ```
    </CodeGroup>

    <Note>
      Send `realtime.function_call_output` only when the AI should use the function result to generate a follow-up reply. Set `data.reply` to `"1"` to trigger that reply. If no follow-up AI interaction is needed, handle the function result in your application and skip this message.
    </Note>
  </Step>
</Steps>

## Common Use Cases

| Use Case            | Example                                           | Implementation                                       |
| ------------------- | ------------------------------------------------- | ---------------------------------------------------- |
| **Weather API**     | User asks: "What's the weather in Beijing?"       | Call weather API with location, return forecast      |
| **Knowledge Base**  | User asks: "How do I integrate function calling?" | Search knowledge base, return relevant documentation |
| **Database Query**  | User asks: "Show me today's sales numbers"        | Query database, return formatted results             |
| **IoT Control**     | User says: "Turn on the lights"                   | Call IoT API to control devices                      |
| **CRM Integration** | User asks: "Update customer status for John"      | Call CRM API to update records                       |

<Note>
  Function calling allows you to extend the digital human's capabilities beyond conversation. Integrate with any REST API, database, or backend service to create powerful, context-aware applications.
</Note>