Function Calling

Integrate external APIs and execute custom functions during conversations

NavTalk supports function calling to integrate digital humans with external systems, enabling the execution of custom functions and API calls during conversations. This allows the digital human to interact with weather APIs, databases, CRM systems, IoT devices, and other external services.

Provider Support:

• OpenAIRealtime: Function calling can be defined and configured directly via API parameters (as shown in this guide). You can pass function definitions programmatically.
• ElevenLabs: Function calling must be configured through the ElevenLabs dashboard. It cannot be defined via API parameters - you need to set up functions in your ElevenLabs account settings before using them.

1. Define Function Tools

   Define the functions that can be called by including them in the tools array. In v2, function tools configuration is sent immediately after the WebSocket connection is established (in the onopen event), not after receiving realtime.session.created:

   JavaScript

   const NavTalkMessageType = Object.freeze({
       REALTIME_INPUT_FUNCTION_CALL: "realtime.input_function_call",
       REALTIME_RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE: "realtime.response.function_call_arguments.done",
       REALTIME_FUNCTION_CALL_OUTPUT: "realtime.function_call_output",
       // ... other event types
   });
   
   // Send function tools configuration when WebSocket connection is established
   function sendFunctionCallConfig() {
     const functions = [
       {
         type: 'function',
         name: 'function_call_judge',
         description: 'Automatically triggers function extension calls when user requests exceed the current conversation capabilities.',
         parameters: {
           type: 'object',
           properties: {
             userInput: {
               type: 'string',
               description: 'Raw user request content to be processed'
             }
           },
           required: ['userInput']
         }
       }
     ];
     
     const functionJson = JSON.stringify(functions);
     const message = {
       type: NavTalkMessageType.REALTIME_INPUT_FUNCTION_CALL,
       data: {
         content: functionJson
       }
     };
     
     socket.send(JSON.stringify(message));
   }
   
   // Call this when WebSocket connection is established
   socket.onopen = () => {
     console.log('WebSocket connection established');
     sendFunctionCallConfig();
   };
   

   NAV_TALK_MESSAGE_TYPE = {
       'REALTIME_INPUT_FUNCTION_CALL': 'realtime.input_function_call',
       'REALTIME_RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE': 'realtime.response.function_call_arguments.done',
       'REALTIME_FUNCTION_CALL_OUTPUT': 'realtime.function_call_output',
       # ... other event types
   }
   
   # Send function tools configuration when WebSocket connection is established
   async def send_function_call_config(websocket):
       functions = [
           {
               'type': 'function',
               'name': 'function_call_judge',
               'description': 'Automatically triggers function extension calls when user requests exceed the current conversation capabilities.',
               'parameters': {
                   'type': 'object',
                   'properties': {
                       'userInput': {
                           'type': 'string',
                           'description': 'Raw user request content to be processed'
                       }
                   },
                   'required': ['userInput']
               }
           }
       ]
       
       function_json = json.dumps(functions)
       message = {
           'type': NAV_TALK_MESSAGE_TYPE['REALTIME_INPUT_FUNCTION_CALL'],
           'data': {
               'content': function_json
           }
       }
       
       await websocket.send(json.dumps(message))
   
   # Call this when WebSocket connection is established
   async def connect_websocket():
       websocket = await websockets.connect(websocket_url)
       await send_function_call_config(websocket)
       return websocket
   

   You can define multiple functions in the tools array. Each function should have a unique name, clear description, and well-defined parameters. Function tools configuration is sent immediately after the WebSocket connection is established, not after receiving realtime.session.created. Connection identity is passed through the WebSocket URL (license plus name or avatarId); per-session configuration such as voice, prompt, and tools is sent with realtime.input_config.

2. Listen for Function Call Events

   Listen for function call events in your WebSocket message handler. When a function needs to be called, you’ll receive a realtime.response.function_call_arguments.done event:

   JavaScript

   const NavTalkMessageType = Object.freeze({
       REALTIME_RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE: "realtime.response.function_call_arguments.done",
       // ... other event types
   });
   
   socket.onmessage = (event) => {
     if (typeof event.data === 'string') {
       const data = JSON.parse(event.data);
       const nav_data = data.data;
       
       switch (data.type) {
         case NavTalkMessageType.REALTIME_RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE:
           handleFunctionCall(data);
           break;
         
         // ... handle other message types
       }
     }
   };
   

   NAV_TALK_MESSAGE_TYPE = {
       'REALTIME_RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE': 'realtime.response.function_call_arguments.done',
       # ... other event types
   }
   
   async def handle_messages(websocket):
       async for message in websocket:
           if isinstance(message, str):
               data = json.loads(message)
               nav_data = data.get('data')
               
               if data.get('type') == NAV_TALK_MESSAGE_TYPE['REALTIME_RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE']:
                   await handle_function_call(data, websocket)
               
               # ... handle other message types
   

   The realtime.response.function_call_arguments.done event contains the following properties:

   data.function_name · string

   Name of the function to be called.

   Example: "function_call_judge"

   data.arguments · string | object

   Function call parameters. May be a JSON string or already parsed object containing the parameter values.

   Example: "{\"userInput\": \"What's the weather in Beijing?\"}" or {"userInput": "What's the weather in Beijing?"}

   data.call_id · string

   Unique identifier for this function call. Required when returning results - use this call_id when sending the function call output back to the AI.

   Example: "call-123456"

3. Handle Function Calls

   Parse the function call arguments and execute your custom logic:

   JavaScript

   function handleFunctionCall(data) {
     try {
       const nav_data = data.data;
       if (!nav_data) {
         console.error('Invalid function call message:', data);
         return;
       }
       
       // Extract function call details
       const functionName = nav_data.function_name;
       const callId = nav_data.call_id;
       let functionCallArgs = {};
       
       // Parse function arguments
       if (nav_data.arguments) {
         try {
           if (typeof nav_data.arguments === 'string') {
             functionCallArgs = JSON.parse(nav_data.arguments);
           } else {
             // arguments is already an object, use it directly
             functionCallArgs = nav_data.arguments;
           }
         } catch (parseError) {
           console.error('Failed to parse function call arguments:', parseError);
         }
       }
       
       // Handle different function calls
       switch (functionName) {
         case 'function_call_judge':
           const userInput = functionCallArgs.userInput;
           if (userInput) {
             // Call your external API or business logic
             callExternalAPI(userInput)
               .then(result => {
                 // Send the result back
                 sendFunctionCallResult(result, callId);
               })
               .catch(error => {
                 console.error('Function call failed:', error);
                 // Send error result
                 sendFunctionCallResult('Error: ' + error.message, callId);
               });
           } else {
             console.error('Missing required parameter: userInput');
           }
           break;
         
         default:
           console.warn('Unhandled function call:', functionName, functionCallArgs);
       }
     } catch (error) {
       console.error('Error parsing function call arguments:', error);
     }
   }
   

   async def handle_function_call(data, websocket):
       try:
           nav_data = data.get('data')
           if not nav_data:
               print(f'Invalid function call message: {data}')
               return
           
           # Extract function call details
           function_name = nav_data.get('function_name')
           call_id = nav_data.get('call_id')
           function_call_args = {}
           
           # Parse function arguments
           arguments_param = nav_data.get('arguments')
           if arguments_param:
               try:
                   if isinstance(arguments_param, str):
                       function_call_args = json.loads(arguments_param)
                   else:
                       # arguments is already a dictionary, use it directly
                       function_call_args = arguments_param
               except json.JSONDecodeError as parse_error:
                   print(f'Failed to parse function call arguments: {parse_error}')
           
           # Handle different function calls
           if function_name == 'function_call_judge':
               user_input = function_call_args.get('userInput')
               if user_input:
                   # Call your external API or business logic
                   try:
                       result = await call_external_api(user_input)
                       # Send the result back
                       await send_function_call_result(result, call_id, websocket)
                   except Exception as error:
                       print(f'Function call failed: {error}')
                       # Send error result
                       await send_function_call_result(f'Error: {str(error)}', call_id, websocket)
               else:
                   print('Missing required parameter: userInput')
           else:
               print(f'Unhandled function call: {function_name}, {function_call_args}')
       except Exception as error:
           print(f'Error handling function call: {error}')
   
4. Integrate External APIs

   Call your external APIs or backend services to handle the business logic:

   JavaScript

   async function callExternalAPI(userInput) {
     return new Promise(async (resolve, reject) => {
       // Example: Call your backend API
       fetch('https://your-api.com/api/function-call', {
         method: 'POST',
         headers: {
           'Content-Type': 'application/json',
         },
         body: JSON.stringify({
           userInput: userInput,
           license: 'your-license-key',
           // ... other parameters
         }),
       })
         .then(response => {
           if (!response.ok) {
             throw new Error(`HTTP error! status: ${response.status}`);
           }
           return response.text(); // or response.json() for JSON responses
         })
         .then(result => {
           console.log('API result:', result);
           resolve(result);
         })
         .catch(error => {
           console.error('API call error:', error);
           reject(error);
         });
     });
   }
   
   // Example: Weather API integration
   async function getWeather(location) {
     const response = await fetch(`https://api.weather.com/forecast?location=${location}`);
     const data = await response.json();
     return `Weather in ${location}: ${data.condition}, ${data.temperature}°C`;
   }
   
   // Example: Knowledge base query
   async function queryKnowledgeBase(query) {
     const response = await fetch('https://your-kb-api.com/search', {
       method: 'POST',
       headers: { 'Content-Type': 'application/json' },
       body: JSON.stringify({ query: query })
     });
     const results = await response.json();
     return results.summary || 'No results found';
   }
   

   import aiohttp
   
   async def call_external_api(user_input):
       """Call your external API or backend service"""
       async with aiohttp.ClientSession() as session:
           async with session.post(
               'https://your-api.com/api/function-call',
               json={
                   'userInput': user_input,
                   'license': 'your-license-key',
                   # ... other parameters
               },
               headers={'Content-Type': 'application/json'}
           ) as response:
               if response.status == 200:
                   result = await response.text()
                   print(f'API result: {result}')
                   return result
               else:
                   raise Exception(f'HTTP error! status: {response.status}')
   
   # Example: Weather API integration
   async def get_weather(location):
       async with aiohttp.ClientSession() as session:
           async with session.get(f'https://api.weather.com/forecast?location={location}') as response:
               data = await response.json()
               return f"Weather in {location}: {data['condition']}, {data['temperature']}°C"
   
   # Example: Knowledge base query
   async def query_knowledge_base(query):
       async with aiohttp.ClientSession() as session:
           async with session.post(
               'https://your-kb-api.com/search',
               json={'query': query}
           ) as response:
               results = await response.json()
               return results.get('summary', 'No results found')
   
5. Optionally Return Results to AI

   After processing the function call, send the result back only when you want the AI to continue the conversation using the function result. If the function call only needs to trigger external business logic and no follow-up AI response is needed, you can omit this step.

   JavaScript

   function sendFunctionCallResult(result, callId) {
     // Send function call output back to NavTalk.
     // reply: "1" asks the AI to continue with this result.
     const resultJson = {
       type: NavTalkMessageType.REALTIME_FUNCTION_CALL_OUTPUT,
       data: {
         content: result,
         call_id: callId,
         reply: "1"
       }
     };
     
     socket.send(JSON.stringify(resultJson));
     console.log('Sent function call result:', resultJson);
   }
   

   async def send_function_call_result(result, call_id, websocket):
       # Send function call output back to NavTalk.
       # reply: "1" asks the AI to continue with this result.
       result_json = {
           'type': NAV_TALK_MESSAGE_TYPE['REALTIME_FUNCTION_CALL_OUTPUT'],
           'data': {
               'content': result,
               'call_id': call_id,
               'reply': '1'
           }
       }
       
       await websocket.send(json.dumps(result_json))
       print(f'Sent function call result: {result_json}')
   

   Send realtime.function_call_output only when the AI should use the function result to generate a follow-up reply. Set data.reply to "1" to trigger that reply. If no follow-up AI interaction is needed, handle the function result in your application and skip this message.

Common Use Cases

Use Case	Example	Implementation
Weather API	User asks: “What’s the weather in Beijing?”	Call weather API with location, return forecast
Knowledge Base	User asks: “How do I integrate function calling?”	Search knowledge base, return relevant documentation
Database Query	User asks: “Show me today’s sales numbers”	Query database, return formatted results
IoT Control	User says: “Turn on the lights”	Call IoT API to control devices
CRM Integration	User asks: “Update customer status for John”	Call CRM API to update records

Function calling allows you to extend the digital human’s capabilities beyond conversation. Integrate with any REST API, database, or backend service to create powerful, context-aware applications.