Function Calling

Integrate external APIs and execute custom functions during conversations

NavTalk supports function calling to integrate digital humans with external systems, enabling the execution of custom functions and API calls during conversations. This allows the digital human to interact with weather APIs, databases, CRM systems, IoT devices, and other external services.

Provider Support:

• OpenAIRealtime: Function calling can be defined and configured directly via API parameters (as shown in this guide). You can pass function definitions programmatically.
• ElevenLabs: Function calling must be configured through the ElevenLabs dashboard. It cannot be defined via API parameters - you need to set up functions in your ElevenLabs account settings before using them.

1. Define Function Tools

   Define the functions that can be called by including them in the tools array. In v2, function tools configuration is sent immediately after the WebSocket connection is established (in the onopen event), not after receiving realtime.session.created:

   JavaScript

   const NavTalkMessageType = Object.freeze({
       REALTIME_INPUT_FUNCTION_CALL: "realtime.input_function_call",
       REALTIME_RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE: "realtime.response.function_call_arguments.done",
       REALTIME_FUNCTION_CALL_OUTPUT: "realtime.function_call_output",
       // ... other event types
   });
   
   // Send function tools configuration when WebSocket connection is established
   function sendFunctionCallConfig() {
     const functions = [
       {
         type: 'function',
         name: 'function_call_judge',
         description: 'Automatically triggers function extension calls when user requests exceed the current conversation capabilities.',
         parameters: {
           type: 'object',
           properties: {
             userInput: {
               type: 'string',
               description: 'Raw user request content to be processed'
             }
           },
           required: ['userInput']
         }
       }
     ];
     
     const functionJson = JSON.stringify(functions);
     const message = {
       type: NavTalkMessageType.REALTIME_INPUT_FUNCTION_CALL,
       data: {
         content: functionJson
       }
     };
     
     socket.send(JSON.stringify(message));
   }
   
   // Call this when WebSocket connection is established
   socket.onopen = () => {
     console.log('WebSocket connection established');
     sendFunctionCallConfig();
   };
   

   You can define multiple functions in the tools array. Each function should have a unique name, clear description, and well-defined parameters. Function tools configuration is sent immediately after WebSocket connection is established, not after receiving realtime.session.created. Session configuration (voice, model, instructions, etc.) is passed through WebSocket connection URL parameters.

2. Listen for Function Call Events

   Listen for function call events in your WebSocket message handler. When a function needs to be called, you’ll receive a realtime.response.function_call_arguments.done event:

   JavaScript

   const NavTalkMessageType = Object.freeze({
       REALTIME_RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE: "realtime.response.function_call_arguments.done",
       // ... other event types
   });
   
   socket.onmessage = (event) => {
     if (typeof event.data === 'string') {
       const data = JSON.parse(event.data);
       const nav_data = data.data;
       
       switch (data.type) {
         case NavTalkMessageType.REALTIME_RESPONSE_FUNCTION_CALL_ARGUMENTS_DONE:
           handleFunctionCall(data);
           break;
         
         // ... handle other message types
       }
     }
   };
   

   The realtime.response.function_call_arguments.done event contains the following properties:

   data.function_name · string

   Name of the function to be called.

   Example: "function_call_judge"

   data.arguments · string | object

   Function call parameters. May be a JSON string or already parsed object containing the parameter values.

   Example: "{\"userInput\": \"What's the weather in Beijing?\"}" or {"userInput": "What's the weather in Beijing?"}

   data.call_id · string

   Unique identifier for this function call. Required when returning results - use this call_id when sending the function call output back to the AI.

   Example: "call-123456"

3. Handle Function Calls

   Parse the function call arguments and execute your custom logic:

   JavaScript

   function handleFunctionCall(data) {
     try {
       const nav_data = data.data;
       if (!nav_data) {
         console.error('Invalid function call message:', data);
         return;
       }
       
       // Extract function call details
       const functionName = nav_data.function_name;
       const callId = nav_data.call_id;
       let functionCallArgs = {};
       
       // Parse function arguments
       if (nav_data.arguments) {
         try {
           if (typeof nav_data.arguments === 'string') {
             functionCallArgs = JSON.parse(nav_data.arguments);
           } else {
             // arguments is already an object, use it directly
             functionCallArgs = nav_data.arguments;
           }
         } catch (parseError) {
           console.error('Failed to parse function call arguments:', parseError);
         }
       }
       
       // Handle different function calls
       switch (functionName) {
         case 'function_call_judge':
           const userInput = functionCallArgs.userInput;
           if (userInput) {
             // Call your external API or business logic
             callExternalAPI(userInput)
               .then(result => {
                 // Send the result back
                 sendFunctionCallResult(result, callId);
               })
               .catch(error => {
                 console.error('Function call failed:', error);
                 // Send error result
                 sendFunctionCallResult('Error: ' + error.message, callId);
               });
           } else {
             console.error('Missing required parameter: userInput');
           }
           break;
         
         default:
           console.warn('Unhandled function call:', functionName, functionCallArgs);
       }
     } catch (error) {
       console.error('Error parsing function call arguments:', error);
     }
   }
   
4. Integrate External APIs

   Call your external APIs or backend services to handle the business logic:

   JavaScript

   async function callExternalAPI(userInput) {
     return new Promise(async (resolve, reject) => {
       // Example: Call your backend API
       fetch('https://your-api.com/api/function-call', {
         method: 'POST',
         headers: {
           'Content-Type': 'application/json',
         },
         body: JSON.stringify({
           userInput: userInput,
           license: 'your-license-key',
           // ... other parameters
         }),
       })
         .then(response => {
           if (!response.ok) {
             throw new Error(`HTTP error! status: ${response.status}`);
           }
           return response.text(); // or response.json() for JSON responses
         })
         .then(result => {
           console.log('API result:', result);
           resolve(result);
         })
         .catch(error => {
           console.error('API call error:', error);
           reject(error);
         });
     });
   }
   
   // Example: Weather API integration
   async function getWeather(location) {
     const response = await fetch(`https://api.weather.com/forecast?location=${location}`);
     const data = await response.json();
     return `Weather in ${location}: ${data.condition}, ${data.temperature}°C`;
   }
   
   // Example: Knowledge base query
   async function queryKnowledgeBase(query) {
     const response = await fetch('https://your-kb-api.com/search', {
       method: 'POST',
       headers: { 'Content-Type': 'application/json' },
       body: JSON.stringify({ query: query })
     });
     const results = await response.json();
     return results.summary || 'No results found';
   }
   
5. Return Results to AI

   After processing the function call, send the result back to the AI so it can continue the conversation:

   JavaScript

   function sendFunctionCallResult(result, callId) {
     // Send function call output with correct format
     const resultJson = {
       type: NavTalkMessageType.REALTIME_FUNCTION_CALL_OUTPUT,
       data: {
         output: result,
         call_id: callId
       }
     };
     
     socket.send(JSON.stringify(resultJson));
     console.log('Sent function call result:', resultJson);
     
     // Actively request a response.create to get the result
     const responseCreateJson = {
       type: "response.create"
     };
     socket.send(JSON.stringify(responseCreateJson));
     console.log('Requested response.create after function call result');
   }
   

   The reply: 1 field in the function call output message indicates that the AI should automatically generate a response based on the function result. The AI will process the function result and continue the conversation without needing a separate response.create message.

Common Use Cases

Use Case	Example	Implementation
Weather API	User asks: “What’s the weather in Beijing?”	Call weather API with location, return forecast
Knowledge Base	User asks: “How do I integrate function calling?”	Search knowledge base, return relevant documentation
Database Query	User asks: “Show me today’s sales numbers”	Query database, return formatted results
IoT Control	User says: “Turn on the lights”	Call IoT API to control devices
CRM Integration	User asks: “Update customer status for John”	Call CRM API to update records

Function calling allows you to extend the digital human’s capabilities beyond conversation. Integrate with any REST API, database, or backend service to create powerful, context-aware applications.