mirror of
https://github.com/katanemo/plano.git
synced 2026-06-17 15:25:17 +02:00
428 lines
27 KiB
ReStructuredText
428 lines
27 KiB
ReStructuredText
@model_server_endpoint = http://localhost:51000
|
|
@archfc_endpoint = http://localhost:5670
|
|
|
|
### Scenario: Single-Turn. Check for single required parameter
|
|
### Tools Used: apply_loan and check_account_balance
|
|
### Expected output(s): "content": "{\"required_functions\": [\"check_account_balance\"], \"clarification\": \"Could you please provide me with your account identifier?\"}",
|
|
### Notes: Tested 10 times. 9/10 times language is consistent and precise. The assistant asks for the missing parameter in a single-turn in the right format
|
|
### Status: Approved
|
|
### Tested By: Salman Paracha
|
|
POST {{archfc_endpoint}}/v1/chat/completions HTTP/1.1
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"model": "Arch-Function",
|
|
"messages": [
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"apply_loan\", \"description\": \"Apply for a personal loan\", \"parameters\": {\"type\": \"object\", \"properties\": {\"amount\": {\"type\": \"number\", \"description\": \"Loan amount in USD\"}, \"term_months\": {\"type\": \"integer\", \"description\": \"Term duration in months\"}, \"credit_score\": {\"type\": \"integer\", \"description\": \"Applicant's credit score\"}}, \"required\": [\"amount\", \"term_months\", \"credit_score\"]}}\n{\"name\": \"check_account_balance\", \"description\": \"Check balance of a financial account\", \"parameters\": {\"type\": \"object\", \"properties\": {\"account_id\": {\"type\": \"string\", \"description\": \"User's account number or identifier\"}}, \"required\": [\"account_id\"]}}\n</tools>\n\nYour task is to decide which functions are needed and collect missing parameters if necessary.\n\nBased on your analysis, provide your response in one of the following JSON formats:\n1. If no functions are needed:\n```\n{\"response\": \"Your response text here\"}\n```\n2. If functions are needed but some required parameters are missing:\n```\n{\"required_functions\": [\"func_name1\", \"func_name2\", ...], \"clarification\": \"Text asking for missing parameters\"}\n```\n3. If functions are needed and all required parameters are available:\n```\n{\"tool_calls\": [{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]}\n```"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "I want to check for my account balance"
|
|
}
|
|
],
|
|
"temperature": 0.6,
|
|
"top_p": 1.0,
|
|
"top_k": 10
|
|
}
|
|
|
|
|
|
### Scenario: Single-Turn. Check for all 3 required parameters.
|
|
### Tools Used: apply_loan and check_account_balance
|
|
### Expected output(s): "{\"required_functions\": [\"apply_loan\"], \"clarification\": \"Could you please provide me with the loan amount, the term duration in months, and your credit score?\"}",
|
|
### Notes: Tested 10 times. 8/10 times the language was consistent and precise. 2/10 the language was to abrupt and not clear and in those turns not all the required parameters were asked for.
|
|
### Status: Needs Work
|
|
### Tested By: Salman Paracha
|
|
POST {{archfc_endpoint}}/v1/chat/completions HTTP/1.1
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"model": "Arch-Function",
|
|
"messages": [
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"apply_loan\", \"description\": \"Apply for a personal loan\", \"parameters\": {\"type\": \"object\", \"properties\": {\"amount\": {\"type\": \"number\", \"description\": \"Loan amount in USD\"}, \"term_months\": {\"type\": \"integer\", \"description\": \"Term duration in months\"}, \"credit_score\": {\"type\": \"integer\", \"description\": \"Applicant's credit score\"}}, \"required\": [\"amount\", \"term_months\", \"credit_score\"]}}\n{\"name\": \"check_account_balance\", \"description\": \"Check balance of a financial account\", \"parameters\": {\"type\": \"object\", \"properties\": {\"account_id\": {\"type\": \"string\", \"description\": \"User's account number or identifier\"}}, \"required\": [\"account_id\"]}}\n</tools>\n\nYour task is to decide which functions are needed and collect missing parameters if necessary.\n\nBased on your analysis, provide your response in one of the following JSON formats:\n1. If no functions are needed:\n```\n{\"response\": \"Your response text here\"}\n```\n2. If functions are needed but some required parameters are missing:\n```\n{\"required_functions\": [\"func_name1\", \"func_name2\", ...], \"clarification\": \"Text asking for missing parameters\"}\n```\n3. If functions are needed and all required parameters are available:\n```\n{\"tool_calls\": [{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]}\n```"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "want to apply for a loan"
|
|
}
|
|
],
|
|
"temperature": 0.6,
|
|
"top_p": 1.0,
|
|
"top_k": 10
|
|
}
|
|
|
|
### Scenario: Multi-Turn(2) Chat History Context.
|
|
### Tools Used: apply_loan, get_stock_price and check_account_balance
|
|
### Expected output(s): "content": "{\"response\": \"You're welcome! If you need any more information, feel free to ask.\"}",
|
|
### Notes: Tested 10 times. Worked every time. The assistant was able to keep track of the chat history and provide the right function call based on the user query.
|
|
### Status: Approved
|
|
### Tested By: Salman Paracha
|
|
|
|
POST {{archfc_endpoint}}/v1/chat/completions HTTP/1.1
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"model": "Arch-Function",
|
|
"messages": [
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"get_stock_price\", \"description\": \"Get the current stock price\", \"parameters\": {\"type\": \"object\", \"properties\": {\"symbol\": {\"type\": \"string\", \"description\": \"The stock symbol\"}}, \"required\": [\"symbol\"]}}\n{\"name\": \"apply_loan\", \"description\": \"Apply for a personal loan\", \"parameters\": {\"type\": \"object\", \"properties\": {\"amount\": {\"type\": \"number\", \"description\": \"Loan amount in USD\"}, \"term_months\": {\"type\": \"integer\", \"description\": \"Term duration in months\"}, \"credit_score\": {\"type\": \"integer\", \"description\": \"Applicant's credit score\"}}, \"required\": [\"amount\", \"term_months\", \"credit_score\"]}}\n{\"name\": \"check_account_balance\", \"description\": \"Check balance of a financial account\", \"parameters\": {\"type\": \"object\", \"properties\": {\"account_id\": {\"type\": \"string\", \"description\": \"User's account number or identifier\"}}, \"required\": [\"account_id\"]}}\n</tools>\n\nYour task is to decide which functions are needed and collect missing parameters if necessary.\n\nBased on your analysis, provide your response in one of the following JSON formats:\n1. If no functions are needed:\n```\n{\"response\": \"Your response text here\"}\n```\n2. If functions are needed but some required parameters are missing:\n```\n{\"required_functions\": [\"func_name1\", \"func_name2\", ...], \"clarification\": \"Text asking for missing parameters\"}\n```\n3. If functions are needed and all required parameters are available:\n```\n{\"tool_calls\": [{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]}\n```"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "can you check the stock price for Amazon?"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"tool_calls\": [{\"name\": \"get_stock_price\", \"arguments\": {\"symbol\": \"AMZN\"}}]}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "<tool_response>\n{'name': 'get_stock_price', 'result': '$247.66'}\n</tool_response>"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"response\": \"The current stock price for AMZN is $247.66.\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "what about Apple"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"tool_calls\": [{\"name\": \"get_stock_price\", \"arguments\": {\"symbol\": \"AAPL\"}}]}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "<tool_response>\n{'name': 'get_stock_price', 'result': '$196.66'}\n</tool_response>"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"response\": \"The current stock price for AAPL is $196.66.\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "Thanks"
|
|
}
|
|
],
|
|
"temperature": 0.6,
|
|
"top_p": 1.0,
|
|
"top_k": 10
|
|
}
|
|
|
|
|
|
### Scenario: Multi-Turn (5) Chat History Context.
|
|
### Tools Used: apply_loan, get_stock_price and check_account_balance
|
|
### Expected output(s): "content": "{\"tool_calls\": [{\"name\": \"get_stock_price\", \"arguments\": {\"symbol\": \"ORCL\"}}]}",
|
|
### Notes: Tested 10 times. The assistant was able to keep track of the chat history and provide the right function call based on the user query.
|
|
### Tested By: Salman Paracha
|
|
### Status: Approved
|
|
|
|
POST {{archfc_endpoint}}/v1/chat/completions HTTP/1.1
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"model": "Arch-Function",
|
|
"messages": [
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"get_stock_price\", \"description\": \"Get the current stock price\", \"parameters\": {\"type\": \"object\", \"properties\": {\"symbol\": {\"type\": \"string\", \"description\": \"The stock symbol\"}}, \"required\": [\"symbol\"]}}\n{\"name\": \"apply_loan\", \"description\": \"Apply for a personal loan\", \"parameters\": {\"type\": \"object\", \"properties\": {\"amount\": {\"type\": \"number\", \"description\": \"Loan amount in USD\"}, \"term_months\": {\"type\": \"integer\", \"description\": \"Term duration in months\"}, \"credit_score\": {\"type\": \"integer\", \"description\": \"Applicant's credit score\"}}, \"required\": [\"amount\", \"term_months\", \"credit_score\"]}}\n{\"name\": \"check_account_balance\", \"description\": \"Check balance of a financial account\", \"parameters\": {\"type\": \"object\", \"properties\": {\"account_id\": {\"type\": \"string\", \"description\": \"User's account number or identifier\"}}, \"required\": [\"account_id\"]}}\n</tools>\n\nYour task is to decide which functions are needed and collect missing parameters if necessary.\n\nBased on your analysis, provide your response in one of the following JSON formats:\n1. If no functions are needed:\n```\n{\"response\": \"Your response text here\"}\n```\n2. If functions are needed but some required parameters are missing:\n```\n{\"required_functions\": [\"func_name1\", \"func_name2\", ...], \"clarification\": \"Text asking for missing parameters\"}\n```\n3. If functions are needed and all required parameters are available:\n```\n{\"tool_calls\": [{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]}\n```"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "can you check the stock price for Amazon?"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"tool_calls\": [{\"name\": \"get_stock_price\", \"arguments\": {\"symbol\": \"AMZN\"}}]}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "<tool_response>\n{'name': 'get_stock_price', 'result': '$247.66'}\n</tool_response>"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"response\": \"The current stock price for AMZN is $247.66.\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "what about Apple"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"tool_calls\": [{\"name\": \"get_stock_price\", \"arguments\": {\"symbol\": \"AAPL\"}}]}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "<tool_response>\n{'name': 'get_stock_price', 'result': '$196.66'}\n</tool_response>"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"response\": \"The current stock price for AAPL is $196.66.\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "How about Toyota"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"tool_calls\": [{\"name\": \"get_stock_price\", \"arguments\": {\"symbol\": \"TM\"}}]}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "<tool_response>\n{'name': 'get_stock_price', 'result': '$24.66'}\n</tool_response>"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"response\": \"The current stock price for TM is $196.66.\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "How about Nike"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"tool_calls\": [{\"name\": \"get_stock_price\", \"arguments\": {\"symbol\": \"NKE\"}}]}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "<tool_response>\n{'name': 'get_stock_price', 'result': '$102.66'}\n</tool_response>"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"response\": \"The current stock price for NKE is $102.66.\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "and what about Oracle"
|
|
}
|
|
],
|
|
"temperature": 0.6,
|
|
"top_p": 1.0,
|
|
"top_k": 10
|
|
}
|
|
|
|
|
|
### Scenario: Multi-Turn(2) Progressive Information Disclosure.
|
|
### Tools Used: apply_loan, get_stock_price and check_account_balance
|
|
### Expected output(s): "content": "{\"required_functions\": [\"apply_loan\"], \"clarification\": \"Could you please confirm the term duration and your credit score?\"}",
|
|
### Notes: Tested 10 times. Worked every time. The assistant was able to keep track of the chat history and provide the right function call based on the user query.
|
|
### Status: Approved
|
|
### Tested By: Salman Paracha
|
|
|
|
POST {{archfc_endpoint}}/v1/chat/completions HTTP/1.1
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"model": "Arch-Function",
|
|
"messages": [
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"get_stock_price\", \"description\": \"Get the current stock price\", \"parameters\": {\"type\": \"object\", \"properties\": {\"symbol\": {\"type\": \"string\", \"description\": \"The stock symbol\"}}, \"required\": [\"symbol\"]}}\n{\"name\": \"apply_loan\", \"description\": \"Apply for a personal loan\", \"parameters\": {\"type\": \"object\", \"properties\": {\"amount\": {\"type\": \"number\", \"description\": \"Loan amount in USD\"}, \"term_months\": {\"type\": \"integer\", \"description\": \"Term duration in months\"}, \"credit_score\": {\"type\": \"integer\", \"description\": \"Applicant's credit score\"}}, \"required\": [\"amount\", \"term_months\", \"credit_score\"]}}\n{\"name\": \"check_account_balance\", \"description\": \"Check balance of a financial account\", \"parameters\": {\"type\": \"object\", \"properties\": {\"account_id\": {\"type\": \"string\", \"description\": \"User's account number or identifier\"}}, \"required\": [\"account_id\"]}}\n</tools>\n\nYour task is to decide which functions are needed and collect missing parameters if necessary.\n\nBased on your analysis, provide your response in one of the following JSON formats:\n1. If no functions are needed:\n```\n{\"response\": \"Your response text here\"}\n```\n2. If functions are needed but some required parameters are missing:\n```\n{\"required_functions\": [\"func_name1\", \"func_name2\", ...], \"clarification\": \"Text asking for missing parameters\"}\n```\n3. If functions are needed and all required parameters are available:\n```\n{\"tool_calls\": [{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]}\n```"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "I want to apply for a loan"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"required_functions\": [\"apply_loan\"], \"clarification\": \"Could you please provide me with the amount you want to loan, the term duration in months, and your credit score?\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "$5000"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"required_functions\": [\"apply_loan\"], \"clarification\": \"Could you please confirm the term duration and your credit score?\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "actually make the loan for $6000"
|
|
}
|
|
],
|
|
"temperature": 0.6,
|
|
"top_p": 1.0,
|
|
"top_k": 10
|
|
}
|
|
|
|
|
|
### Scenario: Multi-Turn(5) Progressive Information Disclosure.
|
|
### Tools Used: apply_loan, get_stock_price and check_account_balance
|
|
### Expected output(s): "content": "{\"response\": \"Have a great day\"}",
|
|
### Notes: Tested 10 times. Worked every time. The assistant was able to keep track of the chat history and provide the right function call based on the user query.
|
|
### Status: Approved
|
|
### Tested By: Salman Paracha
|
|
|
|
POST {{archfc_endpoint}}/v1/chat/completions HTTP/1.1
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"model": "Arch-Function",
|
|
"messages": [
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"apply_loan\", \"description\": \"Apply for a personal loan\", \"parameters\": {\"type\": \"object\", \"properties\": {\"amount\": {\"type\": \"number\", \"description\": \"Loan amount in USD\"}, \"term_months\": {\"type\": \"integer\", \"description\": \"Term duration in months\"}, \"credit_score\": {\"type\": \"integer\", \"description\": \"Applicant's credit score\"}}, \"required\": [\"amount\", \"term_months\", \"credit_score\"]}}\n{\"name\": \"check_account_balance\", \"description\": \"Check balance of a financial account\", \"parameters\": {\"type\": \"object\", \"properties\": {\"account_id\": {\"type\": \"string\", \"description\": \"User's account number or identifier\"}}, \"required\": [\"account_id\"]}}\n</tools>\n\nYour task is to decide which functions are needed and collect missing parameters if necessary.\n\nBased on your analysis, provide your response in one of the following JSON formats:\n1. If no functions are needed:\n```\n{\"response\": \"Your response text here\"}\n```\n2. If functions are needed but some required parameters are missing:\n```\n{\"required_functions\": [\"func_name1\", \"func_name2\", ...], \"clarification\": \"Text asking for missing parameters\"}\n```\n3. If functions are needed and all required parameters are available:\n```\n{\"tool_calls\": [{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]}\n```"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "I want to apply for a loan"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"required_functions\": [\"apply_loan\"], \"clarification\": \"Could you please provide me with the amount you want to loan, the term duration in months, and your credit score?\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "$5000"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"required_functions\": [\"apply_loan\"], \"clarification\": \"Could you please provide me with the term duration in months and your credit score?\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "18 months"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"required_functions\": [\"apply_loan\"], \"clarification\": \"Could you please provide me with your credit score?\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "700"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"tool_calls\": [{\"name\": \"apply_loan\", \"arguments\": {\"amount\": 5000, \"term_months\": 18, \"credit_score\": 700}}]}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "<tool_response>\n{'name': 'apply_loan', 'approved': 'Done'}\n</tool_response>"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "your loan has been approved"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "Thanks"
|
|
}
|
|
],
|
|
"temperature": 0.6,
|
|
"top_p": 1.0,
|
|
"top_k": 10
|
|
}
|
|
|
|
|
|
### Scenario: Multi-Turn Progressive Information Disclosure with multiple function calling.
|
|
### Tools Used: apply_loan, get_stock_price and check_account_balance
|
|
### Tested By: Salman Paracha
|
|
### Expected output(s): "content": "{\"tool_calls\": [{\"name\": \"apply_loan\", \"arguments\": {\"amount\": 6000, \"term_months\": 18, \"credit_score\": 700}}]}",
|
|
### Notes: The assistant doesn't engage in gathering information for the second function call. The assistant should ask for the missing parameter for the second function call.
|
|
### Status: Rejected
|
|
POST {{archfc_endpoint}}/v1/chat/completions HTTP/1.1
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"model": "Arch-Function",
|
|
"messages": [
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"apply_loan\", \"description\": \"Apply for a personal loan\", \"parameters\": {\"type\": \"object\", \"properties\": {\"amount\": {\"type\": \"number\", \"description\": \"Loan amount in USD\"}, \"term_months\": {\"type\": \"integer\", \"description\": \"Term duration in months\"}, \"credit_score\": {\"type\": \"integer\", \"description\": \"Applicant's credit score\"}}, \"required\": [\"amount\", \"term_months\", \"credit_score\"]}}\n{\"name\": \"check_account_balance\", \"description\": \"Check balance of a financial account\", \"parameters\": {\"type\": \"object\", \"properties\": {\"account_id\": {\"type\": \"string\", \"description\": \"User's account number or identifier\"}}, \"required\": [\"account_id\"]}}\n</tools>\n\nYour task is to decide which functions are needed and collect missing parameters if necessary.\n\nBased on your analysis, provide your response in one of the following JSON formats:\n1. If no functions are needed:\n```\n{\"response\": \"Your response text here\"}\n```\n2. If functions are needed but some required parameters are missing:\n```\n{\"required_functions\": [\"func_name1\", \"func_name2\", ...], \"clarification\": \"Text asking for missing parameters\"}\n```\n3. If functions are needed and all required parameters are available:\n```\n{\"tool_calls\": [{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]}\n```"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "I want to apply for a loan and check my account balance"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"required_functions\": [\"apply_loan\", \"check_account_balance\"], \"clarification\": \"Could you please provide me with the amount you want to borrow, the term of the loan in months, and your credit score?\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "amount $5000, 18 months term and credit score of 780"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"tool_calls\": [{\"name\": \"apply_loan\", \"arguments\": {\"amount\": 5000, \"term_months\": 18, \"credit_score\": 700}}]}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "<tool_response>\n{'name': 'apply_loan', 'result': 'Status: Ok'}\n</tool_response>"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"response\": \"Status: Ok. You have sucessfully applied for the loan\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "Ok thanks"
|
|
}
|
|
],
|
|
"temperature": 0.6,
|
|
"top_p": 1.0,
|
|
"top_k": 10
|
|
}
|
|
|
|
### Scenario: Multi-Turn(3) Progressive Information Disclosure; Changing parameter values.
|
|
### Tools Used: apply_loan, get_stock_price and check_account_balance
|
|
### Expected output(s): "content": "{\"tool_calls\": [{\"name\": \"apply_loan\", \"arguments\": {\"amount\": 6000, \"term_months\": 18, \"credit_score\": 700}}]}",
|
|
### Notes: Tested 10 times. Worked every time. The assistant was able to keep track of the chat history and provide the right function call based on the user query.
|
|
### Status: Approved
|
|
### Tested By: Salman Paracha
|
|
|
|
POST {{archfc_endpoint}}/v1/chat/completions HTTP/1.1
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"model": "Arch-Function",
|
|
"messages": [
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"get_stock_price\", \"description\": \"Get the current stock price\", \"parameters\": {\"type\": \"object\", \"properties\": {\"symbol\": {\"type\": \"string\", \"description\": \"The stock symbol\"}}, \"required\": [\"symbol\"]}}\n{\"name\": \"apply_loan\", \"description\": \"Apply for a personal loan\", \"parameters\": {\"type\": \"object\", \"properties\": {\"amount\": {\"type\": \"number\", \"description\": \"Loan amount in USD\"}, \"term_months\": {\"type\": \"integer\", \"description\": \"Term duration in months\"}, \"credit_score\": {\"type\": \"integer\", \"description\": \"Applicant's credit score\"}}, \"required\": [\"amount\", \"term_months\", \"credit_score\"]}}\n{\"name\": \"check_account_balance\", \"description\": \"Check balance of a financial account\", \"parameters\": {\"type\": \"object\", \"properties\": {\"account_id\": {\"type\": \"string\", \"description\": \"User's account number or identifier\"}}, \"required\": [\"account_id\"]}}\n</tools>\n\nYour task is to decide which functions are needed and collect missing parameters if necessary.\n\nBased on your analysis, provide your response in one of the following JSON formats:\n1. If no functions are needed:\n```\n{\"response\": \"Your response text here\"}\n```\n2. If functions are needed but some required parameters are missing:\n```\n{\"required_functions\": [\"func_name1\", \"func_name2\", ...], \"clarification\": \"Text asking for missing parameters\"}\n```\n3. If functions are needed and all required parameters are available:\n```\n{\"tool_calls\": [{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]}\n```"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "I want to apply for a loan"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"required_functions\": [\"apply_loan\"], \"clarification\": \"Could you please provide me with the amount you want to loan, the term duration in months, and your credit score?\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "$5000"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"required_functions\": [\"apply_loan\"], \"clarification\": \"Could you please confirm the term duration and your credit score?\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "actually make the loan for $6000"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"required_functions\": [\"apply_loan\"], \"clarification\": \"Please confirm the term duration and your credit score.\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "for 18 months"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"required_functions\": [\"apply_loan\"], \"clarification\": \"Please confirm your credit score.\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "700"
|
|
}
|
|
],
|
|
"temperature": 0.6,
|
|
"top_p": 1.0,
|
|
"top_k": 10
|
|
}
|