mirror of
https://github.com/katanemo/plano.git
synced 2026-06-17 15:25:17 +02:00
724 lines
54 KiB
ReStructuredText
724 lines
54 KiB
ReStructuredText
@archfc_endpoint = http://localhost:8282
|
|
|
|
# tools = [
|
|
# {
|
|
# "name": "verify_address",
|
|
# "description": "Verify the address",
|
|
# "parameters": {
|
|
# "type": "object",
|
|
# "properties": {
|
|
# "address": {"type": "str", "description": "The address to verify"}
|
|
# },
|
|
# "required": ["address"],
|
|
# },
|
|
# },
|
|
# {
|
|
# "name": "track_shipment",
|
|
# "description": "Track the shipment",
|
|
# "parameters": {
|
|
# "type": "object",
|
|
# "properties": {
|
|
# "tracking_number": {
|
|
# "type": "str",
|
|
# "description": "The tracking number",
|
|
# }
|
|
# },
|
|
# "required": ["tracking_number"],
|
|
# },
|
|
# },
|
|
# {
|
|
# "name": "generate_label",
|
|
# "description": "Generate the shipping label",
|
|
# "parameters": {
|
|
# "type": "object",
|
|
# "properties": {
|
|
# "sender": {"type": "str", "description": "The address to ship from"},
|
|
# "recipient": {
|
|
# "type": "str",
|
|
# "description": "The address to ship to",
|
|
# },
|
|
# "weight": {
|
|
# "type": "int",
|
|
# "description": "The weight of the package",
|
|
# },
|
|
# },
|
|
# "required": ["sender", "recipient", "weight"],
|
|
# },
|
|
# },
|
|
# {
|
|
# "name": "calculate_shipping_rate",
|
|
# "description": "Calculate the shipping rate",
|
|
# "parameters": {
|
|
# "type": "object",
|
|
# "properties": {
|
|
# "sender": {"type": "str", "description": "The address to ship from"},
|
|
# "recipient": {
|
|
# "type": "str",
|
|
# "description": "The address to ship to",
|
|
# },
|
|
# "weight": {
|
|
# "type": "int",
|
|
# "description": "The weight of the package",
|
|
# },
|
|
# },
|
|
# "required": ["sender", "recipient", "weight"],
|
|
# },
|
|
# },
|
|
# ]
|
|
|
|
|
|
# system prompt: "You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"verify_address\", \"description\": \"Verify the address\", \"parameters\": {\"type\": \"object\", \"properties\": {\"address\": {\"type\": \"str\", \"description\": \"The address to verify\"}}, \"required\": [\"address\"]}}\n{\"name\": \"track_shipment\", \"description\": \"Track the shipment\", \"parameters\": {\"type\": \"object\", \"properties\": {\"tracking_number\": {\"type\": \"str\", \"description\": \"The tracking number\"}}, \"required\": [\"tracking_number\"]}}\n{\"name\": \"generate_label\", \"description\": \"Generate the shipping label\", \"parameters\": {\"type\": \"object\", \"properties\": {\"sender\": {\"type\": \"str\", \"description\": \"The address to ship from\"}, \"recipient\": {\"type\": \"str\", \"description\": \"The address to ship to\"}, \"weight\": {\"type\": \"int\", \"description\": \"The weight of the package\"}}, \"required\": [\"sender\", \"recipient\", \"weight\"]}}\n{\"name\": \"calculate_shipping_rate\", \"description\": \"Calculate the shipping rate\", \"parameters\": {\"type\": \"object\", \"properties\": {\"sender\": {\"type\": \"str\", \"description\": \"The address to ship from\"}, \"recipient\": {\"type\": \"str\", \"description\": \"The address to ship to\"}, \"weight\": {\"type\": \"int\", \"description\": \"The weight of the package\"}}, \"required\": [\"sender\", \"recipient\", \"weight\"]}}\n</tools>\n\nYour task is to decide which functions are needed and collect missing parameters if necessary.\n\nBased on your analysis, provide your response in one of the following JSON formats:\n1. If no functions are needed:\n```\n{\"response\": \"Your response text here\"}\n```\n2. If functions are needed but some required parameters are missing:\n```\n{\"required_functions\": [\"func_name1\", \"func_name2\", ...], \"clarification\": \"Text asking for missing parameters\"}\n```\n3. If functions are needed and all required parameters are available:\n```\n{\"tool_calls\": [{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]}\n```"
|
|
|
|
|
|
### scenario: single turn - User request missing one critical parameter
|
|
### tools used: verify_address, track_shipment, generate_label, calculate_shipping_rate,
|
|
### expected sample response: {\"required_functions\": [\"track_shipment\"], \"clarification\": \"Could you please provide me with the tracking number?\"}
|
|
### Status: Approved - worked 10/10
|
|
POST {{archfc_endpoint}}/v1/chat/completions
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"model": "Arch-Function",
|
|
"messages": [
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"verify_address\", \"description\": \"Verify the address\", \"parameters\": {\"type\": \"object\", \"properties\": {\"address\": {\"type\": \"str\", \"description\": \"The address to verify\"}}, \"required\": [\"address\"]}}\n{\"name\": \"track_shipment\", \"description\": \"Track the shipment\", \"parameters\": {\"type\": \"object\", \"properties\": {\"tracking_number\": {\"type\": \"str\", \"description\": \"The tracking number\"}}, \"required\": [\"tracking_number\"]}}\n{\"name\": \"generate_label\", \"description\": \"Generate the shipping label\", \"parameters\": {\"type\": \"object\", \"properties\": {\"sender\": {\"type\": \"str\", \"description\": \"The address to ship from\"}, \"recipient\": {\"type\": \"str\", \"description\": \"The address to ship to\"}, \"weight\": {\"type\": \"int\", \"description\": \"The weight of the package\"}}, \"required\": [\"sender\", \"recipient\", \"weight\"]}}\n{\"name\": \"calculate_shipping_rate\", \"description\": \"Calculate the shipping rate\", \"parameters\": {\"type\": \"object\", \"properties\": {\"sender\": {\"type\": \"str\", \"description\": \"The address to ship from\"}, \"recipient\": {\"type\": \"str\", \"description\": \"The address to ship to\"}, \"weight\": {\"type\": \"int\", \"description\": \"The weight of the package\"}}, \"required\": [\"sender\", \"recipient\", \"weight\"]}}\n</tools>\n\nYour task is to decide which functions are needed and collect missing parameters if necessary.\n\nBased on your analysis, provide your response in one of the following JSON formats:\n1. If no functions are needed:\n```\n{\"response\": \"Your response text here\"}\n```\n2. If functions are needed but some required parameters are missing:\n```\n{\"required_functions\": [\"func_name1\", \"func_name2\", ...], \"clarification\": \"Text asking for missing parameters\"}\n```\n3. If functions are needed and all required parameters are available:\n```\n{\"tool_calls\": [{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]}\n```"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "I want to track my shipment"
|
|
}
|
|
],
|
|
"temperature": 0.6,
|
|
"top_p": 1.0,
|
|
"top_k": 10
|
|
}
|
|
|
|
|
|
### scenario: single turn - User request missing multiple required parameters
|
|
### tools used: verify_address, track_shipment, generate_label, calculate_shipping_rate,
|
|
### expected sample response: {\"required_functions\": [\"generate_label\"], \"clarification\": \"Could you please provide me with the sender's address, recipient's address, and the weight of the package?\"}
|
|
### Status: Approved - worked 10/10
|
|
POST {{archfc_endpoint}}/v1/chat/completions
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"model": "Arch-Function",
|
|
"messages": [
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"verify_address\", \"description\": \"Verify the address\", \"parameters\": {\"type\": \"object\", \"properties\": {\"address\": {\"type\": \"str\", \"description\": \"The address to verify\"}}, \"required\": [\"address\"]}}\n{\"name\": \"track_shipment\", \"description\": \"Track the shipment\", \"parameters\": {\"type\": \"object\", \"properties\": {\"tracking_number\": {\"type\": \"str\", \"description\": \"The tracking number\"}}, \"required\": [\"tracking_number\"]}}\n{\"name\": \"generate_label\", \"description\": \"Generate the shipping label\", \"parameters\": {\"type\": \"object\", \"properties\": {\"sender\": {\"type\": \"str\", \"description\": \"The address to ship from\"}, \"recipient\": {\"type\": \"str\", \"description\": \"The address to ship to\"}, \"weight\": {\"type\": \"int\", \"description\": \"The weight of the package\"}}, \"required\": [\"sender\", \"recipient\", \"weight\"]}}\n{\"name\": \"calculate_shipping_rate\", \"description\": \"Calculate the shipping rate\", \"parameters\": {\"type\": \"object\", \"properties\": {\"sender\": {\"type\": \"str\", \"description\": \"The address to ship from\"}, \"recipient\": {\"type\": \"str\", \"description\": \"The address to ship to\"}, \"weight\": {\"type\": \"int\", \"description\": \"The weight of the package\"}}, \"required\": [\"sender\", \"recipient\", \"weight\"]}}\n</tools>\n\nYour task is to decide which functions are needed and collect missing parameters if necessary.\n\nBased on your analysis, provide your response in one of the following JSON formats:\n1. If no functions are needed:\n```\n{\"response\": \"Your response text here\"}\n```\n2. If functions are needed but some required parameters are missing:\n```\n{\"required_functions\": [\"func_name1\", \"func_name2\", ...], \"clarification\": \"Text asking for missing parameters\"}\n```\n3. If functions are needed and all required parameters are available:\n```\n{\"tool_calls\": [{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]}\n```"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "I want to create a shipping label"
|
|
}
|
|
],
|
|
"temperature": 0.6,
|
|
"top_p": 1.0,
|
|
"top_k": 10
|
|
}
|
|
|
|
|
|
### scenario: single turn - User providing vague information for a required parameter
|
|
### tools used: verify_address, track_shipment, generate_label, calculate_shipping_rate,
|
|
### expected sample response: {\"required_functions\": [\"verify_address\"], \"clarification\": \"I can help you verify the address, please give me your address\"}
|
|
### Status: Not Approved
|
|
### Notes: model keeps sending tool call with incorrect address. Following is an example response from model,
|
|
|
|
### {\"tool_calls\": [{\"name\": \"verify_address\", \"arguments\": {\"address\": \"My new home address\"}}]}
|
|
### {\"tool_calls\": [{\"name\": \"verify_address\", \"arguments\": {\"address\": \"Newly built home address\"}}]}
|
|
|
|
POST {{archfc_endpoint}}/v1/chat/completions
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"model": "Arch-Function",
|
|
"messages": [
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"verify_address\", \"description\": \"Verify the address\", \"parameters\": {\"type\": \"object\", \"properties\": {\"address\": {\"type\": \"str\", \"description\": \"The address to verify\"}}, \"required\": [\"address\"]}}\n{\"name\": \"track_shipment\", \"description\": \"Track the shipment\", \"parameters\": {\"type\": \"object\", \"properties\": {\"tracking_number\": {\"type\": \"str\", \"description\": \"The tracking number\"}}, \"required\": [\"tracking_number\"]}}\n{\"name\": \"generate_label\", \"description\": \"Generate the shipping label\", \"parameters\": {\"type\": \"object\", \"properties\": {\"sender\": {\"type\": \"str\", \"description\": \"The address to ship from\"}, \"recipient\": {\"type\": \"str\", \"description\": \"The address to ship to\"}, \"weight\": {\"type\": \"int\", \"description\": \"The weight of the package\"}}, \"required\": [\"sender\", \"recipient\", \"weight\"]}}\n{\"name\": \"calculate_shipping_rate\", \"description\": \"Calculate the shipping rate\", \"parameters\": {\"type\": \"object\", \"properties\": {\"sender\": {\"type\": \"str\", \"description\": \"The address to ship from\"}, \"recipient\": {\"type\": \"str\", \"description\": \"The address to ship to\"}, \"weight\": {\"type\": \"int\", \"description\": \"The weight of the package\"}}, \"required\": [\"sender\", \"recipient\", \"weight\"]}}\n</tools>\n\nYour task is to decide which functions are needed and collect missing parameters if necessary.\n\nBased on your analysis, provide your response in one of the following JSON formats:\n1. If no functions are needed:\n```\n{\"response\": \"Your response text here\"}\n```\n2. If functions are needed but some required parameters are missing:\n```\n{\"required_functions\": [\"func_name1\", \"func_name2\", ...], \"clarification\": \"Text asking for missing parameters\"}\n```\n3. If functions are needed and all required parameters are available:\n```\n{\"tool_calls\": [{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]}\n```"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "I recently moved to a newly built home but I am not sure if the address is correct or not"
|
|
}
|
|
],
|
|
"temperature": 0.6,
|
|
"top_p": 1.0,
|
|
"top_k": 10
|
|
}
|
|
|
|
|
|
### scenario: two turn - User providing vague information for a required parameter - take 2
|
|
### tools used: verify_address, track_shipment, generate_label, calculate_shipping_rate,
|
|
### expected sample response: {\"required_functions\": [\"verify_address\"], \"clarification\": \"I can help you verify the address, please give me your address\"}
|
|
### Status: Not Approved
|
|
### Notes: model keeps sending tool call with incorrect address. Expected response is to ask for the sender and recipient address. Here is the sample response,
|
|
|
|
### {\"tool_calls\": [{\"name\": \"calculate_shipping_rate\", \"arguments\": {\"sender\": \"your_address\", \"recipient\": \"friend_address\", \"weight\": 15}}]}
|
|
### {\"tool_calls\": [{\"name\": \"calculate_shipping_rate\", \"arguments\": {\"sender\": \"My address\", \"recipient\": \"Friend's address in Germany\", \"weight\": \"10-15 pounds\"}}]}
|
|
|
|
POST {{archfc_endpoint}}/v1/chat/completions
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"model": "Arch-Function",
|
|
"messages": [
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"verify_address\", \"description\": \"Verify the address\", \"parameters\": {\"type\": \"object\", \"properties\": {\"address\": {\"type\": \"str\", \"description\": \"The address to verify\"}}, \"required\": [\"address\"]}}\n{\"name\": \"track_shipment\", \"description\": \"Track the shipment\", \"parameters\": {\"type\": \"object\", \"properties\": {\"tracking_number\": {\"type\": \"str\", \"description\": \"The tracking number\"}}, \"required\": [\"tracking_number\"]}}\n{\"name\": \"generate_label\", \"description\": \"Generate the shipping label\", \"parameters\": {\"type\": \"object\", \"properties\": {\"sender\": {\"type\": \"str\", \"description\": \"The address to ship from\"}, \"recipient\": {\"type\": \"str\", \"description\": \"The address to ship to\"}, \"weight\": {\"type\": \"int\", \"description\": \"The weight of the package\"}}, \"required\": [\"sender\", \"recipient\", \"weight\"]}}\n{\"name\": \"calculate_shipping_rate\", \"description\": \"Calculate the shipping rate\", \"parameters\": {\"type\": \"object\", \"properties\": {\"sender\": {\"type\": \"str\", \"description\": \"The address to ship from\"}, \"recipient\": {\"type\": \"str\", \"description\": \"The address to ship to\"}, \"weight\": {\"type\": \"int\", \"description\": \"The weight of the package\"}}, \"required\": [\"sender\", \"recipient\", \"weight\"]}}\n</tools>\n\nYour task is to decide which functions are needed and collect missing parameters if necessary.\n\nBased on your analysis, provide your response in one of the following JSON formats:\n1. If no functions are needed:\n```\n{\"response\": \"Your response text here\"}\n```\n2. If functions are needed but some required parameters are missing:\n```\n{\"required_functions\": [\"func_name1\", \"func_name2\", ...], \"clarification\": \"Text asking for missing parameters\"}\n```\n3. If functions are needed and all required parameters are available:\n```\n{\"tool_calls\": [{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]}\n```"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "I want to send gift to my friend who lives in germany but I am not sure how much it would cost to ship the package"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"required_functions\": [\"calculate_shipping_rate\"], \"clarification\": \"Could you please provide me with the weight of the package?\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "I am not sure about the weight but it would be around 10-15 pounds"
|
|
}
|
|
],
|
|
"temperature": 0.6,
|
|
"top_p": 1.0,
|
|
"top_k": 10
|
|
}
|
|
|
|
### Domain-specific parameter omission
|
|
|
|
### scenario: single turn - Domain-specific parameter omission
|
|
# tools used: verify_address, track_shipment, generate_label, calculate_shipping_rate,
|
|
# expected sample response: ask for sender address
|
|
# Status: Not Approved
|
|
# Notes: model is not asking for sender and recipient address. Following is an example response from model. There are times when model does ask for sender address. But still falls short of asking for recipient address.
|
|
|
|
# sample response,
|
|
# "{\"required_functions\": [\"generate_label\", \"calculate_shipping_rate\"], \"clarification\": \"Where will the package be sent from?\"}",
|
|
# "{\"required_functions\": [\"generate_label\"], \"clarification\": \"Could you please provide me with the sender's address?\"}",
|
|
# "{\"required_functions\": [\"generate_label\", \"calculate_shipping_rate\"], \"clarification\": \"Could you please provide me with your address?\"}",
|
|
# "{\"tool_calls\": [{\"name\": \"generate_label\", \"arguments\": {\"sender\": \"Your address\", \"recipient\": \"Your friend's address\", \"weight\": 10}}]}",
|
|
# "{\"required_functions\": [\"generate_label\", \"calculate_shipping_rate\"], \"clarification\": \"Could you please provide me with the sender's address?\"}",
|
|
|
|
POST {{archfc_endpoint}}/v1/chat/completions
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"model": "Arch-Function",
|
|
"messages": [
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"verify_address\", \"description\": \"Verify the address\", \"parameters\": {\"type\": \"object\", \"properties\": {\"address\": {\"type\": \"str\", \"description\": \"The address to verify\"}}, \"required\": [\"address\"]}}\n{\"name\": \"track_shipment\", \"description\": \"Track the shipment\", \"parameters\": {\"type\": \"object\", \"properties\": {\"tracking_number\": {\"type\": \"str\", \"description\": \"The tracking number\"}}, \"required\": [\"tracking_number\"]}}\n{\"name\": \"generate_label\", \"description\": \"Generate the shipping label\", \"parameters\": {\"type\": \"object\", \"properties\": {\"sender\": {\"type\": \"str\", \"description\": \"The address to ship from\"}, \"recipient\": {\"type\": \"str\", \"description\": \"The address to ship to\"}, \"weight\": {\"type\": \"int\", \"description\": \"The weight of the package\"}}, \"required\": [\"sender\", \"recipient\", \"weight\"]}}\n{\"name\": \"calculate_shipping_rate\", \"description\": \"Calculate the shipping rate\", \"parameters\": {\"type\": \"object\", \"properties\": {\"sender\": {\"type\": \"str\", \"description\": \"The address to ship from\"}, \"recipient\": {\"type\": \"str\", \"description\": \"The address to ship to\"}, \"weight\": {\"type\": \"int\", \"description\": \"The weight of the package\"}}, \"required\": [\"sender\", \"recipient\", \"weight\"]}}\n</tools>\n\nYour task is to decide which functions are needed and collect missing parameters if necessary.\n\nBased on your analysis, provide your response in one of the following JSON formats:\n1. If no functions are needed:\n```\n{\"response\": \"Your response text here\"}\n```\n2. If functions are needed but some required parameters are missing:\n```\n{\"required_functions\": [\"func_name1\", \"func_name2\", ...], \"clarification\": \"Text asking for missing parameters\"}\n```\n3. If functions are needed and all required parameters are available:\n```\n{\"tool_calls\": [{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]}\n```"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "I want to sending a package to my friend. He lives in New York and the package is 10 pounds"
|
|
}
|
|
],
|
|
"temperature": 0.6,
|
|
"top_p": 1.0,
|
|
"top_k": 10
|
|
}
|
|
|
|
### scenario: 2-turn - omitting domain specific parameter
|
|
# tools used: verify_address, track_shipment, generate_label, calculate_shipping_rate,
|
|
# expected sample response: ask for sender address
|
|
# Status: Not Approved
|
|
# Notes: Model is confused a bit, it keeps asking for recipient address but did not ask details about sender address. And in some occasions it asked about all required parameters. And few times it just returned tool.
|
|
|
|
# sample response,
|
|
# "{\"required_functions\": [\"generate_label\"], \"clarification\": \"Can you provide me with the sender address, recipient address, and the weight of the package?\"}",
|
|
# "{\"required_functions\": [\"generate_label\"], \"clarification\": \"Could you please provide me with the recipient address?\"}",
|
|
# "{\"required_functions\": [\"generate_label\"], \"clarification\": \"Can you please provide me with the recipient's address?\"}",
|
|
# "{\"required_functions\": [\"generate_label\"], \"clarification\": \"Could you please provide me with the recipient address?\"}",
|
|
# "{\"required_functions\": [\"generate_label\"], \"clarification\": \"Could you please provide me with the recipient address?\"}",
|
|
# "{\"required_functions\": [\"generate_label\"], \"clarification\": \"Could you please provide me with the recipient's address?\"}",
|
|
# "{\"required_functions\": [\"generate_label\"], \"clarification\": \"Could you please provide me with the recipient address?\"}",
|
|
# "{\"tool_calls\": [{\"name\": \"generate_label\", \"arguments\": {\"sender\": \"New York\", \"recipient\": \"Friend's address\", \"weight\": 10}}]}",
|
|
# "{\"required_functions\": [\"generate_label\"], \"clarification\": \"Could you please provide me with the sender address and recipient address?\"}",
|
|
|
|
POST {{archfc_endpoint}}/v1/chat/completions
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"model": "Arch-Function",
|
|
"messages": [
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"verify_address\", \"description\": \"Verify the address\", \"parameters\": {\"type\": \"object\", \"properties\": {\"address\": {\"type\": \"str\", \"description\": \"The address to verify\"}}, \"required\": [\"address\"]}}\n{\"name\": \"track_shipment\", \"description\": \"Track the shipment\", \"parameters\": {\"type\": \"object\", \"properties\": {\"tracking_number\": {\"type\": \"str\", \"description\": \"The tracking number\"}}, \"required\": [\"tracking_number\"]}}\n{\"name\": \"generate_label\", \"description\": \"Generate the shipping label\", \"parameters\": {\"type\": \"object\", \"properties\": {\"sender\": {\"type\": \"str\", \"description\": \"The address to ship from\"}, \"recipient\": {\"type\": \"str\", \"description\": \"The address to ship to\"}, \"weight\": {\"type\": \"int\", \"description\": \"The weight of the package\"}}, \"required\": [\"sender\", \"recipient\", \"weight\"]}}\n{\"name\": \"calculate_shipping_rate\", \"description\": \"Calculate the shipping rate\", \"parameters\": {\"type\": \"object\", \"properties\": {\"sender\": {\"type\": \"str\", \"description\": \"The address to ship from\"}, \"recipient\": {\"type\": \"str\", \"description\": \"The address to ship to\"}, \"weight\": {\"type\": \"int\", \"description\": \"The weight of the package\"}}, \"required\": [\"sender\", \"recipient\", \"weight\"]}}\n</tools>\n\nYour task is to decide which functions are needed and collect missing parameters if necessary.\n\nBased on your analysis, provide your response in one of the following JSON formats:\n1. If no functions are needed:\n```\n{\"response\": \"Your response text here\"}\n```\n2. If functions are needed but some required parameters are missing:\n```\n{\"required_functions\": [\"func_name1\", \"func_name2\", ...], \"clarification\": \"Text asking for missing parameters\"}\n```\n3. If functions are needed and all required parameters are available:\n```\n{\"tool_calls\": [{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]}\n```"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "I want to sending a package to my friend."
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"required_functions\": [\"generate_label\"], \"clarification\": \"Could you please provide me with the sender address, recipient address, and the weight of the package?\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "I am not sure about the weight but it would be around 10 pounds and I live in New York"
|
|
}
|
|
],
|
|
"temperature": 0.6,
|
|
"top_p": 1.0,
|
|
"top_k": 10
|
|
}
|
|
|
|
|
|
# tools = [
|
|
# {
|
|
# "name": "account_balance",
|
|
# "description": "Get the account balance",
|
|
# "parameters": {
|
|
# "type": "object",
|
|
# "properties": {
|
|
# "account_id": {"type": "str", "description": "The account ID"}
|
|
# },
|
|
# "required": ["account_id"],
|
|
# },
|
|
# },
|
|
# {
|
|
# "name": "transfer_funds",
|
|
# "description": "Transfer funds between accounts",
|
|
# "parameters": {
|
|
# "type": "object",
|
|
# "properties": {
|
|
# "from_account": {
|
|
# "type": "str",
|
|
# "description": "The account to transfer from",
|
|
# },
|
|
# "to_account": {
|
|
# "type": "str",
|
|
# "description": "The account to transfer to",
|
|
# },
|
|
# "amount": {"type": "int", "description": "The amount to transfer"},
|
|
# },
|
|
# "required": ["from_account", "to_account", "amount"],
|
|
# },
|
|
# },
|
|
# {
|
|
# "name": "account_transactions",
|
|
# "description": "Get the account transactions",
|
|
# "parameters": {
|
|
# "type": "object",
|
|
# "properties": {
|
|
# "account_id": {"type": "str", "description": "The account ID"},
|
|
# "start_date": {
|
|
# "type": "str",
|
|
# "description": "The start date of the transactions",
|
|
# },
|
|
# "end_date": {
|
|
# "type": "str",
|
|
# "description": "The end date of the transactions",
|
|
# },
|
|
# },
|
|
# "required": ["account_id"],
|
|
# },
|
|
# },
|
|
# ]
|
|
|
|
# System Prompt: "You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"account_balance\", \"description\": \"Get the account balance\", \"parameters\": {\"type\": \"object\", \"properties\": {\"account_id\": {\"type\": \"str\", \"description\": \"The account ID\"}}, \"required\": [\"account_id\"]}}\n{\"name\": \"transfer_funds\", \"description\": \"Transfer funds between accounts\", \"parameters\": {\"type\": \"object\", \"properties\": {\"from_account\": {\"type\": \"str\", \"description\": \"The account to transfer from\"}, \"to_account\": {\"type\": \"str\", \"description\": \"The account to transfer to\"}, \"amount\": {\"type\": \"int\", \"description\": \"The amount to transfer\"}}, \"required\": [\"from_account\", \"to_account\", \"amount\"]}}\n{\"name\": \"account_transactions\", \"description\": \"Get the account transactions\", \"parameters\": {\"type\": \"object\", \"properties\": {\"account_id\": {\"type\": \"str\", \"description\": \"The account ID\"}, \"start_date\": {\"type\": \"str\", \"description\": \"The start date of the transactions\"}, \"end_date\": {\"type\": \"str\", \"description\": \"The end date of the transactions\"}}, \"required\": [\"account_id\"]}}\n</tools>\n\nYour task is to decide which functions are needed and collect missing parameters if necessary.\n\nBased on your analysis, provide your response in one of the following JSON formats:\n1. If no functions are needed:\n```\n{\"response\": \"Your response text here\"}\n```\n2. If functions are needed but some required parameters are missing:\n```\n{\"required_functions\": [\"func_name1\", \"func_name2\", ...], \"clarification\": \"Text asking for missing parameters\"}\n```\n3. If functions are needed and all required parameters are available:\n```\n{\"tool_calls\": [{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]}\n```"
|
|
|
|
### scenario: 4-turn - omitting domain specific parameter
|
|
# tools used: account_balance, transfer_funds, account_transactions
|
|
# expected: model should ask for account id where I am transferring funds to
|
|
# Status: Not Approved
|
|
# Notes: model is not asking for account id and instead making a tool call with hallucinated value and asks for confirmation. Sometimes it is asking for account id. And sometimes its asking for all required parameters even though the amount is provided.
|
|
|
|
# sample response,
|
|
# {\"required_functions\": [\"transfer_funds\"], \"clarification\": \"Could you please confirm the following details: You want to transfer 50,000 USD from your account with ID 941123 to an account with ID 123456?\"}
|
|
# {\"required_functions\": [\"transfer_funds\"], \"clarification\": \"Could you please confirm the account IDs for the accounts you want to transfer from and to, as well as the amount you want to transfer?\"}
|
|
# {\"required_functions\": [\"transfer_funds\"], \"clarification\": \"Could you please confirm the account from which you want to transfer the funds and the account to which you want to transfer the funds?\"}
|
|
# {\"required_functions\": [\"transfer_funds\"], \"clarification\": \"Could you please provide me with the account IDs and the amount you want to transfer?\"}
|
|
|
|
POST {{archfc_endpoint}}/v1/chat/completions
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"model": "Arch-Function",
|
|
"messages": [
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"account_balance\", \"description\": \"Get the account balance\", \"parameters\": {\"type\": \"object\", \"properties\": {\"account_id\": {\"type\": \"str\", \"description\": \"The account ID\"}}, \"required\": [\"account_id\"]}}\n{\"name\": \"transfer_funds\", \"description\": \"Transfer funds between accounts\", \"parameters\": {\"type\": \"object\", \"properties\": {\"from_account\": {\"type\": \"str\", \"description\": \"The account to transfer from\"}, \"to_account\": {\"type\": \"str\", \"description\": \"The account to transfer to\"}, \"amount\": {\"type\": \"int\", \"description\": \"The amount to transfer\"}}, \"required\": [\"from_account\", \"to_account\", \"amount\"]}}\n{\"name\": \"account_transactions\", \"description\": \"Get the account transactions\", \"parameters\": {\"type\": \"object\", \"properties\": {\"account_id\": {\"type\": \"str\", \"description\": \"The account ID\"}, \"start_date\": {\"type\": \"str\", \"description\": \"The start date of the transactions\"}, \"end_date\": {\"type\": \"str\", \"description\": \"The end date of the transactions\"}}, \"required\": [\"account_id\"]}}\n</tools>\n\nYour task is to decide which functions are needed and collect missing parameters if necessary.\n\nBased on your analysis, provide your response in one of the following JSON formats:\n1. If no functions are needed:\n```\n{\"response\": \"Your response text here\"}\n```\n2. If functions are needed but some required parameters are missing:\n```\n{\"required_functions\": [\"func_name1\", \"func_name2\", ...], \"clarification\": \"Text asking for missing parameters\"}\n```\n3. If functions are needed and all required parameters are available:\n```\n{\"tool_calls\": [{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]}\n```"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "I want to know my account balance"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"required_functions\": [\"account_balance\"], \"clarification\": \"Could you please provide me with your account ID?\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "account id is 941123"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"tool_calls\": [{\"name\": \"account_balance\", \"arguments\": {\"account_id\": \"941123\"}}]}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "<tool_response>\n{'name': 'account_balance', 'result': '150,000 USD'}\n</tool_response>"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"response\": \"Your account balance is currently 150,000 USD.\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "I want to transfer 50,000 USD"
|
|
}
|
|
],
|
|
"temperature": 0.6,
|
|
"top_p": 1.0,
|
|
"top_k": 10
|
|
}
|
|
|
|
|
|
# tools = [
|
|
# {
|
|
# "name": "account_balance",
|
|
# "description": "Get the account balance",
|
|
# "parameters": {
|
|
# "type": "object",
|
|
# "properties": {
|
|
# "account_id": {"type": "str", "description": "The account ID"}
|
|
# },
|
|
# "required": ["account_id"],
|
|
# },
|
|
# },
|
|
# {
|
|
# "name": "transfer_funds",
|
|
# "description": "Transfer funds between accounts",
|
|
# "parameters": {
|
|
# "type": "object",
|
|
# "properties": {
|
|
# "from_account": {
|
|
# "type": "str",
|
|
# "description": "The account to transfer from",
|
|
# },
|
|
# "to_account": {
|
|
# "type": "str",
|
|
# "description": "The account to transfer to",
|
|
# },
|
|
# "amount": {"type": "int", "description": "The amount to transfer"},
|
|
# "note": {
|
|
# "type": "str",
|
|
# "description": "The note to attach to the transaction",
|
|
# },
|
|
# "transaction_type": {
|
|
# "type": "str",
|
|
# "description": "The type of transaction (e.g. 'transfer', 'payment')",
|
|
# },
|
|
# },
|
|
# "required": ["from_account", "to_account", "amount"],
|
|
# },
|
|
# },
|
|
# {
|
|
# "name": "account_transactions",
|
|
# "description": "Get the account transactions",
|
|
# "parameters": {
|
|
# "type": "object",
|
|
# "properties": {
|
|
# "account_id": {"type": "str", "description": "The account ID"},
|
|
# "start_date": {
|
|
# "type": "str",
|
|
# "description": "The start date of the transactions",
|
|
# },
|
|
# "end_date": {
|
|
# "type": "str",
|
|
# "description": "The end date of the transactions",
|
|
# },
|
|
# },
|
|
# "required": ["account_id"],
|
|
# },
|
|
# },
|
|
# ]
|
|
|
|
### scenario: multi-turn - 3 required and two optional params
|
|
# tools used: account_balance, transfer_funds, account_transactions
|
|
# expected: model should ask for required parameters
|
|
# Status: Not Approved
|
|
# Notes: model is not making tool call even though all parameters have been passed. 3 out of 10 times model asked for confirmation but other times it kept asking for parameters that have been provided already.
|
|
|
|
# sample response,
|
|
# "{\"required_functions\": [\"transfer_funds\"], \"clarification\": \"Please confirm that you want to transfer 25 USD from account 921249 to account 921211.\"}"
|
|
# "{\"required_functions\": [\"transfer_funds\"], \"clarification\": \"How much money do you want to transfer?\"}"
|
|
# "{\"required_functions\": [\"transfer_funds\"], \"clarification\": \"How much money do you want to transfer and do you want to make a payment or transfer?\"}"
|
|
# "{\"required_functions\": [\"transfer_funds\"], \"clarification\": \"You want to transfer 25 USD from account 921249 to account 921211. Is this correct?\"}"
|
|
# "{\"required_functions\": [\"transfer_funds\"], \"clarification\": \"Please confirm the transfer of 25 USD from account 921249 to account 921211.\"}"
|
|
|
|
POST {{archfc_endpoint}}/v1/chat/completions
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"model": "Arch-Function",
|
|
"messages": [
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"account_balance\", \"description\": \"Get the account balance\", \"parameters\": {\"type\": \"object\", \"properties\": {\"account_id\": {\"type\": \"str\", \"description\": \"The account ID\"}}, \"required\": [\"account_id\"]}}\n{\"name\": \"transfer_funds\", \"description\": \"Transfer funds between accounts\", \"parameters\": {\"type\": \"object\", \"properties\": {\"from_account\": {\"type\": \"str\", \"description\": \"The account to transfer from\"}, \"to_account\": {\"type\": \"str\", \"description\": \"The account to transfer to\"}, \"amount\": {\"type\": \"int\", \"description\": \"The amount to transfer\"}, \"note\": {\"type\": \"str\", \"description\": \"The note to attach to the transaction\"}, \"transaction_type\": {\"type\": \"str\", \"description\": \"The type of transaction (e.g. 'transfer', 'payment')\"}}, \"required\": [\"from_account\", \"to_account\", \"amount\"]}}\n{\"name\": \"account_transactions\", \"description\": \"Get the account transactions\", \"parameters\": {\"type\": \"object\", \"properties\": {\"account_id\": {\"type\": \"str\", \"description\": \"The account ID\"}, \"start_date\": {\"type\": \"str\", \"description\": \"The start date of the transactions\"}, \"end_date\": {\"type\": \"str\", \"description\": \"The end date of the transactions\"}}, \"required\": [\"account_id\"]}}\n</tools>\n\nYour task is to decide which functions are needed and collect missing parameters if necessary.\n\nBased on your analysis, provide your response in one of the following JSON formats:\n1. If no functions are needed:\n```\n{\"response\": \"Your response text here\"}\n```\n2. If functions are needed but some required parameters are missing:\n```\n{\"required_functions\": [\"func_name1\", \"func_name2\", ...], \"clarification\": \"Text asking for missing parameters\"}\n```\n3. If functions are needed and all required parameters are available:\n```\n{\"tool_calls\": [{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]}\n```"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "I want to make a transfer of 25 USD"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"required_functions\": [\"transfer_funds\"], \"clarification\": \"From which account do you want to transfer the funds?\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "921249"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"required_functions\": [\"transfer_funds\"], \"clarification\": \"To which account do you want to transfer the funds?\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "921211"
|
|
}
|
|
],
|
|
"temperature": 0.6,
|
|
"top_p": 1.0,
|
|
"top_k": 10
|
|
}
|
|
|
|
|
|
tools = [
|
|
{
|
|
"name": "account_balance",
|
|
"description": "Get the account balance",
|
|
"parameters": {
|
|
"type": "object",
|
|
"properties": {
|
|
"account_id": {"type": "str", "description": "The account ID"}
|
|
},
|
|
"required": ["account_id"],
|
|
},
|
|
},
|
|
{
|
|
"name": "transfer_funds",
|
|
"description": "Transfer funds between accounts",
|
|
"parameters": {
|
|
"type": "object",
|
|
"properties": {
|
|
"from_account": {
|
|
"type": "str",
|
|
"description": "The account to transfer from",
|
|
},
|
|
"to_account": {
|
|
"type": "str",
|
|
"description": "The account to transfer to",
|
|
},
|
|
"amount": {"type": "int", "description": "The amount to transfer"},
|
|
"note": {
|
|
"type": "str",
|
|
"description": "The note to attach to the transaction",
|
|
},
|
|
"transaction_type": {
|
|
"type": "str",
|
|
"description": "The type of transaction (e.g. 'transfer', 'payment')",
|
|
},
|
|
},
|
|
"required": ["from_account", "to_account", "amount", "note"],
|
|
},
|
|
},
|
|
{
|
|
"name": "account_transactions",
|
|
"description": "Get the account transactions",
|
|
"parameters": {
|
|
"type": "object",
|
|
"properties": {
|
|
"account_id": {"type": "str", "description": "The account ID"},
|
|
"start_date": {
|
|
"type": "str",
|
|
"description": "The start date of the transactions",
|
|
},
|
|
"end_date": {
|
|
"type": "str",
|
|
"description": "The end date of the transactions",
|
|
},
|
|
},
|
|
"required": ["account_id"],
|
|
},
|
|
},
|
|
]
|
|
|
|
### scenario: multi-turn - 4 required and one optional params
|
|
# tools used: account_balance, transfer_funds, account_transactions
|
|
# expected: model should ask for required parameters
|
|
# Status: Not Approved
|
|
# Notes: model is not asking for required parameters. It is making tool calls with hallucinated values.
|
|
# 3/10 times tool calls with hallucinated values
|
|
# 2/10 times asks for confirmation
|
|
# rest of the times it asked for required parameter values
|
|
|
|
# sample response,
|
|
|
|
# "{\"tool_calls\": [{\"name\": \"transfer_funds\", \"arguments\": {\"from_account\": \"account_id\", \"to_account\": \"contractor_account_id\", \"amount\": 25, \"note\": \"payment to contractor for home repairs\"}}]}",
|
|
# "{\"required_functions\": [\"transfer_funds\"], \"clarification\": \"Please confirm the following: Transfer 25 USD from your account to the contractor's account with the note 'payment to contractor for home repairs'.\"}",
|
|
# "{\"required_functions\": [\"transfer_funds\"], \"clarification\": \"Who would you like to transfer the funds from?\"}"
|
|
# "{\"required_functions\": [\"transfer_funds\"], \"clarification\": \"What's the account ID for the account you want to transfer the funds from?\"}",
|
|
# "{\"tool_calls\": [{\"name\": \"transfer_funds\", \"arguments\": {\"from_account\": \"my_account\", \"to_account\": \"contractor_account\", \"amount\": 25, \"note\": \"payment to contractor for home repairs\", \"transaction_type\": \"transfer\"}}]}",
|
|
|
|
POST {{archfc_endpoint}}/v1/chat/completions
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"model": "Arch-Function",
|
|
"messages": [
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"account_balance\", \"description\": \"Get the account balance\", \"parameters\": {\"type\": \"object\", \"properties\": {\"account_id\": {\"type\": \"str\", \"description\": \"The account ID\"}}, \"required\": [\"account_id\"]}}\n{\"name\": \"transfer_funds\", \"description\": \"Transfer funds between accounts\", \"parameters\": {\"type\": \"object\", \"properties\": {\"from_account\": {\"type\": \"str\", \"description\": \"The account to transfer from\"}, \"to_account\": {\"type\": \"str\", \"description\": \"The account to transfer to\"}, \"amount\": {\"type\": \"int\", \"description\": \"The amount to transfer\"}, \"note\": {\"type\": \"str\", \"description\": \"The note to attach to the transaction\"}, \"transaction_type\": {\"type\": \"str\", \"description\": \"The type of transaction (e.g. 'transfer', 'payment')\"}}, \"required\": [\"from_account\", \"to_account\", \"amount\", \"note\"]}}\n{\"name\": \"account_transactions\", \"description\": \"Get the account transactions\", \"parameters\": {\"type\": \"object\", \"properties\": {\"account_id\": {\"type\": \"str\", \"description\": \"The account ID\"}, \"start_date\": {\"type\": \"str\", \"description\": \"The start date of the transactions\"}, \"end_date\": {\"type\": \"str\", \"description\": \"The end date of the transactions\"}}, \"required\": [\"account_id\"]}}\n</tools>\n\nYour task is to decide which functions are needed and collect missing parameters if necessary.\n\nBased on your analysis, provide your response in one of the following JSON formats:\n1. If no functions are needed:\n```\n{\"response\": \"Your response text here\"}\n```\n2. If functions are needed but some required parameters are missing:\n```\n{\"required_functions\": [\"func_name1\", \"func_name2\", ...], \"clarification\": \"Text asking for missing parameters\"}\n```\n3. If functions are needed and all required parameters are available:\n```\n{\"tool_calls\": [{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]}\n```"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "I want to make a transfer of 25 USD"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"required_functions\": [\"transfer_funds\"], \"clarification\": \"What's the note you want to attach to the transaction?\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "payment to contractor for home repairs"
|
|
}
|
|
],
|
|
"temperature": 0.6,
|
|
"top_p": 1.0,
|
|
"top_k": 10
|
|
}
|
|
|
|
# tools = [
|
|
# {
|
|
# "name": "account_balance",
|
|
# "description": "Get the account balance",
|
|
# "parameters": {
|
|
# "type": "object",
|
|
# "properties": {
|
|
# "account_id": {"type": "str", "description": "The account ID"}
|
|
# },
|
|
# "required": ["account_id"],
|
|
# },
|
|
# },
|
|
# {
|
|
# "name": "transfer_funds",
|
|
# "description": "Transfer funds between accounts",
|
|
# "parameters": {
|
|
# "type": "object",
|
|
# "properties": {
|
|
# "from_account": {
|
|
# "type": "str",
|
|
# "description": "The account to transfer from",
|
|
# },
|
|
# "to_account": {
|
|
# "type": "str",
|
|
# "description": "The account to transfer to",
|
|
# },
|
|
# "amount": {"type": "int", "description": "The amount to transfer"},
|
|
# "note": {
|
|
# "type": "str",
|
|
# "description": "The note to attach to the transaction",
|
|
# },
|
|
# "transaction_type": {
|
|
# "type": "str",
|
|
# "description": "The type of transaction (e.g. 'transfer', 'payment')",
|
|
# },
|
|
# },
|
|
# "required": [
|
|
# "from_account",
|
|
# "to_account",
|
|
# "amount",
|
|
# "note",
|
|
# "transaction_type",
|
|
# ],
|
|
# },
|
|
# },
|
|
# {
|
|
# "name": "account_transactions",
|
|
# "description": "Get the account transactions",
|
|
# "parameters": {
|
|
# "type": "object",
|
|
# "properties": {
|
|
# "account_id": {"type": "str", "description": "The account ID"},
|
|
# "start_date": {
|
|
# "type": "str",
|
|
# "description": "The start date of the transactions",
|
|
# },
|
|
# "end_date": {
|
|
# "type": "str",
|
|
# "description": "The end date of the transactions",
|
|
# },
|
|
# },
|
|
# "required": ["account_id"],
|
|
# },
|
|
# },
|
|
# ]
|
|
|
|
### scenario: multi-turn - 5 required parameters
|
|
# tools used: account_balance, transfer_funds, account_transactions
|
|
# expected: model should ask for required parameters
|
|
# Status: Not Approved
|
|
# Notes: 7/10 times it made tool call without asking for additional required parameter. It hallucinated value for transaction type
|
|
# 3/10 times it asked for confirmation
|
|
|
|
# sample response,
|
|
# "{\"tool_calls\": [{\"name\": \"transfer_funds\", \"arguments\": {\"from_account\": \"9221124\", \"to_account\": \"921211\", \"amount\": 25, \"note\": \"payment to contractor for home repairs\", \"transaction_type\": \"transfer\"}}]}"
|
|
# "{\"tool_calls\": [{\"name\": \"transfer_funds\", \"arguments\": {\"from_account\": \"9221124\", \"to_account\": \"921211\", \"amount\": 25, \"note\": \"payment to contractor for home repairs\", \"transaction_type\": \"transfer\"}}]}",
|
|
# "{\"required_functions\": [\"transfer_funds\"], \"clarification\": \"Please confirm: transfer 25 USD from account 9221124 to account 921211 with note 'payment to contractor for home repairs' and transaction type 'transfer'.\"}",
|
|
# "{\"tool_calls\": [{\"name\": \"transfer_funds\", \"arguments\": {\"from_account\": \"9221124\", \"to_account\": \"921211\", \"amount\": 25, \"note\": \"payment to contractor for home repairs\", \"transaction_type\": \"transfer\"}}]}",
|
|
# "{\"required_functions\": [\"transfer_funds\"], \"clarification\": \"Please confirm the following: Transfer of 25 USD from account 9221124 to account 921211 for the payment to contractor for home repairs.\"}",
|
|
# "{\"required_functions\": [\"transfer_funds\"], \"clarification\": \"Please confirm the transfer of 25 USD from account 9221124 to account 921211 with the note 'payment to contractor for home repairs'.\"}",
|
|
# "{\"tool_calls\": [{\"name\": \"transfer_funds\", \"arguments\": {\"from_account\": \"9221124\", \"to_account\": \"921211\", \"amount\": 25, \"note\": \"payment to contractor for home repairs\", \"transaction_type\": \"transfer\"}}]}",
|
|
# "{\"tool_calls\": [{\"name\": \"transfer_funds\", \"arguments\": {\"from_account\": \"9221124\", \"to_account\": \"921211\", \"amount\": 25, \"note\": \"payment to contractor for home repairs\", \"transaction_type\": \"transfer\"}}]}",
|
|
# "{\"tool_calls\": [{\"name\": \"transfer_funds\", \"arguments\": {\"from_account\": \"9221124\", \"to_account\": \"921211\", \"amount\": 25, \"note\": \"payment to contractor for home repairs\", \"transaction_type\": \"transfer\"}}]}",
|
|
# "{\"tool_calls\": [{\"name\": \"transfer_funds\", \"arguments\": {\"from_account\": \"9221124\", \"to_account\": \"921211\", \"amount\": 25, \"note\": \"payment to contractor for home repairs\", \"transaction_type\": \"transfer\"}}]}",
|
|
|
|
POST {{archfc_endpoint}}/v1/chat/completions
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"model": "Arch-Function",
|
|
"messages": [
|
|
{
|
|
"role": "system",
|
|
"content": "You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"account_balance\", \"description\": \"Get the account balance\", \"parameters\": {\"type\": \"object\", \"properties\": {\"account_id\": {\"type\": \"str\", \"description\": \"The account ID\"}}, \"required\": [\"account_id\"]}}\n{\"name\": \"transfer_funds\", \"description\": \"Transfer funds between accounts\", \"parameters\": {\"type\": \"object\", \"properties\": {\"from_account\": {\"type\": \"str\", \"description\": \"The account to transfer from\"}, \"to_account\": {\"type\": \"str\", \"description\": \"The account to transfer to\"}, \"amount\": {\"type\": \"int\", \"description\": \"The amount to transfer\"}, \"note\": {\"type\": \"str\", \"description\": \"The note to attach to the transaction\"}, \"transaction_type\": {\"type\": \"str\", \"description\": \"The type of transaction (e.g. 'transfer', 'payment')\"}}, \"required\": [\"from_account\", \"to_account\", \"amount\", \"note\", \"transaction_type\"]}}\n{\"name\": \"account_transactions\", \"description\": \"Get the account transactions\", \"parameters\": {\"type\": \"object\", \"properties\": {\"account_id\": {\"type\": \"str\", \"description\": \"The account ID\"}, \"start_date\": {\"type\": \"str\", \"description\": \"The start date of the transactions\"}, \"end_date\": {\"type\": \"str\", \"description\": \"The end date of the transactions\"}}, \"required\": [\"account_id\"]}}\n</tools>\n\nYour task is to decide which functions are needed and collect missing parameters if necessary.\n\nBased on your analysis, provide your response in one of the following JSON formats:\n1. If no functions are needed:\n```\n{\"response\": \"Your response text here\"}\n```\n2. If functions are needed but some required parameters are missing:\n```\n{\"required_functions\": [\"func_name1\", \"func_name2\", ...], \"clarification\": \"Text asking for missing parameters\"}\n```\n3. If functions are needed and all required parameters are available:\n```\n{\"tool_calls\": [{\"name\": \"func_name1\", \"arguments\": {\"argument1\": \"value1\", \"argument2\": \"value2\"}},... (more tool calls as required)]}\n```"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "I want to make a transfer of 25 USD"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"required_functions\": [\"transfer_funds\"], \"clarification\": \"What's the note you want to attach to the transaction?\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "payment to contractor for home repairs"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"required_functions\": [\"transfer_funds\"], \"clarification\": \"Which account is the transfer coming from?\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "9221124"
|
|
},
|
|
{
|
|
"role": "assistant",
|
|
"content": "{\"required_functions\": [\"transfer_funds\"], \"clarification\": \"And which account is the transfer going to?\"}"
|
|
},
|
|
{
|
|
"role": "user",
|
|
"content": "921211"
|
|
}
|
|
],
|
|
"temperature": 0.6,
|
|
"top_p": 1.0,
|
|
"top_k": 10
|
|
}
|