mirror of
https://github.com/dograh-hq/dograh.git
synced 2026-06-10 08:05:22 +02:00
Merge branch 'main' of https://github.com/xmanindia/dograh
This commit is contained in:
XI
commit
6817117f2b
385 changed files with 27923 additions and 12381 deletions
206
.agents/skills/review-agents-md/SKILL.md
Normal file
206
.agents/skills/review-agents-md/SKILL.md
Normal file
|
|
@ -0,0 +1,206 @@
|
|||
---
|
||||
name: review-agents-md
|
||||
description: Audit Dograh `AGENTS.md` files for drift against the live repo and for bad scope boundaries between parent and child docs. Use when the user asks to review existing AGENTS files, identify stale guidance, decide whether a subtree needs its own `AGENTS.md`, or update the `AGENTS.md` hierarchy under the repo root, `api/`, or `ui/`.
|
||||
---
|
||||
|
||||
# Review AGENTS.md
|
||||
|
||||
Audit first. Report drift, missing coverage, and wrong ownership boundaries before editing docs unless the user explicitly asks for patches.
|
||||
|
||||
## Freshness Rule
|
||||
|
||||
Treat the repo as source of truth.
|
||||
|
||||
- Trust current code and current directory layout over any `AGENTS.md`, `README.md`, or this skill's references.
|
||||
- If prose and code disagree, report the prose as stale.
|
||||
- If a reference file in this skill disagrees with the repo, trust the repo and mention the drift.
|
||||
|
||||
## Workflow
|
||||
|
||||
### 0. Refresh the seam reference before using it
|
||||
|
||||
If subagents are available, refresh `references/dograh-seams.md` before relying on it.
|
||||
|
||||
- Spawn exactly one subagent for this maintenance pass.
|
||||
- Tell the subagent to inspect the live repo.
|
||||
- Limit its patch set to `.agents/skills/review-agents-md/references/dograh-seams.md`.
|
||||
- Also allow `.agents/skills/review-agents-md/scripts/inventory_agents_md.py`, but only if the helper itself needs a repo-specific fix.
|
||||
- Tell the subagent not to recurse into this same seam-refresh workflow. This is a one-level maintenance pass, not an infinite self-audit loop.
|
||||
- Tell the subagent not to review or patch any repo `AGENTS.md` files yet. Its job is only to refresh the seam reference and helper.
|
||||
- After the subagent returns, review its diff quickly before using `dograh-seams.md` in the main audit.
|
||||
|
||||
Use a prompt shaped like:
|
||||
|
||||
```text
|
||||
Review and refresh .agents/skills/review-agents-md/references/dograh-seams.md against the live Dograh repo. Patch only that file, and patch .agents/skills/review-agents-md/scripts/inventory_agents_md.py only if needed. Do not recurse into another seam-refresh pass. Do not review or edit any AGENTS.md files yet.
|
||||
```
|
||||
|
||||
If subagents are not available, do the same seam refresh locally before continuing.
|
||||
|
||||
### 1. Inventory the current hierarchy
|
||||
|
||||
Run the helper first from the repo root:
|
||||
|
||||
```bash
|
||||
python .agents/skills/review-agents-md/scripts/inventory_agents_md.py
|
||||
```
|
||||
|
||||
This prints:
|
||||
|
||||
- every discovered `AGENTS.md`
|
||||
- child `AGENTS.md` ownership boundaries
|
||||
- immediate child directories for each scope
|
||||
- large uncovered subtrees that may deserve their own `AGENTS.md`
|
||||
|
||||
Then confirm with direct file discovery when needed:
|
||||
|
||||
```bash
|
||||
rg --files -g 'AGENTS.md' .
|
||||
find api ui -name AGENTS.md | sort
|
||||
```
|
||||
|
||||
### 2. Read top-down before judging details
|
||||
|
||||
Read in this order:
|
||||
|
||||
1. repo root `AGENTS.md`
|
||||
2. `api/AGENTS.md`
|
||||
3. `ui/AGENTS.md`
|
||||
4. deeper `AGENTS.md` files under those trees
|
||||
|
||||
For each file, write a one-line ownership statement in your notes:
|
||||
|
||||
- what subtree it owns
|
||||
- what shared rules it should contain
|
||||
- which deeper docs, if any, should own implementation details instead
|
||||
|
||||
### 3. Verify each doc against the live code
|
||||
|
||||
Check directory trees, route aggregators, registration points, and extension seams instead of relying on filenames mentioned in prose.
|
||||
|
||||
Dograh files worth checking early:
|
||||
|
||||
- `api/routes/main.py`
|
||||
- `api/routes/telephony.py`
|
||||
- `api/services/integrations/loader.py`
|
||||
- `api/services/integrations/registry.py`
|
||||
- `api/services/pipecat/run_pipeline.py`
|
||||
- `api/tasks/run_integrations.py`
|
||||
- `api/services/telephony/registry.py`
|
||||
- `api/services/telephony/factory.py`
|
||||
- `api/services/telephony/providers/__init__.py`
|
||||
- `ui/src/app/`
|
||||
- `ui/src/components/`
|
||||
- `ui/src/lib/auth/`
|
||||
- `ui/src/client/`
|
||||
|
||||
Read [dograh-seams.md](references/dograh-seams.md) when you need a fast repo-specific starting map.
|
||||
|
||||
### 4. Apply the hierarchy tests
|
||||
|
||||
Use these tests for every scope:
|
||||
|
||||
- `parent-fit`: The parent doc explains immediate child systems, shared invariants, and navigation for the subtree it owns.
|
||||
- `child-fit`: A deeper doc owns local extension contracts, module-specific gotchas, and file-level patterns for its own subtree.
|
||||
- `no-duplication`: The parent does not restate detailed child implementation guidance that should live in the child doc.
|
||||
- `downward-pointing`: The doc should point contributors toward the next relevant subdirectory or deeper `AGENTS.md` instead of trying to explain the whole subtree itself.
|
||||
- `no-gaps`: If a large or extension-heavy subtree has rules the parent cannot explain cleanly in a few lines, flag a missing child `AGENTS.md`.
|
||||
- `no-drift`: File trees, commands, extension points, and architecture claims still match the code.
|
||||
|
||||
### 5. Dograh-specific review heuristics
|
||||
|
||||
#### Root `AGENTS.md`
|
||||
|
||||
Expect root to stay high-level.
|
||||
|
||||
- It should describe the top-level project shape, shared stack, and shared local-development expectations.
|
||||
- It should mention top-level applications and support directories that matter to contributors.
|
||||
- It should not try to document backend-internal extension contracts or frontend component internals.
|
||||
- If a top-level directory materially matters to contributors and is missing from root guidance, report `missing-parent-coverage`.
|
||||
|
||||
#### `api/AGENTS.md`
|
||||
|
||||
Expect `api/AGENTS.md` to orient a contributor across backend domains, not to document every local contract in full.
|
||||
|
||||
- It should point to where routes, services, DB access, schemas, tasks, tests, and security invariants live.
|
||||
- It should accurately describe where workflow execution lives. In current Dograh, that spans `api/services/workflow/`, `api/services/pipecat/`, and post-call work in `api/tasks/run_integrations.py`.
|
||||
- It should accurately describe telephony as a substantial subsystem, not just as one route file.
|
||||
- It should mention integration extensibility and defer package-level rules to `api/services/integrations/AGENTS.md`.
|
||||
- If `api/services/telephony/` or `api/services/workflow/` are complex enough that the parent doc becomes vague or overloaded, report `missing-child-agents`.
|
||||
|
||||
#### `api/services/integrations/AGENTS.md`
|
||||
|
||||
Expect this file to own the integration package contract.
|
||||
|
||||
- It should explain package registration, node model/spec patterns, runtime collection, completion handlers, optional routes, import discipline, and testing expectations.
|
||||
- It should match the live registry/loader path instead of describing central manual wiring that no longer exists.
|
||||
- It should not require edits to `workflow/dto.py`, `run_pipeline.py`, or route aggregation unless the generic framework genuinely changed.
|
||||
|
||||
#### `ui/AGENTS.md`
|
||||
|
||||
Expect `ui/AGENTS.md` to orient contributors across the frontend without documenting individual feature internals.
|
||||
|
||||
- It should describe the App Router layout under `ui/src/app/`.
|
||||
- It should point to reusable feature components under `ui/src/components/`.
|
||||
- It should mention generated API client usage under `ui/src/client/`.
|
||||
- It should mention auth readiness constraints under `ui/src/lib/auth/`.
|
||||
- It should not describe removed folders or outdated stack details.
|
||||
|
||||
### 6. Classify findings
|
||||
|
||||
Use these categories:
|
||||
|
||||
- `stale`: prose mentions files, commands, flows, or architecture that no longer match the repo
|
||||
- `missing-parent-coverage`: a parent scope omits a major subsystem it should orient the reader to
|
||||
- `missing-child-agents`: a deep subtree likely needs its own `AGENTS.md`
|
||||
- `wrong-level`: content belongs in a parent or child scope instead
|
||||
- `extra-detail`: a parent doc is too implementation-specific for its level
|
||||
|
||||
### 7. Report format
|
||||
|
||||
List findings first, ordered by severity.
|
||||
|
||||
Use this shape:
|
||||
|
||||
```text
|
||||
<path>: <category> -> <problem> -> <what should own or replace it>
|
||||
```
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
api/AGENTS.md: missing-child-agents -> telephony is a large extension surface with provider registration, transport, routes, and config rules but has no local AGENTS.md -> add api/services/telephony/AGENTS.md and keep api/AGENTS.md at navigation level
|
||||
api/services/integrations/AGENTS.md: stale -> says central DTO edits are required for new integrations, but registry-based discovery handles node resolution -> update the doc to describe the registry path only
|
||||
ui/AGENTS.md: wrong-level -> describes individual workflow-builder component behavior instead of frontend navigation rules -> move that detail to a deeper doc or remove it
|
||||
```
|
||||
|
||||
After findings, include:
|
||||
|
||||
- open questions or assumptions
|
||||
- optional patch plan, only if the user asked for fixes or clearly wants them next
|
||||
|
||||
## Editing Rules
|
||||
|
||||
If the user wants the docs fixed:
|
||||
|
||||
- patch the smallest set of `AGENTS.md` files that restores a clean hierarchy
|
||||
- add a new `AGENTS.md` only when a subtree has distinct local rules or extension contracts
|
||||
- keep parent docs short and navigational
|
||||
- let child docs own local implementation rules
|
||||
- avoid copying the same guidance into parent and child files
|
||||
- prefer folders over files when writing navigation guidance, unless a file is the only real seam
|
||||
- when adding a new child `AGENTS.md`, start with the shortest useful contract; avoid tutorial-style prose
|
||||
- point the reader downward toward the next relevant subdirectory or child `AGENTS.md`
|
||||
- if a draft feels forced or over-explained, compress it again
|
||||
|
||||
## Useful Commands
|
||||
|
||||
```bash
|
||||
python .agents/skills/review-agents-md/scripts/inventory_agents_md.py
|
||||
rg --files -g 'AGENTS.md' .
|
||||
find api ui -name AGENTS.md | sort
|
||||
find api/services -maxdepth 2 -type d | sort
|
||||
find ui/src -maxdepth 2 -type d | sort
|
||||
rg -n "include_router|all_routers" api/routes/main.py api/services/integrations
|
||||
rg -n "register\\(|ProviderSpec|register_package|create_runtime_sessions|run_completion" api/services/telephony api/services/integrations
|
||||
```
|
||||
4
.agents/skills/review-agents-md/agents/openai.yaml
Normal file
4
.agents/skills/review-agents-md/agents/openai.yaml
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
interface:
|
||||
display_name: "Review AGENTS.md"
|
||||
short_description: "Audit AGENTS.md scope and drift"
|
||||
default_prompt: "Use $review-agents-md to audit AGENTS.md files for drift, missing coverage, and wrong scope boundaries."
|
||||
126
.agents/skills/review-agents-md/references/dograh-seams.md
Normal file
126
.agents/skills/review-agents-md/references/dograh-seams.md
Normal file
|
|
@ -0,0 +1,126 @@
|
|||
# Dograh Seams
|
||||
|
||||
Use this file as a starting map, not as source of truth. Verify every claim against the live repo.
|
||||
|
||||
## Dynamic discovery only
|
||||
|
||||
Do not record the current `AGENTS.md` inventory in this file.
|
||||
|
||||
- discover the live hierarchy with `rg --files -g 'AGENTS.md' .`
|
||||
- use the helper script to identify uncovered hot spots
|
||||
- treat any baked-in inventory of existing `AGENTS.md` files as drift-prone and remove it
|
||||
|
||||
## Root-level anchors
|
||||
|
||||
The repo root still revolves around these contributor-relevant directories:
|
||||
|
||||
- `api/`
|
||||
- `ui/`
|
||||
- `scripts/`
|
||||
- `docs/`
|
||||
- `pipecat/`
|
||||
|
||||
Current code-heavy top-level subtrees that can matter during hierarchy reviews:
|
||||
|
||||
- `evals/` for evaluation tooling, including `evals/visualizer/`
|
||||
- `sdk/` for packaged SDK work, especially `sdk/python/` and `sdk/typescript/`
|
||||
|
||||
Root `AGENTS.md` should stay at this level.
|
||||
|
||||
## Backend anchors
|
||||
|
||||
### Route aggregation
|
||||
|
||||
- REST routers are aggregated in `api/routes/main.py`.
|
||||
- Telephony has its main cross-provider route file at `api/routes/telephony.py`.
|
||||
- Integration package routers are mounted through `api.services.integrations.all_routers()`.
|
||||
- Node-type metadata is exposed from `api/routes/node_types.py`.
|
||||
|
||||
### Workflow execution
|
||||
|
||||
Workflow execution is not a single folder.
|
||||
|
||||
- workflow graph, DTOs, node data, node-spec generation, QA, and tool helpers live under `api/services/workflow/`
|
||||
- live pipeline execution lives under `api/services/pipecat/`
|
||||
- Dograh-specific realtime provider adapters live under `api/services/pipecat/realtime/`
|
||||
- post-call QA, registered integrations, and webhook execution live in `api/tasks/run_integrations.py`
|
||||
|
||||
If `api/AGENTS.md` implies workflow execution lives in only one place, treat that as suspicious.
|
||||
|
||||
### Node spec and SDK seam
|
||||
|
||||
- core node specs are registered lazily from `api/services/workflow/dto.py` by `api/services/workflow/node_specs/__init__.py`
|
||||
- integration node specs are merged through `api.services.integrations.all_node_specs()`
|
||||
- the frontend and SDK-facing node catalog is served from `api/routes/node_types.py`
|
||||
|
||||
### Telephony
|
||||
|
||||
Current telephony architecture is registry-driven.
|
||||
|
||||
- importing `api.services.telephony` eagerly loads `api/services/telephony/providers/` so provider packages self-register
|
||||
- provider registration and `ProviderSpec` live in `api/services/telephony/registry.py`
|
||||
- provider lookup, org-scoped config normalization, inbound matching, and run-scoped resolution live in `api/services/telephony/factory.py`
|
||||
- per-provider HTTP routers live in `api/services/telephony/providers/<name>/routes.py` and are auto-mounted by `api/routes/telephony.py`
|
||||
- provider-local implementations live in `api/services/telephony/providers/<name>/`
|
||||
- current provider packages include `ari`, `cloudonix`, `plivo`, `telnyx`, `twilio`, `vobiz`, and `vonage`
|
||||
- not every provider has an HTTP route module; for example, `ari` is transport-focused and skipped by the auto-mounter
|
||||
|
||||
### Integrations
|
||||
|
||||
Current integrations are also registry-driven.
|
||||
|
||||
- package discovery lives in `api/services/integrations/loader.py` via `pkgutil.iter_modules(...)`
|
||||
- package registration and runtime/completion orchestration live in `api/services/integrations/registry.py`
|
||||
- shared package/session context types live in `api/services/integrations/base.py`
|
||||
- a concrete package example exists at `api/services/integrations/tuner/`
|
||||
|
||||
## Frontend anchors
|
||||
|
||||
### Navigation and pages
|
||||
|
||||
- page routes live under `ui/src/app/`
|
||||
- `ui/src/app/layout.tsx` composes the global frontend providers and `AppLayout`
|
||||
- runtime config handlers live under `ui/src/app/api/config/`
|
||||
- auth/session handlers live under `ui/src/app/api/auth/`
|
||||
- feature coverage should be discovered from the current `ui/src/app/` tree, not maintained as a static list here
|
||||
|
||||
### Components and feature slices
|
||||
|
||||
- shared primitives live under `ui/src/components/ui/`
|
||||
- workflow builder primitives live under `ui/src/components/flow/`
|
||||
- reusable workflow UI lives under `ui/src/components/workflow/`
|
||||
- workflow run UI lives under `ui/src/components/workflow-runs/`
|
||||
- telephony-related UI lives under `ui/src/components/telephony/`
|
||||
- layout components live under `ui/src/components/layout/`
|
||||
- workflow feature code is split between reusable components and route-local code under `ui/src/app/workflow/[workflowId]/`, especially `components/`, `contexts/`, `hooks/`, `stores/`, `utils/`, and nested `run/[runId]/`
|
||||
|
||||
### Client and auth
|
||||
|
||||
- generated API client code lives under `ui/src/client/`, with generated subtrees in `ui/src/client/client/` and `ui/src/client/core/`
|
||||
- auth exports live in `ui/src/lib/auth/index.ts`
|
||||
- auth provider wrappers live under `ui/src/lib/auth/providers/`
|
||||
- server-side auth helpers live in `ui/src/lib/auth/server.ts`
|
||||
- `AuthProvider` chooses between the Stack and local wrappers after fetching `/api/config/auth`, so docs that treat auth as compile-time static are suspicious
|
||||
|
||||
## Known drift example from the audit
|
||||
|
||||
`api/services/telephony/README.md` is still stale in the current repo snapshot:
|
||||
|
||||
- it described flat provider files like `twilio_provider.py` and `vonage_provider.py`
|
||||
- it told contributors to add schemas to `api/schemas/telephony_config.py`
|
||||
- it referenced legacy patterns such as direct `TwilioService` usage
|
||||
|
||||
The live code instead uses provider packages under `providers/<name>/`, registry-driven provider resolution, and route auto-mounting from `api/routes/telephony.py`. Use this as a reminder that prose in adjacent docs may have drifted even when the code is coherent.
|
||||
|
||||
## Hotspot heuristics
|
||||
|
||||
These are review prompts, not frozen conclusions.
|
||||
|
||||
- pay extra attention to deep subtrees that define extension contracts, registration points, or multi-file execution paths
|
||||
- in Dograh, common examples include telephony, workflow execution, generated SDK surfaces, and other service subtrees that span many files
|
||||
|
||||
Ask:
|
||||
|
||||
- does the parent doc have enough room to explain this subtree accurately without becoming overloaded?
|
||||
- does the subtree have distinct extension rules, registration points, or local pitfalls?
|
||||
- would a contributor benefit from a dedicated `AGENTS.md` here?
|
||||
272
.agents/skills/review-agents-md/scripts/inventory_agents_md.py
Normal file
272
.agents/skills/review-agents-md/scripts/inventory_agents_md.py
Normal file
|
|
@ -0,0 +1,272 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Inventory AGENTS.md scopes and large uncovered subtrees.
|
||||
|
||||
Run from the repo root:
|
||||
|
||||
python .agents/skills/review-agents-md/scripts/inventory_agents_md.py
|
||||
|
||||
The output is intentionally heuristic. It helps with discovery; it does not
|
||||
decide by itself whether a subtree must have its own AGENTS.md.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
IGNORE_DIRS = {
|
||||
".git",
|
||||
".hg",
|
||||
".svn",
|
||||
".next",
|
||||
".turbo",
|
||||
".venv",
|
||||
"venv",
|
||||
"node_modules",
|
||||
"__pycache__",
|
||||
".pytest_cache",
|
||||
".ruff_cache",
|
||||
".mypy_cache",
|
||||
".cursor",
|
||||
".claude",
|
||||
".idea",
|
||||
".vscode",
|
||||
"dist",
|
||||
"build",
|
||||
}
|
||||
|
||||
CODE_EXTENSIONS = {
|
||||
".py",
|
||||
".ts",
|
||||
".tsx",
|
||||
".js",
|
||||
".jsx",
|
||||
".mjs",
|
||||
".cjs",
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DirSummary:
|
||||
path: Path
|
||||
code_files: int
|
||||
nested_dirs: int
|
||||
has_agents_here: bool
|
||||
descendant_agents: int
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Summarize AGENTS.md coverage and likely missing deeper scopes.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"roots",
|
||||
nargs="*",
|
||||
default=["."],
|
||||
help="Directories to scan. Defaults to the current directory.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--hotspot-threshold",
|
||||
type=int,
|
||||
default=12,
|
||||
help="Minimum code-file count for a subtree to be listed as a hotspot.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def should_skip_dir(name: str) -> bool:
|
||||
return name in IGNORE_DIRS or name.startswith(".")
|
||||
|
||||
|
||||
def walk_dirs(root: Path):
|
||||
for current_root, dirnames, filenames in os.walk(root):
|
||||
dirnames[:] = sorted(d for d in dirnames if not should_skip_dir(d))
|
||||
yield Path(current_root), dirnames, filenames
|
||||
|
||||
|
||||
def discover_agents(roots: list[Path]) -> list[Path]:
|
||||
found: set[Path] = set()
|
||||
for root in roots:
|
||||
for current_root, _dirnames, filenames in walk_dirs(root):
|
||||
if "AGENTS.md" in filenames:
|
||||
found.add((current_root / "AGENTS.md").resolve())
|
||||
return sorted(found)
|
||||
|
||||
|
||||
def count_nested_dirs(root: Path) -> int:
|
||||
count = 0
|
||||
for current_root, dirnames, _filenames in walk_dirs(root):
|
||||
if Path(current_root) == root:
|
||||
count += len(dirnames)
|
||||
continue
|
||||
count += len(dirnames)
|
||||
return count
|
||||
|
||||
|
||||
def count_code_files(root: Path) -> int:
|
||||
count = 0
|
||||
for _current_root, _dirnames, filenames in walk_dirs(root):
|
||||
for filename in filenames:
|
||||
if Path(filename).suffix in CODE_EXTENSIONS:
|
||||
count += 1
|
||||
return count
|
||||
|
||||
|
||||
def descendant_agents(root: Path, agents: list[Path]) -> list[Path]:
|
||||
root_resolved = root.resolve()
|
||||
return [
|
||||
agent
|
||||
for agent in agents
|
||||
if agent.parent != root_resolved and root_resolved in agent.parents
|
||||
]
|
||||
|
||||
|
||||
def immediate_child_dirs(root: Path) -> list[Path]:
|
||||
children = []
|
||||
for child in sorted(root.iterdir()):
|
||||
if child.is_dir() and not should_skip_dir(child.name):
|
||||
children.append(child)
|
||||
return children
|
||||
|
||||
|
||||
def summarize_child(child: Path, agents: list[Path]) -> DirSummary:
|
||||
desc_agents = descendant_agents(child, agents)
|
||||
return DirSummary(
|
||||
path=child,
|
||||
code_files=count_code_files(child),
|
||||
nested_dirs=count_nested_dirs(child),
|
||||
has_agents_here=(child / "AGENTS.md").exists(),
|
||||
descendant_agents=len(desc_agents),
|
||||
)
|
||||
|
||||
|
||||
def format_path(path: Path, cwd: Path) -> str:
|
||||
try:
|
||||
return str(path.resolve().relative_to(cwd))
|
||||
except ValueError:
|
||||
return str(path.resolve())
|
||||
|
||||
|
||||
def nested_hotspots(summary: DirSummary, agents: list[Path], threshold: int) -> list[DirSummary]:
|
||||
if summary.has_agents_here:
|
||||
return []
|
||||
if summary.code_files < threshold:
|
||||
return []
|
||||
|
||||
nested_summaries = [
|
||||
summarize_child(child, agents) for child in immediate_child_dirs(summary.path)
|
||||
]
|
||||
return [
|
||||
nested
|
||||
for nested in nested_summaries
|
||||
if (
|
||||
not nested.has_agents_here
|
||||
and nested.code_files >= threshold
|
||||
and nested.descendant_agents == 0
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def print_scope(scope_dir: Path, agents: list[Path], cwd: Path, threshold: int) -> None:
|
||||
scope_agent = scope_dir / "AGENTS.md"
|
||||
child_agents = descendant_agents(scope_dir, agents)
|
||||
|
||||
print(f"AGENTS: {format_path(scope_agent, cwd)}")
|
||||
|
||||
if child_agents:
|
||||
print(" child AGENTS:")
|
||||
for agent in child_agents:
|
||||
print(f" - {format_path(agent, cwd)}")
|
||||
else:
|
||||
print(" child AGENTS: none")
|
||||
|
||||
children = immediate_child_dirs(scope_dir)
|
||||
if not children:
|
||||
print(" immediate child dirs: none")
|
||||
print()
|
||||
return
|
||||
|
||||
print(" immediate child dirs:")
|
||||
summaries = [summarize_child(child, agents) for child in children]
|
||||
for summary in summaries:
|
||||
marker = "has AGENTS" if summary.has_agents_here else "no AGENTS"
|
||||
extra = ""
|
||||
if summary.descendant_agents and not summary.has_agents_here:
|
||||
extra = f", {summary.descendant_agents} deeper AGENTS"
|
||||
print(
|
||||
" - "
|
||||
f"{format_path(summary.path, cwd)}/ -> "
|
||||
f"{summary.code_files} code files, "
|
||||
f"{summary.nested_dirs} nested dirs, "
|
||||
f"{marker}{extra}"
|
||||
)
|
||||
|
||||
hotspots = [
|
||||
summary
|
||||
for summary in summaries
|
||||
if (
|
||||
not summary.has_agents_here
|
||||
and summary.code_files >= threshold
|
||||
and summary.descendant_agents == 0
|
||||
)
|
||||
]
|
||||
if hotspots:
|
||||
print(" hotspot children without AGENTS:")
|
||||
for summary in hotspots:
|
||||
print(
|
||||
" - "
|
||||
f"{format_path(summary.path, cwd)}/ -> "
|
||||
f"{summary.code_files} code files, {summary.nested_dirs} nested dirs"
|
||||
)
|
||||
else:
|
||||
print(" hotspot children without AGENTS: none")
|
||||
|
||||
second_level_hotspots = []
|
||||
for summary in summaries:
|
||||
for nested in nested_hotspots(summary, agents, threshold):
|
||||
second_level_hotspots.append((summary.path, nested))
|
||||
|
||||
if second_level_hotspots:
|
||||
print(" nested hotspot children under umbrella dirs:")
|
||||
for parent, nested in second_level_hotspots:
|
||||
print(
|
||||
" - "
|
||||
f"{format_path(nested.path, cwd)}/ -> "
|
||||
f"{nested.code_files} code files, {nested.nested_dirs} nested dirs "
|
||||
f"(under {format_path(parent, cwd)}/)"
|
||||
)
|
||||
else:
|
||||
print(" nested hotspot children under umbrella dirs: none")
|
||||
|
||||
print()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
cwd = Path.cwd().resolve()
|
||||
roots = [Path(root).resolve() for root in args.roots]
|
||||
agents = discover_agents(roots)
|
||||
|
||||
if not agents:
|
||||
print("No AGENTS.md files found.")
|
||||
return 0
|
||||
|
||||
print("# AGENTS inventory")
|
||||
print(f"roots: {', '.join(format_path(root, cwd) for root in roots)}")
|
||||
print()
|
||||
|
||||
scope_dirs = sorted(agent.parent for agent in agents)
|
||||
for scope_dir in scope_dirs:
|
||||
print_scope(scope_dir, agents, cwd, args.hotspot_threshold)
|
||||
|
||||
print("Notes:")
|
||||
print("- This is a heuristic inventory, not an automatic decision engine.")
|
||||
print("- A hotspot is only a candidate for a deeper AGENTS.md.")
|
||||
print("- Always verify architecture claims against live code before reporting drift.")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
285
.agents/skills/review-pr/SKILL.md
Normal file
285
.agents/skills/review-pr/SKILL.md
Normal file
|
|
@ -0,0 +1,285 @@
|
|||
---
|
||||
name: review-pr
|
||||
description: Review a Dograh pull request, branch diff, or pasted patch for repo-specific security and correctness risks that are not obvious from generic FastAPI, Next.js, or Python conventions. Use when the user asks to review a PR, audit a diff, check whether changes are safe to merge, review their own changes, or asks what to look for in a Dograh PR. Focus on tenant isolation, route auth, webhook signing and org derivation, DB layering, worker-sync, migrations, generated SDK usage, and test hazards.
|
||||
---
|
||||
|
||||
# Reviewing PRs (dograh)
|
||||
|
||||
This skill is for reviewing any PR, including PRs written by maintainers. Focus on Dograh-specific regression risks. Skip generic lint, formatting, and type-check comments unless they connect to one of the repo-specific issues below.
|
||||
|
||||
The main failure modes in this repo are:
|
||||
|
||||
- Missing org scoping on request-reachable reads or writes
|
||||
- Authless routes or websockets
|
||||
- Trusting unsigned webhook fields
|
||||
- SQL written outside `api/db/*_client.py`
|
||||
- Per-worker cache state updated without worker sync
|
||||
- UI calls that bypass the generated SDK
|
||||
- Migrations that are not safe on existing production data
|
||||
|
||||
## How to drive the review
|
||||
|
||||
1. Get the diff:
|
||||
- GitHub PR: `gh pr diff <N>` or `gh pr view <N> --json files,additions,deletions`
|
||||
- Local branch: `git diff origin/main...HEAD`
|
||||
2. Bucket changed files into the sections below.
|
||||
3. Read the current repo as source of truth before finalizing findings:
|
||||
- `api/AGENTS.md` for org scoping and worker-sync
|
||||
- `ui/AGENTS.md` for generated client rules
|
||||
- Touched models, DB clients, routes, services, and migrations
|
||||
4. Run only the sections relevant to the changed files.
|
||||
5. Report findings as `<file>:<line> -> <problem> -> <correct pattern>`.
|
||||
|
||||
## Freshness rule
|
||||
|
||||
Treat this file as review policy and navigation, not as a frozen inventory.
|
||||
|
||||
- If the current repo conflicts with this skill, trust the repo and mention the drift.
|
||||
- Do not rely on static allowlists or exact line numbers from this file.
|
||||
- Review against the code in the PR and the current local repo, not against old prose.
|
||||
|
||||
### File to section map
|
||||
|
||||
| Path pattern in diff | Sections to run |
|
||||
| ----------------------------------------------------- | --------------- |
|
||||
| `api/routes/*.py` | 1, 2, 8 |
|
||||
| `api/db/*_client.py`, `api/db/models.py` | 2, 3 |
|
||||
| `api/services/**/*.py` | 2, 3, 4 |
|
||||
| `api/tasks/*.py` | 2, 3, 5 |
|
||||
| `api/alembic/versions/*.py` | 6 |
|
||||
| `api/mcp_server/**`, `api/services/workflow/mcp_*.py` | 1, 2, 7 |
|
||||
| `ui/**` | 9 |
|
||||
| `api/constants.py`, anything `os.getenv` | 10 |
|
||||
| `api/tests/**` | 11 |
|
||||
| `api/schemas/*.py` | 12 |
|
||||
|
||||
---
|
||||
|
||||
## 1. Route authentication (`api/routes/*.py`)
|
||||
|
||||
There is no global auth middleware. Each route declares its own auth behavior. Forgetting one creates a silently public endpoint.
|
||||
|
||||
Common auth deps from `api.services.auth.depends`:
|
||||
|
||||
- `get_user`
|
||||
- `get_user_ws`
|
||||
- `get_superuser`
|
||||
|
||||
Checks:
|
||||
|
||||
- A new `@router.<verb>(...)` handler with no auth dependency is public. Treat that as a finding unless the current file already establishes a deliberate public auth pattern such as public token auth, signed webhook auth, or an equivalent websocket token flow.
|
||||
- `get_user` on an impersonation, cross-org, or global reporting endpoint should usually be `get_superuser`.
|
||||
- A route that reimplements bearer or API-key parsing instead of using the shared auth dependency is a finding.
|
||||
- A websocket handler without `Depends(get_user_ws)` and without a clear public token path is a finding.
|
||||
- A PR tightening `CORSMiddleware` to a fixed origin list needs strong justification. Dograh relies on cross-origin embedding; endpoint auth is the real control.
|
||||
|
||||
Useful commands:
|
||||
|
||||
```bash
|
||||
rg -n "Depends\\((get_user|get_user_ws|get_superuser)\\)" api/routes
|
||||
rg -n "@router\\.(get|post|put|delete|patch|websocket)" api/routes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Organization scoping (the cross-tenant rule)
|
||||
|
||||
This is the highest priority rule in the repo. Every request-reachable read or write of an org-scoped resource must filter or validate by `organization_id`.
|
||||
|
||||
Use `api/AGENTS.md` as the canonical summary.
|
||||
|
||||
Determine scope from the current code:
|
||||
|
||||
- Direct scope: the model has `organization_id`
|
||||
- Indirect scope: the model reaches an org through a parent FK or relationship
|
||||
- Legacy spelling may exist in old migrations or old code, but new runtime code should use `organization_id`
|
||||
|
||||
Checks:
|
||||
|
||||
- Any `*_by_id(...)` call in a route handler is suspicious. If request-reachable and unscoped, it is usually a finding.
|
||||
- New `list_*` or `get_*` endpoints must filter in SQL, not in Python after `.all()`.
|
||||
- If a request writes an FK to another org-scoped resource, the route must first fetch that target row with `user.selected_organization_id` and reject if it does not belong to the org.
|
||||
- Services called from routes must preserve scoping. If a service method or DB client call drops `organization_id`, trace the caller.
|
||||
- Background tasks do not get org context for free. They must reload the parent row and derive org from there.
|
||||
- Webhooks must derive org from a signed or otherwise authenticated identifier, not from caller-supplied body fields like `organization_id`.
|
||||
- New runtime code should use canonical `organization_id`, not `org_id`, `tenant_id`, or `organisation_id`.
|
||||
|
||||
Useful commands:
|
||||
|
||||
```bash
|
||||
rg -n "_by_id\\(" api/routes api/services api/tasks
|
||||
rg -n "db_client\\.get_\\w+\\(" api/routes api/services api/tasks
|
||||
rg -n "organization_id|selected_organization_id" api/routes api/services api/tasks api/db
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. DB query layering (`api/db/` is the only home for SQL)
|
||||
|
||||
Production SQL belongs in `api/db/*_client.py`. Routes, services, and tasks should call DB client methods, not write SQLAlchemy directly.
|
||||
|
||||
Checks:
|
||||
|
||||
- `select`, `update`, `delete`, `insert`, `AsyncSession`, `sessionmaker`, or `async_session` in `api/routes/`, `api/services/`, or `api/tasks/` is a finding.
|
||||
- `api/services/admin_utils/` is the exception. It is not a template for production code.
|
||||
- Session lifecycle belongs inside the DB client.
|
||||
- New DB client parameters should use canonical `organization_id`.
|
||||
|
||||
Useful commands:
|
||||
|
||||
```bash
|
||||
rg -n "(from sqlalchemy|AsyncSession|sessionmaker|async_session)" api/routes api/services api/tasks
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Worker sync - multi-process state coherence (`api/services/worker_sync/`)
|
||||
|
||||
Production runs multiple workers. Per-process mutable caches become stale unless updates are broadcast.
|
||||
|
||||
Use `api/AGENTS.md` as the canonical summary.
|
||||
|
||||
Checks:
|
||||
|
||||
- A new module-level or class-level mutable cache written by an endpoint needs a `WorkerSyncManager` broadcast path.
|
||||
- Local invalidation alone is not enough if other workers can still serve stale state.
|
||||
- If a PR introduces a new cached object, the diff should usually contain all three:
|
||||
- the broadcast call
|
||||
- the event type or equivalent signal definition
|
||||
- the handler registration that reloads fresh state
|
||||
|
||||
---
|
||||
|
||||
## 5. Background tasks (`api/tasks/`, ARQ)
|
||||
|
||||
Checks:
|
||||
|
||||
- User-triggered enqueue paths must validate org ownership before enqueue.
|
||||
- Tasks that accept IDs and reload rows must derive org from those rows, not assume shared context.
|
||||
- Tasks must be idempotent or explicitly retry-safe.
|
||||
- Only real task entrypoints belong in `api/tasks/arq.py::WorkerSettings.functions`.
|
||||
- Secret logging rules from section 10 apply here too.
|
||||
|
||||
---
|
||||
|
||||
## 6. Migrations (`api/alembic/versions/`)
|
||||
|
||||
Checks:
|
||||
|
||||
- `upgrade()` and `downgrade()` should both exist and be meaningfully reversible unless the change truly cannot be reversed.
|
||||
- Adding a `NOT NULL` column to a populated table needs a safe default or a backfill before the constraint.
|
||||
- Tightening nullable to non-nullable needs the backfill before `alter_column(..., nullable=False)`.
|
||||
- New JSON columns should match the table's existing JSON or JSONB conventions.
|
||||
- Large backfills in a migration should be questioned; they often belong out-of-band.
|
||||
- Indexes on large tables need concurrent-safe handling.
|
||||
- Do not turn historical migration naming into a finding by itself. Review the migration being changed, not old untouched migration prose.
|
||||
|
||||
---
|
||||
|
||||
## 7. MCP server (`api/mcp_server/`)
|
||||
|
||||
Checks:
|
||||
|
||||
- New tools should use `authenticate_mcp_request()`, not reimplement API-key validation.
|
||||
- New tool DB lookups should preserve org scoping just like REST routes.
|
||||
- Tools that call external URLs must validate those URLs and consider SSRF.
|
||||
|
||||
---
|
||||
|
||||
## 8. Telephony and webhook handlers
|
||||
|
||||
Checks:
|
||||
|
||||
- New provider webhook flows should implement `verify_inbound_signature()` or the provider equivalent.
|
||||
- Minimal pre-verification work may be required to identify the candidate config, but the route should not do unrelated workflow, user, or stateful work before verification.
|
||||
- Org derivation should come from provider identifiers that are validated by the webhook auth flow, then the derived org/config should drive downstream lookups.
|
||||
- A webhook must not trust raw body `organization_id`.
|
||||
- If a webhook references a phone number, validate that the number exists for the derived org.
|
||||
|
||||
---
|
||||
|
||||
## 9. UI (`ui/`) - generated SDK only
|
||||
|
||||
Use `ui/AGENTS.md` as the canonical summary.
|
||||
|
||||
The frontend should talk to the backend through `ui/src/client/`. Raw `fetch` to internal `/api/v1/` routes is suspicious by default.
|
||||
|
||||
Checks:
|
||||
|
||||
- `fetch('/api/v1/...')` or `fetch(\`${backendUrl}/api/v1/...\`)` in app code is usually a finding unless the current code proves a narrow exception.
|
||||
- Hardcoded backend URLs are a finding.
|
||||
- Manual `Authorization` header construction in regular components is a finding; auth should be injected centrally.
|
||||
- SDK calls fired before auth state is ready are a finding.
|
||||
- Local interfaces that duplicate generated types are a finding.
|
||||
- If backend API shape changed and the UI consumes it, `ui/src/client/` should usually change too.
|
||||
|
||||
Useful commands:
|
||||
|
||||
```bash
|
||||
rg -n "fetch\\(['\"\\`].*api/v1" ui/src
|
||||
rg -n "Authorization" ui/src
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. Logging, secrets, constants
|
||||
|
||||
Checks:
|
||||
|
||||
- New code should use Loguru, not stdlib `logging`.
|
||||
- New `os.getenv(...)` outside `api/constants.py` is a finding.
|
||||
- Do not log API keys, bearer tokens, credentials, full webhook bodies, or PII.
|
||||
|
||||
Common offender shapes:
|
||||
|
||||
- `logger.info(f"config: {config}")`
|
||||
- `logger.debug(request_body)`
|
||||
- Logging raw config or user configuration rows
|
||||
|
||||
---
|
||||
|
||||
## 11. Tests (`api/tests/`)
|
||||
|
||||
Checks:
|
||||
|
||||
- Async waits in tests should use `asyncio.wait_for(...)` or another bounded timeout pattern.
|
||||
- Tests should run against `.env.test`, not `.env`.
|
||||
- Integration tests should not be neutered by replacing real DB behavior with mocks just to make the test pass.
|
||||
- Tests that depend on mutable shared DB state across test cases are suspicious.
|
||||
|
||||
---
|
||||
|
||||
## 12. Schemas (`api/schemas/`)
|
||||
|
||||
Checks:
|
||||
|
||||
- New response schemas should not expose internal FKs or IDs unless the caller genuinely needs them.
|
||||
- Request schemas that accept org-scoped FK values are a trigger to inspect the corresponding route for section 2 ownership validation.
|
||||
|
||||
---
|
||||
|
||||
## Final pass: shape the report
|
||||
|
||||
Present findings in three buckets:
|
||||
|
||||
- **Blocker**
|
||||
- Missing org scope on a request-reachable lookup
|
||||
- Route added without auth and without a proven deliberate public auth mechanism
|
||||
- Webhook without signature verification, or significant unrelated work done before verification
|
||||
- Migration without safe backfill or without a meaningful downgrade
|
||||
- UI bypasses generated SDK for internal API calls
|
||||
- Secrets logged
|
||||
|
||||
- **Should-fix**
|
||||
- Cached state mutated without worker sync
|
||||
- JSON vs JSONB inconsistency
|
||||
- Response schema leaks internal identifiers
|
||||
- Backend API changed without client regen where UI consumes it
|
||||
- Test path can hang indefinitely
|
||||
|
||||
- **Nit**
|
||||
- Naming inconsistencies
|
||||
- Minor convention drift
|
||||
- Low-risk schema or report-shape cleanup
|
||||
|
||||
Cite `file:line` for each finding. Skip anything a formatter, linter, or IDE would already catch unless it connects to one of the repo-specific risks above.
|
||||
|
|
@ -1,3 +1,3 @@
|
|||
{
|
||||
".": "1.29.0"
|
||||
".": "1.31.0"
|
||||
}
|
||||
16
AGENTS.md
16
AGENTS.md
|
|
@ -25,21 +25,7 @@ dograh/
|
|||
|
||||
## Local Development
|
||||
|
||||
### Starting Services
|
||||
|
||||
```bash
|
||||
# Start infrastructure services (postgres, redis, minio)
|
||||
./scripts/start_services_dev.sh
|
||||
|
||||
# Stop all services
|
||||
./scripts/stop_services.sh
|
||||
```
|
||||
|
||||
On Windows (PowerShell):
|
||||
```powershell
|
||||
.\scripts\start_services_dev.ps1
|
||||
.\scripts\stop_services.ps1
|
||||
```
|
||||
Contributor setup and service startup are documented in `docs/contribution/setup.mdx`.
|
||||
|
||||
## Environment Configuration
|
||||
|
||||
|
|
|
|||
50
CHANGELOG.md
50
CHANGELOG.md
|
|
@ -1,5 +1,55 @@
|
|||
# Changelog
|
||||
|
||||
## [1.31.0](https://github.com/dograh-hq/dograh/compare/dograh-v1.30.1...dograh-v1.31.0) (2026-05-21)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* add agent skills to review PR ([#320](https://github.com/dograh-hq/dograh/issues/320)) ([151bf77](https://github.com/dograh-hq/dograh/commit/151bf77e40476b63000c1e88d2f348d5d2791344))
|
||||
* add chat based testing for voice agent ([#308](https://github.com/dograh-hq/dograh/issues/308)) ([d97d1d7](https://github.com/dograh-hq/dograh/commit/d97d1d72cd1a414442b8b9f66d8312950c06978c))
|
||||
* add Review AGENTS.md Skill ([d93d7af](https://github.com/dograh-hq/dograh/commit/d93d7aff4d5308ee17c55855f0ffd1ed9f90449f))
|
||||
* add Tuner Integration to Dograh ([#311](https://github.com/dograh-hq/dograh/issues/311)) ([5f28c1b](https://github.com/dograh-hq/dograh/commit/5f28c1b2a9b17ed19f8a2b4118d1d4eb8c4249a7))
|
||||
* **mcp:** add search_docs tool over docs corpus (closes [#295](https://github.com/dograh-hq/dograh/issues/295)) ([#316](https://github.com/dograh-hq/dograh/issues/316)) ([5762095](https://github.com/dograh-hq/dograh/commit/5762095edfa585fa078ba70d486bc7af14708457))
|
||||
* **mcp:** generic MCP tool source with per-node function filtering ([#301](https://github.com/dograh-hq/dograh/issues/301)) ([75839f9](https://github.com/dograh-hq/dograh/commit/75839f9de5eb26ccc296235af36058e442d10d58))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **security:** bump python-multipart 0.0.20 -> 0.0.27 ([#332](https://github.com/dograh-hq/dograh/issues/332)) ([332754a](https://github.com/dograh-hq/dograh/commit/332754a809ec14b9164c698fb3eff682b1d9d446))
|
||||
* **stt:** align Speechmatics language registry with official transcription codes ([#317](https://github.com/dograh-hq/dograh/issues/317)) ([afa78fe](https://github.com/dograh-hq/dograh/commit/afa78fe859e51d45b12dedd01613f2c24ffc7f65))
|
||||
* **webRTC:** LAN IP filtering ([#333](https://github.com/dograh-hq/dograh/issues/333)) ([af66372](https://github.com/dograh-hq/dograh/commit/af66372b655f05f4fc8e778ec58902e15ce25531))
|
||||
|
||||
|
||||
### Documentation
|
||||
|
||||
* add Simplified Chinese translation of README ([#305](https://github.com/dograh-hq/dograh/issues/305)) ([5b1e398](https://github.com/dograh-hq/dograh/commit/5b1e3980b1982506aa334d19ab594db04ef9e19c))
|
||||
|
||||
## [1.30.1](https://github.com/dograh-hq/dograh/compare/dograh-v1.30.0...dograh-v1.30.1) (2026-05-17)
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* fix race between context init and keepalive for Dograh TTS ([ba7d45f](https://github.com/dograh-hq/dograh/commit/ba7d45fde054e30eb717f7912283d71647bdce2c))
|
||||
|
||||
## [1.30.0](https://github.com/dograh-hq/dograh/compare/dograh-v1.29.0...dograh-v1.30.0) (2026-05-16)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* add openai realtime models ([#298](https://github.com/dograh-hq/dograh/issues/298)) ([2381a80](https://github.com/dograh-hq/dograh/commit/2381a803ade54f6c8d1db572e0f6c3301dd74c20))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* force FORCE_TURN_RELAY for local IPs in setup ([fc04f31](https://github.com/dograh-hq/dograh/commit/fc04f31639e0d326525d6840ca117babe2b25ea8))
|
||||
* provider resolution in telephony cost calculation post workflow integration calls ([0523dcb](https://github.com/dograh-hq/dograh/commit/0523dcb079410803a54deec49afda98cbb96e7bd))
|
||||
|
||||
|
||||
### Documentation
|
||||
|
||||
* add telnyx to telephony providers supporting call transfer ([4ff1f57](https://github.com/dograh-hq/dograh/commit/4ff1f576f0a5e079466318d6e99d27eada6abc9e))
|
||||
* update README.md ([ea13492](https://github.com/dograh-hq/dograh/commit/ea13492a894af11410c7c54500f4bdc6fa0c2cda))
|
||||
|
||||
## [1.29.0](https://github.com/dograh-hq/dograh/compare/dograh-v1.28.0...dograh-v1.29.0) (2026-05-13)
|
||||
|
||||
|
||||
|
|
|
|||
129
README.md
129
README.md
|
|
@ -1,41 +1,71 @@
|
|||
# Dograh AI
|
||||
|
||||
<h3 align="center">⭐ <strong>If you find value in this project, please STAR the Github repository to help others discover our FOSS platform!</strong></h3>
|
||||
**The open-source, self-hostable alternative to Vapi & Retell** — build production voice agents with a drag-and-drop workflow builder. From zero to a working bot in under 2 minutes.
|
||||
|
||||
<p align="center">
|
||||
<a href="https://docs.dograh.com">
|
||||
<img src="https://img.shields.io/badge/docs-https://docs.dograh.com-blue.svg" alt="Docs: https://docs.dograh.com">
|
||||
<a href="https://app.dograh.com">
|
||||
<img src="https://img.shields.io/badge/▶_Try_the_Cloud-app.dograh.com-2563eb?style=for-the-badge" alt="Try the Cloud">
|
||||
</a>
|
||||
<a href="https://deepwiki.com/dograh-hq/dograh">
|
||||
<img src="https://deepwiki.com/badge.svg" alt="Deepwiki: https://deepwiki.com/dograh-hq/dograh">
|
||||
</a>
|
||||
<a href="LICENSE">
|
||||
<img src="https://img.shields.io/badge/license-BSD%202--Clause-blue.svg" alt="License: BSD 2-Clause">
|
||||
|
||||
<a href="#-get-started">
|
||||
<img src="https://img.shields.io/badge/⚡_Self--host_in_60s-One_command-111827?style=for-the-badge" alt="Self-host in 60s">
|
||||
</a>
|
||||
|
||||
<a href="https://join.slack.com/t/dograh-community/shared_invite/zt-3czr47sw5-MSg1J0kJ7IMPOCHF~03auQ">
|
||||
<img src="https://img.shields.io/badge/chat-on%20Slack-4A154B?logo=slack" alt="Slack Community">
|
||||
</a>
|
||||
<a href="https://www.docker.com/">
|
||||
<img src="https://img.shields.io/badge/docker-ready-blue?logo=docker" alt="Docker Ready">
|
||||
<img src="https://img.shields.io/badge/💬_Join_Slack-Community-4A154B?style=for-the-badge&logo=slack" alt="Join Slack">
|
||||
</a>
|
||||
</p>
|
||||
|
||||
**The open-source alternative to Vapi** - Dograh helps you build your own voice agents with an easy drag-and-drop workflow builder. It's the fastest way to build voice AI agents - from zero to working bot in under 2 minutes (our hard SLA standards).
|
||||
<p align="center">
|
||||
<a href="https://docs.dograh.com">📖 Docs</a> ·
|
||||
<a href="LICENSE">📜 BSD 2-Clause</a> ·
|
||||
<a href="README.zh-CN.md">🌐 中文</a>
|
||||
</p>
|
||||
|
||||
- **100% open source**, self-hostable platform - no vendor lock-in, unlike proprietary solutions like Vapi
|
||||
- **Full control & transparency** - every line of code is open, with built-in AI testing personas and flexible LLM/TTS/STT integration
|
||||
- **Maintained by YC alumni and exit founders**, ensuring the future of voice AI stays open, not monopolized
|
||||
<p align="center">
|
||||
<img src="docs/images/hero.gif" alt="Dograh in action — build a workflow, launch a voice agent, talk to it" width="80%">
|
||||
</p>
|
||||
|
||||
## 🎥 Demo Video
|
||||
- **100% open source**, self-hostable — no vendor lock-in, unlike Vapi or Retell
|
||||
- **Full control & transparency** — every line of code is open, with flexible LLM / TTS / STT integration
|
||||
- **Maintained by YC alumni and exit founders**, committed to keeping voice AI open
|
||||
|
||||
## 🎥 Featured
|
||||
|
||||
<div align="center">
|
||||
<a href="https://www.youtube.com/watch?v=xD9JEvfCH9k">
|
||||
<img src="https://img.youtube.com/vi/xD9JEvfCH9k/maxresdefault.jpg" alt="Dograh featured by Better Stack" width="80%" style="border-radius: 8px; box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);">
|
||||
</a>
|
||||
<br>
|
||||
<em>Featured by <strong>Better Stack</strong> — a hands-on look at Dograh</em>
|
||||
</div>
|
||||
|
||||
<details>
|
||||
<summary>📺 Prefer a 2-minute product walkthrough? Click here.</summary>
|
||||
|
||||
<div align="center">
|
||||
<a href="https://youtu.be/9gPneyf9M9w">
|
||||
<img src="docs/images/video_thumbnail_1.png" alt="Watch Dograh AI Demo Video" width="80%" style="border-radius: 8px; box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);">
|
||||
<img src="docs/images/video_thumbnail_1.png" alt="Watch Dograh AI Demo Video" width="70%" style="border-radius: 8px; box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);">
|
||||
</a>
|
||||
<br>
|
||||
<em>Click to watch a 2-minute demo of Dograh AI in action</em>
|
||||
</div>
|
||||
|
||||
</details>
|
||||
|
||||
## ⚖️ Dograh vs Vapi vs Retell
|
||||
|
||||
An honest comparison on the axes that matter most to teams evaluating voice AI platforms.
|
||||
|
||||
| | **Dograh** | **Vapi** | **Retell** |
|
||||
|---|---|---|---|
|
||||
| **License** | BSD 2-Clause (open source) | Proprietary | Proprietary |
|
||||
| **Self-hostable** | ✅ Yes — one Docker command | ❌ SaaS only | ❌ SaaS only |
|
||||
| **Pricing** | Free (self-host) · usage-based (cloud) | Per-minute SaaS | Per-minute SaaS |
|
||||
| **Bring your own LLM / STT / TTS** | ✅ Any provider, or use Dograh's stack | Configurable within their integrations | Configurable within their integrations |
|
||||
| **Source-level customization** | ✅ Every line is yours to modify | ❌ Closed source | ❌ Closed source |
|
||||
| **Data residency** | Your infra, your rules | Their cloud | Their cloud |
|
||||
| **Vendor lock-in** | None | Full | Full |
|
||||
|
||||
|
||||
## 🚀 Get Started
|
||||
|
||||
##### Download and setup Dograh on your Local Machine
|
||||
|
|
@ -56,30 +86,17 @@ curl -o docker-compose.yaml https://raw.githubusercontent.com/dograh-hq/dograh/m
|
|||
|
||||
### 🎙️ Your First Voice Bot
|
||||
|
||||
1. **Open Dashboard**: Launch [http://localhost:3010](http://localhost:3010) on your browser
|
||||
2. **Choose Call Type**: Select **Inbound** or **Outbound** calling.
|
||||
3. **Name Your Bot**: Use a short two-word name (e.g., _Lead Qualification_).
|
||||
4. **Describe Use Case**: In 5–10 words (e.g., _Screen insurance form submissions for purchase intent_).
|
||||
5. **Launch**: Your bot is ready! Open the bot and click **Web Call** to talk to it.
|
||||
6. **No API Keys Needed**: We auto-generate Dograh API keys so you can start immediately. You can switch to your own keys anytime.
|
||||
7. **Default Access**: Includes Dograh’s own LLMs, STT, and TTS stack by default.
|
||||
8. **Bring Your Own Keys**: Optionally connect your own API keys for LLMs, STT, TTS, or telephony providers like Twilio.
|
||||
1. Open [http://localhost:3010](http://localhost:3010) in your browser.
|
||||
2. Pick **Inbound** or **Outbound**, name your bot (e.g. _Lead Qualification_), and describe the use case in 5–10 words (e.g. _Screen insurance form submissions for purchase intent_).
|
||||
3. Click **Web Call** — you're talking to your bot.
|
||||
|
||||
## Quick Summary
|
||||
|
||||
⚡ **Open-source alternative to Vapi** - 2-minute setup with hard SLA standards
|
||||
|
||||
- 🔧 **No vendor lock-in**: Self-hostable platform vs proprietary SaaS solutions
|
||||
- 🤖 **AI Testing Personas**: Test your bots with LoopTalk AI that mimics real customer interactions
|
||||
- 🔓 **100% Open Source**: Every line of code is open - no hidden logic, no black boxes (unlike Vapi)
|
||||
- 🔄 **Flexible Integration**: Bring your own LLM, TTS, or STT - or use Dograh's APIs
|
||||
- ☁️ **Deploy anywhere**: Self-host or use our hosted version at app.dograh.com
|
||||
> 🔑 **No API keys needed.** Dograh ships with auto-generated keys and its own LLM / TTS / STT stack. Connect your own keys for LLM, TTS, STT, or Telephony (e.g. Twilio, Vonage, Telnyx) anytime.
|
||||
|
||||
## Features
|
||||
|
||||
### Voice Capabilities
|
||||
|
||||
- Telephony: Built-in telephony integration like Twilio, Vonage, Vobiz, Cloudonix (easily add others)
|
||||
- Telephony: Built-in telephony integration like Twilio, Vonage, Vobiz, Cloudonix (easily add others), with support for transferring calls to human agents
|
||||
- Languages: English support (expandable to other languages)
|
||||
- Custom Models: Bring your own TTS/STT models
|
||||
- Real-time Processing: Low-latency voice interactions
|
||||
|
|
@ -93,13 +110,9 @@ curl -o docker-compose.yaml https://raw.githubusercontent.com/dograh-hq/dograh/m
|
|||
|
||||
### Testing & Quality
|
||||
|
||||
- LoopTalk (Beta): Create AI personas to test your voice agents
|
||||
- Workflow Testing: Test specific workflow IDs with automated calls
|
||||
- Real-world Simulation: AI personas that mimic actual customer behavior
|
||||
|
||||
## Architecture
|
||||
|
||||
Architecture diagram _(coming soon)_
|
||||
- **Test Mode**: Try your agent end-to-end before publishing, with no production calls or data affected
|
||||
- **In-Dashboard Web Calls**: Talk to your bot directly while building — no telephony setup required
|
||||
- **QA Node**: A built-in workflow node that analyzes prompt quality across your other nodes
|
||||
|
||||
## Deployment Options
|
||||
|
||||
|
|
@ -111,10 +124,6 @@ Refer [Local Setup](https://docs.dograh.com/contribution/setup)
|
|||
|
||||
For detailed deployment instructions including remote server setup with HTTPS, see our [Docker Deployment Guide](https://docs.dograh.com/deployment/docker).
|
||||
|
||||
### Production (Self-Hosted)
|
||||
|
||||
Production guide coming soon. [Drop in a message](https://join.slack.com/t/dograh-community/shared_invite/zt-3czr47sw5-MSg1J0kJ7IMPOCHF~03auQ) for assistance.
|
||||
|
||||
### Cloud Version
|
||||
|
||||
Visit [https://www.dograh.com](https://www.dograh.com/) for our managed cloud offering.
|
||||
|
|
@ -123,14 +132,18 @@ Visit [https://www.dograh.com](https://www.dograh.com/) for our managed cloud of
|
|||
|
||||
You can go to [https://docs.dograh.com](https://docs.dograh.com/) for our documentation.
|
||||
|
||||
## 📦 SDKs
|
||||
|
||||
- **Python SDK** — [pypi.org/project/dograh-sdk](https://pypi.org/project/dograh-sdk/)
|
||||
- **Node SDK** — [npmjs.com/package/@dograh/sdk](https://www.npmjs.com/package/@dograh/sdk)
|
||||
|
||||
## 🤝Community & Support
|
||||
|
||||
- GitHub Issues: Report bugs or request features
|
||||
- Slack: Our Slack community is not just for support — it’s the cornerstone of Dograh AI contributions. Here, you can:
|
||||
- Connect with maintainers and other contributors
|
||||
- Discuss issues and features before coding
|
||||
- Get help with setup and debugging
|
||||
- Stay up to date with contribution sprints
|
||||
> 👋 **Coming from the Better Stack video?** Drop your use case in our [pinned GitHub Discussion](https://github.com/orgs/dograh-hq/discussions/291) — we read every reply and the founders personally onboard early adopters.
|
||||
|
||||
- **Slack** — the cornerstone of Dograh AI contributions. Connect with maintainers, discuss features before coding, get help with setup, and stay current on contribution sprints.
|
||||
- **GitHub Discussions** — share use cases, ask questions, swap workflow recipes.
|
||||
- **GitHub Issues** — report bugs or request features.
|
||||
|
||||
👉 Join us → [Dograh Community Slack](https://join.slack.com/t/dograh-community/shared_invite/zt-3czr47sw5-MSg1J0kJ7IMPOCHF~03auQ)
|
||||
|
||||
|
|
@ -146,6 +159,12 @@ We love contributions! Dograh AI is 100% open source and we intend to keep it th
|
|||
- Push to the branch (git push origin feature/AmazingFeature)
|
||||
- Open a Pull Request
|
||||
|
||||
## ⭐ Star History
|
||||
|
||||
<a href="https://star-history.com/#dograh-hq/dograh&Date">
|
||||
<img src="https://api.star-history.com/svg?repos=dograh-hq/dograh&type=Date" alt="Dograh star history" width="80%">
|
||||
</a>
|
||||
|
||||
## 📄 License
|
||||
|
||||
Dograh AI is licensed under the [BSD 2-Clause License](LICENSE)- the same license as projects that were used in building Dograh AI, ensuring compatibility and freedom to use, modify, and distribute.
|
||||
|
|
|
|||
182
README.zh-CN.md
Normal file
182
README.zh-CN.md
Normal file
|
|
@ -0,0 +1,182 @@
|
|||
# Dograh AI
|
||||
|
||||
> 💡 **Notice**: This documentation is community-maintained. If you spot any translation inaccuracies or content that has drifted from the English version, please feel free to open a PR!
|
||||
>
|
||||
> 💡 **提示**:本文档由社区共同维护。如果您发现翻译不准确,或与英文版本存在出入,欢迎随时提交 PR!
|
||||
|
||||
**开源、可自托管的 Vapi 与 Retell 替代方案** —— 通过拖拽式工作流编辑器构建生产级语音智能体,2 分钟内即可上线一个能用的语音机器人。
|
||||
|
||||
<p align="center">
|
||||
<a href="https://app.dograh.com">
|
||||
<img src="https://img.shields.io/badge/▶_体验云端版本-app.dograh.com-2563eb?style=for-the-badge" alt="体验云端版本">
|
||||
</a>
|
||||
|
||||
<a href="#-快速开始">
|
||||
<img src="https://img.shields.io/badge/⚡_60_秒自托管-一行命令-111827?style=for-the-badge" alt="60 秒自托管">
|
||||
</a>
|
||||
|
||||
<a href="https://join.slack.com/t/dograh-community/shared_invite/zt-3czr47sw5-MSg1J0kJ7IMPOCHF~03auQ">
|
||||
<img src="https://img.shields.io/badge/💬_加入_Slack-社区-4A154B?style=for-the-badge&logo=slack" alt="加入 Slack">
|
||||
</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://docs.dograh.com">📖 文档</a> ·
|
||||
<a href="LICENSE">📜 BSD 2-Clause</a> ·
|
||||
<a href="README.md">🌐 English</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/images/hero.gif" alt="Dograh 实战演示 —— 搭建工作流、启动语音智能体、直接对话" width="80%">
|
||||
</p>
|
||||
|
||||
- **100% 开源**,可自托管 —— 不像 Vapi 或 Retell,没有任何厂商绑定
|
||||
- **完全可控且透明** —— 每一行代码都是开放的,LLM / TTS / STT 集成灵活可换
|
||||
- **由 YC 校友与连续创业者维护**,致力于让语音 AI 始终保持开放
|
||||
|
||||
## 🎥 媒体推荐
|
||||
|
||||
<div align="center">
|
||||
<a href="https://www.youtube.com/watch?v=xD9JEvfCH9k">
|
||||
<img src="https://img.youtube.com/vi/xD9JEvfCH9k/maxresdefault.jpg" alt="Better Stack 介绍 Dograh" width="80%" style="border-radius: 8px; box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);">
|
||||
</a>
|
||||
<br>
|
||||
<em><strong>Better Stack</strong> 上手实测 —— 深入体验 Dograh</em>
|
||||
</div>
|
||||
|
||||
<details>
|
||||
<summary>📺 想看 2 分钟产品快速演示?点这里。</summary>
|
||||
|
||||
<div align="center">
|
||||
<a href="https://youtu.be/9gPneyf9M9w">
|
||||
<img src="docs/images/video_thumbnail_1.png" alt="观看 Dograh AI 演示视频" width="70%" style="border-radius: 8px; box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);">
|
||||
</a>
|
||||
</div>
|
||||
|
||||
</details>
|
||||
|
||||
## ⚖️ Dograh vs Vapi vs Retell
|
||||
|
||||
针对正在评估语音 AI 平台的团队,这里是一份在最关键的维度上诚实的对比。
|
||||
|
||||
| | **Dograh** | **Vapi** | **Retell** |
|
||||
|---|---|---|---|
|
||||
| **协议** | BSD 2-Clause(开源) | 闭源 | 闭源 |
|
||||
| **可自托管** | ✅ 可以 —— 一条 Docker 命令 | ❌ 仅 SaaS | ❌ 仅 SaaS |
|
||||
| **定价** | 免费(自托管)·按用量计费(云端) | 按分钟计费的 SaaS | 按分钟计费的 SaaS |
|
||||
| **自带 LLM / STT / TTS** | ✅ 任意厂商,也可使用 Dograh 自带方案 | 在其集成范围内可配置 | 在其集成范围内可配置 |
|
||||
| **源码级定制** | ✅ 每行代码都可自由修改 | ❌ 闭源 | ❌ 闭源 |
|
||||
| **数据驻留** | 部署在自家基础设施,规则自己定 | 厂商云端 | 厂商云端 |
|
||||
| **厂商绑定** | 无 | 完全绑定 | 完全绑定 |
|
||||
|
||||
|
||||
## 🚀 快速开始
|
||||
|
||||
##### 在本地机器下载并部署 Dograh
|
||||
|
||||
> **提示**
|
||||
> 我们会收集匿名使用数据以改进产品。如需关闭,请在下面的命令中将 `ENABLE_TELEMETRY` 设为 `false`。
|
||||
|
||||
> **提示**
|
||||
> 如果希望在远程服务器上运行该平台,请参考[文档](https://docs.dograh.com/deployment/docker#option-2:-remote-server-deployment)。
|
||||
|
||||
```bash
|
||||
curl -o docker-compose.yaml https://raw.githubusercontent.com/dograh-hq/dograh/main/docker-compose.yaml && REGISTRY=ghcr.io/dograh-hq ENABLE_TELEMETRY=true docker compose up --pull always
|
||||
```
|
||||
|
||||
> **提示**
|
||||
> 首次启动需要 2-3 分钟拉取所有镜像。启动完成后,打开 http://localhost:3010 即可创建你的第一个 AI 语音助手!
|
||||
> 常见问题及解决方案请参见 🔧 **[故障排查](docs/troubleshooting.md)**。
|
||||
|
||||
### 🎙️ 你的第一个语音机器人
|
||||
|
||||
1. 在浏览器中打开 [http://localhost:3010](http://localhost:3010)。
|
||||
2. 选择 **Inbound(呼入)** 或 **Outbound(外呼)**,为机器人命名(例如 _销售线索筛选_),再用 5-10 个词描述用途(例如 _筛选保险表单中的购买意向_)。
|
||||
3. 点击 **Web Call**,直接和你的机器人对话。
|
||||
|
||||
> 🔑 **无需 API Key。** Dograh 自带一套自动生成的密钥,以及内置的 LLM / TTS / STT 栈。你可以随时接入自己的 LLM、TTS、STT 或电信服务商(如 Twilio、Vonage、Telnyx)。
|
||||
|
||||
## 功能特性
|
||||
|
||||
### 语音能力
|
||||
|
||||
- 电信集成:内置 Twilio、Vonage、Vobiz、Cloudonix 等(其他厂商也易于扩展),支持转接到人工坐席
|
||||
- 语言:支持英语(可扩展到其他语言)
|
||||
- 自定义模型:可接入自己的 TTS / STT 模型
|
||||
- 实时处理:低延迟语音交互
|
||||
|
||||
### 开发者体验
|
||||
|
||||
- 零配置启动:自动生成 API Key,即开即用
|
||||
- 基于 Python:基于 Python 构建,便于二次开发
|
||||
- Docker 优先:容器化部署,环境一致
|
||||
- 模块化架构:按需替换各个组件
|
||||
|
||||
### 测试与质量
|
||||
|
||||
- **测试模式**:在发布前端到端试跑你的智能体,既不会产生真实通话,也不会影响生产数据
|
||||
- **面板内 Web 通话**:在搭建过程中直接和机器人对话,无需配置任何电信服务
|
||||
- **QA 节点**:内置的工作流节点,可分析其他节点中提示词的质量
|
||||
|
||||
## 部署方式
|
||||
|
||||
### 本地开发
|
||||
|
||||
参见[本地部署](https://docs.dograh.com/contribution/setup)。
|
||||
|
||||
### 自托管部署
|
||||
|
||||
如需了解远程服务器部署及 HTTPS 配置的详细步骤,请参见我们的 [Docker 部署指南](https://docs.dograh.com/deployment/docker)。
|
||||
|
||||
### 云端版本
|
||||
|
||||
托管云版本请访问 [https://www.dograh.com](https://www.dograh.com/)。
|
||||
|
||||
## 📚 文档
|
||||
|
||||
完整文档请访问 [https://docs.dograh.com](https://docs.dograh.com/)。
|
||||
|
||||
## 🤝 社区与支持
|
||||
|
||||
> 👋 **从 Better Stack 视频过来的朋友?** 欢迎在我们[置顶的 GitHub Discussion](https://github.com/orgs/dograh-hq/discussions/291) 里留下你的使用场景 —— 每一条留言我们都会看,创始团队会亲自对接早期用户。
|
||||
|
||||
- **Slack** —— Dograh AI 协作的主阵地。在这里和维护者交流、在动手前讨论功能、获取部署帮助,并跟进每一轮贡献活动。
|
||||
- **GitHub Discussions** —— 分享使用场景、提问、交流工作流配方。
|
||||
- **GitHub Issues** —— 报告 bug 或提交功能请求。
|
||||
|
||||
👉 加入我们 → [Dograh 社区 Slack](https://join.slack.com/t/dograh-community/shared_invite/zt-3czr47sw5-MSg1J0kJ7IMPOCHF~03auQ)
|
||||
|
||||
## 🙌 参与贡献
|
||||
|
||||
我们欢迎一切贡献!Dograh AI 100% 开源,也会一直保持下去。
|
||||
|
||||
### 入门步骤
|
||||
|
||||
- Fork 本仓库
|
||||
- 创建特性分支(`git checkout -b feature/AmazingFeature`)
|
||||
- 提交你的改动(`git commit -m 'Add some AmazingFeature'`)
|
||||
- 推送到该分支(`git push origin feature/AmazingFeature`)
|
||||
- 提交一个 Pull Request
|
||||
|
||||
## ⭐ Star 历史
|
||||
|
||||
<a href="https://star-history.com/#dograh-hq/dograh&Date">
|
||||
<img src="https://api.star-history.com/svg?repos=dograh-hq/dograh&type=Date" alt="Dograh 的 Star 历史" width="80%">
|
||||
</a>
|
||||
|
||||
## 📄 许可协议
|
||||
|
||||
Dograh AI 基于 [BSD 2-Clause 协议](LICENSE)开源 —— 与构建 Dograh AI 时所采用的项目使用相同的协议,确保兼容性,以及自由使用、修改和分发的权利。
|
||||
|
||||
## 🏢 关于我们
|
||||
|
||||
由 **Dograh**(Zansat Technologies Private Limited)用 ❤️ 打造。
|
||||
创始团队由 YC 校友与连续创业者组成,致力于让语音 AI 始终开放、人人可用。
|
||||
|
||||
<br><br><br>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/dograh-hq/dograh/stargazers">⭐ 给我们一个 Star</a> |
|
||||
<a href="https://app.dograh.com">☁️ 试用云端版本</a> |
|
||||
<a href="https://join.slack.com/t/dograh-community/shared_invite/zt-3czr47sw5-MSg1J0kJ7IMPOCHF~03auQ">💬 加入 Slack</a>
|
||||
</p>
|
||||
|
|
@ -6,35 +6,44 @@ FastAPI backend for the Dograh voice AI platform.
|
|||
|
||||
```
|
||||
api/
|
||||
├── app.py # Application entry point, FastAPI setup
|
||||
├── routes/ # API endpoint handlers
|
||||
├── services/ # Business logic and integrations
|
||||
├── services/ # Domain logic, runtime systems, and extension seams
|
||||
├── db/ # Database models and data access
|
||||
├── schemas/ # Pydantic request/response schemas
|
||||
├── tasks/ # Background jobs (ARQ)
|
||||
├── utils/ # Utility functions
|
||||
├── tasks/ # Background jobs and post-call work
|
||||
├── mcp_server/ # MCP surface exposed by the backend
|
||||
├── utils/ # Shared utilities
|
||||
├── alembic/ # Database migrations
|
||||
├── constants.py # Environment variables and constants
|
||||
└── tests/ # Test suite
|
||||
```
|
||||
|
||||
## Where to Find Things
|
||||
|
||||
| Looking for... | Go to... |
|
||||
| ---------------------- | ------------------------------------------------------------------------ |
|
||||
| API endpoints | `routes/` - each file is a router module, aggregated in `routes/main.py` |
|
||||
| Business logic | `services/` - organized by domain (telephony, workflow, campaign, etc.) |
|
||||
| Database models | `db/models.py` |
|
||||
| Database queries | `db/*_client.py` files (repository pattern) |
|
||||
| Request/response types | `schemas/` |
|
||||
| Background tasks | `tasks/` - uses ARQ for async job processing |
|
||||
| Environment config | `constants.py` |
|
||||
| Looking for... | Go to... |
|
||||
| ---------------------------- | ----------------------------------------------------------------------------- |
|
||||
| API endpoints | `routes/` - domain routers mounted under `/api/v1` |
|
||||
| Workflow graph and node data | `services/workflow/` |
|
||||
| Live pipeline runtime | `services/pipecat/` |
|
||||
| Telephony providers/call flow| `services/telephony/` |
|
||||
| Third-party integrations | `services/integrations/` |
|
||||
| Campaign and other domains | `services/` |
|
||||
| Database access | `db/` |
|
||||
| Request/response types | `schemas/` |
|
||||
| Background jobs | `tasks/` |
|
||||
| MCP backend surface | `mcp_server/` |
|
||||
| Tests | `tests/` |
|
||||
|
||||
## API Structure
|
||||
|
||||
- All routes are mounted at `/api/v1` prefix
|
||||
- Routes are organized by domain (workflow, telephony, campaign, user, etc.)
|
||||
- `routes/main.py` aggregates all routers
|
||||
- Routes are organized by domain under `routes/`
|
||||
- Workflow execution spans `services/workflow/`, `services/pipecat/`, and `tasks/`
|
||||
- Telephony is a full subsystem under `services/telephony/`, with provider-specific packages under `services/telephony/providers/`
|
||||
- Integrations extend through `services/integrations/`; package-specific rules should live in that subtree's own `AGENTS.md`
|
||||
|
||||
## Routes vs Service Layer
|
||||
|
||||
**Keep route handlers thin** — parse/validate the request, resolve auth and `organization_id`, delegate, shape the response. Domain logic (orchestration, business rules, external calls, computation) belongs in `services/`. Before adding logic to a handler, find its home: extend an existing `services/<domain>/` module that owns the concern (see *Where to Find Things*) before adding a focused new module; never a catch-all. Keep DB access in `db/` clients — routes call services, services call DB clients. Litmus test: if `tasks/`, `mcp_server/`, or another route could reuse it, it must live in `services/` to be importable.
|
||||
|
||||
## Database Migrations
|
||||
|
||||
|
|
|
|||
|
|
@ -20,7 +20,7 @@ RUN pip install --user --no-cache-dir -r requirements.txt && \
|
|||
|
||||
# Copy and install pipecat from local submodule
|
||||
COPY pipecat /tmp/pipecat
|
||||
RUN pip install --user --no-cache-dir '/tmp/pipecat[cartesia,deepgram,openai,elevenlabs,groq,google,azure,sarvam,soundfile,silero,webrtc,speechmatics,openrouter,camb]' && \
|
||||
RUN pip install --user --no-cache-dir '/tmp/pipecat[cartesia,deepgram,openai,elevenlabs,groq,google,azure,sarvam,soundfile,silero,webrtc,speechmatics,openrouter,camb,mcp]' && \
|
||||
# Swap opencv-python (pulled by pipecat[webrtc]) for opencv-python-headless
|
||||
# to drop X11/Qt dependencies that otherwise require libxcb etc. in runner.
|
||||
pip uninstall -y opencv-python && \
|
||||
|
|
|
|||
64
api/alembic/versions/0a1b2c3d4e5f_add_mcp_in_toolcategory.py
Normal file
64
api/alembic/versions/0a1b2c3d4e5f_add_mcp_in_toolcategory.py
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
"""add mcp in ToolCategory
|
||||
|
||||
Revision ID: 0a1b2c3d4e5f
|
||||
Revises: 4c1f1e3e8ef2
|
||||
Create Date: 2026-05-16 00:00:00.000000
|
||||
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
from alembic_postgresql_enum import TableReference
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "0a1b2c3d4e5f"
|
||||
down_revision: Union[str, None] = "4c1f1e3e8ef2"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.sync_enum_values(
|
||||
enum_schema="public",
|
||||
enum_name="tool_category",
|
||||
new_values=[
|
||||
"http_api",
|
||||
"end_call",
|
||||
"transfer_call",
|
||||
"calculator",
|
||||
"native",
|
||||
"integration",
|
||||
"mcp",
|
||||
],
|
||||
affected_columns=[
|
||||
TableReference(
|
||||
table_schema="public", table_name="tools", column_name="category"
|
||||
)
|
||||
],
|
||||
enum_values_to_rename=[],
|
||||
)
|
||||
# ### end Alembic commands ###
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.sync_enum_values(
|
||||
enum_schema="public",
|
||||
enum_name="tool_category",
|
||||
new_values=[
|
||||
"http_api",
|
||||
"end_call",
|
||||
"transfer_call",
|
||||
"calculator",
|
||||
"native",
|
||||
"integration",
|
||||
],
|
||||
affected_columns=[
|
||||
TableReference(
|
||||
table_schema="public", table_name="tools", column_name="category"
|
||||
)
|
||||
],
|
||||
enum_values_to_rename=[],
|
||||
)
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
"""rename integrations organisation_id to organization_id
|
||||
|
||||
Revision ID: 19d2a4b6c8ef
|
||||
Revises: 0a1b2c3d4e5f
|
||||
|
||||
Create Date: 2026-05-19 00:00:00.000000
|
||||
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "19d2a4b6c8ef"
|
||||
down_revision: Union[str, None] = "0a1b2c3d4e5f"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.alter_column(
|
||||
"integrations",
|
||||
"organisation_id",
|
||||
new_column_name="organization_id",
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.alter_column(
|
||||
"integrations",
|
||||
"organization_id",
|
||||
new_column_name="organisation_id",
|
||||
)
|
||||
|
|
@ -0,0 +1,64 @@
|
|||
"""add workflow_run_text_sessions
|
||||
|
||||
Revision ID: 2f638891cbb6
|
||||
Revises: 19d2a4b6c8ef
|
||||
Create Date: 2026-05-18 12:58:58.573381
|
||||
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "2f638891cbb6"
|
||||
down_revision: Union[str, None] = "19d2a4b6c8ef"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.create_table(
|
||||
"workflow_run_text_sessions",
|
||||
sa.Column("workflow_run_id", sa.Integer(), nullable=False),
|
||||
sa.Column(
|
||||
"revision", sa.Integer(), server_default=sa.text("0"), nullable=False
|
||||
),
|
||||
sa.Column(
|
||||
"session_data",
|
||||
sa.JSON(),
|
||||
server_default=sa.text("'{}'::json"),
|
||||
nullable=False,
|
||||
),
|
||||
sa.Column(
|
||||
"checkpoint",
|
||||
sa.JSON(),
|
||||
server_default=sa.text("'{}'::json"),
|
||||
nullable=False,
|
||||
),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.ForeignKeyConstraint(
|
||||
["workflow_run_id"], ["workflow_runs.id"], ondelete="CASCADE"
|
||||
),
|
||||
sa.PrimaryKeyConstraint("workflow_run_id"),
|
||||
)
|
||||
op.create_index(
|
||||
"ix_workflow_run_text_sessions_updated_at",
|
||||
"workflow_run_text_sessions",
|
||||
["updated_at"],
|
||||
unique=False,
|
||||
)
|
||||
# ### end Alembic commands ###
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.drop_index(
|
||||
"ix_workflow_run_text_sessions_updated_at",
|
||||
table_name="workflow_run_text_sessions",
|
||||
)
|
||||
op.drop_table("workflow_run_text_sessions")
|
||||
# ### end Alembic commands ###
|
||||
204
api/alembic/versions/4c1f1e3e8ef2_drop_looptalk_tables.py
Normal file
204
api/alembic/versions/4c1f1e3e8ef2_drop_looptalk_tables.py
Normal file
|
|
@ -0,0 +1,204 @@
|
|||
"""drop_looptalk_tables
|
||||
|
||||
Revision ID: 4c1f1e3e8ef2
|
||||
Revises: 6499c608d0f6
|
||||
Create Date: 2026-05-16 14:46:18.296517
|
||||
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "4c1f1e3e8ef2"
|
||||
down_revision: Union[str, None] = "6499c608d0f6"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# Drop child table first so its FK to looptalk_test_sessions is removed before the parent is dropped.
|
||||
op.drop_index(
|
||||
op.f("ix_looptalk_conversations_session_id"),
|
||||
table_name="looptalk_conversations",
|
||||
)
|
||||
op.drop_table("looptalk_conversations")
|
||||
op.drop_index(
|
||||
op.f("ix_looptalk_test_sessions_group_id"), table_name="looptalk_test_sessions"
|
||||
)
|
||||
op.drop_index(
|
||||
op.f("ix_looptalk_test_sessions_load_test_group_id"),
|
||||
table_name="looptalk_test_sessions",
|
||||
)
|
||||
op.drop_index(
|
||||
op.f("ix_looptalk_test_sessions_org_id"), table_name="looptalk_test_sessions"
|
||||
)
|
||||
op.drop_index(
|
||||
op.f("ix_looptalk_test_sessions_status"), table_name="looptalk_test_sessions"
|
||||
)
|
||||
op.drop_table("looptalk_test_sessions")
|
||||
sa.Enum(
|
||||
"pending", "running", "completed", "failed", name="test_session_status"
|
||||
).drop(op.get_bind())
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
sa.Enum(
|
||||
"pending", "running", "completed", "failed", name="test_session_status"
|
||||
).create(op.get_bind())
|
||||
op.create_table(
|
||||
"looptalk_conversations",
|
||||
sa.Column("id", sa.INTEGER(), autoincrement=True, nullable=False),
|
||||
sa.Column("test_session_id", sa.INTEGER(), autoincrement=False, nullable=False),
|
||||
sa.Column("duration_seconds", sa.INTEGER(), autoincrement=False, nullable=True),
|
||||
sa.Column(
|
||||
"actor_recording_url", sa.VARCHAR(), autoincrement=False, nullable=True
|
||||
),
|
||||
sa.Column(
|
||||
"adversary_recording_url", sa.VARCHAR(), autoincrement=False, nullable=True
|
||||
),
|
||||
sa.Column(
|
||||
"combined_recording_url", sa.VARCHAR(), autoincrement=False, nullable=True
|
||||
),
|
||||
sa.Column(
|
||||
"transcript",
|
||||
postgresql.JSON(astext_type=sa.Text()),
|
||||
autoincrement=False,
|
||||
nullable=False,
|
||||
),
|
||||
sa.Column(
|
||||
"metrics",
|
||||
postgresql.JSON(astext_type=sa.Text()),
|
||||
autoincrement=False,
|
||||
nullable=False,
|
||||
),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
postgresql.TIMESTAMP(timezone=True),
|
||||
autoincrement=False,
|
||||
nullable=True,
|
||||
),
|
||||
sa.Column(
|
||||
"ended_at",
|
||||
postgresql.TIMESTAMP(timezone=True),
|
||||
autoincrement=False,
|
||||
nullable=True,
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["test_session_id"],
|
||||
["looptalk_test_sessions.id"],
|
||||
name=op.f("looptalk_conversations_test_session_id_fkey"),
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id", name=op.f("looptalk_conversations_pkey")),
|
||||
)
|
||||
op.create_index(
|
||||
op.f("ix_looptalk_conversations_session_id"),
|
||||
"looptalk_conversations",
|
||||
["test_session_id"],
|
||||
unique=False,
|
||||
)
|
||||
op.create_table(
|
||||
"looptalk_test_sessions",
|
||||
sa.Column("id", sa.INTEGER(), autoincrement=True, nullable=False),
|
||||
sa.Column("organization_id", sa.INTEGER(), autoincrement=False, nullable=False),
|
||||
sa.Column("name", sa.VARCHAR(), autoincrement=False, nullable=False),
|
||||
sa.Column(
|
||||
"status",
|
||||
postgresql.ENUM(
|
||||
"pending",
|
||||
"running",
|
||||
"completed",
|
||||
"failed",
|
||||
name="test_session_status",
|
||||
create_type=False,
|
||||
),
|
||||
autoincrement=False,
|
||||
nullable=False,
|
||||
),
|
||||
sa.Column(
|
||||
"actor_workflow_id", sa.INTEGER(), autoincrement=False, nullable=False
|
||||
),
|
||||
sa.Column(
|
||||
"adversary_workflow_id", sa.INTEGER(), autoincrement=False, nullable=False
|
||||
),
|
||||
sa.Column(
|
||||
"load_test_group_id", sa.VARCHAR(), autoincrement=False, nullable=True
|
||||
),
|
||||
sa.Column("test_index", sa.INTEGER(), autoincrement=False, nullable=True),
|
||||
sa.Column(
|
||||
"config",
|
||||
postgresql.JSON(astext_type=sa.Text()),
|
||||
autoincrement=False,
|
||||
nullable=False,
|
||||
),
|
||||
sa.Column(
|
||||
"results",
|
||||
postgresql.JSON(astext_type=sa.Text()),
|
||||
autoincrement=False,
|
||||
nullable=False,
|
||||
),
|
||||
sa.Column("error", sa.VARCHAR(), autoincrement=False, nullable=True),
|
||||
sa.Column(
|
||||
"created_at",
|
||||
postgresql.TIMESTAMP(timezone=True),
|
||||
autoincrement=False,
|
||||
nullable=True,
|
||||
),
|
||||
sa.Column(
|
||||
"started_at",
|
||||
postgresql.TIMESTAMP(timezone=True),
|
||||
autoincrement=False,
|
||||
nullable=True,
|
||||
),
|
||||
sa.Column(
|
||||
"completed_at",
|
||||
postgresql.TIMESTAMP(timezone=True),
|
||||
autoincrement=False,
|
||||
nullable=True,
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["actor_workflow_id"],
|
||||
["workflows.id"],
|
||||
name=op.f("looptalk_test_sessions_actor_workflow_id_fkey"),
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["adversary_workflow_id"],
|
||||
["workflows.id"],
|
||||
name=op.f("looptalk_test_sessions_adversary_workflow_id_fkey"),
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["organization_id"],
|
||||
["organizations.id"],
|
||||
name=op.f("looptalk_test_sessions_organization_id_fkey"),
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id", name=op.f("looptalk_test_sessions_pkey")),
|
||||
)
|
||||
op.create_index(
|
||||
op.f("ix_looptalk_test_sessions_status"),
|
||||
"looptalk_test_sessions",
|
||||
["status"],
|
||||
unique=False,
|
||||
)
|
||||
op.create_index(
|
||||
op.f("ix_looptalk_test_sessions_org_id"),
|
||||
"looptalk_test_sessions",
|
||||
["organization_id"],
|
||||
unique=False,
|
||||
)
|
||||
op.create_index(
|
||||
op.f("ix_looptalk_test_sessions_load_test_group_id"),
|
||||
"looptalk_test_sessions",
|
||||
["load_test_group_id"],
|
||||
unique=False,
|
||||
)
|
||||
op.create_index(
|
||||
op.f("ix_looptalk_test_sessions_group_id"),
|
||||
"looptalk_test_sessions",
|
||||
["load_test_group_id"],
|
||||
unique=False,
|
||||
)
|
||||
# ### end Alembic commands ###
|
||||
|
|
@ -0,0 +1,61 @@
|
|||
"""add folders and workflow folder_id
|
||||
|
||||
Revision ID: 6bd9f67ec994
|
||||
Revises: 2f638891cbb6
|
||||
Create Date: 2026-05-22 12:52:30.737380
|
||||
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "6bd9f67ec994"
|
||||
down_revision: Union[str, None] = "2f638891cbb6"
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.create_table(
|
||||
"folders",
|
||||
sa.Column("id", sa.Integer(), nullable=False),
|
||||
sa.Column("organization_id", sa.Integer(), nullable=False),
|
||||
sa.Column("name", sa.String(), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.ForeignKeyConstraint(
|
||||
["organization_id"],
|
||||
["organizations.id"],
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("organization_id", "name", name="uq_folder_org_name"),
|
||||
)
|
||||
op.create_index(
|
||||
op.f("ix_folders_organization_id"), "folders", ["organization_id"], unique=False
|
||||
)
|
||||
op.add_column("workflows", sa.Column("folder_id", sa.Integer(), nullable=True))
|
||||
op.create_index(
|
||||
op.f("ix_workflows_folder_id"), "workflows", ["folder_id"], unique=False
|
||||
)
|
||||
op.create_foreign_key(
|
||||
"fk_workflows_folder_id",
|
||||
"workflows",
|
||||
"folders",
|
||||
["folder_id"],
|
||||
["id"],
|
||||
ondelete="SET NULL",
|
||||
)
|
||||
# ### end Alembic commands ###
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.drop_constraint("fk_workflows_folder_id", "workflows", type_="foreignkey")
|
||||
op.drop_index(op.f("ix_workflows_folder_id"), table_name="workflows")
|
||||
op.drop_column("workflows", "folder_id")
|
||||
op.drop_index(op.f("ix_folders_organization_id"), table_name="folders")
|
||||
op.drop_table("folders")
|
||||
# ### end Alembic commands ###
|
||||
|
|
@ -143,11 +143,4 @@ FORCE_TURN_RELAY = os.getenv("FORCE_TURN_RELAY", "false").lower() == "true"
|
|||
OSS_JWT_SECRET = os.getenv("OSS_JWT_SECRET", "change-me-in-production")
|
||||
OSS_JWT_EXPIRY_HOURS = int(os.getenv("OSS_JWT_EXPIRY_HOURS", "720")) # 30 days
|
||||
|
||||
# REMOVE-AFTER 2026-05-15: transitional flag. When True, Telnyx webhook
|
||||
# signature verification is skipped for configs that have no
|
||||
# webhook_public_key set (existing configs predating the field). Set in prod
|
||||
# through 2026-05-15 to give users time to add their key; once removed,
|
||||
# configs without a key will fail signature verification.
|
||||
TELNYX_WEBHOOK_VERIFICATION_OPTIONAL = (
|
||||
os.getenv("TELNYX_WEBHOOK_VERIFICATION_OPTIONAL", "false").lower() == "true"
|
||||
)
|
||||
TUNER_BASE_URL = os.getenv("TUNER_BASE_URL", "https://api.usetuner.ai")
|
||||
|
|
|
|||
|
|
@ -2,9 +2,9 @@ from api.db.agent_trigger_client import AgentTriggerClient
|
|||
from api.db.api_key_client import APIKeyClient
|
||||
from api.db.campaign_client import CampaignClient
|
||||
from api.db.embed_token_client import EmbedTokenClient
|
||||
from api.db.folder_client import FolderClient
|
||||
from api.db.integration_client import IntegrationClient
|
||||
from api.db.knowledge_base_client import KnowledgeBaseClient
|
||||
from api.db.looptalk_client import LoopTalkClient
|
||||
from api.db.organization_client import OrganizationClient
|
||||
from api.db.organization_configuration_client import OrganizationConfigurationClient
|
||||
from api.db.organization_usage_client import OrganizationUsageClient
|
||||
|
|
@ -17,19 +17,20 @@ from api.db.webhook_credential_client import WebhookCredentialClient
|
|||
from api.db.workflow_client import WorkflowClient
|
||||
from api.db.workflow_recording_client import WorkflowRecordingClient
|
||||
from api.db.workflow_run_client import WorkflowRunClient
|
||||
from api.db.workflow_run_text_session_client import WorkflowRunTextSessionClient
|
||||
from api.db.workflow_template_client import WorkflowTemplateClient
|
||||
|
||||
|
||||
class DBClient(
|
||||
WorkflowClient,
|
||||
WorkflowRunClient,
|
||||
WorkflowRunTextSessionClient,
|
||||
UserClient,
|
||||
OrganizationClient,
|
||||
OrganizationConfigurationClient,
|
||||
OrganizationUsageClient,
|
||||
IntegrationClient,
|
||||
WorkflowTemplateClient,
|
||||
LoopTalkClient,
|
||||
CampaignClient,
|
||||
ReportsClient,
|
||||
APIKeyClient,
|
||||
|
|
@ -41,6 +42,7 @@ class DBClient(
|
|||
WorkflowRecordingClient,
|
||||
TelephonyConfigurationClient,
|
||||
TelephonyPhoneNumberClient,
|
||||
FolderClient,
|
||||
):
|
||||
"""
|
||||
Unified database client that combines all specialized database operations.
|
||||
|
|
@ -54,7 +56,6 @@ class DBClient(
|
|||
- OrganizationUsageClient: handles organization usage and quota operations
|
||||
- IntegrationClient: handles integration operations
|
||||
- WorkflowTemplateClient: handles workflow template operations
|
||||
- LoopTalkClient: handles LoopTalk testing operations
|
||||
- CampaignClient: handles campaign operations
|
||||
- ReportsClient: handles reports and analytics operations
|
||||
- APIKeyClient: handles API key operations
|
||||
|
|
@ -63,6 +64,7 @@ class DBClient(
|
|||
- WebhookCredentialClient: handles webhook credential operations
|
||||
- ToolClient: handles tool operations for reusable HTTP API tools
|
||||
- KnowledgeBaseClient: handles knowledge base document and vector search operations
|
||||
- FolderClient: handles folder operations for grouping workflows (agents)
|
||||
"""
|
||||
|
||||
pass
|
||||
|
|
|
|||
115
api/db/folder_client.py
Normal file
115
api/db/folder_client.py
Normal file
|
|
@ -0,0 +1,115 @@
|
|||
from sqlalchemy import func
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.future import select
|
||||
|
||||
from api.db.base_client import BaseDBClient
|
||||
from api.db.models import FolderModel, WorkflowModel
|
||||
from api.enums import WorkflowStatus
|
||||
|
||||
|
||||
class FolderNameConflictError(Exception):
|
||||
"""Raised when a folder name already exists within the organization."""
|
||||
|
||||
|
||||
class FolderClient(BaseDBClient):
|
||||
async def create_folder(self, name: str, organization_id: int) -> FolderModel:
|
||||
async with self.async_session() as session:
|
||||
folder = FolderModel(name=name, organization_id=organization_id)
|
||||
session.add(folder)
|
||||
try:
|
||||
await session.commit()
|
||||
except IntegrityError:
|
||||
await session.rollback()
|
||||
raise FolderNameConflictError(
|
||||
f"A folder named '{name}' already exists."
|
||||
)
|
||||
await session.refresh(folder)
|
||||
return folder
|
||||
|
||||
async def get_folder(
|
||||
self, folder_id: int, organization_id: int
|
||||
) -> FolderModel | None:
|
||||
"""Fetch a single folder scoped to the organization (tenant isolation)."""
|
||||
async with self.async_session() as session:
|
||||
result = await session.execute(
|
||||
select(FolderModel).where(
|
||||
FolderModel.id == folder_id,
|
||||
FolderModel.organization_id == organization_id,
|
||||
)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
async def list_folders(self, organization_id: int) -> list[FolderModel]:
|
||||
async with self.async_session() as session:
|
||||
result = await session.execute(
|
||||
select(FolderModel)
|
||||
.where(FolderModel.organization_id == organization_id)
|
||||
.order_by(FolderModel.name.asc())
|
||||
)
|
||||
return result.scalars().all()
|
||||
|
||||
async def rename_folder(
|
||||
self, folder_id: int, name: str, organization_id: int
|
||||
) -> FolderModel:
|
||||
async with self.async_session() as session:
|
||||
result = await session.execute(
|
||||
select(FolderModel).where(
|
||||
FolderModel.id == folder_id,
|
||||
FolderModel.organization_id == organization_id,
|
||||
)
|
||||
)
|
||||
folder = result.scalar_one_or_none()
|
||||
if folder is None:
|
||||
raise ValueError(f"Folder with id {folder_id} not found")
|
||||
|
||||
folder.name = name
|
||||
try:
|
||||
await session.commit()
|
||||
except IntegrityError:
|
||||
await session.rollback()
|
||||
raise FolderNameConflictError(
|
||||
f"A folder named '{name}' already exists."
|
||||
)
|
||||
await session.refresh(folder)
|
||||
return folder
|
||||
|
||||
async def delete_folder(self, folder_id: int, organization_id: int) -> bool:
|
||||
"""Delete a folder. Member workflows are un-filed (folder_id -> NULL)
|
||||
via the ON DELETE SET NULL foreign key, never deleted.
|
||||
"""
|
||||
async with self.async_session() as session:
|
||||
result = await session.execute(
|
||||
select(FolderModel).where(
|
||||
FolderModel.id == folder_id,
|
||||
FolderModel.organization_id == organization_id,
|
||||
)
|
||||
)
|
||||
folder = result.scalar_one_or_none()
|
||||
if folder is None:
|
||||
return False
|
||||
|
||||
await session.delete(folder)
|
||||
await session.commit()
|
||||
return True
|
||||
|
||||
async def get_active_workflow_counts_by_folder(
|
||||
self, organization_id: int
|
||||
) -> dict[int, int]:
|
||||
"""Return {folder_id: active_workflow_count} for the organization.
|
||||
|
||||
Only counts active (non-archived) workflows with a non-NULL folder_id.
|
||||
"""
|
||||
async with self.async_session() as session:
|
||||
result = await session.execute(
|
||||
select(
|
||||
WorkflowModel.folder_id,
|
||||
func.count(WorkflowModel.id).label("count"),
|
||||
)
|
||||
.where(
|
||||
WorkflowModel.organization_id == organization_id,
|
||||
WorkflowModel.folder_id.is_not(None),
|
||||
WorkflowModel.status == WorkflowStatus.ACTIVE.value,
|
||||
)
|
||||
.group_by(WorkflowModel.folder_id)
|
||||
)
|
||||
return {folder_id: count for folder_id, count in result.all()}
|
||||
|
|
@ -14,7 +14,7 @@ class IntegrationClient(BaseDBClient):
|
|||
async with self.async_session() as session:
|
||||
result = await session.execute(
|
||||
select(IntegrationModel).where(
|
||||
IntegrationModel.organisation_id == organization_id
|
||||
IntegrationModel.organization_id == organization_id
|
||||
)
|
||||
)
|
||||
return result.scalars().all()
|
||||
|
|
@ -23,7 +23,7 @@ class IntegrationClient(BaseDBClient):
|
|||
self,
|
||||
integration_id: str,
|
||||
provider: str,
|
||||
organisation_id: int,
|
||||
organization_id: int,
|
||||
connection_details: dict,
|
||||
created_by: int = None,
|
||||
is_active: bool = True,
|
||||
|
|
@ -32,7 +32,7 @@ class IntegrationClient(BaseDBClient):
|
|||
async with self.async_session() as session:
|
||||
new_integration = IntegrationModel(
|
||||
integration_id=integration_id,
|
||||
organisation_id=organisation_id,
|
||||
organization_id=organization_id,
|
||||
created_by=created_by,
|
||||
is_active=is_active,
|
||||
provider=provider,
|
||||
|
|
@ -96,7 +96,7 @@ class IntegrationClient(BaseDBClient):
|
|||
async with self.async_session() as session:
|
||||
result = await session.execute(
|
||||
select(IntegrationModel).where(
|
||||
IntegrationModel.organisation_id == organization_id,
|
||||
IntegrationModel.organization_id == organization_id,
|
||||
IntegrationModel.is_active == True,
|
||||
)
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,297 +0,0 @@
|
|||
from datetime import UTC, datetime
|
||||
from typing import Any, Dict, List, Optional
|
||||
from uuid import uuid4
|
||||
|
||||
from sqlalchemy.future import select
|
||||
from sqlalchemy.orm import selectinload
|
||||
|
||||
from api.db.base_client import BaseDBClient
|
||||
from api.db.models import (
|
||||
LoopTalkConversation,
|
||||
LoopTalkTestSession,
|
||||
WorkflowModel,
|
||||
)
|
||||
|
||||
|
||||
class LoopTalkClient(BaseDBClient):
|
||||
"""Database client for LoopTalk testing operations."""
|
||||
|
||||
async def create_test_session(
|
||||
self,
|
||||
organization_id: int,
|
||||
name: str,
|
||||
actor_workflow_id: int,
|
||||
adversary_workflow_id: int,
|
||||
config: Dict[str, Any],
|
||||
load_test_group_id: Optional[str] = None,
|
||||
test_index: Optional[int] = None,
|
||||
) -> LoopTalkTestSession:
|
||||
"""Create a new LoopTalk test session."""
|
||||
async with self.async_session() as session:
|
||||
test_session = LoopTalkTestSession(
|
||||
organization_id=organization_id,
|
||||
name=name,
|
||||
actor_workflow_id=actor_workflow_id,
|
||||
adversary_workflow_id=adversary_workflow_id,
|
||||
config=config,
|
||||
load_test_group_id=load_test_group_id,
|
||||
test_index=test_index,
|
||||
status="pending",
|
||||
)
|
||||
session.add(test_session)
|
||||
await session.commit()
|
||||
await session.refresh(test_session)
|
||||
return test_session
|
||||
|
||||
async def get_test_session(
|
||||
self, test_session_id: int, organization_id: int
|
||||
) -> Optional[LoopTalkTestSession]:
|
||||
"""Get a test session by ID."""
|
||||
async with self.async_session() as session:
|
||||
result = await session.execute(
|
||||
select(LoopTalkTestSession)
|
||||
.options(
|
||||
selectinload(LoopTalkTestSession.actor_workflow).selectinload(
|
||||
WorkflowModel.released_definition
|
||||
),
|
||||
selectinload(LoopTalkTestSession.adversary_workflow).selectinload(
|
||||
WorkflowModel.released_definition
|
||||
),
|
||||
selectinload(LoopTalkTestSession.conversations),
|
||||
)
|
||||
.where(
|
||||
LoopTalkTestSession.id == test_session_id,
|
||||
LoopTalkTestSession.organization_id == organization_id,
|
||||
)
|
||||
)
|
||||
return result.scalar_one_or_none()
|
||||
|
||||
async def list_test_sessions(
|
||||
self,
|
||||
organization_id: int,
|
||||
status: Optional[str] = None,
|
||||
load_test_group_id: Optional[str] = None,
|
||||
limit: int = 20,
|
||||
offset: int = 0,
|
||||
) -> List[LoopTalkTestSession]:
|
||||
"""List test sessions with optional filtering."""
|
||||
async with self.async_session() as session:
|
||||
query = select(LoopTalkTestSession).where(
|
||||
LoopTalkTestSession.organization_id == organization_id
|
||||
)
|
||||
|
||||
if status:
|
||||
# "active" is a virtual status used by the UI to represent
|
||||
# both "pending" and "running" sessions. Translate it into
|
||||
# the real enum values stored in the database to avoid
|
||||
# invalid enum casting errors (e.g. asyncpg InvalidTextRepresentationError).
|
||||
if status == "active":
|
||||
query = query.where(
|
||||
LoopTalkTestSession.status.in_(["pending", "running"])
|
||||
)
|
||||
else:
|
||||
query = query.where(LoopTalkTestSession.status == status)
|
||||
|
||||
if load_test_group_id:
|
||||
query = query.where(
|
||||
LoopTalkTestSession.load_test_group_id == load_test_group_id
|
||||
)
|
||||
|
||||
query = (
|
||||
query.order_by(LoopTalkTestSession.created_at.desc())
|
||||
.limit(limit)
|
||||
.offset(offset)
|
||||
)
|
||||
|
||||
result = await session.execute(query)
|
||||
return result.scalars().all()
|
||||
|
||||
async def update_test_session_status(
|
||||
self,
|
||||
test_session_id: int,
|
||||
status: str,
|
||||
error: Optional[str] = None,
|
||||
results: Optional[Dict[str, Any]] = None,
|
||||
) -> LoopTalkTestSession:
|
||||
"""Update test session status and related fields."""
|
||||
async with self.async_session() as session:
|
||||
result = await session.execute(
|
||||
select(LoopTalkTestSession).where(
|
||||
LoopTalkTestSession.id == test_session_id
|
||||
)
|
||||
)
|
||||
test_session = result.scalar_one()
|
||||
|
||||
test_session.status = status
|
||||
|
||||
if status == "running":
|
||||
test_session.started_at = datetime.now(UTC)
|
||||
elif status in ["completed", "failed"]:
|
||||
test_session.completed_at = datetime.now(UTC)
|
||||
|
||||
if error:
|
||||
test_session.error = error
|
||||
|
||||
if results:
|
||||
test_session.results = results
|
||||
|
||||
await session.commit()
|
||||
await session.refresh(test_session)
|
||||
return test_session
|
||||
|
||||
async def create_conversation(self, test_session_id: int) -> LoopTalkConversation:
|
||||
"""Create a new conversation for a test session."""
|
||||
async with self.async_session() as session:
|
||||
conversation = LoopTalkConversation(test_session_id=test_session_id)
|
||||
session.add(conversation)
|
||||
await session.commit()
|
||||
await session.refresh(conversation)
|
||||
return conversation
|
||||
|
||||
async def update_conversation(
|
||||
self,
|
||||
conversation_id: int,
|
||||
duration_seconds: Optional[int] = None,
|
||||
actor_recording_url: Optional[str] = None,
|
||||
adversary_recording_url: Optional[str] = None,
|
||||
combined_recording_url: Optional[str] = None,
|
||||
transcript: Optional[Dict[str, Any]] = None,
|
||||
metrics: Optional[Dict[str, Any]] = None,
|
||||
ended_at: Optional[datetime] = None,
|
||||
) -> LoopTalkConversation:
|
||||
"""Update conversation details."""
|
||||
async with self.async_session() as session:
|
||||
result = await session.execute(
|
||||
select(LoopTalkConversation).where(
|
||||
LoopTalkConversation.id == conversation_id
|
||||
)
|
||||
)
|
||||
conversation = result.scalar_one()
|
||||
|
||||
if duration_seconds is not None:
|
||||
conversation.duration_seconds = duration_seconds
|
||||
if actor_recording_url:
|
||||
conversation.actor_recording_url = actor_recording_url
|
||||
if adversary_recording_url:
|
||||
conversation.adversary_recording_url = adversary_recording_url
|
||||
if combined_recording_url:
|
||||
conversation.combined_recording_url = combined_recording_url
|
||||
if transcript:
|
||||
conversation.transcript = transcript
|
||||
if metrics:
|
||||
conversation.metrics = metrics
|
||||
if ended_at:
|
||||
conversation.ended_at = ended_at
|
||||
|
||||
await session.commit()
|
||||
await session.refresh(conversation)
|
||||
return conversation
|
||||
|
||||
# Note: Turn tracking is handled by Langfuse, not stored in our database
|
||||
|
||||
async def create_load_test_group(
|
||||
self,
|
||||
organization_id: int,
|
||||
name_prefix: str,
|
||||
actor_workflow_id: int,
|
||||
adversary_workflow_id: int,
|
||||
config: Dict[str, Any],
|
||||
test_count: int,
|
||||
) -> List[LoopTalkTestSession]:
|
||||
"""Create multiple test sessions for load testing."""
|
||||
load_test_group_id = str(uuid4())
|
||||
test_sessions = []
|
||||
|
||||
async with self.async_session() as session:
|
||||
for i in range(test_count):
|
||||
test_session = LoopTalkTestSession(
|
||||
organization_id=organization_id,
|
||||
name=f"{name_prefix} - Test {i + 1}",
|
||||
actor_workflow_id=actor_workflow_id,
|
||||
adversary_workflow_id=adversary_workflow_id,
|
||||
config=config,
|
||||
load_test_group_id=load_test_group_id,
|
||||
test_index=i,
|
||||
status="pending",
|
||||
)
|
||||
session.add(test_session)
|
||||
test_sessions.append(test_session)
|
||||
|
||||
await session.commit()
|
||||
|
||||
# Refresh all sessions
|
||||
for test_session in test_sessions:
|
||||
await session.refresh(test_session)
|
||||
|
||||
return test_sessions
|
||||
|
||||
async def get_load_test_group_stats(
|
||||
self, load_test_group_id: str, organization_id: int
|
||||
) -> Dict[str, Any]:
|
||||
"""Get statistics for a load test group."""
|
||||
from sqlalchemy import case, func
|
||||
|
||||
async with self.async_session() as session:
|
||||
# Get status counts using SQL aggregation
|
||||
counts_result = await session.execute(
|
||||
select(
|
||||
func.count().label("total"),
|
||||
func.sum(
|
||||
case((LoopTalkTestSession.status == "pending", 1), else_=0)
|
||||
).label("pending"),
|
||||
func.sum(
|
||||
case((LoopTalkTestSession.status == "running", 1), else_=0)
|
||||
).label("running"),
|
||||
func.sum(
|
||||
case((LoopTalkTestSession.status == "completed", 1), else_=0)
|
||||
).label("completed"),
|
||||
func.sum(
|
||||
case((LoopTalkTestSession.status == "failed", 1), else_=0)
|
||||
).label("failed"),
|
||||
).where(
|
||||
LoopTalkTestSession.load_test_group_id == load_test_group_id,
|
||||
LoopTalkTestSession.organization_id == organization_id,
|
||||
)
|
||||
)
|
||||
counts = counts_result.one()
|
||||
|
||||
# Get session details (still needed for the sessions list)
|
||||
sessions_result = await session.execute(
|
||||
select(
|
||||
LoopTalkTestSession.id,
|
||||
LoopTalkTestSession.name,
|
||||
LoopTalkTestSession.status,
|
||||
LoopTalkTestSession.test_index,
|
||||
LoopTalkTestSession.created_at,
|
||||
LoopTalkTestSession.started_at,
|
||||
LoopTalkTestSession.completed_at,
|
||||
LoopTalkTestSession.error,
|
||||
).where(
|
||||
LoopTalkTestSession.load_test_group_id == load_test_group_id,
|
||||
LoopTalkTestSession.organization_id == organization_id,
|
||||
)
|
||||
)
|
||||
sessions = sessions_result.all()
|
||||
|
||||
stats = {
|
||||
"total": counts.total or 0,
|
||||
"pending": counts.pending or 0,
|
||||
"running": counts.running or 0,
|
||||
"completed": counts.completed or 0,
|
||||
"failed": counts.failed or 0,
|
||||
"sessions": [
|
||||
{
|
||||
"id": s.id,
|
||||
"name": s.name,
|
||||
"status": s.status,
|
||||
"test_index": s.test_index,
|
||||
"created_at": s.created_at,
|
||||
"started_at": s.started_at,
|
||||
"completed_at": s.completed_at,
|
||||
"error": s.error,
|
||||
}
|
||||
for s in sessions
|
||||
],
|
||||
}
|
||||
|
||||
return stats
|
||||
153
api/db/models.py
153
api/db/models.py
|
|
@ -292,8 +292,10 @@ class IntegrationModel(Base):
|
|||
__tablename__ = "integrations"
|
||||
|
||||
id = Column(Integer, primary_key=True, index=True)
|
||||
integration_id = Column(String, nullable=False, index=True) # Nango Connection ID
|
||||
organisation_id = Column(Integer, ForeignKey("organizations.id"), nullable=False)
|
||||
integration_id = Column(
|
||||
String, nullable=False, index=True
|
||||
) # External connection ID
|
||||
organization_id = Column(Integer, ForeignKey("organizations.id"), nullable=False)
|
||||
provider = Column(String, nullable=False)
|
||||
created_by = Column(Integer, ForeignKey("users.id"))
|
||||
is_active = Column(Boolean, default=True, nullable=False)
|
||||
|
|
@ -350,6 +352,32 @@ class WorkflowDefinitionModel(Base):
|
|||
workflow_runs = relationship("WorkflowRunModel", back_populates="definition")
|
||||
|
||||
|
||||
class FolderModel(Base):
|
||||
"""A folder for grouping workflows (agents) within an organization.
|
||||
|
||||
Folders are flat (no nesting) and org-scoped. A workflow belongs to at
|
||||
most one folder via ``WorkflowModel.folder_id``; a NULL folder_id means
|
||||
the workflow is "Uncategorized".
|
||||
"""
|
||||
|
||||
__tablename__ = "folders"
|
||||
|
||||
id = Column(Integer, primary_key=True, index=True)
|
||||
organization_id = Column(
|
||||
Integer, ForeignKey("organizations.id"), nullable=False, index=True
|
||||
)
|
||||
organization = relationship("OrganizationModel")
|
||||
name = Column(String, nullable=False)
|
||||
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(UTC))
|
||||
|
||||
workflows = relationship("WorkflowModel", back_populates="folder")
|
||||
|
||||
# Folder names must be unique within an organization.
|
||||
__table_args__ = (
|
||||
UniqueConstraint("organization_id", "name", name="uq_folder_org_name"),
|
||||
)
|
||||
|
||||
|
||||
class WorkflowModel(Base):
|
||||
__tablename__ = "workflows"
|
||||
id = Column(Integer, primary_key=True, index=True)
|
||||
|
|
@ -364,6 +392,15 @@ class WorkflowModel(Base):
|
|||
user = relationship("UserModel", back_populates="workflows")
|
||||
organization_id = Column(Integer, ForeignKey("organizations.id"), nullable=True)
|
||||
organization = relationship("OrganizationModel")
|
||||
# Optional folder for grouping in the agents list. NULL = "Uncategorized".
|
||||
# ON DELETE SET NULL: deleting a folder un-files its agents, never deletes them.
|
||||
folder_id = Column(
|
||||
Integer,
|
||||
ForeignKey("folders.id", ondelete="SET NULL"),
|
||||
nullable=True,
|
||||
index=True,
|
||||
)
|
||||
folder = relationship("FolderModel", back_populates="workflows")
|
||||
name = Column(String, index=True, nullable=False)
|
||||
status = Column(
|
||||
Enum(*[status.value for status in WorkflowStatus], name="workflow_status"),
|
||||
|
|
@ -482,6 +519,12 @@ class WorkflowRunModel(Base):
|
|||
queued_run_id = Column(Integer, ForeignKey("queued_runs.id"), nullable=True)
|
||||
queued_run = relationship("QueuedRunModel", foreign_keys=[queued_run_id])
|
||||
public_access_token = Column(String(36), nullable=True)
|
||||
text_session = relationship(
|
||||
"WorkflowRunTextSessionModel",
|
||||
back_populates="workflow_run",
|
||||
uselist=False,
|
||||
cascade="all, delete-orphan",
|
||||
)
|
||||
|
||||
# Indexes
|
||||
__table_args__ = (
|
||||
|
|
@ -501,85 +544,41 @@ class WorkflowRunModel(Base):
|
|||
)
|
||||
|
||||
|
||||
# LoopTalk Testing Models
|
||||
class LoopTalkTestSession(Base):
|
||||
__tablename__ = "looptalk_test_sessions"
|
||||
class WorkflowRunTextSessionModel(Base):
|
||||
__tablename__ = "workflow_run_text_sessions"
|
||||
|
||||
id = Column(Integer, primary_key=True, index=True)
|
||||
organization_id = Column(Integer, ForeignKey("organizations.id"), nullable=False)
|
||||
name = Column(String, nullable=False)
|
||||
status = Column(
|
||||
Enum("pending", "running", "completed", "failed", name="test_session_status"),
|
||||
workflow_run_id = Column(
|
||||
Integer,
|
||||
ForeignKey("workflow_runs.id", ondelete="CASCADE"),
|
||||
primary_key=True,
|
||||
)
|
||||
workflow_run = relationship("WorkflowRunModel", back_populates="text_session")
|
||||
revision = Column(
|
||||
Integer,
|
||||
nullable=False,
|
||||
default="pending",
|
||||
default=0,
|
||||
server_default=text("0"),
|
||||
)
|
||||
session_data = Column(
|
||||
JSON,
|
||||
nullable=False,
|
||||
default=dict,
|
||||
server_default=text("'{}'::json"),
|
||||
)
|
||||
checkpoint = Column(
|
||||
JSON,
|
||||
nullable=False,
|
||||
default=dict,
|
||||
server_default=text("'{}'::json"),
|
||||
)
|
||||
|
||||
# Workflow configuration
|
||||
actor_workflow_id = Column(Integer, ForeignKey("workflows.id"), nullable=False)
|
||||
adversary_workflow_id = Column(Integer, ForeignKey("workflows.id"), nullable=False)
|
||||
|
||||
# Load testing configuration
|
||||
load_test_group_id = Column(String, nullable=True, index=True)
|
||||
test_index = Column(Integer, nullable=True)
|
||||
|
||||
# Test metadata
|
||||
config = Column(JSON, nullable=False, default=dict)
|
||||
results = Column(JSON, nullable=False, default=dict)
|
||||
error = Column(String, nullable=True)
|
||||
|
||||
# Timestamps
|
||||
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(UTC))
|
||||
started_at = Column(DateTime(timezone=True), nullable=True)
|
||||
completed_at = Column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
# Relationships
|
||||
organization = relationship("OrganizationModel")
|
||||
actor_workflow = relationship("WorkflowModel", foreign_keys=[actor_workflow_id])
|
||||
adversary_workflow = relationship(
|
||||
"WorkflowModel", foreign_keys=[adversary_workflow_id]
|
||||
)
|
||||
conversations = relationship("LoopTalkConversation", back_populates="test_session")
|
||||
|
||||
# Indexes for performance
|
||||
__table_args__ = (
|
||||
Index("ix_looptalk_test_sessions_org_id", "organization_id"),
|
||||
Index("ix_looptalk_test_sessions_group_id", "load_test_group_id"),
|
||||
Index("ix_looptalk_test_sessions_status", "status"),
|
||||
updated_at = Column(
|
||||
DateTime(timezone=True),
|
||||
default=lambda: datetime.now(UTC),
|
||||
onupdate=lambda: datetime.now(UTC),
|
||||
)
|
||||
|
||||
|
||||
class LoopTalkConversation(Base):
|
||||
__tablename__ = "looptalk_conversations"
|
||||
|
||||
id = Column(Integer, primary_key=True, index=True)
|
||||
test_session_id = Column(
|
||||
Integer, ForeignKey("looptalk_test_sessions.id"), nullable=False
|
||||
)
|
||||
|
||||
# Conversation metadata
|
||||
duration_seconds = Column(Integer, nullable=True)
|
||||
# Note: Turn tracking is handled by Langfuse, not stored here
|
||||
|
||||
# Audio recording URLs
|
||||
actor_recording_url = Column(String, nullable=True)
|
||||
adversary_recording_url = Column(String, nullable=True)
|
||||
combined_recording_url = Column(String, nullable=True)
|
||||
|
||||
# Transcripts (if needed for quick access)
|
||||
transcript = Column(JSON, nullable=False, default=dict)
|
||||
|
||||
# Metrics
|
||||
metrics = Column(JSON, nullable=False, default=dict)
|
||||
|
||||
# Timestamps
|
||||
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(UTC))
|
||||
ended_at = Column(DateTime(timezone=True), nullable=True)
|
||||
|
||||
# Relationships
|
||||
test_session = relationship("LoopTalkTestSession", back_populates="conversations")
|
||||
|
||||
# Indexes
|
||||
__table_args__ = (Index("ix_looptalk_conversations_session_id", "test_session_id"),)
|
||||
__table_args__ = (Index("ix_workflow_run_text_sessions_updated_at", "updated_at"),)
|
||||
|
||||
|
||||
class OrganizationUsageCycleModel(Base):
|
||||
|
|
@ -636,8 +635,8 @@ class CampaignModel(Base):
|
|||
)
|
||||
|
||||
# Source configuration
|
||||
source_type = Column(String, nullable=False, default="google-sheet")
|
||||
source_id = Column(String, nullable=False) # Sheet URL
|
||||
source_type = Column(String, nullable=False, default="csv")
|
||||
source_id = Column(String, nullable=False) # CSV file key
|
||||
|
||||
# State management
|
||||
state = Column(
|
||||
|
|
|
|||
|
|
@ -151,9 +151,9 @@ class OrganizationUsageClient(BaseDBClient):
|
|||
async def update_usage_after_run(
|
||||
self,
|
||||
organization_id: int,
|
||||
actual_tokens: int,
|
||||
duration_seconds: int = 0,
|
||||
charge_usd: float = None,
|
||||
actual_tokens: float,
|
||||
duration_seconds: float = 0,
|
||||
charge_usd: float | None = None,
|
||||
) -> None:
|
||||
"""Update usage after a workflow run completes with actual token count and duration.
|
||||
|
||||
|
|
@ -354,6 +354,7 @@ class OrganizationUsageClient(BaseDBClient):
|
|||
"caller_number": caller_number,
|
||||
"called_number": called_number,
|
||||
"call_type": run.call_type,
|
||||
"mode": run.mode,
|
||||
"disposition": disposition,
|
||||
"initial_context": run.initial_context,
|
||||
"gathered_context": run.gathered_context,
|
||||
|
|
|
|||
|
|
@ -372,6 +372,8 @@ class WorkflowClient(BaseDBClient):
|
|||
WorkflowModel.name,
|
||||
WorkflowModel.status,
|
||||
WorkflowModel.created_at,
|
||||
WorkflowModel.folder_id,
|
||||
WorkflowModel.workflow_uuid,
|
||||
)
|
||||
)
|
||||
|
||||
|
|
@ -425,8 +427,26 @@ class WorkflowClient(BaseDBClient):
|
|||
return result.scalar_one_or_none()
|
||||
|
||||
async def get_workflow(
|
||||
self, workflow_id: int, user_id: int = None, organization_id: int = None
|
||||
self,
|
||||
workflow_id: int,
|
||||
user_id: int | None = None,
|
||||
organization_id: int | None = None,
|
||||
) -> WorkflowModel | None:
|
||||
"""Fetch a workflow by id, scoped to a tenant.
|
||||
|
||||
Scoping is mandatory: pass ``organization_id`` (preferred) or
|
||||
``user_id``. A fully unscoped lookup would let a request-supplied id
|
||||
reach another tenant's workflow. System/runtime paths that only have a
|
||||
``workflow_id`` and derive the org from the workflow itself (e.g.
|
||||
inbound telephony routing) must call ``get_workflow_by_id`` instead —
|
||||
the explicit unscoped variant.
|
||||
"""
|
||||
if user_id is None and organization_id is None:
|
||||
raise ValueError(
|
||||
"get_workflow requires organization_id (preferred) or user_id "
|
||||
"for tenant scoping; use get_workflow_by_id for unscoped "
|
||||
"system lookups."
|
||||
)
|
||||
async with self.async_session() as session:
|
||||
query = (
|
||||
select(WorkflowModel)
|
||||
|
|
@ -448,6 +468,13 @@ class WorkflowClient(BaseDBClient):
|
|||
return result.scalars().first()
|
||||
|
||||
async def get_workflow_by_id(self, workflow_id: int) -> WorkflowModel | None:
|
||||
"""Fetch a workflow by id WITHOUT tenant scoping.
|
||||
|
||||
Explicit unscoped variant of ``get_workflow``. Only for system/runtime
|
||||
contexts that legitimately have just a workflow_id and derive the org
|
||||
from the workflow itself (e.g. inbound telephony). Never call this with
|
||||
a request-supplied id on a user-facing path.
|
||||
"""
|
||||
async with self.async_session() as session:
|
||||
result = await session.execute(
|
||||
select(WorkflowModel)
|
||||
|
|
@ -609,7 +636,7 @@ class WorkflowClient(BaseDBClient):
|
|||
self,
|
||||
workflow_id: int,
|
||||
status: str,
|
||||
organization_id: int = None,
|
||||
organization_id: int,
|
||||
) -> WorkflowModel:
|
||||
"""
|
||||
Update the status of a workflow.
|
||||
|
|
@ -617,7 +644,9 @@ class WorkflowClient(BaseDBClient):
|
|||
Args:
|
||||
workflow_id: The ID of the workflow to update
|
||||
status: The new status (active/archived)
|
||||
organization_id: The organization ID
|
||||
organization_id: The organization ID. Required and always filtered
|
||||
on: this is a mutation, so an unscoped query would let a caller
|
||||
archive another org's workflow (tenant-isolation bypass).
|
||||
|
||||
Returns:
|
||||
The updated WorkflowModel
|
||||
|
|
@ -632,12 +661,12 @@ class WorkflowClient(BaseDBClient):
|
|||
selectinload(WorkflowModel.current_definition),
|
||||
selectinload(WorkflowModel.released_definition),
|
||||
)
|
||||
.where(WorkflowModel.id == workflow_id)
|
||||
.where(
|
||||
WorkflowModel.id == workflow_id,
|
||||
WorkflowModel.organization_id == organization_id,
|
||||
)
|
||||
)
|
||||
|
||||
if organization_id:
|
||||
query = query.where(WorkflowModel.organization_id == organization_id)
|
||||
|
||||
result = await session.execute(query)
|
||||
workflow = result.scalars().first()
|
||||
|
||||
|
|
@ -654,6 +683,47 @@ class WorkflowClient(BaseDBClient):
|
|||
await session.refresh(workflow)
|
||||
return workflow
|
||||
|
||||
async def move_workflow_to_folder(
|
||||
self,
|
||||
workflow_id: int,
|
||||
folder_id: int | None,
|
||||
organization_id: int,
|
||||
) -> WorkflowModel:
|
||||
"""Set (or clear) a workflow's folder.
|
||||
|
||||
Pass ``folder_id=None`` to move the workflow to "Uncategorized". The
|
||||
caller must validate that ``folder_id`` belongs to ``organization_id``
|
||||
before calling (the FK only proves the folder exists, not ownership).
|
||||
|
||||
``organization_id`` is required and always filtered on: this is a
|
||||
mutation, so an unscoped query would let a caller move another org's
|
||||
workflow (tenant-isolation bypass).
|
||||
|
||||
Raises:
|
||||
ValueError: If the workflow is not found within the organization.
|
||||
"""
|
||||
async with self.async_session() as session:
|
||||
query = select(WorkflowModel).where(
|
||||
WorkflowModel.id == workflow_id,
|
||||
WorkflowModel.organization_id == organization_id,
|
||||
)
|
||||
|
||||
result = await session.execute(query)
|
||||
workflow = result.scalars().first()
|
||||
|
||||
if not workflow:
|
||||
raise ValueError(f"Workflow with ID {workflow_id} not found")
|
||||
|
||||
workflow.folder_id = folder_id
|
||||
|
||||
try:
|
||||
await session.commit()
|
||||
except Exception as e:
|
||||
await session.rollback()
|
||||
raise e
|
||||
await session.refresh(workflow)
|
||||
return workflow
|
||||
|
||||
async def get_workflow_run_count(self, workflow_id: int) -> int:
|
||||
"""Get the count of runs for a workflow."""
|
||||
async with self.async_session() as session:
|
||||
|
|
|
|||
|
|
@ -32,16 +32,22 @@ class WorkflowRunClient(BaseDBClient):
|
|||
campaign_id: int = None,
|
||||
queued_run_id: int = None,
|
||||
use_draft: bool = False,
|
||||
organization_id: int | None = None,
|
||||
) -> WorkflowRunModel:
|
||||
async with self.async_session() as session:
|
||||
# Get workflow and user to check organization
|
||||
workflow = await session.execute(
|
||||
workflow_query = (
|
||||
select(WorkflowModel)
|
||||
.options(joinedload(WorkflowModel.user))
|
||||
.where(
|
||||
WorkflowModel.id == workflow_id, WorkflowModel.user_id == user_id
|
||||
)
|
||||
)
|
||||
if organization_id is not None:
|
||||
workflow_query = workflow_query.where(
|
||||
WorkflowModel.organization_id == organization_id
|
||||
)
|
||||
|
||||
workflow = await session.execute(workflow_query)
|
||||
workflow = workflow.scalars().first()
|
||||
if not workflow:
|
||||
raise ValueError(f"Workflow with ID {workflow_id} not found")
|
||||
|
|
|
|||
124
api/db/workflow_run_text_session_client.py
Normal file
124
api/db/workflow_run_text_session_client.py
Normal file
|
|
@ -0,0 +1,124 @@
|
|||
from sqlalchemy.future import select
|
||||
from sqlalchemy.orm import joinedload
|
||||
|
||||
from api.db.base_client import BaseDBClient
|
||||
from api.db.models import (
|
||||
WorkflowModel,
|
||||
WorkflowRunModel,
|
||||
WorkflowRunTextSessionModel,
|
||||
)
|
||||
|
||||
|
||||
class WorkflowRunTextSessionRevisionConflictError(Exception):
|
||||
def __init__(self, expected_revision: int, actual_revision: int):
|
||||
self.expected_revision = expected_revision
|
||||
self.actual_revision = actual_revision
|
||||
super().__init__(
|
||||
"Workflow run text session revision conflict: "
|
||||
f"expected {expected_revision}, found {actual_revision}"
|
||||
)
|
||||
|
||||
|
||||
class WorkflowRunTextSessionClient(BaseDBClient):
|
||||
async def ensure_workflow_run_text_session(
|
||||
self,
|
||||
workflow_run_id: int,
|
||||
session_data: dict | None = None,
|
||||
checkpoint: dict | None = None,
|
||||
) -> WorkflowRunTextSessionModel:
|
||||
async with self.async_session() as session:
|
||||
result = await session.execute(
|
||||
select(WorkflowRunTextSessionModel)
|
||||
.where(WorkflowRunTextSessionModel.workflow_run_id == workflow_run_id)
|
||||
.with_for_update()
|
||||
)
|
||||
text_session = result.scalars().first()
|
||||
if text_session:
|
||||
return text_session
|
||||
|
||||
run_result = await session.execute(
|
||||
select(WorkflowRunModel).where(WorkflowRunModel.id == workflow_run_id)
|
||||
)
|
||||
workflow_run = run_result.scalars().first()
|
||||
if not workflow_run:
|
||||
raise ValueError(f"Workflow run with ID {workflow_run_id} not found")
|
||||
|
||||
text_session = WorkflowRunTextSessionModel(
|
||||
workflow_run_id=workflow_run_id,
|
||||
session_data=session_data or {},
|
||||
checkpoint=checkpoint or {},
|
||||
)
|
||||
session.add(text_session)
|
||||
try:
|
||||
await session.commit()
|
||||
except Exception as e:
|
||||
await session.rollback()
|
||||
raise e
|
||||
await session.refresh(text_session)
|
||||
return text_session
|
||||
|
||||
async def get_workflow_run_text_session(
|
||||
self,
|
||||
workflow_run_id: int,
|
||||
*,
|
||||
organization_id: int,
|
||||
) -> WorkflowRunTextSessionModel | None:
|
||||
async with self.async_session() as session:
|
||||
query = (
|
||||
select(WorkflowRunTextSessionModel)
|
||||
.options(
|
||||
joinedload(WorkflowRunTextSessionModel.workflow_run).joinedload(
|
||||
WorkflowRunModel.workflow
|
||||
)
|
||||
)
|
||||
.join(WorkflowRunTextSessionModel.workflow_run)
|
||||
.join(WorkflowRunModel.workflow)
|
||||
.where(WorkflowRunTextSessionModel.workflow_run_id == workflow_run_id)
|
||||
.where(WorkflowModel.organization_id == organization_id)
|
||||
)
|
||||
|
||||
result = await session.execute(query)
|
||||
return result.scalars().first()
|
||||
|
||||
async def update_workflow_run_text_session(
|
||||
self,
|
||||
workflow_run_id: int,
|
||||
*,
|
||||
session_data: dict | None = None,
|
||||
checkpoint: dict | None = None,
|
||||
expected_revision: int | None = None,
|
||||
) -> WorkflowRunTextSessionModel:
|
||||
async with self.async_session() as session:
|
||||
result = await session.execute(
|
||||
select(WorkflowRunTextSessionModel)
|
||||
.where(WorkflowRunTextSessionModel.workflow_run_id == workflow_run_id)
|
||||
.with_for_update()
|
||||
)
|
||||
text_session = result.scalars().first()
|
||||
if not text_session:
|
||||
raise ValueError(
|
||||
f"Workflow run text session with run ID {workflow_run_id} not found"
|
||||
)
|
||||
|
||||
if (
|
||||
expected_revision is not None
|
||||
and text_session.revision != expected_revision
|
||||
):
|
||||
raise WorkflowRunTextSessionRevisionConflictError(
|
||||
expected_revision=expected_revision,
|
||||
actual_revision=text_session.revision,
|
||||
)
|
||||
|
||||
if session_data is not None:
|
||||
text_session.session_data = session_data
|
||||
if checkpoint is not None:
|
||||
text_session.checkpoint = checkpoint
|
||||
text_session.revision += 1
|
||||
|
||||
try:
|
||||
await session.commit()
|
||||
except Exception as e:
|
||||
await session.rollback()
|
||||
raise e
|
||||
await session.refresh(text_session)
|
||||
return text_session
|
||||
|
|
@ -27,6 +27,7 @@ class WorkflowRunMode(Enum):
|
|||
TELNYX = "telnyx"
|
||||
WEBRTC = "webrtc"
|
||||
SMALLWEBRTC = "smallwebrtc"
|
||||
TEXTCHAT = "textchat"
|
||||
|
||||
# Historical, not used anymore. Don't
|
||||
# use and don't remove
|
||||
|
|
@ -133,6 +134,7 @@ class ToolCategory(Enum):
|
|||
CALCULATOR = "calculator" # Built-in calculator tool
|
||||
NATIVE = "native" # Built-in integrations (future: dtmf_input)
|
||||
INTEGRATION = "integration" # Third-party integrations (future: Google Calendar, Salesforce, etc.)
|
||||
MCP = "mcp" # Customer-provided MCP server exposing a tool catalog
|
||||
|
||||
|
||||
class ToolStatus(Enum):
|
||||
|
|
|
|||
|
|
@ -18,7 +18,9 @@ async def authenticate_mcp_request() -> UserModel:
|
|||
the `langfuse.user.id` / `langfuse.session.id` attributes make the
|
||||
span filterable in the Langfuse UI.
|
||||
"""
|
||||
headers = get_http_headers()
|
||||
# FastMCP strips Authorization by default unless explicitly included.
|
||||
# Preserve it here so Bearer API keys work for MCP tool invocations.
|
||||
headers = get_http_headers(include={"authorization"})
|
||||
api_key = headers.get("x-api-key")
|
||||
if not api_key:
|
||||
auth = headers.get("authorization", "")
|
||||
|
|
|
|||
|
|
@ -7,6 +7,14 @@ handling, hard constraints). Design-level per-field guidance belongs in
|
|||
each `PropertySpec.llm_hint`; it flows out through `get_node_type` and
|
||||
doesn't need to be repeated here.
|
||||
|
||||
Tool names, parameters, and per-tool `error_code` values are NOT
|
||||
authoritative here — they reach the model dynamically via `tools/list`
|
||||
from each tool's own signature and docstring. Reference tools by bare
|
||||
name and describe orchestration; do not restate signatures (they drift)
|
||||
or re-enumerate error codes (document those on the tool itself).
|
||||
`test_mcp_instructions_drift.py` fails if this guide names a tool that
|
||||
is not registered, or if a tool's error codes aren't in its docstring.
|
||||
|
||||
Extend based on real LLM failures — every bullet below ideally maps to a
|
||||
mistake the system has seen at least once.
|
||||
"""
|
||||
|
|
@ -16,18 +24,23 @@ You build and edit Dograh voice-AI workflows by emitting TypeScript that uses th
|
|||
|
||||
## Call order
|
||||
|
||||
### Reading documentation
|
||||
1. `search_docs` — use first for keyword or acronym lookup when the user is asking how Dograh works or how to configure something.
|
||||
2. `read_doc` — fetch the full page once one result looks likely. Prefer this over reasoning from search summaries alone.
|
||||
3. `list_docs` — use when the user wants to browse a topic area or when search terms are too vague. Call it with no arguments for the top-level sections; returned section paths feed back into `list_docs`, returned page paths feed into `read_doc`.
|
||||
|
||||
### Editing an existing workflow
|
||||
1. `list_workflows` — locate the target workflow.
|
||||
2. `get_workflow_code(workflow_id)` — fetch the current source.
|
||||
3. (optional) `list_node_types` / `get_node_type(name)` — consult before adding or editing a node type whose fields aren't already visible in the current code.
|
||||
2. `get_workflow_code` — fetch the current source for that workflow.
|
||||
3. (optional) `list_node_types` / `get_node_type` — consult before adding or editing a node type whose fields aren't already visible in the current code.
|
||||
4. Mutate the code in place. Preserve existing nodes, edges, and variable names unless the task requires removing or renaming them.
|
||||
5. `save_workflow(workflow_id, code)` — persist as a new draft. The published version is untouched.
|
||||
5. `save_workflow` — persist as a new draft. The published version is untouched.
|
||||
|
||||
### Creating a new workflow
|
||||
1. Create a simple 1-node workflow with only `startCall`. The user can iteratively add complexity by editing it.
|
||||
2. `list_node_types` / `get_node_type(name)` — consult to learn the fields available on the node types you intend to use.
|
||||
2. `list_node_types` / `get_node_type` — consult to learn the fields available on the node types you intend to use.
|
||||
3. Author SDK TypeScript from scratch. The `new Workflow({ name: "..." })` call is required — `name` becomes the workflow's display name.
|
||||
4. `create_workflow(code)` — persists a new workflow as version 1 (published). Returns the new `workflow_id`. For subsequent edits use `save_workflow(workflow_id, code)` (which writes a draft).
|
||||
4. `create_workflow` — persists a new workflow as version 1 (published). Returns the new `workflow_id`. For subsequent edits use `save_workflow` (which writes a draft).
|
||||
|
||||
## Allowed source shape
|
||||
|
||||
|
|
@ -68,14 +81,7 @@ Example:
|
|||
|
||||
## Iterating on errors
|
||||
|
||||
`save_workflow` and `create_workflow` return one of:
|
||||
- `parse_error` — Disallowed construct (see grammar above) or malformed TypeScript.
|
||||
- `validation_error` — Node data failed spec validation (unknown field, missing required, wrong type, bad `options` value).
|
||||
- `graph_validation` — Structural rule broken (missing startCall, unreachable node, edge to/from wrong node type).
|
||||
- `missing_name` — (`create_workflow` only) `new Workflow({ name })` is absent or empty.
|
||||
- `bridge_error` — Internal; retry once, then surface to the user.
|
||||
|
||||
Every error carries `line` and `column`. Fix at that location and resubmit the **complete source** — this tool does not accept patches.
|
||||
A failed `save_workflow` / `create_workflow` returns a result with `saved`/`created` set to false, a machine-readable `error_code`, and a human-readable `error` message — carrying `line` and `column` when the problem is locatable in your source. The full set of `error_code` values and their meanings is documented on each tool (visible in its description). Read the `error` message, fix at the reported location, and resubmit the **complete source** — these tools do not accept patches. If a failure looks internal or transient rather than a problem with your code, retry once before surfacing it to the user.
|
||||
|
||||
## Field conventions
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
from fastmcp import FastMCP
|
||||
from mcp.types import ToolAnnotations
|
||||
|
||||
from api.mcp_server.instructions import DOGRAH_MCP_INSTRUCTIONS
|
||||
from api.mcp_server.tools.catalog import (
|
||||
|
|
@ -8,6 +9,7 @@ from api.mcp_server.tools.catalog import (
|
|||
list_tools,
|
||||
)
|
||||
from api.mcp_server.tools.create_workflow import create_workflow
|
||||
from api.mcp_server.tools.docs_search import list_docs, read_doc, search_docs
|
||||
from api.mcp_server.tools.get_workflow_code import get_workflow_code
|
||||
from api.mcp_server.tools.node_types import get_node_type, list_node_types
|
||||
from api.mcp_server.tools.save_workflow import save_workflow
|
||||
|
|
@ -29,3 +31,13 @@ for _tool in (
|
|||
save_workflow,
|
||||
):
|
||||
mcp.tool(_tool)
|
||||
|
||||
_DOCS_TOOL_ANNOTATIONS = ToolAnnotations(
|
||||
readOnlyHint=True,
|
||||
idempotentHint=True,
|
||||
destructiveHint=False,
|
||||
openWorldHint=False,
|
||||
)
|
||||
|
||||
for _tool in (list_docs, read_doc, search_docs):
|
||||
mcp.tool(_tool, annotations=_DOCS_TOOL_ANNOTATIONS)
|
||||
|
|
|
|||
55
api/mcp_server/tools/_workflow_projection.py
Normal file
55
api/mcp_server/tools/_workflow_projection.py
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Literal
|
||||
|
||||
from api.db import db_client
|
||||
from api.mcp_server.ts_bridge import generate_code
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class WorkflowProjectionSource:
|
||||
payload: dict[str, Any] | None
|
||||
version: Literal["draft", "published", "legacy"]
|
||||
version_number: int | None
|
||||
|
||||
|
||||
async def select_workflow_projection_source(workflow: Any) -> WorkflowProjectionSource:
|
||||
"""Choose the same working copy across read and save MCP tools.
|
||||
|
||||
Draft wins over published because that's what a human editor would
|
||||
be mutating. Legacy `workflow_definition` is the final fallback for
|
||||
older rows that predate versioned definitions.
|
||||
"""
|
||||
draft = await db_client.get_draft_version(workflow.id)
|
||||
if draft is not None and draft.workflow_json:
|
||||
return WorkflowProjectionSource(
|
||||
payload=draft.workflow_json,
|
||||
version="draft",
|
||||
version_number=draft.version_number,
|
||||
)
|
||||
|
||||
released = workflow.released_definition
|
||||
if released is not None and released.workflow_json:
|
||||
return WorkflowProjectionSource(
|
||||
payload=released.workflow_json,
|
||||
version="published",
|
||||
version_number=released.version_number,
|
||||
)
|
||||
|
||||
return WorkflowProjectionSource(
|
||||
payload=workflow.workflow_definition or None,
|
||||
version="legacy",
|
||||
version_number=None,
|
||||
)
|
||||
|
||||
|
||||
async def project_workflow_to_sdk_view(workflow: Any) -> dict[str, Any]:
|
||||
source = await select_workflow_projection_source(workflow)
|
||||
code = await generate_code(source.payload or {}, workflow_name=workflow.name or "")
|
||||
return {
|
||||
"name": workflow.name or "",
|
||||
"version": source.version,
|
||||
"version_number": source.version_number,
|
||||
"code": code,
|
||||
}
|
||||
|
|
@ -12,10 +12,10 @@ Execution flow mirrors `save_workflow`:
|
|||
4. Persist via `db_client.create_workflow` — workflow row + v1
|
||||
published definition in a single transaction.
|
||||
|
||||
Error codes surfaced to the LLM match `save_workflow`. An additional
|
||||
`missing_name` error is returned when the source omits
|
||||
`new Workflow({ name: "..." })` — the name is required and there is no
|
||||
prior workflow to fall back to.
|
||||
Each failure path returns an `error_code` via `_error_result`. Those
|
||||
codes and their meanings are documented in the `create_workflow`
|
||||
docstring (the description shipped to the LLM via `tools/list`); keep the
|
||||
two in sync — `test_mcp_instructions_drift.py` enforces it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -34,6 +34,10 @@ from api.mcp_server.ts_bridge import TsBridgeError, parse_code
|
|||
from api.services.posthog_client import capture_event
|
||||
from api.services.workflow.dto import ReactFlowDTO
|
||||
from api.services.workflow.layout import reconcile_positions
|
||||
from api.services.workflow.trigger_paths import (
|
||||
extract_trigger_paths,
|
||||
validate_trigger_paths,
|
||||
)
|
||||
from api.services.workflow.workflow_graph import WorkflowGraph
|
||||
|
||||
|
||||
|
|
@ -53,20 +57,6 @@ def _format_errors(errors: list[dict[str, Any]]) -> str:
|
|||
return "\n".join(parts)
|
||||
|
||||
|
||||
def _extract_trigger_paths(workflow_definition: dict) -> list[str]:
|
||||
"""Mirror of `routes.workflow.extract_trigger_paths` — kept local so the
|
||||
MCP layer doesn't depend on the route module."""
|
||||
if not workflow_definition:
|
||||
return []
|
||||
paths: list[str] = []
|
||||
for node in workflow_definition.get("nodes") or []:
|
||||
if node.get("type") == "trigger":
|
||||
trigger_path = (node.get("data") or {}).get("trigger_path")
|
||||
if trigger_path:
|
||||
paths.append(trigger_path)
|
||||
return paths
|
||||
|
||||
|
||||
@traced_tool
|
||||
async def create_workflow(code: str) -> dict[str, Any]:
|
||||
"""Parse SDK TypeScript and create a new published workflow.
|
||||
|
|
@ -86,6 +76,22 @@ async def create_workflow(code: str) -> dict[str, Any]:
|
|||
On success the new workflow is published as version 1. Use
|
||||
`save_workflow(workflow_id, code)` for subsequent edits — those go to
|
||||
a draft.
|
||||
|
||||
On failure the result has `created: false`, a machine-readable
|
||||
`error_code`, and a human-readable `error` (with file:line:column
|
||||
where the problem is locatable). Resubmit the full corrected source —
|
||||
patches are not accepted. Possible `error_code` values:
|
||||
- `parse_error` — disallowed construct or malformed TypeScript.
|
||||
- `validation_error` — node data failed spec validation (unknown
|
||||
field, missing required, wrong type, option out of range).
|
||||
- `schema_validation` — wire-format (DTO) rejection; rare.
|
||||
- `graph_validation` — structural rule broken (e.g. no start node,
|
||||
unreachable node, edge to/from the wrong node type).
|
||||
- `missing_name` — `new Workflow({ name })` is absent or empty; the
|
||||
name is required and there is no prior workflow to fall back to.
|
||||
- `trigger_path_conflict` — a trigger node's path is already used by
|
||||
another workflow in this organization; rename it and resubmit.
|
||||
- `bridge_error` — internal/transient; retry once, then surface it.
|
||||
"""
|
||||
user = await authenticate_mcp_request()
|
||||
|
||||
|
|
@ -113,6 +119,12 @@ async def create_workflow(code: str) -> dict[str, Any]:
|
|||
# 1b. New workflow — no prior version to reconcile against; layout
|
||||
# places new nodes adjacent to their first incoming neighbor.
|
||||
payload = reconcile_positions(payload, None)
|
||||
trigger_path_issues = validate_trigger_paths(payload)
|
||||
if trigger_path_issues:
|
||||
return _error_result(
|
||||
"validation_error",
|
||||
"\n".join(issue.message for issue in trigger_path_issues),
|
||||
)
|
||||
|
||||
# 2. Pydantic shape check (defence in depth — parser is spec-driven).
|
||||
try:
|
||||
|
|
@ -128,7 +140,7 @@ async def create_workflow(code: str) -> dict[str, Any]:
|
|||
|
||||
# 4. Reject upfront if any trigger path collides with another workflow's
|
||||
# trigger in this org so we don't leave an orphan workflow record.
|
||||
trigger_paths = _extract_trigger_paths(payload)
|
||||
trigger_paths = extract_trigger_paths(payload)
|
||||
if trigger_paths:
|
||||
try:
|
||||
await db_client.assert_trigger_paths_available(
|
||||
|
|
|
|||
704
api/mcp_server/tools/docs_search.py
Normal file
704
api/mcp_server/tools/docs_search.py
Normal file
|
|
@ -0,0 +1,704 @@
|
|||
"""MCP docs discovery tools over the Mintlify docs tree.
|
||||
|
||||
The docs surface is intentionally split into three steps:
|
||||
|
||||
- ``list_docs`` for lightweight navigation over the published hierarchy
|
||||
- ``search_docs`` for keyword lookup across the visible docs catalog
|
||||
- ``read_doc`` for the full content of one chosen page (or one section)
|
||||
|
||||
The runtime index is derived from ``docs/docs.json`` plus the referenced
|
||||
``.mdx``/``.md`` files. That keeps navigation, ordering, and visibility in
|
||||
sync with the published docs rather than indexing every file under ``docs/``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from collections import Counter
|
||||
from dataclasses import dataclass, replace
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
from fastapi import HTTPException
|
||||
|
||||
from api.mcp_server.auth import authenticate_mcp_request
|
||||
from api.mcp_server.tracing import traced_tool
|
||||
|
||||
DOCS_SEARCH_MAX_LIMIT = 25
|
||||
DOCS_LIST_MAX_DEPTH = 3
|
||||
_ROOT_SECTION_PATH = "__root__"
|
||||
|
||||
_TOKEN_RE = re.compile(r"[A-Za-z0-9_]+")
|
||||
_FRONTMATTER_RE = re.compile(r"\A---\s*\n(.*?)\n---\s*\n?", re.DOTALL)
|
||||
_HEADING_RE = re.compile(r"^(#{1,6})\s+(.*?)\s*$", re.MULTILINE)
|
||||
_STOPWORDS = {
|
||||
"a",
|
||||
"an",
|
||||
"and",
|
||||
"are",
|
||||
"at",
|
||||
"be",
|
||||
"by",
|
||||
"can",
|
||||
"do",
|
||||
"for",
|
||||
"from",
|
||||
"how",
|
||||
"i",
|
||||
"if",
|
||||
"in",
|
||||
"is",
|
||||
"it",
|
||||
"me",
|
||||
"my",
|
||||
"of",
|
||||
"on",
|
||||
"or",
|
||||
"the",
|
||||
"to",
|
||||
"what",
|
||||
"when",
|
||||
"where",
|
||||
"with",
|
||||
"you",
|
||||
"your",
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DocSection:
|
||||
title: str
|
||||
slug: str
|
||||
level: int
|
||||
content: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DocPage:
|
||||
path: str
|
||||
file_path: str
|
||||
title: str
|
||||
description: str
|
||||
llm_hint: str
|
||||
aliases: tuple[str, ...]
|
||||
breadcrumb: tuple[str, ...]
|
||||
content: str
|
||||
sections: tuple[DocSection, ...]
|
||||
order: int
|
||||
|
||||
def breadcrumb_text(self) -> str:
|
||||
return " > ".join(self.breadcrumb)
|
||||
|
||||
def routing_hint(self) -> str:
|
||||
return self.llm_hint or self.description
|
||||
|
||||
def to_catalog_dict(self, section: DocSection | None = None) -> dict:
|
||||
data = {
|
||||
"kind": "page",
|
||||
"path": self.path,
|
||||
"title": self.title,
|
||||
"breadcrumb": self.breadcrumb_text(),
|
||||
"llm_hint": self.routing_hint(),
|
||||
}
|
||||
if section is not None:
|
||||
data["section_title"] = section.title
|
||||
data["section_slug"] = section.slug
|
||||
return _compact_dict(data)
|
||||
|
||||
def to_read_dict(self, section: DocSection | None = None) -> dict:
|
||||
active_section = section
|
||||
content = self.content
|
||||
if active_section is not None:
|
||||
content = active_section.content
|
||||
|
||||
return _compact_dict(
|
||||
{
|
||||
"path": self.path,
|
||||
"title": self.title,
|
||||
"breadcrumb": self.breadcrumb_text(),
|
||||
"llm_hint": self.routing_hint(),
|
||||
"section_title": active_section.title if active_section else None,
|
||||
"section_slug": active_section.slug if active_section else None,
|
||||
"content": content,
|
||||
"sections": [
|
||||
{"title": sec.title, "slug": sec.slug}
|
||||
for sec in self.sections
|
||||
if sec.title and sec.slug
|
||||
],
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class NavSection:
|
||||
path: str
|
||||
title: str
|
||||
breadcrumb: tuple[str, ...]
|
||||
children: tuple[tuple[str, str], ...]
|
||||
descendant_page_count: int = 0
|
||||
|
||||
def breadcrumb_text(self) -> str:
|
||||
return " > ".join(self.breadcrumb)
|
||||
|
||||
def to_mcp_dict(self) -> dict:
|
||||
hint = None
|
||||
if self.descendant_page_count:
|
||||
hint = f"Browse {self.descendant_page_count} docs in this section."
|
||||
return _compact_dict(
|
||||
{
|
||||
"kind": "section",
|
||||
"path": self.path,
|
||||
"title": self.title,
|
||||
"breadcrumb": self.breadcrumb_text(),
|
||||
"llm_hint": hint,
|
||||
"has_children": bool(self.children),
|
||||
"child_count": len(self.children),
|
||||
"page_count": self.descendant_page_count,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DocsIndex:
|
||||
pages_by_path: dict[str, DocPage]
|
||||
sections_by_path: dict[str, NavSection]
|
||||
|
||||
|
||||
def _compact_dict(data: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
key: value for key, value in data.items() if value not in (None, "", [], (), {})
|
||||
}
|
||||
|
||||
|
||||
def _slugify(value: str) -> str:
|
||||
slug = re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-")
|
||||
return slug or "section"
|
||||
|
||||
|
||||
def _coerce_docs_root(candidate: Path) -> Path | None:
|
||||
candidate = candidate.expanduser().resolve()
|
||||
if (candidate / "docs.json").is_file():
|
||||
return candidate
|
||||
nested = candidate / "docs"
|
||||
if (nested / "docs.json").is_file():
|
||||
return nested
|
||||
return None
|
||||
|
||||
|
||||
def _resolve_docs_root() -> Path | None:
|
||||
"""Return the path to the on-disk docs tree, or None if not found."""
|
||||
override = os.environ.get("DOGRAH_DOCS_PATH")
|
||||
if override:
|
||||
resolved = _coerce_docs_root(Path(override))
|
||||
if resolved is not None:
|
||||
return resolved
|
||||
|
||||
docker_default = _coerce_docs_root(Path("/app/docs"))
|
||||
if docker_default is not None:
|
||||
return docker_default
|
||||
|
||||
for parent in Path(__file__).resolve().parents:
|
||||
resolved = _coerce_docs_root(parent / "docs")
|
||||
if resolved is not None:
|
||||
return resolved
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def _split_frontmatter(contents: str) -> tuple[dict[str, Any], str]:
|
||||
match = _FRONTMATTER_RE.match(contents)
|
||||
if not match:
|
||||
return {}, contents
|
||||
try:
|
||||
frontmatter = yaml.safe_load(match.group(1)) or {}
|
||||
except yaml.YAMLError:
|
||||
return {}, contents
|
||||
if not isinstance(frontmatter, dict):
|
||||
frontmatter = {}
|
||||
return frontmatter, contents[match.end() :].lstrip("\n")
|
||||
|
||||
|
||||
def _strip_frontmatter(contents: str) -> str:
|
||||
"""Drop the YAML frontmatter block from a docs page body."""
|
||||
return _split_frontmatter(contents)[1]
|
||||
|
||||
|
||||
def _clean_heading_text(raw: str) -> str:
|
||||
text = re.sub(r"\s*\{#.*\}\s*$", "", raw.strip())
|
||||
return " ".join(text.split())
|
||||
|
||||
|
||||
def _extract_page_title(contents: str, fallback: str) -> str:
|
||||
"""Pull a human-readable title for a docs page."""
|
||||
frontmatter, body = _split_frontmatter(contents)
|
||||
title = frontmatter.get("title")
|
||||
if isinstance(title, str) and title.strip():
|
||||
return title.strip()
|
||||
|
||||
match = _HEADING_RE.search(body)
|
||||
if match:
|
||||
return _clean_heading_text(match.group(2))
|
||||
|
||||
return fallback
|
||||
|
||||
|
||||
def _normalize_text(value: Any) -> str:
|
||||
if isinstance(value, str):
|
||||
return " ".join(value.strip().split())
|
||||
return ""
|
||||
|
||||
|
||||
def _normalize_aliases(value: Any) -> tuple[str, ...]:
|
||||
if isinstance(value, str):
|
||||
aliases = [value]
|
||||
elif isinstance(value, list):
|
||||
aliases = [item for item in value if isinstance(item, str)]
|
||||
else:
|
||||
aliases = []
|
||||
return tuple(alias.strip() for alias in aliases if alias.strip())
|
||||
|
||||
|
||||
def _extract_sections(body: str) -> tuple[DocSection, ...]:
|
||||
matches = list(_HEADING_RE.finditer(body))
|
||||
stripped_body = body.strip()
|
||||
if not matches:
|
||||
if not stripped_body:
|
||||
return ()
|
||||
return (
|
||||
DocSection(
|
||||
title="Overview",
|
||||
slug="overview",
|
||||
level=1,
|
||||
content=stripped_body,
|
||||
),
|
||||
)
|
||||
|
||||
sections: list[DocSection] = []
|
||||
preamble = body[: matches[0].start()].strip()
|
||||
if preamble:
|
||||
sections.append(
|
||||
DocSection(
|
||||
title="Overview",
|
||||
slug="overview",
|
||||
level=1,
|
||||
content=preamble,
|
||||
)
|
||||
)
|
||||
|
||||
for index, match in enumerate(matches):
|
||||
start = match.start()
|
||||
end = matches[index + 1].start() if index + 1 < len(matches) else len(body)
|
||||
title = _clean_heading_text(match.group(2))
|
||||
sections.append(
|
||||
DocSection(
|
||||
title=title or "Section",
|
||||
slug=_slugify(title or "section"),
|
||||
level=len(match.group(1)),
|
||||
content=body[start:end].strip(),
|
||||
)
|
||||
)
|
||||
return tuple(sections)
|
||||
|
||||
|
||||
def _tokenize_text(text: str) -> list[str]:
|
||||
return [
|
||||
token
|
||||
for token in _TOKEN_RE.findall(text.lower())
|
||||
if len(token) >= 2 and token not in _STOPWORDS
|
||||
]
|
||||
|
||||
|
||||
def _tokenize_query(query: str) -> list[str]:
|
||||
"""Split a user query into lowercased keyword terms."""
|
||||
seen: set[str] = set()
|
||||
terms: list[str] = []
|
||||
for token in _TOKEN_RE.findall(query.lower()):
|
||||
if len(token) < 2 or token in _STOPWORDS or token in seen:
|
||||
continue
|
||||
seen.add(token)
|
||||
terms.append(token)
|
||||
return terms
|
||||
|
||||
|
||||
def _resolve_doc_file(root: Path, route_path: str) -> Path | None:
|
||||
candidates = (
|
||||
root / f"{route_path}.mdx",
|
||||
root / f"{route_path}.md",
|
||||
root / route_path / "index.mdx",
|
||||
root / route_path / "index.md",
|
||||
)
|
||||
for candidate in candidates:
|
||||
if candidate.is_file():
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def _build_doc_page(
|
||||
root: Path,
|
||||
route_path: str,
|
||||
*,
|
||||
breadcrumb: tuple[str, ...],
|
||||
order: int,
|
||||
) -> DocPage | None:
|
||||
file_path = _resolve_doc_file(root, route_path)
|
||||
if file_path is None:
|
||||
return None
|
||||
try:
|
||||
contents = file_path.read_text(encoding="utf-8")
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return None
|
||||
|
||||
frontmatter, body = _split_frontmatter(contents)
|
||||
fallback = route_path.rsplit("/", 1)[-1].replace("-", " ").title()
|
||||
title = _extract_page_title(contents, fallback=fallback)
|
||||
description = _normalize_text(frontmatter.get("description"))
|
||||
llm_hint = _normalize_text(frontmatter.get("llm_hint"))
|
||||
aliases = _normalize_aliases(frontmatter.get("aliases"))
|
||||
content = body.strip()
|
||||
|
||||
return DocPage(
|
||||
path=route_path,
|
||||
file_path=file_path.relative_to(root).as_posix(),
|
||||
title=title,
|
||||
description=description,
|
||||
llm_hint=llm_hint,
|
||||
aliases=aliases,
|
||||
breadcrumb=breadcrumb,
|
||||
content=content,
|
||||
sections=_extract_sections(content),
|
||||
order=order,
|
||||
)
|
||||
|
||||
|
||||
def _score_counter(counter: Counter[str], term: str, *, weight: int, cap: int) -> int:
|
||||
return min(counter.get(term, 0), cap) * weight
|
||||
|
||||
|
||||
def _normalized_phrase(text: str) -> str:
|
||||
return " ".join(_tokenize_text(text))
|
||||
|
||||
|
||||
def _score_section(section: DocSection, terms: list[str]) -> int:
|
||||
title_counts = Counter(_tokenize_text(section.title))
|
||||
body_counts = Counter(_tokenize_text(section.content))
|
||||
score = 0
|
||||
matched_terms = 0
|
||||
for term in terms:
|
||||
term_score = _score_counter(
|
||||
title_counts, term, weight=7, cap=2
|
||||
) + _score_counter(body_counts, term, weight=1, cap=4)
|
||||
if term_score:
|
||||
matched_terms += 1
|
||||
score += term_score
|
||||
score += matched_terms * 4
|
||||
|
||||
phrase = " ".join(terms)
|
||||
if phrase and phrase in _normalized_phrase(section.content):
|
||||
score += 6
|
||||
return score
|
||||
|
||||
|
||||
def _score_page(page: DocPage, terms: list[str]) -> tuple[int, DocSection | None]:
|
||||
if not terms:
|
||||
return 0, None
|
||||
|
||||
path_counts = Counter(_tokenize_text(page.path))
|
||||
title_counts = Counter(_tokenize_text(page.title))
|
||||
breadcrumb_counts = Counter(_tokenize_text(" ".join(page.breadcrumb)))
|
||||
hint_counts = Counter(_tokenize_text(page.routing_hint()))
|
||||
alias_counts = Counter(_tokenize_text(" ".join(page.aliases)))
|
||||
|
||||
score = 0
|
||||
matched_terms = 0
|
||||
for term in terms:
|
||||
term_score = (
|
||||
_score_counter(path_counts, term, weight=6, cap=3)
|
||||
+ _score_counter(title_counts, term, weight=10, cap=2)
|
||||
+ _score_counter(breadcrumb_counts, term, weight=4, cap=2)
|
||||
+ _score_counter(hint_counts, term, weight=7, cap=3)
|
||||
+ _score_counter(alias_counts, term, weight=7, cap=3)
|
||||
)
|
||||
if term_score:
|
||||
matched_terms += 1
|
||||
score += term_score
|
||||
|
||||
best_section = None
|
||||
best_section_score = 0
|
||||
for section in page.sections:
|
||||
section_score = _score_section(section, terms)
|
||||
if section_score > best_section_score:
|
||||
best_section = section
|
||||
best_section_score = section_score
|
||||
|
||||
if score == 0 and best_section_score == 0:
|
||||
return 0, None
|
||||
|
||||
score += matched_terms * 8 + best_section_score
|
||||
|
||||
phrase = " ".join(terms)
|
||||
if phrase:
|
||||
if phrase in _normalized_phrase(page.title):
|
||||
score += 12
|
||||
elif phrase in _normalized_phrase(page.routing_hint()):
|
||||
score += 8
|
||||
elif phrase in _normalized_phrase(page.path):
|
||||
score += 8
|
||||
elif best_section is not None and phrase in _normalized_phrase(
|
||||
best_section.content
|
||||
):
|
||||
score += 4
|
||||
|
||||
return score, best_section
|
||||
|
||||
|
||||
def _set_descendant_counts(
|
||||
sections_by_path: dict[str, NavSection],
|
||||
section_path: str,
|
||||
) -> int:
|
||||
section = sections_by_path[section_path]
|
||||
page_count = 0
|
||||
for child_kind, child_path in section.children:
|
||||
if child_kind == "page":
|
||||
page_count += 1
|
||||
else:
|
||||
page_count += _set_descendant_counts(sections_by_path, child_path)
|
||||
sections_by_path[section_path] = replace(section, descendant_page_count=page_count)
|
||||
return page_count
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _docs_index() -> DocsIndex:
|
||||
root = _resolve_docs_root()
|
||||
if root is None:
|
||||
return DocsIndex(pages_by_path={}, sections_by_path={})
|
||||
|
||||
try:
|
||||
docs_config = json.loads((root / "docs.json").read_text(encoding="utf-8"))
|
||||
except (OSError, UnicodeDecodeError, json.JSONDecodeError):
|
||||
return DocsIndex(pages_by_path={}, sections_by_path={})
|
||||
|
||||
pages_by_path: dict[str, DocPage] = {}
|
||||
sections_by_path: dict[str, NavSection] = {}
|
||||
page_order = 0
|
||||
|
||||
def ensure_unique_section_path(base_path: str) -> str:
|
||||
if base_path not in sections_by_path:
|
||||
return base_path
|
||||
suffix = 2
|
||||
while f"{base_path}-{suffix}" in sections_by_path:
|
||||
suffix += 1
|
||||
return f"{base_path}-{suffix}"
|
||||
|
||||
def walk_pages(
|
||||
items: list[Any],
|
||||
*,
|
||||
section_path: str,
|
||||
section_title: str,
|
||||
ancestor_breadcrumb: tuple[str, ...],
|
||||
) -> None:
|
||||
nonlocal page_order
|
||||
children: list[tuple[str, str]] = []
|
||||
page_breadcrumb = ancestor_breadcrumb + (section_title,)
|
||||
|
||||
for item in items:
|
||||
if isinstance(item, str):
|
||||
route_path = item.strip("/")
|
||||
if not route_path:
|
||||
continue
|
||||
if route_path not in pages_by_path:
|
||||
page = _build_doc_page(
|
||||
root,
|
||||
route_path,
|
||||
breadcrumb=page_breadcrumb,
|
||||
order=page_order,
|
||||
)
|
||||
if page is not None:
|
||||
pages_by_path[route_path] = page
|
||||
page_order += 1
|
||||
if route_path in pages_by_path:
|
||||
children.append(("page", route_path))
|
||||
continue
|
||||
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
group_title = str(item.get("group", "")).strip()
|
||||
nested_pages = item.get("pages")
|
||||
if not group_title or not isinstance(nested_pages, list):
|
||||
continue
|
||||
|
||||
child_path = ensure_unique_section_path(
|
||||
f"{section_path}/{_slugify(group_title)}"
|
||||
)
|
||||
walk_pages(
|
||||
nested_pages,
|
||||
section_path=child_path,
|
||||
section_title=group_title,
|
||||
ancestor_breadcrumb=page_breadcrumb,
|
||||
)
|
||||
children.append(("section", child_path))
|
||||
|
||||
sections_by_path[section_path] = NavSection(
|
||||
path=section_path,
|
||||
title=section_title,
|
||||
breadcrumb=ancestor_breadcrumb,
|
||||
children=tuple(children),
|
||||
)
|
||||
|
||||
root_children: list[tuple[str, str]] = []
|
||||
tabs = docs_config.get("navigation", {}).get("tabs", [])
|
||||
for tab in tabs:
|
||||
if not isinstance(tab, dict):
|
||||
continue
|
||||
tab_title = str(tab.get("tab", "")).strip() or "Docs"
|
||||
for group in tab.get("groups", []):
|
||||
if not isinstance(group, dict):
|
||||
continue
|
||||
group_title = str(group.get("group", "")).strip()
|
||||
group_pages = group.get("pages")
|
||||
if not group_title or not isinstance(group_pages, list):
|
||||
continue
|
||||
top_level_path = ensure_unique_section_path(
|
||||
f"{_slugify(tab_title)}/{_slugify(group_title)}"
|
||||
)
|
||||
walk_pages(
|
||||
group_pages,
|
||||
section_path=top_level_path,
|
||||
section_title=group_title,
|
||||
ancestor_breadcrumb=(tab_title,),
|
||||
)
|
||||
root_children.append(("section", top_level_path))
|
||||
|
||||
sections_by_path[_ROOT_SECTION_PATH] = NavSection(
|
||||
path=_ROOT_SECTION_PATH,
|
||||
title="Docs",
|
||||
breadcrumb=(),
|
||||
children=tuple(root_children),
|
||||
)
|
||||
_set_descendant_counts(sections_by_path, _ROOT_SECTION_PATH)
|
||||
|
||||
return DocsIndex(pages_by_path=pages_by_path, sections_by_path=sections_by_path)
|
||||
|
||||
|
||||
def _get_page_or_404(path: str) -> DocPage:
|
||||
page = _docs_index().pages_by_path.get(path.strip("/"))
|
||||
if page is None:
|
||||
raise HTTPException(status_code=404, detail=f"Unknown docs page: {path!r}")
|
||||
return page
|
||||
|
||||
|
||||
def _find_section(page: DocPage, section: str) -> DocSection | None:
|
||||
target = section.strip().lower()
|
||||
for candidate in page.sections:
|
||||
if candidate.slug.lower() == target or candidate.title.lower() == target:
|
||||
return candidate
|
||||
return None
|
||||
|
||||
|
||||
def _expand_nav_entries(
|
||||
index: DocsIndex,
|
||||
section_path: str,
|
||||
depth: int,
|
||||
) -> list[dict]:
|
||||
section = index.sections_by_path[section_path]
|
||||
results: list[dict] = []
|
||||
for child_kind, child_path in section.children:
|
||||
if child_kind == "section":
|
||||
child_section = index.sections_by_path[child_path]
|
||||
results.append(child_section.to_mcp_dict())
|
||||
if depth > 1:
|
||||
results.extend(_expand_nav_entries(index, child_path, depth - 1))
|
||||
else:
|
||||
results.append(index.pages_by_path[child_path].to_catalog_dict())
|
||||
return results
|
||||
|
||||
|
||||
@traced_tool
|
||||
async def list_docs(path: str | None = None, depth: int = 1) -> list[dict]:
|
||||
"""Browse the Dograh docs hierarchy before reading a page in full.
|
||||
|
||||
``path`` addresses navigation sections exposed by this tool. Page paths
|
||||
returned by ``search_docs`` and ``read_doc`` are the published docs routes
|
||||
instead, for example ``voice-agent/tools/mcp-tool``.
|
||||
"""
|
||||
await authenticate_mcp_request()
|
||||
|
||||
if depth < 1 or depth > DOCS_LIST_MAX_DEPTH:
|
||||
raise ValueError(f"`depth` must be between 1 and {DOCS_LIST_MAX_DEPTH}.")
|
||||
|
||||
index = _docs_index()
|
||||
if not index.sections_by_path:
|
||||
return []
|
||||
|
||||
if path is None:
|
||||
return _expand_nav_entries(index, _ROOT_SECTION_PATH, depth)
|
||||
|
||||
normalized = path.strip("/")
|
||||
if normalized in index.sections_by_path:
|
||||
return _expand_nav_entries(index, normalized, depth)
|
||||
if normalized in index.pages_by_path:
|
||||
return [index.pages_by_path[normalized].to_catalog_dict()]
|
||||
|
||||
raise HTTPException(status_code=404, detail=f"Unknown docs section: {path!r}")
|
||||
|
||||
|
||||
@traced_tool
|
||||
async def read_doc(path: str, section: str | None = None) -> dict:
|
||||
"""Read one docs page after you have narrowed to a likely match."""
|
||||
await authenticate_mcp_request()
|
||||
|
||||
if not isinstance(path, str) or not path.strip():
|
||||
raise ValueError("`path` must be a non-empty string.")
|
||||
|
||||
page = _get_page_or_404(path)
|
||||
active_section = None
|
||||
if section is not None:
|
||||
active_section = _find_section(page, section)
|
||||
if active_section is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Unknown section {section!r} for docs page {path!r}",
|
||||
)
|
||||
return page.to_read_dict(section=active_section)
|
||||
|
||||
|
||||
@traced_tool
|
||||
async def search_docs(query: str, limit: int = 5) -> list[dict]:
|
||||
"""Search the Dograh documentation and return a lean ranked shortlist.
|
||||
|
||||
Use this first for keyword or acronym lookup. Once the right page looks
|
||||
likely, call ``read_doc(path)`` instead of reasoning from summaries alone.
|
||||
"""
|
||||
await authenticate_mcp_request()
|
||||
|
||||
if not isinstance(query, str) or not query.strip():
|
||||
raise ValueError("`query` must be a non-empty string.")
|
||||
if limit < 1:
|
||||
raise ValueError("`limit` must be at least 1.")
|
||||
|
||||
terms = _tokenize_query(query)
|
||||
if not terms:
|
||||
raise ValueError(
|
||||
"`query` must contain at least one non-stopword alphanumeric term."
|
||||
)
|
||||
|
||||
index = _docs_index()
|
||||
if not index.pages_by_path:
|
||||
return []
|
||||
|
||||
capped_limit = min(limit, DOCS_SEARCH_MAX_LIMIT)
|
||||
ranked: list[tuple[int, int, DocPage, DocSection | None]] = []
|
||||
for page in index.pages_by_path.values():
|
||||
score, best_section = _score_page(page, terms)
|
||||
if score <= 0:
|
||||
continue
|
||||
ranked.append((score, page.order, page, best_section))
|
||||
|
||||
ranked.sort(key=lambda item: (-item[0], item[1], item[2].path))
|
||||
return [
|
||||
page.to_catalog_dict(section=best_section)
|
||||
for _, _, page, best_section in ranked[:capped_limit]
|
||||
]
|
||||
|
|
@ -18,8 +18,9 @@ from fastapi import HTTPException
|
|||
|
||||
from api.db import db_client
|
||||
from api.mcp_server.auth import authenticate_mcp_request
|
||||
from api.mcp_server.tools._workflow_projection import project_workflow_to_sdk_view
|
||||
from api.mcp_server.tracing import traced_tool
|
||||
from api.mcp_server.ts_bridge import TsBridgeError, generate_code
|
||||
from api.mcp_server.ts_bridge import TsBridgeError
|
||||
|
||||
|
||||
@traced_tool
|
||||
|
|
@ -39,31 +40,14 @@ async def get_workflow_code(workflow_id: int) -> dict[str, Any]:
|
|||
if not workflow:
|
||||
raise HTTPException(status_code=404, detail=f"Workflow {workflow_id} not found")
|
||||
|
||||
# Draft wins over published — editing a draft is the normal flow.
|
||||
# `current_definition` (is_current=True) is the published row, so we
|
||||
# fetch the draft explicitly. If the latest draft was just published,
|
||||
# no draft row exists and we fall through to `released_definition`.
|
||||
draft = await db_client.get_draft_version(workflow_id)
|
||||
released = workflow.released_definition
|
||||
|
||||
if draft is not None and draft.workflow_json:
|
||||
payload = draft.workflow_json
|
||||
source = "draft"
|
||||
elif released is not None and released.workflow_json:
|
||||
payload = released.workflow_json
|
||||
source = "published"
|
||||
else:
|
||||
payload = workflow.workflow_definition or {}
|
||||
source = "legacy"
|
||||
|
||||
try:
|
||||
code = await generate_code(payload, workflow_name=workflow.name or "")
|
||||
view = await project_workflow_to_sdk_view(workflow)
|
||||
except TsBridgeError as e:
|
||||
raise HTTPException(status_code=500, detail=f"Failed to generate code: {e}")
|
||||
|
||||
return {
|
||||
"workflow_id": workflow_id,
|
||||
"name": workflow.name or "",
|
||||
"version": source,
|
||||
"code": code,
|
||||
"name": view["name"],
|
||||
"version": view["version"],
|
||||
"code": view["code"],
|
||||
}
|
||||
|
|
|
|||
|
|
@ -40,15 +40,17 @@ async def list_node_types() -> dict:
|
|||
|
||||
@traced_tool
|
||||
async def get_node_type(name: str) -> dict:
|
||||
"""Fetch the full schema for a node type, including every property's
|
||||
type, default, conditional visibility rules, and LLM-readable
|
||||
description, plus worked examples.
|
||||
"""Fetch the authoring schema for a node type: each property's name,
|
||||
type, default, requiredness, enum options, validation bounds, and
|
||||
LLM-readable description, plus worked examples and graph constraints.
|
||||
|
||||
Use the property `description` and the `examples` list to understand
|
||||
semantics — types alone are not enough.
|
||||
UI-only metadata (display labels, placeholders, conditional visibility
|
||||
rules, renderer hints) is intentionally omitted — set only the fields
|
||||
you need. Use the property `description`/`llm_hint` and the `examples`
|
||||
list to understand semantics; types alone are not enough.
|
||||
"""
|
||||
await authenticate_mcp_request()
|
||||
spec = get_spec(name)
|
||||
if spec is None:
|
||||
raise HTTPException(status_code=404, detail=f"Unknown node type: {name!r}")
|
||||
return spec.model_dump(mode="json")
|
||||
return spec.to_mcp_dict()
|
||||
|
|
|
|||
|
|
@ -10,16 +10,12 @@ Execution flow:
|
|||
4. Save as a new draft via `db_client.save_workflow_draft` — the
|
||||
published version stays intact, so edits are rollback-safe.
|
||||
|
||||
Error codes surfaced to the LLM:
|
||||
parse_error — TS parse failed or a disallowed construct was used
|
||||
validation_error — node data failed spec validation (unknown field,
|
||||
missing required, wrong type, option out of range)
|
||||
schema_validation — ReactFlowDTO Pydantic rejection (rare; parser bug)
|
||||
graph_validation — semantic graph rule broken (e.g. no start node)
|
||||
bridge_error — Node subprocess failed before returning JSON
|
||||
|
||||
All LLM-facing errors include file:line:column where available so the
|
||||
LLM can correct its code directly.
|
||||
Each failure path returns an `error_code` via `_error_result`. Those
|
||||
codes and their meanings are documented in the `save_workflow` docstring
|
||||
(the description shipped to the LLM via `tools/list`); keep the two in
|
||||
sync — `test_mcp_instructions_drift.py` enforces it. All LLM-facing
|
||||
errors include file:line:column where available so the LLM can correct
|
||||
its code directly.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -32,28 +28,21 @@ from pydantic import ValidationError as PydanticValidationError
|
|||
|
||||
from api.db import db_client
|
||||
from api.mcp_server.auth import authenticate_mcp_request
|
||||
from api.mcp_server.tools._workflow_projection import (
|
||||
select_workflow_projection_source,
|
||||
)
|
||||
from api.mcp_server.tracing import traced_tool
|
||||
from api.mcp_server.ts_bridge import TsBridgeError, parse_code
|
||||
from api.services.workflow.dto import ReactFlowDTO
|
||||
from api.services.workflow.layout import reconcile_positions
|
||||
from api.services.workflow.trigger_paths import validate_trigger_paths
|
||||
from api.services.workflow.workflow_graph import WorkflowGraph
|
||||
|
||||
|
||||
async def _previous_workflow_json(workflow: Any) -> dict[str, Any] | None:
|
||||
"""Same selection priority as `get_workflow_code` — the version the
|
||||
LLM saw is the version we reconcile against.
|
||||
|
||||
`current_definition` (is_current=True) is the published row, so the
|
||||
draft must be fetched explicitly. If no draft exists (e.g. the last
|
||||
draft was just published), fall through to `released_definition`.
|
||||
"""
|
||||
draft = await db_client.get_draft_version(workflow.id)
|
||||
if draft is not None and draft.workflow_json:
|
||||
return draft.workflow_json
|
||||
released = workflow.released_definition
|
||||
if released is not None and released.workflow_json:
|
||||
return released.workflow_json
|
||||
return workflow.workflow_definition or None
|
||||
"""Match the agent-facing read tools' source selection."""
|
||||
source = await select_workflow_projection_source(workflow)
|
||||
return source.payload
|
||||
|
||||
|
||||
def _error_result(code: str, message: str, **extra: Any) -> dict[str, Any]:
|
||||
|
|
@ -91,6 +80,18 @@ async def save_workflow(workflow_id: int, code: str) -> dict[str, Any]:
|
|||
|
||||
On success the draft version is saved; the published version is
|
||||
untouched.
|
||||
|
||||
On failure the result has `saved: false`, a machine-readable
|
||||
`error_code`, and a human-readable `error` (with file:line:column
|
||||
where the problem is locatable). Resubmit the full corrected source —
|
||||
patches are not accepted. Possible `error_code` values:
|
||||
- `parse_error` — disallowed construct or malformed TypeScript.
|
||||
- `validation_error` — node data failed spec validation (unknown
|
||||
field, missing required, wrong type, option out of range).
|
||||
- `schema_validation` — wire-format (DTO) rejection; rare.
|
||||
- `graph_validation` — structural rule broken (e.g. no start node,
|
||||
unreachable node, edge to/from the wrong node type).
|
||||
- `bridge_error` — internal/transient; retry once, then surface it.
|
||||
"""
|
||||
user = await authenticate_mcp_request()
|
||||
|
||||
|
|
@ -121,6 +122,12 @@ async def save_workflow(workflow_id: int, code: str) -> dict[str, Any]:
|
|||
# here we fill them back in from what was there before, and pick
|
||||
# approximate placements for newly-introduced nodes.
|
||||
payload = reconcile_positions(payload, await _previous_workflow_json(workflow))
|
||||
trigger_path_issues = validate_trigger_paths(payload)
|
||||
if trigger_path_issues:
|
||||
return _error_result(
|
||||
"validation_error",
|
||||
"\n".join(issue.message for issue in trigger_path_issues),
|
||||
)
|
||||
|
||||
# 2. Pydantic shape check (defence in depth — parser is spec-driven).
|
||||
try:
|
||||
|
|
|
|||
|
|
@ -2,7 +2,9 @@ from fastapi import HTTPException
|
|||
|
||||
from api.db import db_client
|
||||
from api.mcp_server.auth import authenticate_mcp_request
|
||||
from api.mcp_server.tools._workflow_projection import project_workflow_to_sdk_view
|
||||
from api.mcp_server.tracing import traced_tool
|
||||
from api.mcp_server.ts_bridge import TsBridgeError
|
||||
|
||||
|
||||
@traced_tool
|
||||
|
|
@ -10,9 +12,9 @@ async def list_workflows(status: str | None = "active") -> list[dict]:
|
|||
"""List agents (workflows) in the caller's organization.
|
||||
|
||||
Returns id, name, status, and created_at for each agent. Use
|
||||
`get_workflow` to fetch a single agent's full definition. Defaults
|
||||
to active agents; pass `status="archived"` to list archived agents,
|
||||
or `status=None` to list all.
|
||||
`get_workflow` to fetch a single agent's current SDK view and
|
||||
metadata. Defaults to active agents; pass `status="archived"` to
|
||||
list archived agents, or `status=None` to list all.
|
||||
"""
|
||||
user = await authenticate_mcp_request()
|
||||
workflows = await db_client.get_all_workflows_for_listing(
|
||||
|
|
@ -32,7 +34,11 @@ async def list_workflows(status: str | None = "active") -> list[dict]:
|
|||
|
||||
@traced_tool
|
||||
async def get_workflow(workflow_id: int) -> dict:
|
||||
"""Fetch a single agent by id, including its current published definition."""
|
||||
"""Fetch a single agent by id, projected into the SDK code view.
|
||||
|
||||
Output shape:
|
||||
{"id": int, "name": str, "status": str, "version": "draft" | "published" | "legacy", "version_number": int | None, "code": "<TS source>"}
|
||||
"""
|
||||
user = await authenticate_mcp_request()
|
||||
workflow = await db_client.get_workflow(
|
||||
workflow_id, organization_id=user.selected_organization_id
|
||||
|
|
@ -40,11 +46,16 @@ async def get_workflow(workflow_id: int) -> dict:
|
|||
if not workflow:
|
||||
raise HTTPException(status_code=404, detail=f"Workflow {workflow_id} not found")
|
||||
|
||||
current = workflow.current_definition
|
||||
try:
|
||||
view = await project_workflow_to_sdk_view(workflow)
|
||||
except TsBridgeError as e:
|
||||
raise HTTPException(status_code=500, detail=f"Failed to generate code: {e}")
|
||||
|
||||
return {
|
||||
"id": workflow.id,
|
||||
"name": workflow.name,
|
||||
"name": view["name"],
|
||||
"status": workflow.status,
|
||||
"definition": current.workflow_json if current else None,
|
||||
"version_number": current.version_number if current else None,
|
||||
"version": view["version"],
|
||||
"version_number": view["version_number"],
|
||||
"code": view["code"],
|
||||
}
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@ import json
|
|||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from api.services.workflow.dto import EdgeDataDTO
|
||||
from api.services.workflow.node_specs import all_specs
|
||||
|
||||
_VALIDATOR_ENTRY = Path(__file__).resolve().parent / "ts_validator" / "src" / "index.ts"
|
||||
|
|
@ -31,6 +32,10 @@ def _specs_payload() -> list[dict[str, Any]]:
|
|||
return [s.model_dump(mode="json") for s in all_specs()]
|
||||
|
||||
|
||||
def _edge_field_names() -> list[str]:
|
||||
return list(EdgeDataDTO.model_fields.keys())
|
||||
|
||||
|
||||
async def _invoke(request: dict[str, Any]) -> dict[str, Any]:
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
"node",
|
||||
|
|
@ -65,6 +70,7 @@ async def generate_code(workflow: dict[str, Any], *, workflow_name: str = "") ->
|
|||
"command": "generate",
|
||||
"workflow": workflow,
|
||||
"specs": _specs_payload(),
|
||||
"edgeFieldNames": _edge_field_names(),
|
||||
"workflowName": workflow_name,
|
||||
}
|
||||
)
|
||||
|
|
@ -89,5 +95,6 @@ async def parse_code(code: str) -> dict[str, Any]:
|
|||
"command": "parse",
|
||||
"code": code,
|
||||
"specs": _specs_payload(),
|
||||
"edgeFieldNames": _edge_field_names(),
|
||||
}
|
||||
)
|
||||
|
|
|
|||
|
|
@ -14,9 +14,18 @@ import type {
|
|||
export function generateCode(
|
||||
workflow: WireWorkflow,
|
||||
specs: NodeSpec[],
|
||||
opts: { workflowName?: string } = {},
|
||||
opts: { workflowName?: string; edgeFieldNames?: string[] } = {},
|
||||
): GenerateResult {
|
||||
const specByName = new Map(specs.map((s) => [s.name, s]));
|
||||
const edgeFieldNames = new Set(
|
||||
opts.edgeFieldNames ?? [
|
||||
"label",
|
||||
"condition",
|
||||
"transition_speech",
|
||||
"transition_speech_type",
|
||||
"transition_speech_recording_id",
|
||||
],
|
||||
);
|
||||
|
||||
// Catch unknown node types up-front — otherwise we'd emit an import
|
||||
// line for a factory that doesn't exist.
|
||||
|
|
@ -97,7 +106,7 @@ export function generateCode(
|
|||
],
|
||||
};
|
||||
}
|
||||
const cleanedEdge = pickEdgeFields(edge.data);
|
||||
const cleanedEdge = pickEdgeFields(edge.data, edgeFieldNames);
|
||||
const edgeOpts = renderObject(cleanedEdge, 0);
|
||||
lines.push(`wf.edge(${src}, ${tgt}, ${edgeOpts});`);
|
||||
}
|
||||
|
|
@ -210,22 +219,13 @@ function stripUnknown(
|
|||
return out;
|
||||
}
|
||||
|
||||
// Edge schema is fixed (no NodeSpec for edges). Mirrors the allowed
|
||||
// fields on `Workflow.edge(...)` in both SDKs.
|
||||
const KNOWN_EDGE_FIELDS = new Set([
|
||||
"label",
|
||||
"condition",
|
||||
"transition_speech",
|
||||
"transition_speech_type",
|
||||
"transition_speech_recording_id",
|
||||
]);
|
||||
|
||||
function pickEdgeFields(
|
||||
data: Record<string, unknown>,
|
||||
knownEdgeFields: Set<string>,
|
||||
): Record<string, unknown> {
|
||||
const out: Record<string, unknown> = {};
|
||||
for (const [k, v] of Object.entries(data)) {
|
||||
if (KNOWN_EDGE_FIELDS.has(k)) out[k] = v;
|
||||
if (knownEdgeFields.has(k)) out[k] = v;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -11,6 +11,7 @@ interface GenerateRequest {
|
|||
command: "generate";
|
||||
workflow: WireWorkflow;
|
||||
specs: NodeSpec[];
|
||||
edgeFieldNames: string[];
|
||||
workflowName?: string;
|
||||
}
|
||||
|
||||
|
|
@ -18,6 +19,7 @@ interface ParseRequest {
|
|||
command: "parse";
|
||||
code: string;
|
||||
specs: NodeSpec[];
|
||||
edgeFieldNames: string[];
|
||||
}
|
||||
|
||||
type Request = GenerateRequest | ParseRequest;
|
||||
|
|
@ -49,11 +51,16 @@ async function main(): Promise<void> {
|
|||
}
|
||||
|
||||
if (req.command === "generate") {
|
||||
writeResult(generateCode(req.workflow, req.specs, { workflowName: req.workflowName }));
|
||||
writeResult(
|
||||
generateCode(req.workflow, req.specs, {
|
||||
workflowName: req.workflowName,
|
||||
edgeFieldNames: req.edgeFieldNames,
|
||||
}),
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (req.command === "parse") {
|
||||
writeResult(parseCode(req.code, req.specs));
|
||||
writeResult(parseCode(req.code, req.specs, req.edgeFieldNames));
|
||||
return;
|
||||
}
|
||||
writeResult({
|
||||
|
|
|
|||
|
|
@ -25,8 +25,19 @@ import type {
|
|||
WireNode,
|
||||
} from "./types.ts";
|
||||
|
||||
export function parseCode(code: string, specs: NodeSpec[]): ParseResult {
|
||||
export function parseCode(
|
||||
code: string,
|
||||
specs: NodeSpec[],
|
||||
edgeFieldNames: string[] = [
|
||||
"label",
|
||||
"condition",
|
||||
"transition_speech",
|
||||
"transition_speech_type",
|
||||
"transition_speech_recording_id",
|
||||
],
|
||||
): ParseResult {
|
||||
const specByName = new Map(specs.map((s) => [s.name, s]));
|
||||
const allowedEdgeFieldNames = new Set(edgeFieldNames);
|
||||
const sourceFile = ts.createSourceFile(
|
||||
"workflow.ts",
|
||||
code,
|
||||
|
|
@ -335,6 +346,12 @@ export function parseCode(code: string, specs: NodeSpec[]): ParseResult {
|
|||
addError(stmt, "`edge` requires a non-empty `condition` string.");
|
||||
return;
|
||||
}
|
||||
for (const key of Object.keys(optsObj)) {
|
||||
if (!allowedEdgeFieldNames.has(key)) {
|
||||
addError(stmt, `Unknown edge field: \`${key}\`.`);
|
||||
return;
|
||||
}
|
||||
}
|
||||
edges.push({
|
||||
id: `${src.id}-${tgt.id}`,
|
||||
source: src.id,
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
[project]
|
||||
name = "dograh-api"
|
||||
version = "1.29.0"
|
||||
version = "1.31.0"
|
||||
description = "Backend API for Dograh voice AI platform"
|
||||
requires-python = ">=3.12"
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ arq==0.26.3
|
|||
twilio==9.8.0
|
||||
minio==7.2.16
|
||||
alembic-postgresql-enum==1.8.0
|
||||
python-multipart==0.0.20
|
||||
python-multipart==0.0.27
|
||||
sentry-sdk[fastapi]==2.38.0
|
||||
sqlalchemy[asyncio]==2.0.43
|
||||
msgpack==1.1.2
|
||||
|
|
@ -18,4 +18,5 @@ bcrypt==5.0.0
|
|||
email-validator==2.3.0
|
||||
posthog==7.11.1
|
||||
fastmcp==3.2.4
|
||||
tuner-pipecat-sdk==0.2.0
|
||||
PyNaCl==1.6.2
|
||||
|
|
|
|||
|
|
@ -152,8 +152,8 @@ class CircuitBreakerConfigResponse(BaseModel):
|
|||
class CreateCampaignRequest(BaseModel):
|
||||
name: str = Field(..., min_length=1, max_length=255)
|
||||
workflow_id: int
|
||||
source_type: str = Field(..., pattern="^(google-sheet|csv)$")
|
||||
source_id: str # Google Sheet URL or CSV file key
|
||||
source_type: str = Field(..., pattern="^csv$")
|
||||
source_id: str # CSV file key
|
||||
# Optional during the legacy → multi-config migration window. Required in
|
||||
# a follow-up. When omitted, the dispatcher falls back to the org's
|
||||
# default config.
|
||||
|
|
@ -929,8 +929,6 @@ async def get_campaign_source_download_url(
|
|||
user: UserModel = Depends(get_user),
|
||||
) -> CampaignSourceDownloadResponse:
|
||||
"""Get presigned download URL for campaign CSV source file
|
||||
|
||||
Only works for CSV source type. For Google Sheets, use the source_id directly.
|
||||
Validates that the campaign belongs to the user's organization for security.
|
||||
"""
|
||||
# Verify campaign exists and belongs to organization
|
||||
|
|
|
|||
99
api/routes/folder.py
Normal file
99
api/routes/folder.py
Normal file
|
|
@ -0,0 +1,99 @@
|
|||
from datetime import datetime
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
|
||||
from api.db import db_client
|
||||
from api.db.folder_client import FolderNameConflictError
|
||||
from api.db.models import UserModel
|
||||
from api.services.auth.depends import get_user
|
||||
|
||||
router = APIRouter(prefix="/folder")
|
||||
|
||||
|
||||
class FolderResponse(BaseModel):
|
||||
id: int
|
||||
name: str
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class CreateFolderRequest(BaseModel):
|
||||
name: str = Field(..., min_length=1, max_length=100)
|
||||
|
||||
@field_validator("name")
|
||||
@classmethod
|
||||
def strip_name(cls, v: str) -> str:
|
||||
v = v.strip()
|
||||
if not v:
|
||||
raise ValueError("Folder name cannot be empty")
|
||||
return v
|
||||
|
||||
|
||||
class UpdateFolderRequest(CreateFolderRequest):
|
||||
pass
|
||||
|
||||
|
||||
@router.get("/")
|
||||
async def list_folders(
|
||||
user: UserModel = Depends(get_user),
|
||||
) -> list[FolderResponse]:
|
||||
"""List all folders in the authenticated user's organization."""
|
||||
folders = await db_client.list_folders(
|
||||
organization_id=user.selected_organization_id
|
||||
)
|
||||
return [
|
||||
FolderResponse(id=f.id, name=f.name, created_at=f.created_at) for f in folders
|
||||
]
|
||||
|
||||
|
||||
@router.post("/")
|
||||
async def create_folder(
|
||||
request: CreateFolderRequest,
|
||||
user: UserModel = Depends(get_user),
|
||||
) -> FolderResponse:
|
||||
"""Create a new folder in the authenticated user's organization."""
|
||||
try:
|
||||
folder = await db_client.create_folder(
|
||||
name=request.name,
|
||||
organization_id=user.selected_organization_id,
|
||||
)
|
||||
except FolderNameConflictError as e:
|
||||
raise HTTPException(status_code=409, detail=str(e))
|
||||
return FolderResponse(id=folder.id, name=folder.name, created_at=folder.created_at)
|
||||
|
||||
|
||||
@router.put("/{folder_id}")
|
||||
async def rename_folder(
|
||||
folder_id: int,
|
||||
request: UpdateFolderRequest,
|
||||
user: UserModel = Depends(get_user),
|
||||
) -> FolderResponse:
|
||||
"""Rename a folder owned by the authenticated user's organization."""
|
||||
try:
|
||||
folder = await db_client.rename_folder(
|
||||
folder_id=folder_id,
|
||||
name=request.name,
|
||||
organization_id=user.selected_organization_id,
|
||||
)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=404, detail=str(e))
|
||||
except FolderNameConflictError as e:
|
||||
raise HTTPException(status_code=409, detail=str(e))
|
||||
return FolderResponse(id=folder.id, name=folder.name, created_at=folder.created_at)
|
||||
|
||||
|
||||
@router.delete("/{folder_id}")
|
||||
async def delete_folder(
|
||||
folder_id: int,
|
||||
user: UserModel = Depends(get_user),
|
||||
) -> dict[str, bool]:
|
||||
"""Delete a folder. Member agents are moved to "Uncategorized", not deleted."""
|
||||
deleted = await db_client.delete_folder(
|
||||
folder_id=folder_id,
|
||||
organization_id=user.selected_organization_id,
|
||||
)
|
||||
if not deleted:
|
||||
raise HTTPException(
|
||||
status_code=404, detail=f"Folder with id {folder_id} not found"
|
||||
)
|
||||
return {"success": True}
|
||||
|
|
@ -1,266 +0,0 @@
|
|||
"""
|
||||
Route for 3rd party integrations. Currently being backed by nango.
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Dict, List, Optional, TypedDict
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from loguru import logger
|
||||
from pydantic import BaseModel
|
||||
|
||||
from api.db import db_client
|
||||
from api.db.models import UserModel
|
||||
from api.services.auth.depends import get_user
|
||||
from api.services.integrations.nango import nango_service
|
||||
|
||||
router = APIRouter(prefix="/integration")
|
||||
|
||||
|
||||
@dataclass
|
||||
class IntegrationResponse:
|
||||
id: int
|
||||
integration_id: str
|
||||
organisation_id: int
|
||||
created_by: Optional[int]
|
||||
provider: str
|
||||
is_active: bool
|
||||
created_at: str
|
||||
action: str
|
||||
provider_data: dict
|
||||
|
||||
|
||||
class SessionResponse(TypedDict):
|
||||
session_token: str
|
||||
expires_at: str
|
||||
|
||||
|
||||
class WebhookResponse(TypedDict):
|
||||
status: str
|
||||
message: str
|
||||
|
||||
|
||||
class UpdateIntegrationRequest(BaseModel):
|
||||
selected_files: List[Dict[str, Any]]
|
||||
|
||||
|
||||
class AccessTokenResponse(BaseModel):
|
||||
access_token: Optional[str]
|
||||
refresh_token: Optional[str]
|
||||
expires_at: Optional[str]
|
||||
connection_id: str
|
||||
|
||||
|
||||
def build_integration_response(integration) -> IntegrationResponse:
|
||||
"""Build a standardized integration response with provider-specific data."""
|
||||
provider_data = {}
|
||||
|
||||
if integration.provider == "google-sheet":
|
||||
# For Google Sheets, include selected_files
|
||||
provider_data["selected_files"] = integration.connection_details.get(
|
||||
"selected_files", []
|
||||
)
|
||||
elif integration.provider == "slack":
|
||||
# For Slack, include channel information
|
||||
channel = integration.connection_details.get("connection_config", {}).get(
|
||||
"incoming_webhook.channel"
|
||||
)
|
||||
if channel:
|
||||
provider_data["channel"] = channel
|
||||
|
||||
return IntegrationResponse(
|
||||
id=integration.id,
|
||||
integration_id=integration.integration_id,
|
||||
organisation_id=integration.organisation_id,
|
||||
created_by=integration.created_by,
|
||||
provider=integration.provider,
|
||||
is_active=integration.is_active,
|
||||
created_at=integration.created_at.isoformat(),
|
||||
action=integration.action,
|
||||
provider_data=provider_data,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/")
|
||||
async def get_integrations(
|
||||
user: UserModel = Depends(get_user),
|
||||
) -> list[IntegrationResponse]:
|
||||
"""
|
||||
Get all integrations for the user's selected organization.
|
||||
|
||||
Returns:
|
||||
List of integrations associated with the user's selected organization
|
||||
"""
|
||||
if not user.selected_organization_id:
|
||||
raise HTTPException(
|
||||
status_code=400, detail="No organization selected for the user"
|
||||
)
|
||||
|
||||
integrations = await db_client.get_integrations_by_organization_id(
|
||||
user.selected_organization_id
|
||||
)
|
||||
|
||||
return [build_integration_response(integration) for integration in integrations]
|
||||
|
||||
|
||||
@router.post("/session")
|
||||
async def create_session(
|
||||
user: UserModel = Depends(get_user),
|
||||
) -> SessionResponse:
|
||||
"""
|
||||
Create a Nango session for the user's selected organization.
|
||||
|
||||
Returns:
|
||||
Session token and ID for the created session
|
||||
"""
|
||||
if not user.selected_organization_id:
|
||||
raise HTTPException(
|
||||
status_code=400, detail="No organization selected for the user"
|
||||
)
|
||||
|
||||
try:
|
||||
session_data = await nango_service.create_session(
|
||||
user_id=str(user.id), organization_id=user.selected_organization_id
|
||||
)
|
||||
|
||||
return {
|
||||
"session_token": session_data["data"]["token"],
|
||||
"expires_at": session_data["data"]["expires_at"],
|
||||
}
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
except Exception as e:
|
||||
raise HTTPException(
|
||||
status_code=500, detail=f"Failed to create session: {str(e)}"
|
||||
)
|
||||
|
||||
|
||||
@router.put("/{integration_id}")
|
||||
async def update_integration(
|
||||
integration_id: int,
|
||||
request: UpdateIntegrationRequest,
|
||||
user: UserModel = Depends(get_user),
|
||||
) -> IntegrationResponse:
|
||||
"""
|
||||
Update an integration's selected files (for Google Sheets).
|
||||
|
||||
Args:
|
||||
integration_id: The ID of the integration to update
|
||||
request: The update request containing selected files
|
||||
user: The authenticated user
|
||||
|
||||
Returns:
|
||||
Updated integration details
|
||||
"""
|
||||
if not user.selected_organization_id:
|
||||
raise HTTPException(
|
||||
status_code=400, detail="No organization selected for the user"
|
||||
)
|
||||
|
||||
# Get the integration first to verify ownership
|
||||
integrations = await db_client.get_integrations_by_organization_id(
|
||||
user.selected_organization_id
|
||||
)
|
||||
|
||||
integration = next((i for i in integrations if i.id == integration_id), None)
|
||||
if not integration:
|
||||
raise HTTPException(status_code=404, detail="Integration not found")
|
||||
|
||||
# Only allow updating selected_files for google-sheet provider
|
||||
if integration.provider != "google-sheet":
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="This endpoint only supports updating Google Sheet integrations",
|
||||
)
|
||||
|
||||
# Update the connection_details with the new selected_files
|
||||
updated_connection_details = integration.connection_details.copy()
|
||||
updated_connection_details["selected_files"] = request.selected_files
|
||||
|
||||
# Update the integration
|
||||
updated_integration = await db_client.update_integration_connection_details(
|
||||
integration_id=integration_id, connection_details=updated_connection_details
|
||||
)
|
||||
|
||||
if not updated_integration:
|
||||
raise HTTPException(status_code=500, detail="Failed to update integration")
|
||||
|
||||
return build_integration_response(updated_integration)
|
||||
|
||||
|
||||
@router.get("/{integration_id}/access-token")
|
||||
async def get_integration_access_token(
|
||||
integration_id: int,
|
||||
user: UserModel = Depends(get_user),
|
||||
) -> AccessTokenResponse:
|
||||
"""
|
||||
Get the latest access token for an integration from Nango.
|
||||
|
||||
Args:
|
||||
integration_id: The ID of the integration
|
||||
user: The authenticated user
|
||||
|
||||
Returns:
|
||||
Dict containing access token and expiration info
|
||||
"""
|
||||
if not user.selected_organization_id:
|
||||
raise HTTPException(
|
||||
status_code=400, detail="No organization selected for the user"
|
||||
)
|
||||
|
||||
# Get the integration to verify ownership and get connection details
|
||||
integrations = await db_client.get_integrations_by_organization_id(
|
||||
user.selected_organization_id
|
||||
)
|
||||
|
||||
integration = next((i for i in integrations if i.id == integration_id), None)
|
||||
if not integration:
|
||||
raise HTTPException(status_code=404, detail="Integration not found")
|
||||
|
||||
try:
|
||||
# Fetch the latest access token from Nango
|
||||
token_data = await nango_service.get_access_token(
|
||||
connection_id=integration.integration_id,
|
||||
provider_config_key=integration.provider,
|
||||
)
|
||||
|
||||
# Extract relevant fields
|
||||
return AccessTokenResponse(
|
||||
access_token=token_data.get("credentials", {}).get("access_token"),
|
||||
refresh_token=token_data.get("credentials", {}).get("refresh_token"),
|
||||
expires_at=token_data.get("credentials", {}).get("expires_at"),
|
||||
connection_id=integration.integration_id,
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to get access token: {str(e)}")
|
||||
raise HTTPException(
|
||||
status_code=500, detail=f"Failed to fetch access token: {str(e)}"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/webhook", include_in_schema=False)
|
||||
async def handle_nango_webhook(
|
||||
request: Request,
|
||||
) -> WebhookResponse:
|
||||
"""
|
||||
Handle Nango integration webhook requests.
|
||||
|
||||
Processes webhook events from Nango when integrations are created/updated
|
||||
and stores the integration details in the database.
|
||||
|
||||
Args:
|
||||
request: The raw FastAPI request object
|
||||
|
||||
Returns:
|
||||
WebhookResponse with status and message
|
||||
"""
|
||||
raw_body = await request.body()
|
||||
|
||||
# Get signature from headers (you may need to adjust the header name)
|
||||
signature = request.headers.get("X-Nango-Signature")
|
||||
|
||||
# Use the nango service to process the webhook
|
||||
result = await nango_service.process_webhook(raw_body, signature)
|
||||
|
||||
return result
|
||||
|
|
@ -1,316 +0,0 @@
|
|||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from fastapi import (
|
||||
APIRouter,
|
||||
BackgroundTasks,
|
||||
Depends,
|
||||
HTTPException,
|
||||
WebSocket,
|
||||
)
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from api.db import db_client
|
||||
from api.db.models import UserModel
|
||||
from api.services.auth.depends import get_user
|
||||
from api.services.looptalk.orchestrator import LoopTalkTestOrchestrator
|
||||
|
||||
router = APIRouter(prefix="/looptalk")
|
||||
|
||||
|
||||
# Request/Response Models
|
||||
class CreateTestSessionRequest(BaseModel):
|
||||
name: str
|
||||
actor_workflow_id: int
|
||||
adversary_workflow_id: int
|
||||
config: Dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class StartTestSessionRequest(BaseModel):
|
||||
test_session_id: int
|
||||
|
||||
|
||||
class CreateLoadTestRequest(BaseModel):
|
||||
name_prefix: str
|
||||
actor_workflow_id: int
|
||||
adversary_workflow_id: int
|
||||
test_count: int = Field(ge=1, le=10)
|
||||
config: Dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class TestSessionResponse(BaseModel):
|
||||
id: int
|
||||
name: str
|
||||
status: str
|
||||
actor_workflow_id: int
|
||||
adversary_workflow_id: int
|
||||
load_test_group_id: Optional[str]
|
||||
test_index: Optional[int]
|
||||
config: Dict[str, Any]
|
||||
results: Optional[Dict[str, Any]]
|
||||
error: Optional[str]
|
||||
created_at: datetime
|
||||
started_at: Optional[datetime]
|
||||
completed_at: Optional[datetime]
|
||||
|
||||
|
||||
class ConversationResponse(BaseModel):
|
||||
id: int
|
||||
test_session_id: int
|
||||
duration_seconds: Optional[int]
|
||||
actor_recording_url: Optional[str]
|
||||
adversary_recording_url: Optional[str]
|
||||
combined_recording_url: Optional[str]
|
||||
transcript: Optional[Dict[str, Any]]
|
||||
metrics: Optional[Dict[str, Any]]
|
||||
created_at: datetime
|
||||
ended_at: Optional[datetime]
|
||||
|
||||
|
||||
# Note: Turn tracking is handled by Langfuse, not exposed via API
|
||||
|
||||
|
||||
class LoadTestStatsResponse(BaseModel):
|
||||
total: int
|
||||
pending: int
|
||||
running: int
|
||||
completed: int
|
||||
failed: int
|
||||
sessions: List[Dict[str, Any]]
|
||||
|
||||
|
||||
# Singleton orchestrator instance
|
||||
_orchestrator: Optional[LoopTalkTestOrchestrator] = None
|
||||
|
||||
|
||||
def get_orchestrator() -> LoopTalkTestOrchestrator:
|
||||
"""Get or create the LoopTalk orchestrator instance."""
|
||||
global _orchestrator
|
||||
if _orchestrator is None:
|
||||
_orchestrator = LoopTalkTestOrchestrator(db_client=db_client)
|
||||
return _orchestrator
|
||||
|
||||
|
||||
@router.post("/test-sessions", response_model=TestSessionResponse)
|
||||
async def create_test_session(
|
||||
request: CreateTestSessionRequest, user: UserModel = Depends(get_user)
|
||||
):
|
||||
"""Create a new LoopTalk test session."""
|
||||
|
||||
# Verify user has access to both workflows
|
||||
actor_workflow = await db_client.get_workflow(request.actor_workflow_id, user.id)
|
||||
if not actor_workflow:
|
||||
raise HTTPException(status_code=404, detail="Actor workflow not found")
|
||||
|
||||
adversary_workflow = await db_client.get_workflow(
|
||||
request.adversary_workflow_id, user.id
|
||||
)
|
||||
if not adversary_workflow:
|
||||
raise HTTPException(status_code=404, detail="Adversary workflow not found")
|
||||
|
||||
# Create test session
|
||||
test_session = await db_client.create_test_session(
|
||||
organization_id=user.selected_organization_id,
|
||||
name=request.name,
|
||||
actor_workflow_id=request.actor_workflow_id,
|
||||
adversary_workflow_id=request.adversary_workflow_id,
|
||||
config=request.config,
|
||||
)
|
||||
|
||||
return test_session
|
||||
|
||||
|
||||
@router.get("/test-sessions", response_model=List[TestSessionResponse])
|
||||
async def list_test_sessions(
|
||||
status: Optional[str] = None,
|
||||
load_test_group_id: Optional[str] = None,
|
||||
limit: int = 20,
|
||||
offset: int = 0,
|
||||
user: UserModel = Depends(get_user),
|
||||
):
|
||||
"""List LoopTalk test sessions."""
|
||||
|
||||
test_sessions = await db_client.list_test_sessions(
|
||||
organization_id=user.selected_organization_id,
|
||||
status=status,
|
||||
load_test_group_id=load_test_group_id,
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
)
|
||||
|
||||
return test_sessions
|
||||
|
||||
|
||||
@router.get("/test-sessions/{test_session_id}", response_model=TestSessionResponse)
|
||||
async def get_test_session(test_session_id: int, user: UserModel = Depends(get_user)):
|
||||
"""Get a specific test session."""
|
||||
|
||||
test_session = await db_client.get_test_session(
|
||||
test_session_id=test_session_id, organization_id=user.selected_organization_id
|
||||
)
|
||||
|
||||
if not test_session:
|
||||
raise HTTPException(status_code=404, detail="Test session not found")
|
||||
|
||||
return test_session
|
||||
|
||||
|
||||
@router.post("/test-sessions/{test_session_id}/start")
|
||||
async def start_test_session(
|
||||
test_session_id: int,
|
||||
background_tasks: BackgroundTasks,
|
||||
user: UserModel = Depends(get_user),
|
||||
orchestrator: LoopTalkTestOrchestrator = Depends(get_orchestrator),
|
||||
):
|
||||
"""Start a LoopTalk test session."""
|
||||
|
||||
# Verify test session exists and user has access
|
||||
test_session = await db_client.get_test_session(
|
||||
test_session_id=test_session_id, organization_id=user.selected_organization_id
|
||||
)
|
||||
|
||||
if not test_session:
|
||||
raise HTTPException(status_code=404, detail="Test session not found")
|
||||
|
||||
if test_session.status != "pending":
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Test session is {test_session.status}, not pending",
|
||||
)
|
||||
|
||||
# Start test session in background
|
||||
background_tasks.add_task(
|
||||
orchestrator.start_test_session,
|
||||
test_session_id=test_session_id,
|
||||
organization_id=user.selected_organization_id,
|
||||
)
|
||||
|
||||
return {"message": "Test session starting", "test_session_id": test_session_id}
|
||||
|
||||
|
||||
@router.post("/test-sessions/{test_session_id}/stop")
|
||||
async def stop_test_session(
|
||||
test_session_id: int,
|
||||
user: UserModel = Depends(get_user),
|
||||
orchestrator: LoopTalkTestOrchestrator = Depends(get_orchestrator),
|
||||
):
|
||||
"""Stop a running test session."""
|
||||
|
||||
# Verify test session exists and user has access
|
||||
test_session = await db_client.get_test_session(
|
||||
test_session_id=test_session_id, organization_id=user.selected_organization_id
|
||||
)
|
||||
|
||||
if not test_session:
|
||||
raise HTTPException(status_code=404, detail="Test session not found")
|
||||
|
||||
if test_session.status != "running":
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Test session is {test_session.status}, not running",
|
||||
)
|
||||
|
||||
# Stop test session
|
||||
result = await orchestrator.stop_test_session(test_session_id=test_session_id)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
@router.get("/test-sessions/{test_session_id}/conversation")
|
||||
async def get_test_session_conversation(
|
||||
test_session_id: int, user: UserModel = Depends(get_user)
|
||||
):
|
||||
"""Get conversation details for a test session."""
|
||||
|
||||
# Verify test session exists and user has access
|
||||
test_session = await db_client.get_test_session(
|
||||
test_session_id=test_session_id, organization_id=user.selected_organization_id
|
||||
)
|
||||
|
||||
if not test_session:
|
||||
raise HTTPException(status_code=404, detail="Test session not found")
|
||||
|
||||
# Get conversation
|
||||
if test_session.conversations:
|
||||
conversation = test_session.conversations[
|
||||
0
|
||||
] # For now, one conversation per session
|
||||
|
||||
# Note: Turn details are available in Langfuse, not here
|
||||
return {
|
||||
"conversation": conversation,
|
||||
"message": "Turn details are tracked in Langfuse",
|
||||
}
|
||||
|
||||
return {"conversation": None}
|
||||
|
||||
|
||||
@router.post("/load-tests", response_model=Dict[str, Any])
|
||||
async def create_load_test(
|
||||
request: CreateLoadTestRequest,
|
||||
background_tasks: BackgroundTasks,
|
||||
user: UserModel = Depends(get_user),
|
||||
orchestrator: LoopTalkTestOrchestrator = Depends(get_orchestrator),
|
||||
):
|
||||
"""Create and start a load test."""
|
||||
|
||||
# Verify user has access to both workflows
|
||||
actor_workflow = await db_client.get_workflow(request.actor_workflow_id, user.id)
|
||||
if not actor_workflow:
|
||||
raise HTTPException(status_code=404, detail="Actor workflow not found")
|
||||
|
||||
adversary_workflow = await db_client.get_workflow(
|
||||
request.adversary_workflow_id, user.id
|
||||
)
|
||||
if not adversary_workflow:
|
||||
raise HTTPException(status_code=404, detail="Adversary workflow not found")
|
||||
|
||||
# Start load test in background
|
||||
result = await orchestrator.start_load_test(
|
||||
organization_id=user.selected_organization_id,
|
||||
name_prefix=request.name_prefix,
|
||||
actor_workflow_id=request.actor_workflow_id,
|
||||
adversary_workflow_id=request.adversary_workflow_id,
|
||||
config=request.config,
|
||||
test_count=request.test_count,
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
@router.get(
|
||||
"/load-tests/{load_test_group_id}/stats", response_model=LoadTestStatsResponse
|
||||
)
|
||||
async def get_load_test_stats(
|
||||
load_test_group_id: str, user: UserModel = Depends(get_user)
|
||||
):
|
||||
"""Get statistics for a load test group."""
|
||||
|
||||
stats = await db_client.get_load_test_group_stats(
|
||||
load_test_group_id=load_test_group_id,
|
||||
organization_id=user.selected_organization_id,
|
||||
)
|
||||
|
||||
return stats
|
||||
|
||||
|
||||
@router.get("/active-tests")
|
||||
async def get_active_tests(
|
||||
orchestrator: LoopTalkTestOrchestrator = Depends(get_orchestrator),
|
||||
user: UserModel = Depends(get_user),
|
||||
):
|
||||
"""Get information about currently active test sessions."""
|
||||
|
||||
return orchestrator.get_active_test_info()
|
||||
|
||||
|
||||
@router.websocket("/test-sessions/{test_session_id}/audio-stream")
|
||||
async def audio_stream_websocket(
|
||||
websocket: WebSocket,
|
||||
test_session_id: int,
|
||||
role: str = "mixed", # "actor", "adversary", or "mixed"
|
||||
token: Optional[str] = None,
|
||||
):
|
||||
"""WebSocket endpoint for real-time audio streaming from LoopTalk test sessions."""
|
||||
# TODO: to be implemented
|
||||
pass
|
||||
|
|
@ -6,9 +6,8 @@ from api.routes.agent_stream import router as agent_stream_router
|
|||
from api.routes.auth import router as auth_router
|
||||
from api.routes.campaign import router as campaign_router
|
||||
from api.routes.credentials import router as credentials_router
|
||||
from api.routes.integration import router as integration_router
|
||||
from api.routes.folder import router as folder_router
|
||||
from api.routes.knowledge_base import router as knowledge_base_router
|
||||
from api.routes.looptalk import router as looptalk_router
|
||||
from api.routes.node_types import router as node_types_router
|
||||
from api.routes.organization import router as organization_router
|
||||
from api.routes.organization_usage import router as organization_usage_router
|
||||
|
|
@ -27,6 +26,8 @@ from api.routes.webrtc_signaling import router as webrtc_signaling_router
|
|||
from api.routes.workflow import router as workflow_router
|
||||
from api.routes.workflow_embed import router as workflow_embed_router
|
||||
from api.routes.workflow_recording import router as workflow_recording_router
|
||||
from api.routes.workflow_text_chat import router as workflow_text_chat_router
|
||||
from api.services.integrations import all_routers
|
||||
|
||||
router = APIRouter(
|
||||
tags=["main"],
|
||||
|
|
@ -36,15 +37,14 @@ router = APIRouter(
|
|||
router.include_router(telephony_router)
|
||||
router.include_router(superuser_router)
|
||||
router.include_router(workflow_router)
|
||||
router.include_router(workflow_text_chat_router)
|
||||
router.include_router(user_router)
|
||||
router.include_router(campaign_router)
|
||||
router.include_router(credentials_router)
|
||||
router.include_router(tool_router)
|
||||
router.include_router(integration_router)
|
||||
router.include_router(organization_router)
|
||||
router.include_router(s3_router)
|
||||
router.include_router(service_keys_router)
|
||||
router.include_router(looptalk_router)
|
||||
router.include_router(organization_usage_router)
|
||||
router.include_router(reports_router)
|
||||
router.include_router(webrtc_signaling_router)
|
||||
|
|
@ -55,10 +55,14 @@ router.include_router(public_download_router)
|
|||
router.include_router(workflow_embed_router)
|
||||
router.include_router(knowledge_base_router)
|
||||
router.include_router(workflow_recording_router)
|
||||
router.include_router(folder_router)
|
||||
router.include_router(auth_router)
|
||||
router.include_router(node_types_router)
|
||||
router.include_router(agent_stream_router)
|
||||
|
||||
for _integration_router in all_routers():
|
||||
router.include_router(_integration_router)
|
||||
|
||||
|
||||
class HealthResponse(BaseModel):
|
||||
status: str
|
||||
|
|
|
|||
|
|
@ -57,6 +57,7 @@ class WorkflowRunUsageResponse(BaseModel):
|
|||
caller_number: Optional[str] = None
|
||||
called_number: Optional[str] = None
|
||||
call_type: Optional[str] = None
|
||||
mode: Optional[str] = None
|
||||
disposition: Optional[str] = None
|
||||
initial_context: Optional[Dict[str, Any]] = None
|
||||
gathered_context: Optional[Dict[str, Any]] = None
|
||||
|
|
|
|||
|
|
@ -1,18 +1,19 @@
|
|||
"""Public API endpoints for agent triggers.
|
||||
"""Public API endpoints for public agent execution.
|
||||
|
||||
These endpoints are accessible with API key authentication and allow
|
||||
external systems to programmatically trigger phone calls.
|
||||
"""
|
||||
|
||||
import random
|
||||
from typing import Optional
|
||||
from dataclasses import dataclass
|
||||
from typing import Awaitable, Callable, Optional
|
||||
|
||||
from fastapi import APIRouter, Header, HTTPException
|
||||
from loguru import logger
|
||||
from pydantic import BaseModel
|
||||
|
||||
from api.db import db_client
|
||||
from api.enums import TriggerState
|
||||
from api.enums import TriggerState, WorkflowStatus
|
||||
from api.services.quota_service import check_dograh_quota_by_user_id
|
||||
from api.services.telephony.factory import (
|
||||
get_default_telephony_provider,
|
||||
|
|
@ -39,6 +40,14 @@ class TriggerCallResponse(BaseModel):
|
|||
workflow_run_name: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class ResolvedAgentTarget:
|
||||
workflow: object
|
||||
organization_id: int
|
||||
identifier_type: str
|
||||
identifier_value: str
|
||||
|
||||
|
||||
def trigger_exists_in_workflow(workflow_definition: dict, trigger_path: str) -> bool:
|
||||
"""Check if trigger node exists in workflow definition.
|
||||
|
||||
|
|
@ -57,72 +66,133 @@ def trigger_exists_in_workflow(workflow_definition: dict, trigger_path: str) ->
|
|||
return False
|
||||
|
||||
|
||||
async def _initiate_call(
|
||||
uuid: str,
|
||||
request: TriggerCallRequest,
|
||||
x_api_key: str,
|
||||
*,
|
||||
use_draft: bool,
|
||||
) -> TriggerCallResponse:
|
||||
"""Shared core for production and test trigger endpoints.
|
||||
|
||||
When ``use_draft`` is True the latest draft definition is executed;
|
||||
otherwise the published (released) definition is used.
|
||||
"""
|
||||
# 1. Validate API key
|
||||
async def _validate_api_key(x_api_key: str):
|
||||
"""Validate the org API key used to invoke a public agent endpoint."""
|
||||
api_key = await db_client.validate_api_key(x_api_key)
|
||||
if not api_key:
|
||||
raise HTTPException(status_code=401, detail="Invalid API key")
|
||||
return api_key
|
||||
|
||||
# 2. Lookup agent trigger by UUID
|
||||
trigger = await db_client.get_agent_trigger_by_path(uuid)
|
||||
|
||||
def _ensure_workflow_is_active(workflow) -> None:
|
||||
if workflow.status != WorkflowStatus.ACTIVE.value:
|
||||
raise HTTPException(status_code=404, detail="Workflow is not active")
|
||||
|
||||
|
||||
def _get_execution_user_id(workflow) -> int:
|
||||
if workflow.user_id is None:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail="Workflow has no execution owner",
|
||||
)
|
||||
return workflow.user_id
|
||||
|
||||
|
||||
async def _get_workflow_definition_for_execution(workflow, *, use_draft: bool) -> dict:
|
||||
"""Return the definition that would execute for this public agent request."""
|
||||
if use_draft:
|
||||
draft = await db_client.get_draft_version(workflow.id)
|
||||
if draft:
|
||||
return draft.workflow_json
|
||||
|
||||
if workflow.released_definition is None:
|
||||
raise HTTPException(
|
||||
status_code=404, detail="Workflow has no published definition"
|
||||
)
|
||||
|
||||
return workflow.released_definition.workflow_json
|
||||
|
||||
|
||||
async def _resolve_trigger_target(
|
||||
trigger_path: str,
|
||||
organization_id: int,
|
||||
*,
|
||||
use_draft: bool,
|
||||
) -> ResolvedAgentTarget:
|
||||
"""Resolve a trigger UUID to a workflow, scoped to the API key's org."""
|
||||
trigger = await db_client.get_agent_trigger_by_path(trigger_path)
|
||||
if not trigger:
|
||||
raise HTTPException(status_code=404, detail="Agent trigger not found")
|
||||
|
||||
# 3. Validate organization match (API key org must match trigger org)
|
||||
if api_key.organization_id != trigger.organization_id:
|
||||
if organization_id != trigger.organization_id:
|
||||
raise HTTPException(status_code=403, detail="Access denied")
|
||||
|
||||
# 4. Validate trigger is active
|
||||
if trigger.state != TriggerState.ACTIVE.value:
|
||||
raise HTTPException(status_code=404, detail="Agent trigger is not active")
|
||||
|
||||
# 4.5 Check Dograh quota before initiating the call (apply the trigger's
|
||||
# workflow's model_overrides so we evaluate the keys this run will use).
|
||||
workflow = await db_client.get_workflow(
|
||||
trigger.workflow_id,
|
||||
organization_id=organization_id,
|
||||
)
|
||||
if not workflow:
|
||||
raise HTTPException(status_code=404, detail="Workflow not found")
|
||||
|
||||
_ensure_workflow_is_active(workflow)
|
||||
workflow_definition = await _get_workflow_definition_for_execution(
|
||||
workflow,
|
||||
use_draft=use_draft,
|
||||
)
|
||||
if not trigger_exists_in_workflow(workflow_definition, trigger_path):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail="Trigger not found in the selected Agent",
|
||||
)
|
||||
|
||||
return ResolvedAgentTarget(
|
||||
workflow=workflow,
|
||||
organization_id=organization_id,
|
||||
identifier_type="trigger_path",
|
||||
identifier_value=trigger_path,
|
||||
)
|
||||
|
||||
|
||||
async def _resolve_workflow_uuid_target(
|
||||
workflow_uuid: str,
|
||||
organization_id: int,
|
||||
*,
|
||||
use_draft: bool,
|
||||
) -> ResolvedAgentTarget:
|
||||
"""Resolve a workflow UUID directly, scoped to the API key's org."""
|
||||
workflow = await db_client.get_workflow_by_uuid(workflow_uuid, organization_id)
|
||||
if not workflow:
|
||||
raise HTTPException(status_code=404, detail="Workflow not found")
|
||||
|
||||
_ensure_workflow_is_active(workflow)
|
||||
await _get_workflow_definition_for_execution(workflow, use_draft=use_draft)
|
||||
|
||||
return ResolvedAgentTarget(
|
||||
workflow=workflow,
|
||||
organization_id=organization_id,
|
||||
identifier_type="workflow_uuid",
|
||||
identifier_value=workflow_uuid,
|
||||
)
|
||||
|
||||
|
||||
async def _execute_resolved_target(
|
||||
target: ResolvedAgentTarget,
|
||||
request: TriggerCallRequest,
|
||||
*,
|
||||
use_draft: bool,
|
||||
api_key_id: int | None,
|
||||
api_key_created_by: int | None,
|
||||
) -> TriggerCallResponse:
|
||||
"""Shared execution path once the target workflow has been resolved."""
|
||||
execution_user_id = _get_execution_user_id(target.workflow)
|
||||
|
||||
# Check Dograh quota using the workflow owner's config and model overrides.
|
||||
quota_result = await check_dograh_quota_by_user_id(
|
||||
api_key.created_by, workflow_id=trigger.workflow_id
|
||||
execution_user_id,
|
||||
workflow_id=target.workflow.id,
|
||||
)
|
||||
if not quota_result.has_quota:
|
||||
raise HTTPException(status_code=402, detail=quota_result.error_message)
|
||||
|
||||
# 5. Get workflow and resolve the definition (published vs draft)
|
||||
workflow = await db_client.get_workflow_by_id(trigger.workflow_id)
|
||||
if not workflow:
|
||||
raise HTTPException(status_code=404, detail="Workflow not found")
|
||||
|
||||
if use_draft:
|
||||
draft = await db_client.get_draft_version(trigger.workflow_id)
|
||||
# Fall back to the published definition when no draft exists, so the
|
||||
# test URL always runs *something* — typically the same agent the
|
||||
# production URL would run.
|
||||
workflow_definition = (
|
||||
draft.workflow_json if draft else workflow.released_definition.workflow_json
|
||||
)
|
||||
else:
|
||||
workflow_definition = workflow.released_definition.workflow_json
|
||||
|
||||
# Validate trigger node still exists in the resolved definition
|
||||
if not trigger_exists_in_workflow(workflow_definition, uuid):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail="Trigger not found in the published Agent",
|
||||
)
|
||||
|
||||
# 6. Get telephony provider — either the caller-specified config (validated
|
||||
# against the trigger's org) or the org's default config.
|
||||
# Get telephony provider — either the caller-specified config (validated
|
||||
# against the workflow's org) or the org's default config.
|
||||
if request.telephony_configuration_id is not None:
|
||||
cfg = await db_client.get_telephony_configuration_for_org(
|
||||
request.telephony_configuration_id, trigger.organization_id
|
||||
request.telephony_configuration_id,
|
||||
target.organization_id,
|
||||
)
|
||||
if not cfg:
|
||||
raise HTTPException(
|
||||
|
|
@ -130,7 +200,7 @@ async def _initiate_call(
|
|||
)
|
||||
try:
|
||||
provider = await get_telephony_provider_by_id(
|
||||
cfg.id, trigger.organization_id
|
||||
cfg.id, target.organization_id
|
||||
)
|
||||
except ValueError:
|
||||
raise HTTPException(
|
||||
|
|
@ -140,14 +210,14 @@ async def _initiate_call(
|
|||
resolved_cfg_id = cfg.id
|
||||
else:
|
||||
try:
|
||||
provider = await get_default_telephony_provider(trigger.organization_id)
|
||||
provider = await get_default_telephony_provider(target.organization_id)
|
||||
except ValueError:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="Telephony provider not configured for this organization",
|
||||
)
|
||||
default_cfg = await db_client.get_default_telephony_configuration(
|
||||
trigger.organization_id
|
||||
target.organization_id
|
||||
)
|
||||
resolved_cfg_id = default_cfg.id if default_cfg else None
|
||||
|
||||
|
|
@ -164,24 +234,36 @@ async def _initiate_call(
|
|||
# 8. Create workflow run
|
||||
mode_label = "TEST" if use_draft else "API"
|
||||
workflow_run_name = f"WR-{mode_label}-{random.randint(1000, 9999)}"
|
||||
initial_context = {
|
||||
"provider": provider.PROVIDER_NAME,
|
||||
"phone_number": request.phone_number,
|
||||
"trigger_mode": "test" if use_draft else "production",
|
||||
"telephony_configuration_id": resolved_cfg_id,
|
||||
"agent_identifier": target.identifier_value,
|
||||
"agent_identifier_type": target.identifier_type,
|
||||
"workflow_uuid": target.workflow.workflow_uuid,
|
||||
}
|
||||
if target.identifier_type == "trigger_path":
|
||||
initial_context["agent_uuid"] = target.identifier_value
|
||||
if api_key_id is not None:
|
||||
initial_context["api_key_id"] = api_key_id
|
||||
if api_key_created_by is not None:
|
||||
initial_context["api_key_created_by"] = api_key_created_by
|
||||
initial_context.update(request.initial_context or {})
|
||||
|
||||
workflow_run = await db_client.create_workflow_run(
|
||||
name=workflow_run_name,
|
||||
workflow_id=trigger.workflow_id,
|
||||
workflow_id=target.workflow.id,
|
||||
mode=workflow_run_mode,
|
||||
initial_context={
|
||||
"provider": provider.PROVIDER_NAME,
|
||||
"phone_number": request.phone_number,
|
||||
"agent_uuid": uuid,
|
||||
"trigger_mode": "test" if use_draft else "production",
|
||||
"telephony_configuration_id": resolved_cfg_id,
|
||||
**(request.initial_context or {}),
|
||||
},
|
||||
user_id=api_key.created_by,
|
||||
initial_context=initial_context,
|
||||
user_id=execution_user_id,
|
||||
use_draft=use_draft,
|
||||
organization_id=target.organization_id,
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"Created workflow run {workflow_run.id} for API trigger {uuid} "
|
||||
f"Created workflow run {workflow_run.id} for public agent "
|
||||
f"{target.identifier_type}={target.identifier_value} "
|
||||
f"(mode={'test' if use_draft else 'production'}) "
|
||||
f"to phone number {request.phone_number}"
|
||||
)
|
||||
|
|
@ -192,10 +274,10 @@ async def _initiate_call(
|
|||
|
||||
webhook_url = (
|
||||
f"{backend_endpoint}/api/v1/telephony/{webhook_endpoint}"
|
||||
f"?workflow_id={trigger.workflow_id}"
|
||||
f"&user_id={api_key.created_by}"
|
||||
f"?workflow_id={target.workflow.id}"
|
||||
f"&user_id={execution_user_id}"
|
||||
f"&workflow_run_id={workflow_run.id}"
|
||||
f"&organization_id={trigger.organization_id}"
|
||||
f"&organization_id={target.organization_id}"
|
||||
)
|
||||
|
||||
# 10. Initiate call via telephony provider. workflow_id and user_id are
|
||||
|
|
@ -207,8 +289,8 @@ async def _initiate_call(
|
|||
to_number=request.phone_number,
|
||||
webhook_url=webhook_url,
|
||||
workflow_run_id=workflow_run.id,
|
||||
workflow_id=trigger.workflow_id,
|
||||
user_id=api_key.created_by,
|
||||
workflow_id=target.workflow.id,
|
||||
user_id=execution_user_id,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
|
|
@ -221,7 +303,7 @@ async def _initiate_call(
|
|||
|
||||
logger.info(
|
||||
f"Call initiated successfully for workflow run {workflow_run.id} "
|
||||
f"via trigger {uuid}"
|
||||
f"via {target.identifier_type}={target.identifier_value}"
|
||||
)
|
||||
|
||||
return TriggerCallResponse(
|
||||
|
|
@ -231,6 +313,30 @@ async def _initiate_call(
|
|||
)
|
||||
|
||||
|
||||
async def _initiate_call(
|
||||
identifier: str,
|
||||
request: TriggerCallRequest,
|
||||
x_api_key: str,
|
||||
*,
|
||||
use_draft: bool,
|
||||
target_resolver: Callable[..., Awaitable[ResolvedAgentTarget]],
|
||||
) -> TriggerCallResponse:
|
||||
"""Resolve the requested public target, then execute the common call flow."""
|
||||
api_key = await _validate_api_key(x_api_key)
|
||||
target = await target_resolver(
|
||||
identifier,
|
||||
api_key.organization_id,
|
||||
use_draft=use_draft,
|
||||
)
|
||||
return await _execute_resolved_target(
|
||||
target,
|
||||
request,
|
||||
use_draft=use_draft,
|
||||
api_key_id=api_key.id,
|
||||
api_key_created_by=api_key.created_by,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/{uuid}", response_model=TriggerCallResponse)
|
||||
async def initiate_call(
|
||||
uuid: str,
|
||||
|
|
@ -241,7 +347,13 @@ async def initiate_call(
|
|||
|
||||
Executes the workflow's currently released definition.
|
||||
"""
|
||||
return await _initiate_call(uuid, request, x_api_key, use_draft=False)
|
||||
return await _initiate_call(
|
||||
uuid,
|
||||
request,
|
||||
x_api_key,
|
||||
use_draft=False,
|
||||
target_resolver=_resolve_trigger_target,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/test/{uuid}", response_model=TriggerCallResponse)
|
||||
|
|
@ -255,4 +367,42 @@ async def initiate_call_test(
|
|||
Useful for verifying changes before publishing. Falls back to the
|
||||
published definition when no draft exists.
|
||||
"""
|
||||
return await _initiate_call(uuid, request, x_api_key, use_draft=True)
|
||||
return await _initiate_call(
|
||||
uuid,
|
||||
request,
|
||||
x_api_key,
|
||||
use_draft=True,
|
||||
target_resolver=_resolve_trigger_target,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/workflow/{workflow_uuid}", response_model=TriggerCallResponse)
|
||||
async def initiate_call_by_workflow_uuid(
|
||||
workflow_uuid: str,
|
||||
request: TriggerCallRequest,
|
||||
x_api_key: str = Header(..., alias="X-API-Key"),
|
||||
):
|
||||
"""Initiate a phone call against the published workflow identified by UUID."""
|
||||
return await _initiate_call(
|
||||
workflow_uuid,
|
||||
request,
|
||||
x_api_key,
|
||||
use_draft=False,
|
||||
target_resolver=_resolve_workflow_uuid_target,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/test/workflow/{workflow_uuid}", response_model=TriggerCallResponse)
|
||||
async def initiate_call_test_by_workflow_uuid(
|
||||
workflow_uuid: str,
|
||||
request: TriggerCallRequest,
|
||||
x_api_key: str = Header(..., alias="X-API-Key"),
|
||||
):
|
||||
"""Initiate a phone call against the latest draft of the workflow by UUID."""
|
||||
return await _initiate_call(
|
||||
workflow_uuid,
|
||||
request,
|
||||
x_api_key,
|
||||
use_draft=True,
|
||||
target_resolver=_resolve_workflow_uuid_target,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,8 +1,9 @@
|
|||
"""Public download endpoints for workflow recordings and transcripts.
|
||||
|
||||
These endpoints provide secure, token-based public access to workflow artifacts
|
||||
without requiring authentication. Tokens are generated on-demand when webhooks
|
||||
are executed and included in the webhook payload.
|
||||
without requiring authentication. Tokens are generated on-demand during
|
||||
post-call processing for runs that execute integrations, QA, or campaign
|
||||
reporting.
|
||||
"""
|
||||
|
||||
from typing import Literal
|
||||
|
|
|
|||
|
|
@ -79,7 +79,7 @@ async def _validate_and_extract_workflow_run_id(
|
|||
|
||||
Args:
|
||||
key: S3 object key
|
||||
allow_special_paths: If True, allows looptalk/voicemail paths
|
||||
allow_special_paths: If True, allows voicemail paths
|
||||
|
||||
Returns:
|
||||
workflow_run_id if found, None for special paths (when allowed)
|
||||
|
|
@ -91,10 +91,7 @@ async def _validate_and_extract_workflow_run_id(
|
|||
run_id_str = key[len("transcripts/") : -4] # strip prefix & suffix
|
||||
elif key.startswith("recordings/") and key.endswith(".wav"):
|
||||
run_id_str = key[len("recordings/") : -4]
|
||||
elif allow_special_paths and (
|
||||
key.startswith("looptalk/") or key.startswith("voicemail_detections/")
|
||||
):
|
||||
# Allow looptalk and voicemail paths for debugging (only if explicitly allowed)
|
||||
elif allow_special_paths and key.startswith("voicemail_detections/"):
|
||||
return None # Skip validation for these paths
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail="Invalid key format")
|
||||
|
|
@ -258,7 +255,7 @@ async def get_file_metadata(
|
|||
f"METADATA: Using stored {backend} for metadata request - key: {key}"
|
||||
)
|
||||
else:
|
||||
# Fallback to current storage for legacy records or looptalk/voicemail files
|
||||
# Fallback to current storage for legacy records or voicemail files
|
||||
storage = storage_fs
|
||||
current_backend = StorageBackend.get_current_backend()
|
||||
logger.warning(
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ from api.enums import CallType, WorkflowRunState
|
|||
from api.errors.telephony_errors import TelephonyError
|
||||
from api.sdk_expose import sdk_expose
|
||||
from api.services.auth.depends import get_user
|
||||
from api.services.quota_service import check_dograh_quota, check_dograh_quota_by_user_id
|
||||
from api.services.quota_service import check_dograh_quota_by_user_id
|
||||
from api.services.telephony.call_transfer_manager import get_call_transfer_manager
|
||||
from api.services.telephony.factory import (
|
||||
get_all_telephony_providers,
|
||||
|
|
@ -60,6 +60,15 @@ class InitiateCallRequest(BaseModel):
|
|||
from_phone_number_id: int | None = None
|
||||
|
||||
|
||||
def _get_execution_user_id(workflow) -> int:
|
||||
if workflow.user_id is None:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail="Workflow has no execution owner",
|
||||
)
|
||||
return workflow.user_id
|
||||
|
||||
|
||||
@router.post(
|
||||
"/initiate-call",
|
||||
**sdk_expose(
|
||||
|
|
@ -107,15 +116,6 @@ async def initiate_call(
|
|||
detail="telephony_not_configured",
|
||||
)
|
||||
|
||||
# Check Dograh quota before initiating the call (apply per-workflow
|
||||
# model_overrides so the keys we will actually use are the ones checked).
|
||||
quota_result = await check_dograh_quota(user, workflow_id=request.workflow_id)
|
||||
if not quota_result.has_quota:
|
||||
raise HTTPException(status_code=402, detail=quota_result.error_message)
|
||||
|
||||
# Determine the workflow run mode based on provider type
|
||||
workflow_run_mode = provider.PROVIDER_NAME
|
||||
|
||||
phone_number = request.phone_number or user_configuration.test_phone_number
|
||||
|
||||
if not phone_number:
|
||||
|
|
@ -125,25 +125,38 @@ async def initiate_call(
|
|||
"configuration",
|
||||
)
|
||||
|
||||
workflow = await db_client.get_workflow(
|
||||
request.workflow_id, organization_id=user.selected_organization_id
|
||||
)
|
||||
if not workflow:
|
||||
raise HTTPException(status_code=404, detail="Workflow not found")
|
||||
execution_user_id = _get_execution_user_id(workflow)
|
||||
|
||||
# Check Dograh quota before initiating the call (apply per-workflow
|
||||
# model_overrides so the keys we will actually use are the ones checked).
|
||||
quota_result = await check_dograh_quota_by_user_id(
|
||||
execution_user_id, workflow_id=workflow.id
|
||||
)
|
||||
if not quota_result.has_quota:
|
||||
raise HTTPException(status_code=402, detail=quota_result.error_message)
|
||||
|
||||
# Determine the workflow run mode based on provider type
|
||||
workflow_run_mode = provider.PROVIDER_NAME
|
||||
|
||||
workflow_run_id = request.workflow_run_id
|
||||
|
||||
if not workflow_run_id:
|
||||
# Fetch workflow to merge template context variables (e.g. caller_number,
|
||||
# called_number set in workflow settings for testing pre-call data fetch)
|
||||
workflow = await db_client.get_workflow(
|
||||
request.workflow_id, organization_id=user.selected_organization_id
|
||||
)
|
||||
if not workflow:
|
||||
raise HTTPException(status_code=404, detail="Workflow not found")
|
||||
# Merge template context variables (e.g. caller_number, called_number
|
||||
# set in workflow settings for testing pre-call data fetch).
|
||||
template_vars = workflow.template_context_variables or {}
|
||||
|
||||
numeric_suffix = int(str(uuid.uuid4()).replace("-", "")[:8], 16) % 100000000
|
||||
workflow_run_name = f"WR-TEL-OUT-{numeric_suffix:08d}"
|
||||
workflow_run = await db_client.create_workflow_run(
|
||||
workflow_run_name,
|
||||
request.workflow_id,
|
||||
workflow.id,
|
||||
workflow_run_mode,
|
||||
user_id=user.id,
|
||||
user_id=execution_user_id,
|
||||
call_type=CallType.OUTBOUND,
|
||||
initial_context={
|
||||
**template_vars,
|
||||
|
|
@ -153,12 +166,20 @@ async def initiate_call(
|
|||
"telephony_configuration_id": telephony_configuration_id,
|
||||
},
|
||||
use_draft=True,
|
||||
organization_id=user.selected_organization_id,
|
||||
)
|
||||
workflow_run_id = workflow_run.id
|
||||
else:
|
||||
workflow_run = await db_client.get_workflow_run(workflow_run_id, user.id)
|
||||
workflow_run = await db_client.get_workflow_run(
|
||||
workflow_run_id, organization_id=user.selected_organization_id
|
||||
)
|
||||
if not workflow_run:
|
||||
raise HTTPException(status_code=400, detail="Workflow run not found")
|
||||
if workflow_run.workflow_id != workflow.id:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="workflow_run_workflow_mismatch",
|
||||
)
|
||||
workflow_run_name = workflow_run.name
|
||||
|
||||
# Construct webhook URL based on provider type
|
||||
|
|
@ -168,13 +189,13 @@ async def initiate_call(
|
|||
|
||||
webhook_url = (
|
||||
f"{backend_endpoint}/api/v1/telephony/{webhook_endpoint}"
|
||||
f"?workflow_id={request.workflow_id}"
|
||||
f"&user_id={user.id}"
|
||||
f"?workflow_id={workflow.id}"
|
||||
f"&user_id={execution_user_id}"
|
||||
f"&workflow_run_id={workflow_run_id}"
|
||||
f"&organization_id={user.selected_organization_id}"
|
||||
)
|
||||
|
||||
keywords = {"workflow_id": request.workflow_id, "user_id": user.id}
|
||||
keywords = {"workflow_id": workflow.id, "user_id": execution_user_id}
|
||||
|
||||
# Resolve optional caller-ID. The config has already been validated against
|
||||
# the user's organization, so filtering by config_id is sufficient for
|
||||
|
|
@ -292,6 +313,7 @@ async def _detect_provider(webhook_data: dict, headers: dict):
|
|||
|
||||
async def _validate_inbound_request(
|
||||
workflow_id: int,
|
||||
webhook_url: str,
|
||||
provider_class,
|
||||
normalized_data,
|
||||
webhook_data: dict,
|
||||
|
|
@ -304,7 +326,9 @@ async def _validate_inbound_request(
|
|||
"""
|
||||
from api.services.telephony import registry as telephony_registry
|
||||
|
||||
workflow = await db_client.get_workflow(workflow_id)
|
||||
# System lookup: inbound routing only has the workflow_id and derives the
|
||||
# org/user from the workflow itself, so use the explicit unscoped variant.
|
||||
workflow = await db_client.get_workflow_by_id(workflow_id)
|
||||
if not workflow:
|
||||
return False, TelephonyError.WORKFLOW_NOT_FOUND, {}, None
|
||||
|
||||
|
|
@ -361,8 +385,6 @@ async def _validate_inbound_request(
|
|||
# Verify webhook signature using the matched config's credentials. The
|
||||
# provider extracts its own signature/timestamp/nonce headers from the
|
||||
# dict, so this dispatcher stays generic.
|
||||
backend_endpoint, _ = await get_backend_endpoints()
|
||||
webhook_url = f"{backend_endpoint}/api/v1/telephony/inbound/{workflow_id}"
|
||||
provider_instance = await get_telephony_provider_by_id(
|
||||
telephony_configuration_id, organization_id
|
||||
)
|
||||
|
|
@ -527,8 +549,9 @@ async def _handle_telephony_websocket(
|
|||
await websocket.close(code=4404, reason="Workflow run not found")
|
||||
return
|
||||
|
||||
# Get workflow for organization info
|
||||
workflow = await db_client.get_workflow(workflow_id)
|
||||
# Get workflow for organization info. System lookup keyed only on the
|
||||
# workflow_id (org is derived below) — use the explicit unscoped variant.
|
||||
workflow = await db_client.get_workflow_by_id(workflow_id)
|
||||
if not workflow:
|
||||
logger.error(f"Workflow {workflow_id} not found")
|
||||
await websocket.close(code=4404, reason="Workflow not found")
|
||||
|
|
@ -697,13 +720,11 @@ async def handle_inbound_run(request: Request):
|
|||
user_id = workflow.user_id
|
||||
|
||||
# 3. Verify webhook signature against the matched config's credentials.
|
||||
backend_endpoint, wss_backend_endpoint = await get_backend_endpoints()
|
||||
webhook_url = f"{backend_endpoint}/api/v1/telephony/inbound/run"
|
||||
provider_instance = await get_telephony_provider_by_id(
|
||||
telephony_configuration_id, config.organization_id
|
||||
)
|
||||
signature_valid = await provider_instance.verify_inbound_signature(
|
||||
webhook_url, webhook_data, headers, raw_body
|
||||
str(request.url), webhook_data, headers, raw_body
|
||||
)
|
||||
if not signature_valid:
|
||||
logger.warning(
|
||||
|
|
@ -736,6 +757,7 @@ async def handle_inbound_run(request: Request):
|
|||
from_phone_number_id=phone_row.id,
|
||||
)
|
||||
|
||||
backend_endpoint, wss_backend_endpoint = await get_backend_endpoints()
|
||||
websocket_url = (
|
||||
f"{wss_backend_endpoint}/api/v1/telephony/ws/"
|
||||
f"{workflow_id}/{user_id}/{workflow_run_id}"
|
||||
|
|
@ -836,6 +858,7 @@ async def handle_inbound_telephony(
|
|||
provider_instance,
|
||||
) = await _validate_inbound_request(
|
||||
workflow_id,
|
||||
str(request.url),
|
||||
provider_class,
|
||||
normalized_data,
|
||||
webhook_data,
|
||||
|
|
|
|||
|
|
@ -1,10 +1,12 @@
|
|||
"""API routes for managing tools."""
|
||||
|
||||
import asyncio
|
||||
import re
|
||||
from datetime import datetime
|
||||
from typing import Annotated, Any, Dict, List, Literal, Optional, Union
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from loguru import logger
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
|
||||
from api.db import db_client
|
||||
|
|
@ -13,9 +15,23 @@ from api.enums import PostHogEvent, ToolCategory, ToolStatus
|
|||
from api.sdk_expose import sdk_expose
|
||||
from api.services.auth.depends import get_user
|
||||
from api.services.posthog_client import capture_event
|
||||
from api.services.workflow.mcp_tool_session import discover_mcp_tools
|
||||
from api.services.workflow.tools.mcp_tool import (
|
||||
McpDefinitionError,
|
||||
validate_mcp_definition,
|
||||
)
|
||||
from api.services.workflow.tools.mcp_tool import (
|
||||
McpToolConfig as SharedMcpToolConfig,
|
||||
)
|
||||
from api.services.workflow.tools.mcp_tool import (
|
||||
McpToolDefinition as SharedMcpToolDefinition,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/tools")
|
||||
|
||||
McpToolConfig = SharedMcpToolConfig
|
||||
McpToolDefinition = SharedMcpToolDefinition
|
||||
|
||||
|
||||
# Request/Response schemas
|
||||
class ToolParameter(BaseModel):
|
||||
|
|
@ -29,6 +45,20 @@ class ToolParameter(BaseModel):
|
|||
)
|
||||
|
||||
|
||||
class PresetToolParameter(BaseModel):
|
||||
"""A parameter injected by Dograh at runtime."""
|
||||
|
||||
name: str = Field(description="Parameter name (used as key in request body)")
|
||||
type: str = Field(description="Parameter type: string, number, or boolean")
|
||||
value_template: str = Field(
|
||||
description="Fixed value or template, e.g. {{initial_context.phone_number}}"
|
||||
)
|
||||
required: bool = Field(
|
||||
default=True,
|
||||
description="Whether the parameter must resolve to a non-empty value",
|
||||
)
|
||||
|
||||
|
||||
class HttpApiConfig(BaseModel):
|
||||
"""Configuration for HTTP API tools."""
|
||||
|
||||
|
|
@ -43,6 +73,10 @@ class HttpApiConfig(BaseModel):
|
|||
parameters: Optional[List[ToolParameter]] = Field(
|
||||
default=None, description="Parameters that the tool accepts from LLM"
|
||||
)
|
||||
preset_parameters: Optional[List[PresetToolParameter]] = Field(
|
||||
default=None,
|
||||
description="Parameters injected by Dograh from fixed values or workflow context templates",
|
||||
)
|
||||
timeout_ms: Optional[int] = Field(
|
||||
default=5000, description="Request timeout in milliseconds"
|
||||
)
|
||||
|
|
@ -165,6 +199,7 @@ ToolDefinition = Annotated[
|
|||
EndCallToolDefinition,
|
||||
TransferCallToolDefinition,
|
||||
CalculatorToolDefinition,
|
||||
McpToolDefinition,
|
||||
],
|
||||
Field(discriminator="type"),
|
||||
]
|
||||
|
|
@ -230,6 +265,14 @@ class ToolResponse(BaseModel):
|
|||
from_attributes = True
|
||||
|
||||
|
||||
class McpRefreshResponse(BaseModel):
|
||||
"""Result of re-discovering an MCP server's tool catalog."""
|
||||
|
||||
tool_uuid: str
|
||||
discovered_tools: list = Field(default_factory=list)
|
||||
error: Optional[str] = None
|
||||
|
||||
|
||||
def build_tool_response(tool, include_created_by: bool = False) -> ToolResponse:
|
||||
"""Build a response from a tool model."""
|
||||
created_by = None
|
||||
|
|
@ -318,6 +361,52 @@ async def list_tools(
|
|||
return [build_tool_response(tool) for tool in tools]
|
||||
|
||||
|
||||
async def _fetch_credential(credential_uuid: Optional[str], organization_id: int):
|
||||
"""Best-effort credential lookup for MCP auth. A missing/failed credential
|
||||
degrades to ``None`` (unauthenticated) rather than failing the request."""
|
||||
if not credential_uuid:
|
||||
return None
|
||||
try:
|
||||
return await db_client.get_credential_by_uuid(credential_uuid, organization_id)
|
||||
except Exception as e: # noqa: BLE001
|
||||
logger.warning(f"MCP: credential fetch failed: {e}")
|
||||
return None
|
||||
|
||||
|
||||
async def _populate_discovered_tools(definition: dict, *, organization_id: int) -> dict:
|
||||
"""Best-effort: for an MCP definition, connect to the server, list its
|
||||
tools, and overwrite ``config.discovered_tools``. Never raises and never
|
||||
blocks tool save — a dead server yields ``discovered_tools: []``. Non-MCP
|
||||
definitions pass through untouched."""
|
||||
if not isinstance(definition, dict) or definition.get("type") != "mcp":
|
||||
return definition
|
||||
try:
|
||||
cfg = validate_mcp_definition(definition)
|
||||
except McpDefinitionError:
|
||||
return definition
|
||||
|
||||
credential = await _fetch_credential(cfg.get("credential_uuid"), organization_id)
|
||||
|
||||
# Run discovery in an isolated asyncio task so an anyio cancel-scope
|
||||
# CancelledError doesn't bleed into the parent task and corrupt the
|
||||
# subsequent DB write. _run() never raises (degrades to []).
|
||||
async def _run() -> list:
|
||||
try:
|
||||
return await discover_mcp_tools(
|
||||
url=cfg["url"],
|
||||
credential=credential,
|
||||
timeout_secs=cfg["timeout_secs"],
|
||||
sse_read_timeout_secs=cfg["sse_read_timeout_secs"],
|
||||
)
|
||||
except BaseException as e: # noqa: BLE001
|
||||
logger.warning(f"MCP discovery failed; caching empty list: {e}")
|
||||
return []
|
||||
|
||||
discovered = await asyncio.ensure_future(_run())
|
||||
definition["config"]["discovered_tools"] = discovered
|
||||
return definition
|
||||
|
||||
|
||||
@router.post("/")
|
||||
async def create_tool(
|
||||
request: CreateToolRequest,
|
||||
|
|
@ -339,11 +428,16 @@ async def create_tool(
|
|||
|
||||
validate_category(request.category)
|
||||
|
||||
definition = await _populate_discovered_tools(
|
||||
request.definition.model_dump(),
|
||||
organization_id=user.selected_organization_id,
|
||||
)
|
||||
|
||||
tool = await db_client.create_tool(
|
||||
organization_id=user.selected_organization_id,
|
||||
user_id=user.id,
|
||||
name=request.name,
|
||||
definition=request.definition.model_dump(),
|
||||
definition=definition,
|
||||
category=request.category,
|
||||
description=request.description,
|
||||
icon=request.icon,
|
||||
|
|
@ -392,6 +486,67 @@ async def get_tool(
|
|||
return build_tool_response(tool, include_created_by=True)
|
||||
|
||||
|
||||
@router.post("/{tool_uuid}/mcp/refresh")
|
||||
async def refresh_mcp_tools(
|
||||
tool_uuid: str,
|
||||
user: UserModel = Depends(get_user),
|
||||
) -> McpRefreshResponse:
|
||||
"""Re-discover an MCP tool's server catalog and overwrite the cached
|
||||
``definition.config.discovered_tools``. Server down → 200 with error
|
||||
(cache not overwritten on transient failure)."""
|
||||
if not user.selected_organization_id:
|
||||
raise HTTPException(
|
||||
status_code=400, detail="No organization selected for the user"
|
||||
)
|
||||
|
||||
tool = await db_client.get_tool_by_uuid(
|
||||
tool_uuid, user.selected_organization_id, include_archived=True
|
||||
)
|
||||
if not tool:
|
||||
raise HTTPException(status_code=404, detail="Tool not found")
|
||||
if tool.category != ToolCategory.MCP.value:
|
||||
raise HTTPException(status_code=400, detail="Tool is not an MCP tool")
|
||||
|
||||
try:
|
||||
cfg = validate_mcp_definition(tool.definition)
|
||||
except McpDefinitionError as e:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid MCP definition: {e}")
|
||||
|
||||
credential = await _fetch_credential(
|
||||
cfg.get("credential_uuid"), user.selected_organization_id
|
||||
)
|
||||
|
||||
try:
|
||||
discovered = await discover_mcp_tools(
|
||||
url=cfg["url"],
|
||||
credential=credential,
|
||||
timeout_secs=cfg["timeout_secs"],
|
||||
sse_read_timeout_secs=cfg["sse_read_timeout_secs"],
|
||||
)
|
||||
except Exception as e: # noqa: BLE001
|
||||
logger.warning(f"MCP refresh discovery failed: {e}")
|
||||
discovered = []
|
||||
|
||||
if not discovered:
|
||||
error = (
|
||||
f"Could not reach the MCP server at {cfg['url']} "
|
||||
f"(or it exposes no tools). Previously cached list retained."
|
||||
)
|
||||
# Do NOT clobber a previously-good cache with [] on a transient outage.
|
||||
return McpRefreshResponse(tool_uuid=tool_uuid, discovered_tools=[], error=error)
|
||||
|
||||
new_def = dict(tool.definition or {})
|
||||
new_def["config"] = {**new_def.get("config", {}), "discovered_tools": discovered}
|
||||
await db_client.update_tool(
|
||||
tool_uuid=tool_uuid,
|
||||
organization_id=user.selected_organization_id,
|
||||
definition=new_def,
|
||||
)
|
||||
return McpRefreshResponse(
|
||||
tool_uuid=tool_uuid, discovered_tools=discovered, error=None
|
||||
)
|
||||
|
||||
|
||||
@router.put("/{tool_uuid}")
|
||||
async def update_tool(
|
||||
tool_uuid: str,
|
||||
|
|
@ -416,12 +571,21 @@ async def update_tool(
|
|||
if request.status:
|
||||
validate_status(request.status)
|
||||
|
||||
definition = (
|
||||
await _populate_discovered_tools(
|
||||
request.definition.model_dump(),
|
||||
organization_id=user.selected_organization_id,
|
||||
)
|
||||
if request.definition
|
||||
else None
|
||||
)
|
||||
|
||||
tool = await db_client.update_tool(
|
||||
tool_uuid=tool_uuid,
|
||||
organization_id=user.selected_organization_id,
|
||||
name=request.name,
|
||||
description=request.description,
|
||||
definition=request.definition.model_dump() if request.definition else None,
|
||||
definition=definition,
|
||||
icon=request.icon,
|
||||
icon_color=request.icon_color,
|
||||
status=request.status,
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@ import asyncio
|
|||
import ipaddress
|
||||
import os
|
||||
from datetime import UTC, datetime
|
||||
from enum import Enum
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
from aiortc import RTCIceServer
|
||||
|
|
@ -49,6 +50,63 @@ from api.services.quota_service import check_dograh_quota
|
|||
router = APIRouter(prefix="/ws")
|
||||
|
||||
|
||||
class NonRelayFilterPolicy(Enum):
|
||||
"""What to filter from non-relay ICE candidates. Relay candidates always pass."""
|
||||
|
||||
NONE = "none" # filter nothing — pass all candidates
|
||||
PRIVATE = "private" # filter non-relay candidates with private/CGNAT IPs
|
||||
ALL = "all" # filter all non-relay candidates (relay-only mode)
|
||||
|
||||
|
||||
def is_local_or_cgnat_ip(ip_str: str) -> bool:
|
||||
"""Return True for RFC1918, loopback, link-local, and CGNAT addresses."""
|
||||
|
||||
try:
|
||||
ip = ipaddress.ip_address(ip_str)
|
||||
except ValueError:
|
||||
return False
|
||||
|
||||
is_cgnat = ip.version == 4 and ip in ipaddress.ip_network("100.64.0.0/10")
|
||||
return ip.is_private or ip.is_loopback or ip.is_link_local or is_cgnat
|
||||
|
||||
|
||||
def resolve_ice_filter_policies(
|
||||
environment: str,
|
||||
force_turn_relay: bool,
|
||||
server_ip: str,
|
||||
) -> tuple[NonRelayFilterPolicy, NonRelayFilterPolicy]:
|
||||
"""Resolve outbound and inbound non-relay filtering for this deployment."""
|
||||
|
||||
private_lan_deployment = (
|
||||
environment != Environment.LOCAL.value and is_local_or_cgnat_ip(server_ip)
|
||||
)
|
||||
|
||||
if force_turn_relay:
|
||||
# Relay-only diagnostics stay explicit. On private LAN deployments we
|
||||
# must still accept inbound private candidates for relay<->host pairs.
|
||||
outbound_policy = NonRelayFilterPolicy.ALL
|
||||
inbound_policy = (
|
||||
NonRelayFilterPolicy.NONE
|
||||
if private_lan_deployment
|
||||
else NonRelayFilterPolicy.PRIVATE
|
||||
)
|
||||
return outbound_policy, inbound_policy
|
||||
|
||||
if environment == Environment.LOCAL.value or private_lan_deployment:
|
||||
return NonRelayFilterPolicy.NONE, NonRelayFilterPolicy.NONE
|
||||
|
||||
# Public remote deployment: drop private-IP host candidates to avoid
|
||||
# coturn denied-peer-ip errors against Docker bridge and LAN interfaces.
|
||||
return NonRelayFilterPolicy.PRIVATE, NonRelayFilterPolicy.PRIVATE
|
||||
|
||||
|
||||
ICE_OUTBOUND_POLICY, ICE_INBOUND_POLICY = resolve_ice_filter_policies(
|
||||
ENVIRONMENT,
|
||||
FORCE_TURN_RELAY,
|
||||
os.getenv("SERVER_IP", ""),
|
||||
)
|
||||
|
||||
|
||||
def is_private_ip_candidate(candidate_str: str) -> bool:
|
||||
"""Check if ICE candidate contains a private IP address or CGNAT IP Address.
|
||||
|
||||
|
|
@ -69,61 +127,58 @@ def is_private_ip_candidate(candidate_str: str) -> bool:
|
|||
if "typ" in parts:
|
||||
typ_index = parts.index("typ")
|
||||
ip_str = parts[typ_index - 2]
|
||||
ip = ipaddress.ip_address(ip_str)
|
||||
is_cgnat = ip in ipaddress.ip_network("100.64.0.0/10")
|
||||
return ip.is_private or is_cgnat
|
||||
return is_local_or_cgnat_ip(ip_str)
|
||||
except (ValueError, IndexError):
|
||||
pass
|
||||
return False
|
||||
|
||||
|
||||
def filter_outbound_sdp(sdp: str) -> str:
|
||||
"""Strip ICE candidates from an outbound answer SDP based on env config.
|
||||
def _keep_candidate(candidate_str: str, policy: NonRelayFilterPolicy) -> bool:
|
||||
"""Return True if this ICE candidate should be kept under the given policy.
|
||||
|
||||
Two filters apply:
|
||||
|
||||
1. In non-LOCAL environments, drop host candidates with private/CGNAT IPs.
|
||||
aiortc gathers host candidates from every interface on the box, including
|
||||
Docker bridges (172.17.0.1, 172.18.0.1). Advertising those to the browser
|
||||
causes coturn "peer IP X denied" errors when the browser asks TURN to
|
||||
permit them.
|
||||
|
||||
2. When FORCE_TURN_RELAY is set, drop every non-relay candidate so the
|
||||
only path the browser can use is via TURN. Lets you verify TURN
|
||||
connectivity end-to-end — if TURN is broken, the call simply fails.
|
||||
Relay candidates always pass — a relay with a private IP (LAN TURN server)
|
||||
must never be dropped regardless of policy.
|
||||
"""
|
||||
if ENVIRONMENT == Environment.LOCAL.value and not FORCE_TURN_RELAY:
|
||||
if " typ relay" in candidate_str:
|
||||
return True
|
||||
if policy == NonRelayFilterPolicy.NONE:
|
||||
return True
|
||||
if policy == NonRelayFilterPolicy.ALL:
|
||||
return False
|
||||
# PRIVATE: drop non-relay candidates with private/CGNAT IPs
|
||||
return not is_private_ip_candidate(candidate_str)
|
||||
|
||||
|
||||
def filter_outbound_sdp(sdp: str) -> str:
|
||||
"""Strip ICE candidates from an outbound answer SDP based on ICE_OUTBOUND_POLICY."""
|
||||
if ICE_OUTBOUND_POLICY == NonRelayFilterPolicy.NONE:
|
||||
return sdp
|
||||
|
||||
lines = sdp.split("\r\n")
|
||||
filtered: List[str] = []
|
||||
dropped_non_relay = 0
|
||||
dropped = 0
|
||||
kept_relay = 0
|
||||
for line in lines:
|
||||
if line.startswith("a=candidate:"):
|
||||
candidate_str = line[2:]
|
||||
if FORCE_TURN_RELAY and " typ relay" not in candidate_str:
|
||||
dropped_non_relay += 1
|
||||
if not _keep_candidate(candidate_str, ICE_OUTBOUND_POLICY):
|
||||
dropped += 1
|
||||
continue
|
||||
if ENVIRONMENT != Environment.LOCAL.value and is_private_ip_candidate(
|
||||
candidate_str
|
||||
):
|
||||
continue
|
||||
if FORCE_TURN_RELAY:
|
||||
if " typ relay" in candidate_str:
|
||||
kept_relay += 1
|
||||
filtered.append(line)
|
||||
|
||||
if FORCE_TURN_RELAY:
|
||||
if ICE_OUTBOUND_POLICY == NonRelayFilterPolicy.ALL:
|
||||
if kept_relay == 0:
|
||||
logger.warning(
|
||||
"FORCE_TURN_RELAY is on but the answer SDP has no relay candidates "
|
||||
f"(dropped {dropped_non_relay} non-relay). TURN may be unreachable; "
|
||||
f"(dropped {dropped} non-relay). TURN may be unreachable; "
|
||||
"the connection will fail."
|
||||
)
|
||||
else:
|
||||
logger.info(
|
||||
f"FORCE_TURN_RELAY: kept {kept_relay} relay candidates, "
|
||||
f"dropped {dropped_non_relay} non-relay"
|
||||
f"dropped {dropped} non-relay"
|
||||
)
|
||||
|
||||
return "\r\n".join(filtered)
|
||||
|
|
@ -370,9 +425,7 @@ class SignalingManager:
|
|||
Uses SmallWebRTC's native ICE trickling support via add_ice_candidate().
|
||||
Candidates are parsed using aiortc's candidate_from_sdp() for proper formatting,
|
||||
consistent with SmallWebRTCRequestHandler.handle_patch_request().
|
||||
|
||||
In non-local environments, private IP candidates are filtered out to prevent
|
||||
TURN relay errors when coturn blocks private IP ranges (denied-peer-ip).
|
||||
Candidates are filtered according to ICE_INBOUND_POLICY before being added.
|
||||
"""
|
||||
pc_id = payload.get("pc_id")
|
||||
candidate_data = payload.get("candidate")
|
||||
|
|
@ -389,13 +442,9 @@ class SignalingManager:
|
|||
if candidate_data:
|
||||
candidate_str = candidate_data.get("candidate", "")
|
||||
|
||||
# Filter out private IP candidates in non-local environments
|
||||
# This prevents TURN relay errors when coturn blocks private IP ranges
|
||||
if ENVIRONMENT != Environment.LOCAL.value and is_private_ip_candidate(
|
||||
candidate_str
|
||||
):
|
||||
if not _keep_candidate(candidate_str, ICE_INBOUND_POLICY):
|
||||
logger.debug(
|
||||
f"Skipping private IP candidate in {ENVIRONMENT}: {candidate_str[:50]}..."
|
||||
f"Dropping inbound candidate per policy ({ICE_INBOUND_POLICY.value}): {candidate_str[:50]}..."
|
||||
)
|
||||
return
|
||||
|
||||
|
|
|
|||
|
|
@ -32,99 +32,16 @@ from api.services.storage import storage_fs
|
|||
from api.services.workflow.dto import ReactFlowDTO, sanitize_workflow_definition
|
||||
from api.services.workflow.duplicate import duplicate_workflow
|
||||
from api.services.workflow.errors import ItemKind, WorkflowError
|
||||
from api.services.workflow.trigger_paths import (
|
||||
TriggerPathIssue,
|
||||
ensure_trigger_paths,
|
||||
extract_trigger_paths,
|
||||
regenerate_trigger_uuids,
|
||||
trigger_path_to_node_id,
|
||||
validate_trigger_paths,
|
||||
)
|
||||
from api.services.workflow.workflow_graph import WorkflowGraph
|
||||
|
||||
|
||||
def extract_trigger_paths(workflow_definition: dict) -> List[str]:
|
||||
"""Extract trigger UUIDs from workflow definition.
|
||||
|
||||
Args:
|
||||
workflow_definition: The workflow definition JSON
|
||||
|
||||
Returns:
|
||||
List of trigger UUIDs found in the workflow
|
||||
"""
|
||||
if not workflow_definition:
|
||||
return []
|
||||
|
||||
nodes = workflow_definition.get("nodes", [])
|
||||
trigger_paths = []
|
||||
|
||||
for node in nodes:
|
||||
if node.get("type") == "trigger":
|
||||
trigger_path = node.get("data", {}).get("trigger_path")
|
||||
if trigger_path:
|
||||
trigger_paths.append(trigger_path)
|
||||
|
||||
return trigger_paths
|
||||
|
||||
|
||||
def _trigger_path_to_node_id(workflow_definition: dict) -> dict[str, str]:
|
||||
"""Map each trigger node's trigger_path to its node id."""
|
||||
if not workflow_definition:
|
||||
return {}
|
||||
out: dict[str, str] = {}
|
||||
for node in workflow_definition.get("nodes", []):
|
||||
if node.get("type") == "trigger":
|
||||
tp = node.get("data", {}).get("trigger_path")
|
||||
if tp:
|
||||
out[tp] = node.get("id")
|
||||
return out
|
||||
|
||||
|
||||
def regenerate_trigger_uuids(workflow_definition: dict) -> dict:
|
||||
"""Regenerate UUIDs for all trigger nodes in a workflow definition.
|
||||
|
||||
This should be called when creating a new workflow from a template or
|
||||
duplicating a workflow to avoid trigger UUID conflicts.
|
||||
|
||||
Args:
|
||||
workflow_definition: The workflow definition JSON
|
||||
|
||||
Returns:
|
||||
Updated workflow definition with new trigger UUIDs
|
||||
"""
|
||||
if not workflow_definition:
|
||||
return workflow_definition
|
||||
|
||||
# Deep copy to avoid modifying the original
|
||||
import copy
|
||||
|
||||
updated_definition = copy.deepcopy(workflow_definition)
|
||||
|
||||
nodes = updated_definition.get("nodes", [])
|
||||
for node in nodes:
|
||||
if node.get("type") == "trigger":
|
||||
# Generate a new UUID for this trigger
|
||||
if "data" not in node:
|
||||
node["data"] = {}
|
||||
node["data"]["trigger_path"] = str(uuid.uuid4())
|
||||
|
||||
return updated_definition
|
||||
|
||||
|
||||
def ensure_trigger_paths(workflow_definition: Optional[dict]) -> Optional[dict]:
|
||||
"""Mint a UUID for any trigger node that's missing ``data.trigger_path``.
|
||||
|
||||
Trigger nodes that already carry a non-empty trigger_path are left
|
||||
untouched so stable IDs survive edits. The input is not mutated; the
|
||||
returned dict is what should be persisted and echoed in the response.
|
||||
"""
|
||||
if not workflow_definition:
|
||||
return workflow_definition
|
||||
|
||||
import copy
|
||||
|
||||
out = copy.deepcopy(workflow_definition)
|
||||
for node in out.get("nodes") or []:
|
||||
if node.get("type") != "trigger":
|
||||
continue
|
||||
data = node.setdefault("data", {})
|
||||
if not data.get("trigger_path"):
|
||||
data["trigger_path"] = str(uuid.uuid4())
|
||||
return out
|
||||
|
||||
|
||||
router = APIRouter(prefix="/workflow")
|
||||
|
||||
|
||||
|
|
@ -139,7 +56,7 @@ def _trigger_conflict_http_exception(
|
|||
"""Build a 409 with the same detail shape as validate's 422 so the editor
|
||||
can highlight the offending trigger node(s) using the same code path."""
|
||||
path_to_node = (
|
||||
_trigger_path_to_node_id(workflow_definition) if workflow_definition else {}
|
||||
trigger_path_to_node_id(workflow_definition) if workflow_definition else {}
|
||||
)
|
||||
errors: list[WorkflowError] = [
|
||||
WorkflowError(
|
||||
|
|
@ -159,6 +76,24 @@ def _trigger_conflict_http_exception(
|
|||
)
|
||||
|
||||
|
||||
def _trigger_path_validation_http_exception(
|
||||
issues: list[TriggerPathIssue],
|
||||
) -> HTTPException:
|
||||
errors = [
|
||||
WorkflowError(
|
||||
kind=ItemKind.node,
|
||||
id=issue.node_id,
|
||||
field="data.trigger_path",
|
||||
message=issue.message,
|
||||
)
|
||||
for issue in issues
|
||||
]
|
||||
return HTTPException(
|
||||
status_code=422,
|
||||
detail=ValidateWorkflowResponse(is_valid=False, errors=errors).model_dump(),
|
||||
)
|
||||
|
||||
|
||||
async def _validate_workflow_definition(
|
||||
workflow_definition: Optional[dict],
|
||||
exclude_workflow_id: Optional[int] = None,
|
||||
|
|
@ -187,6 +122,17 @@ async def _validate_workflow_definition(
|
|||
except ValueError as e:
|
||||
errors.extend(e.args[0])
|
||||
|
||||
# ----------- Trigger Path Format Check ------------
|
||||
for issue in validate_trigger_paths(workflow_definition):
|
||||
errors.append(
|
||||
WorkflowError(
|
||||
kind=ItemKind.node,
|
||||
id=issue.node_id,
|
||||
field="data.trigger_path",
|
||||
message=issue.message,
|
||||
)
|
||||
)
|
||||
|
||||
# ----------- Trigger Path Conflict Check ------------
|
||||
trigger_paths = extract_trigger_paths(workflow_definition)
|
||||
if trigger_paths:
|
||||
|
|
@ -195,7 +141,7 @@ async def _validate_workflow_definition(
|
|||
exclude_workflow_id=exclude_workflow_id,
|
||||
)
|
||||
if conflicts:
|
||||
path_to_node = _trigger_path_to_node_id(workflow_definition)
|
||||
path_to_node = trigger_path_to_node_id(workflow_definition)
|
||||
for conflicting_path in conflicts:
|
||||
errors.append(
|
||||
WorkflowError(
|
||||
|
|
@ -251,6 +197,14 @@ class WorkflowListResponse(BaseModel):
|
|||
status: str
|
||||
created_at: datetime
|
||||
total_runs: int
|
||||
folder_id: int | None = None
|
||||
workflow_uuid: str | None = None
|
||||
|
||||
|
||||
class MoveWorkflowToFolderRequest(BaseModel):
|
||||
"""Move a workflow into a folder, or to "Uncategorized" when null."""
|
||||
|
||||
folder_id: int | None = None
|
||||
|
||||
|
||||
class WorkflowCountResponse(BaseModel):
|
||||
|
|
@ -404,6 +358,9 @@ async def create_workflow(
|
|||
# Auto-mint trigger_path for any trigger node that didn't ship one so
|
||||
# clients don't need to generate UUIDs themselves.
|
||||
workflow_definition = ensure_trigger_paths(request.workflow_definition)
|
||||
trigger_path_issues = validate_trigger_paths(workflow_definition)
|
||||
if trigger_path_issues:
|
||||
raise _trigger_path_validation_http_exception(trigger_path_issues)
|
||||
|
||||
# Validate trigger path uniqueness BEFORE creating the workflow so we
|
||||
# don't leave an orphaned workflow record when the trigger conflicts.
|
||||
|
|
@ -641,6 +598,8 @@ async def get_workflows(
|
|||
status=workflow.status,
|
||||
created_at=workflow.created_at,
|
||||
total_runs=run_counts.get(workflow.id, 0),
|
||||
folder_id=workflow.folder_id,
|
||||
workflow_uuid=workflow.workflow_uuid,
|
||||
)
|
||||
for workflow in workflows
|
||||
]
|
||||
|
|
@ -883,6 +842,48 @@ async def update_workflow_status(
|
|||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.put("/{workflow_id}/folder")
|
||||
async def move_workflow_to_folder(
|
||||
workflow_id: int,
|
||||
request: MoveWorkflowToFolderRequest,
|
||||
user: UserModel = Depends(get_user),
|
||||
) -> WorkflowListResponse:
|
||||
"""Move a workflow into a folder, or to "Uncategorized" (folder_id=null).
|
||||
|
||||
Validates that the target folder belongs to the caller's organization —
|
||||
the FK alone proves the folder exists, not that the caller may use it.
|
||||
"""
|
||||
# Validate target folder ownership (tenant isolation) unless un-filing.
|
||||
if request.folder_id is not None:
|
||||
folder = await db_client.get_folder(
|
||||
request.folder_id, organization_id=user.selected_organization_id
|
||||
)
|
||||
if folder is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Folder with id {request.folder_id} not found",
|
||||
)
|
||||
|
||||
try:
|
||||
workflow = await db_client.move_workflow_to_folder(
|
||||
workflow_id=workflow_id,
|
||||
folder_id=request.folder_id,
|
||||
organization_id=user.selected_organization_id,
|
||||
)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=404, detail=str(e))
|
||||
|
||||
run_count = await db_client.get_workflow_run_count(workflow.id)
|
||||
return WorkflowListResponse(
|
||||
id=workflow.id,
|
||||
name=workflow.name,
|
||||
status=workflow.status,
|
||||
created_at=workflow.created_at,
|
||||
total_runs=run_count,
|
||||
folder_id=workflow.folder_id,
|
||||
)
|
||||
|
||||
|
||||
@router.put(
|
||||
"/{workflow_id}",
|
||||
**sdk_expose(
|
||||
|
|
@ -917,6 +918,9 @@ async def update_workflow(
|
|||
# response echoes workflow_definition so the client picks up the new
|
||||
# UUID without a refetch.
|
||||
workflow_definition = ensure_trigger_paths(workflow_definition)
|
||||
trigger_path_issues = validate_trigger_paths(workflow_definition)
|
||||
if trigger_path_issues:
|
||||
raise _trigger_path_validation_http_exception(trigger_path_issues)
|
||||
if workflow_definition:
|
||||
existing_workflow = await db_client.get_workflow(
|
||||
workflow_id, organization_id=user.selected_organization_id
|
||||
|
|
@ -1081,7 +1085,12 @@ async def create_workflow_run(
|
|||
user: The user to create the workflow run for
|
||||
"""
|
||||
run = await db_client.create_workflow_run(
|
||||
request.name, workflow_id, request.mode, user.id, use_draft=True
|
||||
request.name,
|
||||
workflow_id,
|
||||
request.mode,
|
||||
user.id,
|
||||
use_draft=True,
|
||||
organization_id=user.selected_organization_id,
|
||||
)
|
||||
return {
|
||||
"id": run.id,
|
||||
|
|
|
|||
282
api/routes/workflow_text_chat.py
Normal file
282
api/routes/workflow_text_chat.py
Normal file
|
|
@ -0,0 +1,282 @@
|
|||
from datetime import datetime
|
||||
from typing import Any, Dict
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pipecat.utils.run_context import set_current_run_id
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from api.db import db_client
|
||||
from api.db.models import UserModel, WorkflowRunTextSessionModel
|
||||
from api.enums import WorkflowRunMode
|
||||
from api.services.auth.depends import get_user
|
||||
from api.services.quota_service import check_dograh_quota
|
||||
from api.services.workflow.text_chat_session_service import (
|
||||
TextChatPendingTurnLostError,
|
||||
TextChatSessionExecutionError,
|
||||
TextChatSessionRevisionConflictError,
|
||||
TextChatTurnNotFoundError,
|
||||
append_text_chat_user_message,
|
||||
default_text_chat_checkpoint,
|
||||
default_text_chat_session_data,
|
||||
execute_pending_text_chat_turn,
|
||||
initialize_text_chat_session,
|
||||
normalize_text_chat_checkpoint,
|
||||
normalize_text_chat_session_data,
|
||||
rewind_text_chat_session_state,
|
||||
)
|
||||
|
||||
router = APIRouter(prefix="/workflow", tags=["workflow-text-chat"])
|
||||
|
||||
|
||||
class CreateTextChatSessionRequest(BaseModel):
|
||||
name: str | None = None
|
||||
initial_context: Dict[str, Any] | None = None
|
||||
annotations: Dict[str, Any] | None = None
|
||||
|
||||
|
||||
class AppendTextChatMessageRequest(BaseModel):
|
||||
text: str = Field(min_length=1)
|
||||
expected_revision: int | None = None
|
||||
|
||||
|
||||
class RewindTextChatSessionRequest(BaseModel):
|
||||
cursor_turn_id: str | None = None
|
||||
expected_revision: int | None = None
|
||||
|
||||
|
||||
class WorkflowRunTextSessionResponse(BaseModel):
|
||||
workflow_run_id: int
|
||||
workflow_id: int
|
||||
name: str
|
||||
mode: str
|
||||
state: str
|
||||
is_completed: bool
|
||||
revision: int
|
||||
initial_context: Dict[str, Any] | None = None
|
||||
gathered_context: Dict[str, Any] | None = None
|
||||
annotations: Dict[str, Any] | None = None
|
||||
session_data: Dict[str, Any]
|
||||
checkpoint: Dict[str, Any]
|
||||
created_at: datetime
|
||||
updated_at: datetime | None = None
|
||||
|
||||
|
||||
def _get_state_value(state: Any) -> str:
|
||||
return state.value if hasattr(state, "value") else str(state)
|
||||
|
||||
|
||||
def _build_response(
|
||||
text_session: WorkflowRunTextSessionModel,
|
||||
) -> WorkflowRunTextSessionResponse:
|
||||
workflow_run = text_session.workflow_run
|
||||
return WorkflowRunTextSessionResponse(
|
||||
workflow_run_id=workflow_run.id,
|
||||
workflow_id=workflow_run.workflow_id,
|
||||
name=workflow_run.name,
|
||||
mode=workflow_run.mode,
|
||||
state=_get_state_value(workflow_run.state),
|
||||
is_completed=workflow_run.is_completed,
|
||||
revision=text_session.revision,
|
||||
initial_context=workflow_run.initial_context,
|
||||
gathered_context=workflow_run.gathered_context,
|
||||
annotations=workflow_run.annotations,
|
||||
session_data=normalize_text_chat_session_data(text_session.session_data),
|
||||
checkpoint=normalize_text_chat_checkpoint(text_session.checkpoint),
|
||||
created_at=text_session.created_at,
|
||||
updated_at=text_session.updated_at,
|
||||
)
|
||||
|
||||
|
||||
def _revision_conflict_detail(e: Any) -> dict[str, Any]:
|
||||
return {
|
||||
"message": "Text chat session revision conflict",
|
||||
"expected_revision": e.expected_revision,
|
||||
"actual_revision": e.actual_revision,
|
||||
}
|
||||
|
||||
|
||||
def _require_selected_organization_id(user: UserModel) -> int:
|
||||
if user.selected_organization_id is None:
|
||||
raise HTTPException(status_code=403, detail="Organization context is required")
|
||||
return user.selected_organization_id
|
||||
|
||||
|
||||
async def _ensure_text_chat_quota(user: UserModel, workflow_id: int) -> None:
|
||||
quota_result = await check_dograh_quota(user, workflow_id=workflow_id)
|
||||
if not quota_result.has_quota:
|
||||
raise HTTPException(status_code=402, detail=quota_result.error_message)
|
||||
|
||||
|
||||
async def _load_text_session_or_404(
|
||||
workflow_id: int,
|
||||
run_id: int,
|
||||
user: UserModel,
|
||||
) -> WorkflowRunTextSessionModel:
|
||||
set_current_run_id(run_id)
|
||||
organization_id = _require_selected_organization_id(user)
|
||||
text_session = await db_client.get_workflow_run_text_session(
|
||||
run_id, organization_id=organization_id
|
||||
)
|
||||
if not text_session or not text_session.workflow_run:
|
||||
raise HTTPException(status_code=404, detail="Text chat session not found")
|
||||
if text_session.workflow_run.workflow_id != workflow_id:
|
||||
raise HTTPException(status_code=404, detail="Text chat session not found")
|
||||
if text_session.workflow_run.mode != WorkflowRunMode.TEXTCHAT.value:
|
||||
raise HTTPException(
|
||||
status_code=400, detail="Workflow run is not a text chat session"
|
||||
)
|
||||
return text_session
|
||||
|
||||
|
||||
async def _execute_pending_turn_response(
|
||||
*,
|
||||
workflow_id: int,
|
||||
run_id: int,
|
||||
text_session: WorkflowRunTextSessionModel,
|
||||
) -> WorkflowRunTextSessionResponse:
|
||||
try:
|
||||
updated_text_session = await execute_pending_text_chat_turn(
|
||||
workflow_id=workflow_id,
|
||||
run_id=run_id,
|
||||
text_session=text_session,
|
||||
)
|
||||
except TextChatSessionRevisionConflictError as e:
|
||||
raise HTTPException(status_code=409, detail=_revision_conflict_detail(e))
|
||||
except TextChatPendingTurnLostError as e:
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
except TextChatSessionExecutionError as e:
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
return _build_response(updated_text_session)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/{workflow_id}/text-chat/sessions",
|
||||
response_model=WorkflowRunTextSessionResponse,
|
||||
)
|
||||
async def create_text_chat_session(
|
||||
workflow_id: int,
|
||||
request: CreateTextChatSessionRequest,
|
||||
user: UserModel = Depends(get_user),
|
||||
) -> WorkflowRunTextSessionResponse:
|
||||
organization_id = _require_selected_organization_id(user)
|
||||
await _ensure_text_chat_quota(user, workflow_id)
|
||||
|
||||
session_name = request.name or f"WR-TEXT-{uuid4().hex[:6].upper()}"
|
||||
try:
|
||||
workflow_run = await db_client.create_workflow_run(
|
||||
name=session_name,
|
||||
workflow_id=workflow_id,
|
||||
mode=WorkflowRunMode.TEXTCHAT.value,
|
||||
user_id=user.id,
|
||||
initial_context=request.initial_context,
|
||||
use_draft=True,
|
||||
organization_id=organization_id,
|
||||
)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=404, detail=str(e))
|
||||
|
||||
set_current_run_id(workflow_run.id)
|
||||
|
||||
annotations = {
|
||||
"tester": {
|
||||
"source": "workflow_editor",
|
||||
"modality": "text",
|
||||
}
|
||||
}
|
||||
if request.annotations:
|
||||
annotations = {**annotations, **request.annotations}
|
||||
workflow_run = await db_client.update_workflow_run(
|
||||
workflow_run.id,
|
||||
annotations=annotations,
|
||||
)
|
||||
|
||||
text_session = await db_client.ensure_workflow_run_text_session(
|
||||
workflow_run.id,
|
||||
session_data=default_text_chat_session_data(),
|
||||
checkpoint=default_text_chat_checkpoint(),
|
||||
)
|
||||
|
||||
try:
|
||||
text_session = await initialize_text_chat_session(
|
||||
run_id=workflow_run.id,
|
||||
text_session=text_session,
|
||||
)
|
||||
except TextChatSessionRevisionConflictError as e:
|
||||
raise HTTPException(status_code=409, detail=_revision_conflict_detail(e))
|
||||
|
||||
return await _execute_pending_turn_response(
|
||||
workflow_id=workflow_id,
|
||||
run_id=workflow_run.id,
|
||||
text_session=text_session,
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/{workflow_id}/text-chat/sessions/{run_id}",
|
||||
response_model=WorkflowRunTextSessionResponse,
|
||||
)
|
||||
async def get_text_chat_session(
|
||||
workflow_id: int,
|
||||
run_id: int,
|
||||
user: UserModel = Depends(get_user),
|
||||
) -> WorkflowRunTextSessionResponse:
|
||||
text_session = await _load_text_session_or_404(workflow_id, run_id, user)
|
||||
return _build_response(text_session)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/{workflow_id}/text-chat/sessions/{run_id}/messages",
|
||||
response_model=WorkflowRunTextSessionResponse,
|
||||
)
|
||||
async def append_text_chat_message(
|
||||
workflow_id: int,
|
||||
run_id: int,
|
||||
request: AppendTextChatMessageRequest,
|
||||
user: UserModel = Depends(get_user),
|
||||
) -> WorkflowRunTextSessionResponse:
|
||||
text_session = await _load_text_session_or_404(workflow_id, run_id, user)
|
||||
await _ensure_text_chat_quota(user, workflow_id)
|
||||
|
||||
try:
|
||||
text_session = await append_text_chat_user_message(
|
||||
run_id=run_id,
|
||||
text_session=text_session,
|
||||
user_text=request.text,
|
||||
expected_revision=request.expected_revision,
|
||||
)
|
||||
except TextChatSessionRevisionConflictError as e:
|
||||
raise HTTPException(status_code=409, detail=_revision_conflict_detail(e))
|
||||
|
||||
return await _execute_pending_turn_response(
|
||||
workflow_id=workflow_id,
|
||||
run_id=run_id,
|
||||
text_session=text_session,
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/{workflow_id}/text-chat/sessions/{run_id}/rewind",
|
||||
response_model=WorkflowRunTextSessionResponse,
|
||||
)
|
||||
async def rewind_text_chat_session(
|
||||
workflow_id: int,
|
||||
run_id: int,
|
||||
request: RewindTextChatSessionRequest,
|
||||
user: UserModel = Depends(get_user),
|
||||
) -> WorkflowRunTextSessionResponse:
|
||||
text_session = await _load_text_session_or_404(workflow_id, run_id, user)
|
||||
try:
|
||||
text_session = await rewind_text_chat_session_state(
|
||||
run_id=run_id,
|
||||
text_session=text_session,
|
||||
cursor_turn_id=request.cursor_turn_id,
|
||||
expected_revision=request.expected_revision,
|
||||
)
|
||||
except TextChatTurnNotFoundError as e:
|
||||
raise HTTPException(status_code=404, detail=str(e))
|
||||
except TextChatSessionRevisionConflictError as e:
|
||||
raise HTTPException(status_code=409, detail=_revision_conflict_detail(e))
|
||||
|
||||
return _build_response(text_session)
|
||||
|
|
@ -129,22 +129,6 @@ async def get_user(
|
|||
return user_model
|
||||
|
||||
|
||||
async def get_user_optional(
|
||||
authorization: Annotated[str | None, Header()] = None,
|
||||
x_api_key: Annotated[str | None, Header(alias="X-API-Key")] = None,
|
||||
) -> UserModel | None:
|
||||
"""
|
||||
Same as get_user but returns None instead of raising 401 if unauthorized.
|
||||
Useful for endpoints that need to work both with and without auth.
|
||||
"""
|
||||
try:
|
||||
return await get_user(authorization, x_api_key)
|
||||
except HTTPException as e:
|
||||
if e.status_code == 401:
|
||||
return None
|
||||
raise
|
||||
|
||||
|
||||
async def _handle_oss_auth(authorization: str | None) -> UserModel:
|
||||
"""
|
||||
Handle authentication for OSS deployment mode.
|
||||
|
|
|
|||
|
|
@ -20,8 +20,8 @@ from api.utils.common import get_backend_endpoints
|
|||
if TYPE_CHECKING:
|
||||
# Type-only — importing api.services.telephony eagerly triggers the
|
||||
# provider package init, which can pull in this module via the routes
|
||||
# chain and create a circular import. Runtime calls below go through
|
||||
# ``factory.get_telephony_provider`` (lazy import inside the method).
|
||||
# chain and create a circular import. Runtime calls below lazy-import the
|
||||
# factory helpers inside methods instead.
|
||||
from api.services.telephony.base import TelephonyProvider
|
||||
|
||||
|
||||
|
|
@ -31,12 +31,6 @@ class CampaignCallDispatcher:
|
|||
def __init__(self):
|
||||
self.default_concurrent_limit = int(DEFAULT_ORG_CONCURRENCY_LIMIT)
|
||||
|
||||
async def get_telephony_provider(self, organization_id: int) -> "TelephonyProvider":
|
||||
"""Get telephony provider instance for specific organization (default config)."""
|
||||
from api.services.telephony.factory import get_default_telephony_provider
|
||||
|
||||
return await get_default_telephony_provider(organization_id)
|
||||
|
||||
async def get_provider_for_campaign(self, campaign) -> "TelephonyProvider":
|
||||
"""Get the telephony provider pinned to this campaign's config. Falls back
|
||||
to the org's default config for legacy campaigns whose
|
||||
|
|
@ -302,7 +296,6 @@ class CampaignCallDispatcher:
|
|||
f"?workflow_id={campaign.workflow_id}"
|
||||
f"&user_id={campaign.created_by}"
|
||||
f"&workflow_run_id={workflow_run.id}"
|
||||
f"&campaign_id={campaign.id}"
|
||||
f"&organization_id={campaign.organization_id}"
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -183,9 +183,7 @@ class CampaignSourceSyncService(ABC):
|
|||
async def get_source_credentials(
|
||||
self, organization_id: int, source_type: str
|
||||
) -> Dict[str, Any]:
|
||||
"""Gets OAuth tokens or API credentials via Nango"""
|
||||
# This would be implemented to work with Nango service
|
||||
# For now, returning placeholder
|
||||
"""Gets source credentials when a sync service requires them."""
|
||||
logger.info(
|
||||
f"Getting credentials for org {organization_id}, source {source_type}"
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,15 +1,12 @@
|
|||
from api.services.campaign.source_sync import CampaignSourceSyncService
|
||||
from api.services.campaign.sources.csv import CSVSyncService
|
||||
from api.services.campaign.sources.google_sheets import GoogleSheetsSyncService
|
||||
|
||||
|
||||
def get_sync_service(source_type: str) -> CampaignSourceSyncService:
|
||||
"""Returns appropriate sync service based on source type"""
|
||||
|
||||
services = {
|
||||
"google-sheet": GoogleSheetsSyncService,
|
||||
"csv": CSVSyncService,
|
||||
# Add more as needed: "hubspot": HubSpotSyncService,
|
||||
}
|
||||
|
||||
service_class = services.get(source_type)
|
||||
|
|
|
|||
|
|
@ -1,5 +1,3 @@
|
|||
"""Campaign source sync services"""
|
||||
|
||||
from .google_sheets import GoogleSheetsSyncService
|
||||
|
||||
__all__ = ["GoogleSheetsSyncService"]
|
||||
__all__: list[str] = []
|
||||
|
|
|
|||
|
|
@ -1,224 +0,0 @@
|
|||
import re
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
import httpx
|
||||
from loguru import logger
|
||||
|
||||
from api.db import db_client
|
||||
from api.services.campaign.source_sync import (
|
||||
CampaignSourceSyncService,
|
||||
ValidationError,
|
||||
ValidationResult,
|
||||
)
|
||||
from api.services.integrations.nango import NangoService
|
||||
|
||||
|
||||
class GoogleSheetsSyncService(CampaignSourceSyncService):
|
||||
"""Implementation for Google Sheets synchronization"""
|
||||
|
||||
def __init__(self):
|
||||
self.nango_service = NangoService()
|
||||
self.sheets_api_base = "https://sheets.googleapis.com/v4/spreadsheets"
|
||||
|
||||
async def _get_access_token(self, organization_id: int) -> str:
|
||||
"""Get OAuth access token for Google Sheets via Nango."""
|
||||
integrations = await db_client.get_integrations_by_organization_id(
|
||||
organization_id
|
||||
)
|
||||
integration = None
|
||||
for intg in integrations:
|
||||
if intg.provider == "google-sheet" and intg.is_active:
|
||||
integration = intg
|
||||
break
|
||||
|
||||
if not integration:
|
||||
raise ValueError("Google Sheets integration not found or inactive")
|
||||
|
||||
token_data = await self.nango_service.get_access_token(
|
||||
connection_id=integration.integration_id, provider_config_key="google-sheet"
|
||||
)
|
||||
return token_data["credentials"]["access_token"]
|
||||
|
||||
async def _fetch_all_sheet_data(
|
||||
self, sheet_url: str, organization_id: int
|
||||
) -> List[List[str]]:
|
||||
"""Fetch all data from a Google Sheet. Returns all rows including header."""
|
||||
access_token = await self._get_access_token(organization_id)
|
||||
sheet_id = self._extract_sheet_id(sheet_url)
|
||||
|
||||
metadata = await self._get_sheet_metadata(sheet_id, access_token)
|
||||
if not metadata.get("sheets"):
|
||||
raise ValueError("No sheets found in the spreadsheet")
|
||||
|
||||
sheet_name = metadata["sheets"][0]["properties"]["title"]
|
||||
|
||||
return await self._fetch_sheet_data(sheet_id, f"{sheet_name}!A:Z", access_token)
|
||||
|
||||
async def validate_source(
|
||||
self, source_id: str, organization_id: Optional[int] = None
|
||||
) -> ValidationResult:
|
||||
"""Validate a Google Sheet source for campaign creation."""
|
||||
if organization_id is None:
|
||||
return ValidationResult(
|
||||
is_valid=False,
|
||||
error=ValidationError(
|
||||
message="Organization ID is required for Google Sheets validation"
|
||||
),
|
||||
)
|
||||
|
||||
# Validate URL format first
|
||||
pattern = r"/spreadsheets/d/([a-zA-Z0-9-_]+)"
|
||||
if not re.search(pattern, source_id):
|
||||
return ValidationResult(
|
||||
is_valid=False,
|
||||
error=ValidationError(
|
||||
message=f"Invalid Google Sheets URL: {source_id}"
|
||||
),
|
||||
)
|
||||
|
||||
try:
|
||||
rows = await self._fetch_all_sheet_data(source_id, organization_id)
|
||||
except ValueError as e:
|
||||
return ValidationResult(
|
||||
is_valid=False,
|
||||
error=ValidationError(message=str(e)),
|
||||
)
|
||||
except httpx.HTTPStatusError as e:
|
||||
logger.error(f"HTTP error fetching Google Sheet: {e.response.status_code}")
|
||||
return ValidationResult(
|
||||
is_valid=False,
|
||||
error=ValidationError(
|
||||
message=f"Failed to fetch Google Sheet data: {e.response.status_code}"
|
||||
),
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Error fetching Google Sheet: {e}")
|
||||
return ValidationResult(
|
||||
is_valid=False,
|
||||
error=ValidationError(message="Failed to fetch Google Sheet data"),
|
||||
)
|
||||
|
||||
if not rows or len(rows) < 2:
|
||||
return ValidationResult(
|
||||
is_valid=False,
|
||||
error=ValidationError(
|
||||
message="Google Sheet must have a header row and at least one data row"
|
||||
),
|
||||
)
|
||||
|
||||
headers = rows[0]
|
||||
data_rows = rows[1:]
|
||||
|
||||
return self.validate_source_data(headers, data_rows)
|
||||
|
||||
async def sync_source_data(self, campaign_id: int) -> int:
|
||||
"""
|
||||
Fetches data from Google Sheets and creates queued_runs
|
||||
"""
|
||||
# Get campaign
|
||||
campaign = await db_client.get_campaign_by_id(campaign_id)
|
||||
if not campaign:
|
||||
raise ValueError(f"Campaign {campaign_id} not found")
|
||||
|
||||
rows = await self._fetch_all_sheet_data(
|
||||
campaign.source_id, campaign.organization_id
|
||||
)
|
||||
|
||||
if not rows or len(rows) < 2:
|
||||
logger.warning(f"No data found in sheet for campaign {campaign_id}")
|
||||
return 0
|
||||
|
||||
headers = self.normalize_headers(rows[0])
|
||||
data_rows = rows[1:]
|
||||
|
||||
sheet_id = self._extract_sheet_id(campaign.source_id)
|
||||
|
||||
queued_runs = []
|
||||
for idx, row_values in enumerate(data_rows, 1):
|
||||
# Pad row to match headers length
|
||||
padded_row = row_values + [""] * (len(headers) - len(row_values))
|
||||
|
||||
# Create context variables dict
|
||||
context_vars = dict(zip(headers, padded_row))
|
||||
|
||||
# Skip if no phone number
|
||||
if not context_vars.get("phone_number"):
|
||||
logger.debug(f"Skipping row {idx}: no phone_number")
|
||||
continue
|
||||
|
||||
# Generate unique source UUID
|
||||
source_uuid = f"sheet_{sheet_id}_row_{idx}"
|
||||
|
||||
queued_runs.append(
|
||||
{
|
||||
"campaign_id": campaign_id,
|
||||
"source_uuid": source_uuid,
|
||||
"context_variables": context_vars,
|
||||
"state": "queued",
|
||||
}
|
||||
)
|
||||
|
||||
# Bulk insert
|
||||
if queued_runs:
|
||||
await db_client.bulk_create_queued_runs(queued_runs)
|
||||
logger.info(
|
||||
f"Created {len(queued_runs)} queued runs for campaign {campaign_id}"
|
||||
)
|
||||
|
||||
# Update campaign total_rows
|
||||
await db_client.update_campaign(
|
||||
campaign_id=campaign_id,
|
||||
total_rows=len(queued_runs),
|
||||
source_sync_status="completed",
|
||||
)
|
||||
|
||||
return len(queued_runs)
|
||||
|
||||
async def _fetch_sheet_data(
|
||||
self, sheet_id: str, range: str, access_token: str
|
||||
) -> List[List[str]]:
|
||||
"""Fetch data from Google Sheets API"""
|
||||
url = f"{self.sheets_api_base}/{sheet_id}/values/{range}"
|
||||
headers = {"Authorization": f"Bearer {access_token}"}
|
||||
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get(url, headers=headers)
|
||||
response.raise_for_status()
|
||||
|
||||
data = response.json()
|
||||
return data.get("values", [])
|
||||
|
||||
async def _get_sheet_metadata(
|
||||
self, sheet_id: str, access_token: str
|
||||
) -> Dict[str, Any]:
|
||||
"""Get sheet metadata including sheet names"""
|
||||
url = f"{self.sheets_api_base}/{sheet_id}"
|
||||
headers = {"Authorization": f"Bearer {access_token}"}
|
||||
|
||||
logger.debug(f"Fetching sheet metadata from URL: {url}")
|
||||
logger.debug(f"Using sheet_id: {sheet_id}")
|
||||
|
||||
async with httpx.AsyncClient() as client:
|
||||
try:
|
||||
response = await client.get(url, headers=headers)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
except httpx.HTTPStatusError as e:
|
||||
logger.error(f"HTTP error {e.response.status_code} for URL: {url}")
|
||||
logger.error(f"Response body: {e.response.text}")
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error fetching sheet metadata: {e}")
|
||||
raise
|
||||
|
||||
def _extract_sheet_id(self, sheet_url: str) -> str:
|
||||
"""
|
||||
Extract sheet ID from various Google Sheets URL formats:
|
||||
- https://docs.google.com/spreadsheets/d/{id}/edit
|
||||
- https://docs.google.com/spreadsheets/d/{id}/edit#gid=0
|
||||
"""
|
||||
pattern = r"/spreadsheets/d/([a-zA-Z0-9-_]+)"
|
||||
match = re.search(pattern, sheet_url)
|
||||
if match:
|
||||
return match.group(1)
|
||||
raise ValueError(f"Invalid Google Sheets URL: {sheet_url}")
|
||||
|
|
@ -47,11 +47,16 @@ class UserConfigurationValidator:
|
|||
ServiceProviders.CAMB.value: self._check_camb_api_key,
|
||||
ServiceProviders.AWS_BEDROCK.value: self._check_aws_bedrock_api_key,
|
||||
ServiceProviders.SPEACHES.value: self._check_speaches_api_key,
|
||||
ServiceProviders.GOOGLE_VERTEX.value: self._check_google_vertex_llm_api_key,
|
||||
ServiceProviders.OPENAI_REALTIME.value: self._check_openai_api_key,
|
||||
ServiceProviders.GROK_REALTIME.value: self._check_grok_realtime_api_key,
|
||||
ServiceProviders.ULTRAVOX_REALTIME.value: self._check_ultravox_realtime_api_key,
|
||||
ServiceProviders.GOOGLE_REALTIME.value: self._check_google_api_key,
|
||||
ServiceProviders.GOOGLE_VERTEX_REALTIME.value: self._check_google_vertex_realtime_api_key,
|
||||
ServiceProviders.ASSEMBLYAI.value: self._check_assemblyai_api_key,
|
||||
ServiceProviders.GLADIA.value: self._check_gladia_api_key,
|
||||
ServiceProviders.RIME.value: self._check_rime_api_key,
|
||||
ServiceProviders.MINIMAX.value: self._check_minimax_api_key,
|
||||
}
|
||||
|
||||
async def validate(
|
||||
|
|
@ -116,6 +121,36 @@ class UserConfigurationValidator:
|
|||
return [{"model": service_name, "message": str(e)}]
|
||||
return []
|
||||
|
||||
# Vertex Realtime uses service-account credentials (or ADC) instead of api_key
|
||||
if provider == ServiceProviders.GOOGLE_VERTEX_REALTIME.value:
|
||||
try:
|
||||
if not self._check_google_vertex_realtime_api_key(
|
||||
provider, service_config
|
||||
):
|
||||
return [
|
||||
{
|
||||
"model": service_name,
|
||||
"message": f"Invalid {provider} configuration",
|
||||
}
|
||||
]
|
||||
except ValueError as e:
|
||||
return [{"model": service_name, "message": str(e)}]
|
||||
return []
|
||||
|
||||
# Vertex LLM uses service-account credentials (or ADC) instead of api_key
|
||||
if provider == ServiceProviders.GOOGLE_VERTEX.value:
|
||||
try:
|
||||
if not self._check_google_vertex_llm_api_key(provider, service_config):
|
||||
return [
|
||||
{
|
||||
"model": service_name,
|
||||
"message": f"Invalid {provider} configuration",
|
||||
}
|
||||
]
|
||||
except ValueError as e:
|
||||
return [{"model": service_name, "message": str(e)}]
|
||||
return []
|
||||
|
||||
# AWS Bedrock uses AWS credentials instead of api_key
|
||||
if provider == ServiceProviders.AWS_BEDROCK.value:
|
||||
try:
|
||||
|
|
@ -130,6 +165,19 @@ class UserConfigurationValidator:
|
|||
return [{"model": service_name, "message": str(e)}]
|
||||
return []
|
||||
|
||||
# MiniMax TTS requires a group_id alongside the API key.
|
||||
# LLM configs don't expose group_id, so only check when the field exists.
|
||||
if provider == ServiceProviders.MINIMAX.value and hasattr(
|
||||
service_config, "group_id"
|
||||
):
|
||||
if not getattr(service_config, "group_id", None):
|
||||
return [
|
||||
{
|
||||
"model": service_name,
|
||||
"message": "group_id is required for MiniMax TTS",
|
||||
}
|
||||
]
|
||||
|
||||
api_key = service_config.api_key
|
||||
|
||||
try:
|
||||
|
|
@ -205,6 +253,12 @@ class UserConfigurationValidator:
|
|||
def _check_openrouter_api_key(self, model: str, api_key: str) -> bool:
|
||||
return True
|
||||
|
||||
def _check_grok_realtime_api_key(self, model: str, api_key: str) -> bool:
|
||||
return True
|
||||
|
||||
def _check_ultravox_realtime_api_key(self, model: str, api_key: str) -> bool:
|
||||
return True
|
||||
|
||||
def _check_speechmatics_api_key(self, model: str, api_key: str) -> bool:
|
||||
return True
|
||||
|
||||
|
|
@ -216,6 +270,20 @@ class UserConfigurationValidator:
|
|||
raise ValueError("base_url is required for Speaches services")
|
||||
return True
|
||||
|
||||
def _check_google_vertex_realtime_api_key(self, model: str, service_config) -> bool:
|
||||
if not getattr(service_config, "project_id", None):
|
||||
raise ValueError("project_id is required for Google Vertex Realtime")
|
||||
if not getattr(service_config, "location", None):
|
||||
raise ValueError("location is required for Google Vertex Realtime")
|
||||
return True
|
||||
|
||||
def _check_google_vertex_llm_api_key(self, model: str, service_config) -> bool:
|
||||
if not getattr(service_config, "project_id", None):
|
||||
raise ValueError("project_id is required for Google Vertex")
|
||||
if not getattr(service_config, "location", None):
|
||||
raise ValueError("location is required for Google Vertex")
|
||||
return True
|
||||
|
||||
def _check_aws_bedrock_api_key(self, model: str, service_config) -> bool:
|
||||
if not service_config.aws_access_key or not service_config.aws_secret_key:
|
||||
raise ValueError("AWS access key and secret key are required for Bedrock")
|
||||
|
|
@ -229,3 +297,8 @@ class UserConfigurationValidator:
|
|||
|
||||
def _check_rime_api_key(self, model: str, api_key: str) -> bool:
|
||||
return True
|
||||
|
||||
def _check_minimax_api_key(self, model: str, api_key: str) -> bool:
|
||||
# MiniMax doesn't publish a cheap key-validation endpoint; trust the key
|
||||
# at save time and surface auth errors at first call (same as Rime/Sarvam).
|
||||
return True
|
||||
|
|
|
|||
|
|
@ -13,31 +13,40 @@ from typing import Any, Dict, Optional
|
|||
|
||||
from api.schemas.user_configuration import UserConfiguration
|
||||
from api.services.configuration.registry import ServiceConfig
|
||||
from api.services.integrations import get_node_secret_fields
|
||||
|
||||
VISIBLE_CHARS = 4 # number of trailing characters to reveal
|
||||
MASK_CHAR = "*"
|
||||
MASK_MARKER = "***" # substring that indicates a masked key
|
||||
SERVICE_SECRET_FIELDS = ("api_key", "credentials", "aws_access_key", "aws_secret_key")
|
||||
|
||||
|
||||
def contains_masked_key(api_key: str | list[str] | None) -> bool:
|
||||
"""Return True if *api_key* looks like a masked placeholder."""
|
||||
if api_key is None:
|
||||
def contains_masked_key(value: str | list[str] | None) -> bool:
|
||||
"""Return True if *value* looks like a masked placeholder."""
|
||||
if value is None:
|
||||
return False
|
||||
keys = api_key if isinstance(api_key, list) else [api_key]
|
||||
keys = value if isinstance(value, list) else [value]
|
||||
return any(MASK_MARKER in k for k in keys)
|
||||
|
||||
|
||||
def check_for_masked_keys(config: "UserConfiguration") -> None:
|
||||
"""Raise ValueError if any service in *config* still has a masked API key."""
|
||||
"""Raise ValueError if any service in *config* still has a masked secret."""
|
||||
for field in ("llm", "tts", "stt", "embeddings", "realtime"):
|
||||
service = getattr(config, field, None)
|
||||
if service is None:
|
||||
continue
|
||||
if contains_masked_key(service.get_all_api_keys()):
|
||||
raise ValueError(
|
||||
f"The {field} api_key appears to be masked. "
|
||||
"Please provide the actual API key, not the masked value."
|
||||
)
|
||||
for secret_field in SERVICE_SECRET_FIELDS:
|
||||
if not hasattr(service, secret_field):
|
||||
continue
|
||||
if secret_field == "api_key" and hasattr(service, "get_all_api_keys"):
|
||||
secret_value = service.get_all_api_keys()
|
||||
else:
|
||||
secret_value = getattr(service, secret_field, None)
|
||||
if contains_masked_key(secret_value):
|
||||
raise ValueError(
|
||||
f"The {field} {secret_field} appears to be masked. "
|
||||
"Please provide the actual value, not the masked value."
|
||||
)
|
||||
|
||||
|
||||
def mask_key(real_key: str, visible: int = VISIBLE_CHARS) -> str:
|
||||
|
|
@ -104,12 +113,14 @@ def _mask_service(service_cfg: Optional[ServiceConfig]) -> Optional[Dict[str, An
|
|||
|
||||
# Work on a dict copy so we don't mutate original models
|
||||
data = service_cfg.model_dump()
|
||||
if "api_key" in data and data["api_key"]:
|
||||
raw = data["api_key"]
|
||||
for secret_field in SERVICE_SECRET_FIELDS:
|
||||
if secret_field not in data or not data[secret_field]:
|
||||
continue
|
||||
raw = data[secret_field]
|
||||
if isinstance(raw, list):
|
||||
data["api_key"] = [mask_key(k) for k in raw]
|
||||
data[secret_field] = [mask_key(k) for k in raw]
|
||||
else:
|
||||
data["api_key"] = mask_key(raw)
|
||||
data[secret_field] = mask_key(raw)
|
||||
return data
|
||||
|
||||
|
||||
|
|
@ -129,14 +140,22 @@ def mask_user_config(config: UserConfiguration) -> Dict[str, Any]:
|
|||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Workflow definition helpers – mask / merge QA-node API keys
|
||||
# Workflow definition helpers – mask / merge node API keys
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_QA_API_KEY_FIELD = "qa_api_key"
|
||||
_NODE_SECRET_FIELDS: dict[str, tuple[str, ...]] = {
|
||||
"qa": ("qa_api_key",),
|
||||
}
|
||||
|
||||
|
||||
def _secret_fields_for_node_type(node_type: str | None) -> tuple[str, ...]:
|
||||
if not node_type:
|
||||
return ()
|
||||
return _NODE_SECRET_FIELDS.get(node_type, ()) or get_node_secret_fields(node_type)
|
||||
|
||||
|
||||
def mask_workflow_definition(workflow_definition: Optional[Dict]) -> Optional[Dict]:
|
||||
"""Return a *shallow copy* of *workflow_definition* with QA-node API keys masked."""
|
||||
"""Return a copy of *workflow_definition* with node secret fields masked."""
|
||||
if not workflow_definition:
|
||||
return workflow_definition
|
||||
|
||||
|
|
@ -144,47 +163,46 @@ def mask_workflow_definition(workflow_definition: Optional[Dict]) -> Optional[Di
|
|||
|
||||
masked = copy.deepcopy(workflow_definition)
|
||||
for node in masked.get("nodes", []):
|
||||
if node.get("type") != "qa":
|
||||
secret_fields = _secret_fields_for_node_type(node.get("type"))
|
||||
if not secret_fields:
|
||||
continue
|
||||
data = node.get("data", {})
|
||||
raw_key = data.get(_QA_API_KEY_FIELD)
|
||||
if raw_key:
|
||||
data[_QA_API_KEY_FIELD] = mask_key(raw_key)
|
||||
for field in secret_fields:
|
||||
raw_key = data.get(field)
|
||||
if raw_key:
|
||||
data[field] = mask_key(raw_key)
|
||||
return masked
|
||||
|
||||
|
||||
def merge_workflow_api_keys(
|
||||
incoming_definition: Optional[Dict], existing_definition: Optional[Dict]
|
||||
) -> Optional[Dict]:
|
||||
"""Preserve real QA-node API keys when the incoming value is a masked placeholder.
|
||||
|
||||
For each QA node in *incoming_definition*, if its ``qa_api_key`` equals
|
||||
the masked form of the corresponding node in *existing_definition*, the
|
||||
real key is restored so it is never lost.
|
||||
"""
|
||||
"""Preserve real node secret fields when the incoming value is masked."""
|
||||
if not incoming_definition or not existing_definition:
|
||||
return incoming_definition
|
||||
|
||||
# Build lookup: node-id → data for existing QA nodes
|
||||
existing_qa: Dict[str, Dict] = {}
|
||||
existing_nodes: Dict[str, Dict] = {}
|
||||
for node in existing_definition.get("nodes", []):
|
||||
if node.get("type") == "qa":
|
||||
existing_qa[node["id"]] = node.get("data", {})
|
||||
if _secret_fields_for_node_type(node.get("type")):
|
||||
existing_nodes[node["id"]] = node.get("data", {})
|
||||
|
||||
for node in incoming_definition.get("nodes", []):
|
||||
if node.get("type") != "qa":
|
||||
secret_fields = _secret_fields_for_node_type(node.get("type"))
|
||||
if not secret_fields:
|
||||
continue
|
||||
data = node.get("data", {})
|
||||
incoming_key = data.get(_QA_API_KEY_FIELD)
|
||||
if not incoming_key:
|
||||
continue
|
||||
|
||||
old_data = existing_qa.get(node["id"])
|
||||
old_data = existing_nodes.get(node["id"])
|
||||
if not old_data:
|
||||
continue
|
||||
|
||||
old_key = old_data.get(_QA_API_KEY_FIELD, "")
|
||||
if old_key and is_mask_of(incoming_key, old_key):
|
||||
data[_QA_API_KEY_FIELD] = old_key
|
||||
for field in secret_fields:
|
||||
incoming_key = data.get(field)
|
||||
if not incoming_key:
|
||||
continue
|
||||
|
||||
old_key = old_data.get(field, "")
|
||||
if old_key and is_mask_of(incoming_key, old_key):
|
||||
data[field] = old_key
|
||||
|
||||
return incoming_definition
|
||||
|
|
|
|||
|
|
@ -7,7 +7,10 @@ stored, while honouring masked API keys.
|
|||
from typing import Dict
|
||||
|
||||
from api.schemas.user_configuration import UserConfiguration
|
||||
from api.services.configuration.masking import resolve_masked_api_keys
|
||||
from api.services.configuration.masking import (
|
||||
SERVICE_SECRET_FIELDS,
|
||||
resolve_masked_api_keys,
|
||||
)
|
||||
|
||||
SERVICE_FIELDS = ("llm", "tts", "stt", "embeddings", "realtime")
|
||||
|
||||
|
|
@ -45,18 +48,16 @@ def merge_user_configurations(
|
|||
and incoming_cfg.get("provider") != old_cfg.get("provider")
|
||||
)
|
||||
|
||||
incoming_api_key = incoming_cfg.get("api_key")
|
||||
|
||||
if not provider_changed:
|
||||
# conditional preservation of api_key
|
||||
if incoming_api_key is not None:
|
||||
if old_cfg and "api_key" in old_cfg:
|
||||
incoming_cfg["api_key"] = resolve_masked_api_keys(
|
||||
incoming_api_key, old_cfg["api_key"]
|
||||
)
|
||||
else:
|
||||
if "api_key" in old_cfg:
|
||||
incoming_cfg["api_key"] = old_cfg["api_key"]
|
||||
for secret_field in SERVICE_SECRET_FIELDS:
|
||||
incoming_secret = incoming_cfg.get(secret_field)
|
||||
if incoming_secret is not None:
|
||||
if old_cfg and secret_field in old_cfg:
|
||||
incoming_cfg[secret_field] = resolve_masked_api_keys(
|
||||
incoming_secret, old_cfg[secret_field]
|
||||
)
|
||||
elif secret_field in old_cfg:
|
||||
incoming_cfg[secret_field] = old_cfg[secret_field]
|
||||
|
||||
merged[service_name] = incoming_cfg
|
||||
|
||||
|
|
|
|||
49
api/services/configuration/options/__init__.py
Normal file
49
api/services/configuration/options/__init__.py
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
from .deepgram import DEEPGRAM_LANGUAGES, DEEPGRAM_STT_MODELS
|
||||
from .gladia import GLADIA_STT_LANGUAGES, GLADIA_STT_MODELS
|
||||
from .google import (
|
||||
GOOGLE_MODELS,
|
||||
GOOGLE_REALTIME_LANGUAGES,
|
||||
GOOGLE_REALTIME_MODELS,
|
||||
GOOGLE_REALTIME_VOICES,
|
||||
GOOGLE_STT_LANGUAGES,
|
||||
GOOGLE_STT_MODELS,
|
||||
GOOGLE_TTS_LANGUAGES,
|
||||
GOOGLE_TTS_MODELS,
|
||||
GOOGLE_TTS_VOICES,
|
||||
GOOGLE_VERTEX_REALTIME_LANGUAGES,
|
||||
GOOGLE_VERTEX_REALTIME_MODELS,
|
||||
GOOGLE_VERTEX_REALTIME_VOICES,
|
||||
)
|
||||
from .sarvam import (
|
||||
SARVAM_LANGUAGES,
|
||||
SARVAM_STT_MODELS,
|
||||
SARVAM_TTS_MODELS,
|
||||
SARVAM_V2_VOICES,
|
||||
SARVAM_V3_VOICES,
|
||||
)
|
||||
from .speechmatics import SPEECHMATICS_STT_LANGUAGES
|
||||
|
||||
__all__ = [
|
||||
"DEEPGRAM_LANGUAGES",
|
||||
"DEEPGRAM_STT_MODELS",
|
||||
"GLADIA_STT_LANGUAGES",
|
||||
"GLADIA_STT_MODELS",
|
||||
"GOOGLE_MODELS",
|
||||
"GOOGLE_REALTIME_LANGUAGES",
|
||||
"GOOGLE_REALTIME_MODELS",
|
||||
"GOOGLE_REALTIME_VOICES",
|
||||
"GOOGLE_STT_LANGUAGES",
|
||||
"GOOGLE_STT_MODELS",
|
||||
"GOOGLE_TTS_LANGUAGES",
|
||||
"GOOGLE_TTS_MODELS",
|
||||
"GOOGLE_TTS_VOICES",
|
||||
"GOOGLE_VERTEX_REALTIME_LANGUAGES",
|
||||
"GOOGLE_VERTEX_REALTIME_MODELS",
|
||||
"GOOGLE_VERTEX_REALTIME_VOICES",
|
||||
"SARVAM_LANGUAGES",
|
||||
"SARVAM_STT_MODELS",
|
||||
"SARVAM_TTS_MODELS",
|
||||
"SARVAM_V2_VOICES",
|
||||
"SARVAM_V3_VOICES",
|
||||
"SPEECHMATICS_STT_LANGUAGES",
|
||||
]
|
||||
84
api/services/configuration/options/deepgram.py
Normal file
84
api/services/configuration/options/deepgram.py
Normal file
|
|
@ -0,0 +1,84 @@
|
|||
DEEPGRAM_STT_MODELS = ("nova-3-general", "flux-general-en", "flux-general-multi")
|
||||
DEEPGRAM_LANGUAGES = (
|
||||
"multi",
|
||||
"ar",
|
||||
"ar-AE",
|
||||
"ar-SA",
|
||||
"ar-QA",
|
||||
"ar-KW",
|
||||
"ar-SY",
|
||||
"ar-LB",
|
||||
"ar-PS",
|
||||
"ar-JO",
|
||||
"ar-EG",
|
||||
"ar-SD",
|
||||
"ar-TD",
|
||||
"ar-MA",
|
||||
"ar-DZ",
|
||||
"ar-TN",
|
||||
"ar-IQ",
|
||||
"ar-IR",
|
||||
"be",
|
||||
"bn",
|
||||
"bs",
|
||||
"bg",
|
||||
"ca",
|
||||
"cs",
|
||||
"da",
|
||||
"da-DK",
|
||||
"de",
|
||||
"de-CH",
|
||||
"el",
|
||||
"en",
|
||||
"en-US",
|
||||
"en-AU",
|
||||
"en-GB",
|
||||
"en-IN",
|
||||
"en-NZ",
|
||||
"es",
|
||||
"es-419",
|
||||
"et",
|
||||
"fa",
|
||||
"fi",
|
||||
"fr",
|
||||
"fr-CA",
|
||||
"he",
|
||||
"hi",
|
||||
"hr",
|
||||
"hu",
|
||||
"id",
|
||||
"it",
|
||||
"ja",
|
||||
"kn",
|
||||
"ko",
|
||||
"ko-KR",
|
||||
"lt",
|
||||
"lv",
|
||||
"mk",
|
||||
"mr",
|
||||
"ms",
|
||||
"nl",
|
||||
"nl-BE",
|
||||
"no",
|
||||
"pl",
|
||||
"pt",
|
||||
"pt-BR",
|
||||
"pt-PT",
|
||||
"ro",
|
||||
"ru",
|
||||
"sk",
|
||||
"sl",
|
||||
"sr",
|
||||
"sv",
|
||||
"sv-SE",
|
||||
"ta",
|
||||
"te",
|
||||
"th",
|
||||
"tl",
|
||||
"tr",
|
||||
"uk",
|
||||
"ur",
|
||||
"vi",
|
||||
"zh-CN",
|
||||
"zh-TW",
|
||||
)
|
||||
103
api/services/configuration/options/gladia.py
Normal file
103
api/services/configuration/options/gladia.py
Normal file
|
|
@ -0,0 +1,103 @@
|
|||
GLADIA_STT_MODELS = ("solaria-1",)
|
||||
GLADIA_STT_LANGUAGES = (
|
||||
"af",
|
||||
"am",
|
||||
"ar",
|
||||
"as",
|
||||
"az",
|
||||
"ba",
|
||||
"be",
|
||||
"bg",
|
||||
"bn",
|
||||
"bo",
|
||||
"br",
|
||||
"bs",
|
||||
"ca",
|
||||
"cs",
|
||||
"cy",
|
||||
"da",
|
||||
"de",
|
||||
"el",
|
||||
"en",
|
||||
"es",
|
||||
"et",
|
||||
"eu",
|
||||
"fa",
|
||||
"fi",
|
||||
"fo",
|
||||
"fr",
|
||||
"gl",
|
||||
"gu",
|
||||
"ha",
|
||||
"haw",
|
||||
"he",
|
||||
"hi",
|
||||
"hr",
|
||||
"ht",
|
||||
"hu",
|
||||
"hy",
|
||||
"id",
|
||||
"is",
|
||||
"it",
|
||||
"ja",
|
||||
"jw",
|
||||
"ka",
|
||||
"kk",
|
||||
"km",
|
||||
"kn",
|
||||
"ko",
|
||||
"la",
|
||||
"lb",
|
||||
"ln",
|
||||
"lo",
|
||||
"lt",
|
||||
"lv",
|
||||
"mg",
|
||||
"mi",
|
||||
"mk",
|
||||
"ml",
|
||||
"mn",
|
||||
"mr",
|
||||
"ms",
|
||||
"mt",
|
||||
"my",
|
||||
"ne",
|
||||
"nl",
|
||||
"nn",
|
||||
"no",
|
||||
"oc",
|
||||
"pa",
|
||||
"pl",
|
||||
"ps",
|
||||
"pt",
|
||||
"ro",
|
||||
"ru",
|
||||
"sa",
|
||||
"sd",
|
||||
"si",
|
||||
"sk",
|
||||
"sl",
|
||||
"sn",
|
||||
"so",
|
||||
"sq",
|
||||
"sr",
|
||||
"su",
|
||||
"sv",
|
||||
"sw",
|
||||
"ta",
|
||||
"te",
|
||||
"tg",
|
||||
"th",
|
||||
"tk",
|
||||
"tl",
|
||||
"tr",
|
||||
"tt",
|
||||
"uk",
|
||||
"ur",
|
||||
"uz",
|
||||
"vi",
|
||||
"wo",
|
||||
"yi",
|
||||
"yo",
|
||||
"zh",
|
||||
)
|
||||
281
api/services/configuration/options/google.py
Normal file
281
api/services/configuration/options/google.py
Normal file
|
|
@ -0,0 +1,281 @@
|
|||
GOOGLE_MODELS = (
|
||||
"gemini-2.0-flash",
|
||||
"gemini-2.0-flash-lite",
|
||||
"gemini-2.5-flash",
|
||||
"gemini-2.5-flash-lite",
|
||||
"gemini-3.5-flash",
|
||||
"gemini-3.5-flash-lite",
|
||||
)
|
||||
GOOGLE_VERTEX_MODELS = (
|
||||
"gemini-2.5-flash",
|
||||
"gemini-2.5-flash-lite",
|
||||
"gemini-3.1-flash-lite",
|
||||
"gemini-3.5-flash",
|
||||
)
|
||||
|
||||
GOOGLE_REALTIME_MODELS = ("gemini-3.1-flash-live-preview",)
|
||||
GOOGLE_REALTIME_VOICES = ("Puck", "Charon", "Kore", "Fenrir", "Aoede")
|
||||
GOOGLE_REALTIME_LANGUAGES = (
|
||||
"ar",
|
||||
"bn",
|
||||
"de",
|
||||
"en",
|
||||
"es",
|
||||
"fr",
|
||||
"gu",
|
||||
"hi",
|
||||
"id",
|
||||
"it",
|
||||
"ja",
|
||||
"kn",
|
||||
"ko",
|
||||
"ml",
|
||||
"mr",
|
||||
"nl",
|
||||
"pl",
|
||||
"pt",
|
||||
"ru",
|
||||
"ta",
|
||||
"te",
|
||||
"th",
|
||||
"tr",
|
||||
"vi",
|
||||
"zh",
|
||||
)
|
||||
|
||||
GOOGLE_VERTEX_REALTIME_MODELS = ("google/gemini-live-2.5-flash-native-audio",)
|
||||
GOOGLE_VERTEX_REALTIME_VOICES = GOOGLE_REALTIME_VOICES
|
||||
GOOGLE_VERTEX_REALTIME_LANGUAGES = GOOGLE_REALTIME_LANGUAGES
|
||||
|
||||
GOOGLE_STT_MODELS = ("latest_long", "latest_short", "chirp_3")
|
||||
# Docs-derived from Google Cloud Speech-to-Text V2 supported languages.
|
||||
GOOGLE_STT_LANGUAGES = (
|
||||
"af-ZA",
|
||||
"am-ET",
|
||||
"ar-AE",
|
||||
"ar-BH",
|
||||
"ar-DZ",
|
||||
"ar-EG",
|
||||
"ar-IL",
|
||||
"ar-IQ",
|
||||
"ar-JO",
|
||||
"ar-KW",
|
||||
"ar-LB",
|
||||
"ar-MA",
|
||||
"ar-MR",
|
||||
"ar-OM",
|
||||
"ar-PS",
|
||||
"ar-QA",
|
||||
"ar-SA",
|
||||
"ar-SY",
|
||||
"ar-TN",
|
||||
"ar-XA",
|
||||
"ar-YE",
|
||||
"as-IN",
|
||||
"ast-ES",
|
||||
"az-AZ",
|
||||
"be-BY",
|
||||
"bg-BG",
|
||||
"bn-BD",
|
||||
"bn-IN",
|
||||
"bs-BA",
|
||||
"ca-ES",
|
||||
"ceb-PH",
|
||||
"ckb-IQ",
|
||||
"cmn-Hans-CN",
|
||||
"cmn-Hant-TW",
|
||||
"cs-CZ",
|
||||
"cy-GB",
|
||||
"da-DK",
|
||||
"de-AT",
|
||||
"de-CH",
|
||||
"de-DE",
|
||||
"el-GR",
|
||||
"en-AU",
|
||||
"en-GB",
|
||||
"en-HK",
|
||||
"en-IE",
|
||||
"en-IN",
|
||||
"en-NZ",
|
||||
"en-PH",
|
||||
"en-PK",
|
||||
"en-SG",
|
||||
"en-US",
|
||||
"es-419",
|
||||
"es-AR",
|
||||
"es-BO",
|
||||
"es-CL",
|
||||
"es-CO",
|
||||
"es-CR",
|
||||
"es-DO",
|
||||
"es-EC",
|
||||
"es-ES",
|
||||
"es-GT",
|
||||
"es-HN",
|
||||
"es-MX",
|
||||
"es-NI",
|
||||
"es-PA",
|
||||
"es-PE",
|
||||
"es-PR",
|
||||
"es-SV",
|
||||
"es-US",
|
||||
"es-UY",
|
||||
"es-VE",
|
||||
"et-EE",
|
||||
"eu-ES",
|
||||
"fa-IR",
|
||||
"ff-SN",
|
||||
"fi-FI",
|
||||
"fil-PH",
|
||||
"fr-BE",
|
||||
"fr-CA",
|
||||
"fr-CH",
|
||||
"fr-FR",
|
||||
"ga-IE",
|
||||
"gl-ES",
|
||||
"gu-IN",
|
||||
"ha-NG",
|
||||
"hi-IN",
|
||||
"hr-HR",
|
||||
"hu-HU",
|
||||
"hy-AM",
|
||||
"id-ID",
|
||||
"ig-NG",
|
||||
"is-IS",
|
||||
"it-CH",
|
||||
"it-IT",
|
||||
"iw-IL",
|
||||
"ja-JP",
|
||||
"jv-ID",
|
||||
"ka-GE",
|
||||
"kam-KE",
|
||||
"kea-CV",
|
||||
"kk-KZ",
|
||||
"km-KH",
|
||||
"kn-IN",
|
||||
"ko-KR",
|
||||
"ky-KG",
|
||||
"lb-LU",
|
||||
"lg-UG",
|
||||
"ln-CD",
|
||||
"lo-LA",
|
||||
"lt-LT",
|
||||
"luo-KE",
|
||||
"lv-LV",
|
||||
"mi-NZ",
|
||||
"mk-MK",
|
||||
"ml-IN",
|
||||
"mn-MN",
|
||||
"mr-IN",
|
||||
"ms-MY",
|
||||
"mt-MT",
|
||||
"my-MM",
|
||||
"ne-NP",
|
||||
"nl-BE",
|
||||
"nl-NL",
|
||||
"no-NO",
|
||||
"nso-ZA",
|
||||
"ny-MW",
|
||||
"oc-FR",
|
||||
"om-ET",
|
||||
"or-IN",
|
||||
"pa-Guru-IN",
|
||||
"pl-PL",
|
||||
"ps-AF",
|
||||
"pt-BR",
|
||||
"pt-PT",
|
||||
"ro-RO",
|
||||
"ru-RU",
|
||||
"rup-BG",
|
||||
"rw-RW",
|
||||
"sd-IN",
|
||||
"si-LK",
|
||||
"sk-SK",
|
||||
"sl-SI",
|
||||
"sn-ZW",
|
||||
"so-SO",
|
||||
"sq-AL",
|
||||
"sr-RS",
|
||||
"ss-Latn-ZA",
|
||||
"st-ZA",
|
||||
"su-ID",
|
||||
"sv-SE",
|
||||
"sw",
|
||||
"sw-KE",
|
||||
"ta-IN",
|
||||
"te-IN",
|
||||
"tg-TJ",
|
||||
"th-TH",
|
||||
"tn-Latn-ZA",
|
||||
"tr-TR",
|
||||
"ts-ZA",
|
||||
"uk-UA",
|
||||
"umb-AO",
|
||||
"ur-PK",
|
||||
"uz-UZ",
|
||||
"ve-ZA",
|
||||
"vi-VN",
|
||||
"wo-SN",
|
||||
"xh-ZA",
|
||||
"yo-NG",
|
||||
"yue-Hant-HK",
|
||||
"zu-ZA",
|
||||
)
|
||||
|
||||
GOOGLE_TTS_MODELS = ("chirp_3_hd",)
|
||||
GOOGLE_TTS_VOICES = ("en-US-Chirp3-HD-Charon",)
|
||||
GOOGLE_TTS_LANGUAGES = (
|
||||
"ar-XA",
|
||||
"bn-IN",
|
||||
"bg-BG",
|
||||
"yue-HK",
|
||||
"hr-HR",
|
||||
"cs-CZ",
|
||||
"da-DK",
|
||||
"nl-BE",
|
||||
"nl-NL",
|
||||
"en-AU",
|
||||
"en-IN",
|
||||
"en-GB",
|
||||
"en-US",
|
||||
"et-EE",
|
||||
"fi-FI",
|
||||
"fr-CA",
|
||||
"fr-FR",
|
||||
"de-DE",
|
||||
"el-GR",
|
||||
"gu-IN",
|
||||
"he-IL",
|
||||
"hi-IN",
|
||||
"hu-HU",
|
||||
"id-ID",
|
||||
"it-IT",
|
||||
"ja-JP",
|
||||
"kn-IN",
|
||||
"ko-KR",
|
||||
"lv-LV",
|
||||
"lt-LT",
|
||||
"ml-IN",
|
||||
"cmn-CN",
|
||||
"mr-IN",
|
||||
"nb-NO",
|
||||
"pl-PL",
|
||||
"pt-BR",
|
||||
"pa-IN",
|
||||
"ro-RO",
|
||||
"ru-RU",
|
||||
"sr-RS",
|
||||
"sk-SK",
|
||||
"sl-SI",
|
||||
"es-ES",
|
||||
"es-US",
|
||||
"sw-KE",
|
||||
"sv-SE",
|
||||
"ta-IN",
|
||||
"te-IN",
|
||||
"th-TH",
|
||||
"tr-TR",
|
||||
"uk-UA",
|
||||
"ur-IN",
|
||||
"vi-VN",
|
||||
)
|
||||
66
api/services/configuration/options/sarvam.py
Normal file
66
api/services/configuration/options/sarvam.py
Normal file
|
|
@ -0,0 +1,66 @@
|
|||
SARVAM_TTS_MODELS = ("bulbul:v2", "bulbul:v3")
|
||||
SARVAM_V2_VOICES = (
|
||||
"anushka",
|
||||
"manisha",
|
||||
"vidya",
|
||||
"arya",
|
||||
"abhilash",
|
||||
"karun",
|
||||
"hitesh",
|
||||
)
|
||||
SARVAM_V3_VOICES = (
|
||||
"shubh",
|
||||
"aditya",
|
||||
"ritu",
|
||||
"priya",
|
||||
"neha",
|
||||
"rahul",
|
||||
"pooja",
|
||||
"rohan",
|
||||
"simran",
|
||||
"kavya",
|
||||
"amit",
|
||||
"dev",
|
||||
"ishita",
|
||||
"shreya",
|
||||
"ratan",
|
||||
"varun",
|
||||
"manan",
|
||||
"sumit",
|
||||
"roopa",
|
||||
"kabir",
|
||||
"aayan",
|
||||
"ashutosh",
|
||||
"advait",
|
||||
"amelia",
|
||||
"sophia",
|
||||
"anand",
|
||||
"tanya",
|
||||
"tarun",
|
||||
"sunny",
|
||||
"mani",
|
||||
"gokul",
|
||||
"vijay",
|
||||
"shruti",
|
||||
"suhani",
|
||||
"mohit",
|
||||
"kavitha",
|
||||
"rehan",
|
||||
"soham",
|
||||
"rupali",
|
||||
)
|
||||
SARVAM_LANGUAGES = (
|
||||
"bn-IN",
|
||||
"en-IN",
|
||||
"gu-IN",
|
||||
"hi-IN",
|
||||
"kn-IN",
|
||||
"ml-IN",
|
||||
"mr-IN",
|
||||
"od-IN",
|
||||
"pa-IN",
|
||||
"ta-IN",
|
||||
"te-IN",
|
||||
"as-IN",
|
||||
)
|
||||
SARVAM_STT_MODELS = ("saarika:v2.5", "saaras:v2")
|
||||
63
api/services/configuration/options/speechmatics.py
Normal file
63
api/services/configuration/options/speechmatics.py
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
SPEECHMATICS_STT_LANGUAGES = (
|
||||
"ar",
|
||||
"ar_en",
|
||||
"ba",
|
||||
"eu",
|
||||
"be",
|
||||
"bn",
|
||||
"bg",
|
||||
"yue",
|
||||
"ca",
|
||||
"hr",
|
||||
"cs",
|
||||
"da",
|
||||
"nl",
|
||||
"en",
|
||||
"eo",
|
||||
"et",
|
||||
"fi",
|
||||
"fr",
|
||||
"gl",
|
||||
"de",
|
||||
"el",
|
||||
"he",
|
||||
"hi",
|
||||
"hu",
|
||||
"id",
|
||||
"ia",
|
||||
"ga",
|
||||
"it",
|
||||
"ja",
|
||||
"ko",
|
||||
"lv",
|
||||
"lt",
|
||||
"ms",
|
||||
"en_ms",
|
||||
"mt",
|
||||
"cmn",
|
||||
"cmn_en",
|
||||
"cmn_en_ms_ta",
|
||||
"mr",
|
||||
"mn",
|
||||
"no",
|
||||
"fa",
|
||||
"pl",
|
||||
"pt",
|
||||
"ro",
|
||||
"ru",
|
||||
"sk",
|
||||
"sl",
|
||||
"es",
|
||||
"sw",
|
||||
"sv",
|
||||
"tl",
|
||||
"ta",
|
||||
"en_ta",
|
||||
"th",
|
||||
"tr",
|
||||
"uk",
|
||||
"ur",
|
||||
"ug",
|
||||
"vi",
|
||||
"cy",
|
||||
)
|
||||
File diff suppressed because it is too large
Load diff
239
api/services/integrations/AGENTS.md
Normal file
239
api/services/integrations/AGENTS.md
Normal file
|
|
@ -0,0 +1,239 @@
|
|||
# Integrations - Plugin Contract
|
||||
|
||||
`api/services/integrations/` is the extension seam for third-party integrations.
|
||||
New integrations should be self-contained here. Do not bleed integration-specific
|
||||
logic into `workflow/dto.py`, `workflow/node_specs/`, `run_pipeline.py`,
|
||||
`event_handlers.py`, or `run_integrations.py` unless you are changing the generic
|
||||
framework itself.
|
||||
|
||||
## Golden Path
|
||||
|
||||
Create a package:
|
||||
|
||||
```text
|
||||
api/services/integrations/<name>/
|
||||
├── __init__.py
|
||||
├── node.py
|
||||
├── runtime.py # optional
|
||||
├── completion.py # optional
|
||||
├── routes.py # optional
|
||||
└── client.py # optional
|
||||
```
|
||||
|
||||
The package self-registers on import via `register_package(...)`. Discovery is
|
||||
automatic: `api/services/integrations/loader.py` imports every submodule under
|
||||
`api.services.integrations` except the reserved internal names `base`, `loader`,
|
||||
and `registry`.
|
||||
|
||||
## Registration Pattern
|
||||
|
||||
`__init__.py` should register one `IntegrationPackageSpec`, following the
|
||||
existing integration packages in this directory.
|
||||
|
||||
Use:
|
||||
|
||||
```python
|
||||
PACKAGE = register_package(
|
||||
IntegrationPackageSpec(
|
||||
name="<package_name>",
|
||||
nodes=(NODE,),
|
||||
create_runtime_sessions=create_runtime_sessions, # optional
|
||||
run_completion=run_completion, # optional
|
||||
routers=(router,), # optional
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
The package name is the registry key. The node `type_name` is the workflow node
|
||||
type string and must stay stable once exposed.
|
||||
|
||||
## Node Model + Spec
|
||||
|
||||
For integration nodes, the Pydantic model is the source of truth. The serialized
|
||||
`NodeSpec` is derived from it.
|
||||
|
||||
Refer to an existing integration node for the overall structure:
|
||||
|
||||
- Define one Pydantic model per node, inheriting
|
||||
`api/services/workflow/node_data.py:BaseNodeData`.
|
||||
- Annotate it with `@node_spec(...)`.
|
||||
- Define fields with `spec_field(...)`.
|
||||
- Generate the external spec with `SPEC = build_spec(ModelClass)`.
|
||||
- Register the node with `IntegrationNodeRegistration(...)`.
|
||||
|
||||
Important rules:
|
||||
|
||||
- Put runtime validation in the model, not in the generated spec.
|
||||
Example: conditional requiredness belongs in `@model_validator(mode="after")`.
|
||||
- Keep `@node_spec(name=...)` and `IntegrationNodeRegistration.type_name`
|
||||
identical. They are the same workflow node type string.
|
||||
- Put wire constraints in the field itself where possible.
|
||||
Example: `gt=0`, `min_length=1`, `pattern=...`.
|
||||
- Put UI/export-only differences in `field_overrides`.
|
||||
Use this for `display_name`, `description`, `required`, `spec_default`,
|
||||
`display_options`, or property ordering.
|
||||
- Use `spec_exclude=True` for internal fields that must exist in persisted data
|
||||
but must not show up in `/api/v1/node-types`.
|
||||
- Set `property_order=(...)` in `@node_spec(...)` when the editor field order
|
||||
must remain stable.
|
||||
|
||||
Typical workflow graph constraints for configuration-only integration nodes:
|
||||
|
||||
```python
|
||||
GraphConstraints(min_incoming=0, max_incoming=0, min_outgoing=0, max_outgoing=0)
|
||||
```
|
||||
|
||||
These constraints control how the node can be connected in the workflow graph.
|
||||
Use them for configuration nodes that are not conversational graph steps.
|
||||
|
||||
## Secret Fields
|
||||
|
||||
If the node stores secrets, register them in
|
||||
`IntegrationNodeRegistration.sensitive_fields`.
|
||||
|
||||
That is enough for generic masking / masked round-trip preservation via
|
||||
`api/services/configuration/masking.py`. Do not add new integration-specific
|
||||
masking branches unless you are changing the shared masking framework.
|
||||
|
||||
## No Central DTO Edits
|
||||
|
||||
Do not add integration node classes to `api/services/workflow/dto.py`.
|
||||
|
||||
Integration nodes are resolved dynamically through:
|
||||
|
||||
- `get_node_data_model()` in `workflow/dto.py`
|
||||
- `get_node_spec()` / `all_node_specs()` in `services/integrations/registry.py`
|
||||
|
||||
`RFNodeDTO` validates integration nodes by `type` through the registry. That is
|
||||
the intended extension path.
|
||||
|
||||
## Live Call Path
|
||||
|
||||
If the integration needs live call data, implement `create_runtime_sessions(...)`
|
||||
in `runtime.py` and return `IntegrationRuntimeSession` objects.
|
||||
|
||||
The generic wiring is already in `api/services/pipecat/run_pipeline.py`:
|
||||
|
||||
- `create_runtime_sessions(IntegrationRuntimeContext(...))` is called before the
|
||||
pipeline task starts.
|
||||
- Each returned session gets `session.attach(task)` called.
|
||||
|
||||
Use this only for lightweight live collection:
|
||||
|
||||
- attach task observers
|
||||
- read context messages
|
||||
- capture timing / turn / tool events
|
||||
- build an in-memory snapshot
|
||||
|
||||
Do not do outbound network I/O in the live path unless there is a very strong
|
||||
reason. Prefer the standard pattern: collect live, deliver after the call.
|
||||
|
||||
`IntegrationRuntimeContext` gives you:
|
||||
|
||||
- `workflow_run_id`
|
||||
- `workflow_run`
|
||||
- `workflow_graph`
|
||||
- `run_definition`
|
||||
- `user_config`
|
||||
- `is_realtime`
|
||||
- `context_messages_provider`
|
||||
|
||||
Typical runtime pattern:
|
||||
|
||||
- scan `context.workflow_graph.nodes.values()` for enabled nodes of your type
|
||||
- if none are enabled, return `[]`
|
||||
- build one collector/session per workflow run, not per node, unless the
|
||||
integration truly needs multiple independent collectors
|
||||
|
||||
## Call-Finish Snapshot Path
|
||||
|
||||
`api/services/pipecat/event_handlers.py` finalizes runtime sessions before the
|
||||
engine is cleaned up.
|
||||
|
||||
The generic flow:
|
||||
|
||||
1. `on_pipeline_finished` builds `gathered_context`
|
||||
2. each runtime session gets `await session.on_call_finished(...)`
|
||||
3. returned dicts are merged into `integration_logs`
|
||||
4. those logs are persisted into `workflow_run.logs`
|
||||
|
||||
Use `on_call_finished(...)` to emit a compact, serializable snapshot that the
|
||||
post-call completion handler can consume later. Return `None` if there is nothing
|
||||
to persist.
|
||||
|
||||
This is the handoff between the live call path and the post-call task path.
|
||||
|
||||
## Post-Call Completion Path
|
||||
|
||||
If the integration needs durable artifacts, public URLs, retries, or external
|
||||
delivery, implement `run_completion(nodes, context)` in `completion.py`.
|
||||
|
||||
The generic orchestration is already in `api/tasks/run_integrations.py`:
|
||||
|
||||
1. load the pinned workflow definition from the workflow run
|
||||
2. create a public token if post-call work exists
|
||||
3. run QA nodes first
|
||||
4. run registered integration completion handlers
|
||||
5. run webhook nodes last
|
||||
|
||||
Your handler receives:
|
||||
|
||||
- `nodes`: raw workflow node dicts for your node types only
|
||||
- `IntegrationCompletionContext`:
|
||||
- `workflow_run_id`
|
||||
- `workflow_run`
|
||||
- `workflow_definition`
|
||||
- `definition_id`
|
||||
- `organization_id`
|
||||
- `public_token`
|
||||
|
||||
Expected completion handler pattern:
|
||||
|
||||
- validate each node with `YourNodeData.model_validate(node.get("data", {}))`
|
||||
- skip disabled nodes
|
||||
- read any runtime snapshot from `context.workflow_run.logs`
|
||||
- build durable URLs using `public_token` when appropriate
|
||||
- perform external delivery
|
||||
- return a result dict keyed per node, usually with `node_id` embedded
|
||||
|
||||
Returned data is merged into `workflow_run.annotations`.
|
||||
|
||||
Do not assume completion runs inside the live pipeline process. Treat it as a
|
||||
separate post-call worker step.
|
||||
|
||||
## Optional Routes
|
||||
|
||||
If an integration exposes HTTP routes, put them in `routes.py` and include the
|
||||
router in `IntegrationPackageSpec.routers`.
|
||||
|
||||
Routers are mounted automatically by `api/routes/main.py` through `all_routers()`.
|
||||
Do not edit `routes/main.py` for per-integration route wiring.
|
||||
|
||||
## Import Discipline
|
||||
|
||||
Keep package import side effects light.
|
||||
|
||||
The integration loader runs during:
|
||||
|
||||
- node-type/spec enumeration
|
||||
- tests
|
||||
- route startup
|
||||
- registry access
|
||||
|
||||
So avoid top-level imports that require environment variables, network access,
|
||||
or heavyweight initialization when possible. Prefer lazy imports inside
|
||||
`run_completion()` / `create_runtime_sessions()` if the dependency is optional or
|
||||
environment-sensitive.
|
||||
|
||||
## Testing Expectations
|
||||
|
||||
At minimum, new integrations should add coverage for:
|
||||
|
||||
- node model validation
|
||||
- generated spec/example validity
|
||||
- secret masking + masked round-trip preservation if secrets exist
|
||||
- runtime snapshot creation if live collectors exist
|
||||
- completion handler happy path and disabled-node skip path
|
||||
|
||||
If you change shared integration machinery, test the framework in the generic
|
||||
code path, not only the concrete integration.
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
from api.services.integrations.base import (
|
||||
IntegrationCompletionContext,
|
||||
IntegrationNodeRegistration,
|
||||
IntegrationPackageSpec,
|
||||
IntegrationRuntimeContext,
|
||||
IntegrationRuntimeSession,
|
||||
)
|
||||
from api.services.integrations.registry import (
|
||||
all_node_specs,
|
||||
all_packages,
|
||||
all_routers,
|
||||
create_runtime_sessions,
|
||||
get_node_data_model,
|
||||
get_node_registration,
|
||||
get_node_secret_fields,
|
||||
get_node_spec,
|
||||
has_completion_handlers,
|
||||
register_package,
|
||||
run_completion_handlers,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"IntegrationCompletionContext",
|
||||
"IntegrationNodeRegistration",
|
||||
"IntegrationPackageSpec",
|
||||
"IntegrationRuntimeContext",
|
||||
"IntegrationRuntimeSession",
|
||||
"all_node_specs",
|
||||
"all_packages",
|
||||
"all_routers",
|
||||
"create_runtime_sessions",
|
||||
"get_node_data_model",
|
||||
"get_node_registration",
|
||||
"get_node_secret_fields",
|
||||
"get_node_spec",
|
||||
"has_completion_handlers",
|
||||
"register_package",
|
||||
"run_completion_handlers",
|
||||
]
|
||||
69
api/services/integrations/base.py
Normal file
69
api/services/integrations/base.py
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Awaitable, Callable, Protocol
|
||||
|
||||
from fastapi import APIRouter
|
||||
|
||||
from api.services.workflow.node_data import BaseNodeData
|
||||
from api.services.workflow.node_specs._base import NodeSpec
|
||||
|
||||
|
||||
class IntegrationRuntimeSession(Protocol):
|
||||
name: str
|
||||
|
||||
def attach(self, task: Any) -> None: ...
|
||||
|
||||
async def on_call_finished(
|
||||
self,
|
||||
*,
|
||||
gathered_context: dict[str, Any],
|
||||
) -> dict[str, Any] | None: ...
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class IntegrationRuntimeContext:
|
||||
workflow_run_id: int
|
||||
workflow_run: Any
|
||||
workflow_graph: Any
|
||||
run_definition: Any
|
||||
user_config: Any
|
||||
is_realtime: bool
|
||||
context_messages_provider: Callable[[], list[dict[str, Any]]]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class IntegrationCompletionContext:
|
||||
workflow_run_id: int
|
||||
workflow_run: Any
|
||||
workflow_definition: dict[str, Any]
|
||||
definition_id: int | None
|
||||
organization_id: int
|
||||
public_token: str | None
|
||||
|
||||
|
||||
RuntimeFactory = Callable[
|
||||
[IntegrationRuntimeContext],
|
||||
list[IntegrationRuntimeSession],
|
||||
]
|
||||
CompletionHandler = Callable[
|
||||
[list[dict[str, Any]], IntegrationCompletionContext],
|
||||
Awaitable[dict[str, Any]],
|
||||
]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class IntegrationNodeRegistration:
|
||||
type_name: str
|
||||
data_model: type[BaseNodeData]
|
||||
node_spec: NodeSpec
|
||||
sensitive_fields: tuple[str, ...] = ()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class IntegrationPackageSpec:
|
||||
name: str
|
||||
nodes: tuple[IntegrationNodeRegistration, ...] = ()
|
||||
routers: tuple[APIRouter, ...] = ()
|
||||
create_runtime_sessions: RuntimeFactory | None = None
|
||||
run_completion: CompletionHandler | None = None
|
||||
21
api/services/integrations/loader.py
Normal file
21
api/services/integrations/loader.py
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import pkgutil
|
||||
|
||||
_INTERNAL_MODULES = {"base", "loader", "registry"}
|
||||
_loaded = False
|
||||
|
||||
|
||||
def ensure_integrations_loaded() -> None:
|
||||
global _loaded
|
||||
if _loaded:
|
||||
return
|
||||
|
||||
package = importlib.import_module("api.services.integrations")
|
||||
for module_info in pkgutil.iter_modules(package.__path__):
|
||||
if module_info.name in _INTERNAL_MODULES:
|
||||
continue
|
||||
importlib.import_module(f"{package.__name__}.{module_info.name}")
|
||||
|
||||
_loaded = True
|
||||
|
|
@ -1,253 +0,0 @@
|
|||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from typing import Any, Dict
|
||||
|
||||
import httpx
|
||||
from fastapi import HTTPException
|
||||
from loguru import logger
|
||||
from pydantic import BaseModel
|
||||
|
||||
from api.db import db_client
|
||||
|
||||
NANGO_ALLOWED_INTEGRATIONS = [
|
||||
i.strip() for i in os.environ.get("NANGO_ALLOWED_INTEGRATIONS", "slack").split(",")
|
||||
]
|
||||
|
||||
|
||||
class NangoWebhookRequest(BaseModel):
|
||||
type: str
|
||||
connectionId: str
|
||||
providerConfigKey: str
|
||||
authMode: str
|
||||
provider: str
|
||||
environment: str
|
||||
operation: str
|
||||
endUser: dict # Contains endUserId and organizationId
|
||||
success: bool
|
||||
|
||||
|
||||
class NangoService:
|
||||
def __init__(self):
|
||||
self.base_url = "https://api.nango.dev"
|
||||
self.secret_key = os.getenv("NANGO_API_KEY")
|
||||
|
||||
def _verify_webhook_signature(
|
||||
self, request_body: str, signature: str = None
|
||||
) -> bool:
|
||||
"""
|
||||
Verify the webhook signature using SHA256 hash.
|
||||
|
||||
Args:
|
||||
request_body: The raw request body as string
|
||||
signature: The signature from request headers (optional for now)
|
||||
|
||||
Returns:
|
||||
True if signature is valid
|
||||
"""
|
||||
expected_signature = self.secret_key + request_body
|
||||
expected_hash = hashlib.sha256(expected_signature.encode("utf-8")).hexdigest()
|
||||
return expected_hash == signature
|
||||
|
||||
async def create_session(
|
||||
self, user_id: str, organization_id: int
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Create a Nango session for the given user and organization.
|
||||
|
||||
Args:
|
||||
user_id: The end user ID
|
||||
organization_id: The organization ID
|
||||
|
||||
Returns:
|
||||
Response from Nango API
|
||||
"""
|
||||
if not self.secret_key:
|
||||
raise ValueError("NANGO_SECRET_KEY environment variable is not set")
|
||||
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.secret_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
payload = {
|
||||
"end_user": {"id": user_id},
|
||||
"organization": {"id": str(organization_id)},
|
||||
"allowed_integrations": NANGO_ALLOWED_INTEGRATIONS,
|
||||
}
|
||||
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.post(
|
||||
f"{self.base_url}/connect/sessions", headers=headers, json=payload
|
||||
)
|
||||
|
||||
if response.status_code != 201:
|
||||
raise httpx.HTTPStatusError(
|
||||
f"Nango API error: {response.status_code}",
|
||||
request=response.request,
|
||||
response=response,
|
||||
)
|
||||
|
||||
return response.json()
|
||||
|
||||
async def process_webhook(
|
||||
self, raw_body: bytes, signature: str = None
|
||||
) -> Dict[str, str]:
|
||||
"""
|
||||
Process incoming Nango webhook request.
|
||||
|
||||
Args:
|
||||
raw_body: The raw request body as bytes
|
||||
signature: Optional signature from request headers
|
||||
|
||||
Returns:
|
||||
Dict with status and message
|
||||
"""
|
||||
# Decode and parse the request body
|
||||
try:
|
||||
body_text = raw_body.decode("utf-8")
|
||||
webhook_json = json.loads(body_text) if body_text else {}
|
||||
logger.debug(f"received webhook from nango: {webhook_json}")
|
||||
except json.JSONDecodeError as e:
|
||||
logger.error(f"JSON decode error: {e} body_text: {body_text}")
|
||||
raise HTTPException(status_code=400, detail=f"Invalid JSON: {str(e)}")
|
||||
|
||||
# Verify webhook signature
|
||||
if not self._verify_webhook_signature(body_text, signature):
|
||||
raise HTTPException(status_code=401, detail="Invalid webhook signature")
|
||||
|
||||
# Parse webhook data
|
||||
try:
|
||||
webhook_data = NangoWebhookRequest(**webhook_json)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to parse webhook data: {e}")
|
||||
raise HTTPException(
|
||||
status_code=400, detail=f"Invalid webhook format: {str(e)}"
|
||||
)
|
||||
|
||||
# Extract user and organization IDs from the webhook payload
|
||||
end_user = webhook_data.endUser
|
||||
if (
|
||||
not end_user
|
||||
or "endUserId" not in end_user
|
||||
or "organizationId" not in end_user
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=400, detail="Missing endUser information in webhook payload"
|
||||
)
|
||||
|
||||
user_id = int(end_user["endUserId"])
|
||||
organization_id = int(end_user["organizationId"])
|
||||
|
||||
# Use the connectionId as the integration_id since it's unique per integration
|
||||
integration_id = webhook_data.connectionId
|
||||
|
||||
# Initialize connection_details
|
||||
connection_details = {}
|
||||
|
||||
# Fetch connection details if type is auth and provider is slack
|
||||
if webhook_data.type == "auth":
|
||||
connection_details = await self._fetch_connection_details(
|
||||
integration_id, webhook_data.provider
|
||||
)
|
||||
|
||||
# Create the integration in the database
|
||||
integration = await db_client.create_integration(
|
||||
integration_id=integration_id,
|
||||
organisation_id=organization_id,
|
||||
provider=webhook_data.provider,
|
||||
created_by=user_id,
|
||||
is_active=True,
|
||||
connection_details=connection_details,
|
||||
)
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"message": f"Integration created successfully with ID: {integration.id}",
|
||||
}
|
||||
|
||||
async def _fetch_connection_details(
|
||||
self, connection_id: str, provider_key: str
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Fetch connection details from Nango API for a given connection ID.
|
||||
|
||||
Args:
|
||||
connection_id: The connection ID from the webhook
|
||||
|
||||
Returns:
|
||||
Connection details as a dictionary
|
||||
"""
|
||||
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.secret_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
url = f"{self.base_url}/connection/{connection_id}/?provider_config_key={provider_key}"
|
||||
|
||||
async with httpx.AsyncClient() as client:
|
||||
try:
|
||||
response = await client.get(url, headers=headers)
|
||||
|
||||
if response.status_code != 200:
|
||||
logger.error(
|
||||
f"Failed to fetch connection details: {response.status_code} - {response.text}"
|
||||
)
|
||||
raise httpx.HTTPStatusError(
|
||||
f"Nango API error while fetching connection: {response.status_code}",
|
||||
request=response.request,
|
||||
response=response,
|
||||
)
|
||||
|
||||
connection_details = response.json()
|
||||
return connection_details
|
||||
|
||||
except httpx.HTTPError as e:
|
||||
logger.error(f"HTTP error while fetching connection details: {e}")
|
||||
# Return empty dict if API call fails, but log the error
|
||||
return {}
|
||||
|
||||
async def get_access_token(
|
||||
self, connection_id: str, provider_config_key: str
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Get the latest access token for a connection from Nango.
|
||||
|
||||
Args:
|
||||
connection_id: The connection ID
|
||||
provider_config_key: The provider config key (e.g., 'google-sheet')
|
||||
|
||||
Returns:
|
||||
Dict containing access token and other connection details
|
||||
"""
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.secret_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
url = f"{self.base_url}/connection/{connection_id}?provider_config_key={provider_config_key}"
|
||||
|
||||
async with httpx.AsyncClient() as client:
|
||||
try:
|
||||
response = await client.get(url, headers=headers)
|
||||
|
||||
if response.status_code != 200:
|
||||
logger.error(
|
||||
f"Failed to get access token: {response.status_code} - {response.text}"
|
||||
)
|
||||
raise httpx.HTTPStatusError(
|
||||
f"Nango API error: {response.status_code}",
|
||||
request=response.request,
|
||||
response=response,
|
||||
)
|
||||
|
||||
return response.json()
|
||||
|
||||
except httpx.HTTPError as e:
|
||||
logger.error(f"HTTP error while getting access token: {e}")
|
||||
raise
|
||||
|
||||
|
||||
# Create a singleton instance
|
||||
nango_service = NangoService()
|
||||
128
api/services/integrations/registry.py
Normal file
128
api/services/integrations/registry.py
Normal file
|
|
@ -0,0 +1,128 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from api.services.integrations.base import (
|
||||
IntegrationCompletionContext,
|
||||
IntegrationNodeRegistration,
|
||||
IntegrationPackageSpec,
|
||||
IntegrationRuntimeContext,
|
||||
)
|
||||
from api.services.workflow.node_data import BaseNodeData
|
||||
|
||||
_PACKAGE_REGISTRY: dict[str, IntegrationPackageSpec] = {}
|
||||
|
||||
|
||||
def register_package(spec: IntegrationPackageSpec) -> IntegrationPackageSpec:
|
||||
existing = _PACKAGE_REGISTRY.get(spec.name)
|
||||
if existing is not None and existing is not spec:
|
||||
raise ValueError(
|
||||
f"Duplicate integration package registration for {spec.name!r}"
|
||||
)
|
||||
_PACKAGE_REGISTRY[spec.name] = spec
|
||||
return spec
|
||||
|
||||
|
||||
def _ensure_loaded() -> None:
|
||||
from api.services.integrations.loader import ensure_integrations_loaded
|
||||
|
||||
ensure_integrations_loaded()
|
||||
|
||||
|
||||
def all_packages() -> list[IntegrationPackageSpec]:
|
||||
_ensure_loaded()
|
||||
return [_PACKAGE_REGISTRY[name] for name in sorted(_PACKAGE_REGISTRY)]
|
||||
|
||||
|
||||
def get_package(name: str) -> IntegrationPackageSpec | None:
|
||||
_ensure_loaded()
|
||||
return _PACKAGE_REGISTRY.get(name)
|
||||
|
||||
|
||||
def get_node_registration(type_name: str) -> IntegrationNodeRegistration | None:
|
||||
_ensure_loaded()
|
||||
for package in _PACKAGE_REGISTRY.values():
|
||||
for node in package.nodes:
|
||||
if node.type_name == type_name:
|
||||
return node
|
||||
return None
|
||||
|
||||
|
||||
def get_node_data_model(type_name: str) -> type[BaseNodeData] | None:
|
||||
registration = get_node_registration(type_name)
|
||||
return registration.data_model if registration else None
|
||||
|
||||
|
||||
def get_node_spec(type_name: str):
|
||||
registration = get_node_registration(type_name)
|
||||
return registration.node_spec if registration else None
|
||||
|
||||
|
||||
def get_node_secret_fields(type_name: str) -> tuple[str, ...]:
|
||||
registration = get_node_registration(type_name)
|
||||
return registration.sensitive_fields if registration else ()
|
||||
|
||||
|
||||
def all_node_specs():
|
||||
_ensure_loaded()
|
||||
specs = []
|
||||
for package in all_packages():
|
||||
specs.extend(node.node_spec for node in package.nodes)
|
||||
return specs
|
||||
|
||||
|
||||
def all_routers():
|
||||
_ensure_loaded()
|
||||
routers = []
|
||||
for package in all_packages():
|
||||
routers.extend(package.routers)
|
||||
return routers
|
||||
|
||||
|
||||
def create_runtime_sessions(
|
||||
context: IntegrationRuntimeContext,
|
||||
):
|
||||
_ensure_loaded()
|
||||
sessions = []
|
||||
for package in all_packages():
|
||||
if package.create_runtime_sessions is None:
|
||||
continue
|
||||
sessions.extend(package.create_runtime_sessions(context))
|
||||
return sessions
|
||||
|
||||
|
||||
def iter_completion_packages(
|
||||
workflow_definition: dict[str, Any],
|
||||
):
|
||||
_ensure_loaded()
|
||||
nodes = workflow_definition.get("nodes", []) if workflow_definition else []
|
||||
for package in all_packages():
|
||||
node_types = {node.type_name for node in package.nodes}
|
||||
package_nodes = [
|
||||
node
|
||||
for node in nodes
|
||||
if isinstance(node, dict) and node.get("type") in node_types
|
||||
]
|
||||
if package_nodes:
|
||||
yield package, package_nodes
|
||||
|
||||
|
||||
def has_completion_handlers(workflow_definition: dict[str, Any]) -> bool:
|
||||
return any(
|
||||
package.run_completion is not None
|
||||
for package, _nodes in iter_completion_packages(workflow_definition)
|
||||
)
|
||||
|
||||
|
||||
async def run_completion_handlers(
|
||||
*,
|
||||
context: IntegrationCompletionContext,
|
||||
) -> dict[str, Any]:
|
||||
results: dict[str, Any] = {}
|
||||
for package, nodes in iter_completion_packages(context.workflow_definition):
|
||||
if package.run_completion is None:
|
||||
continue
|
||||
package_result = await package.run_completion(nodes, context)
|
||||
if package_result:
|
||||
results.update(package_result)
|
||||
return results
|
||||
19
api/services/integrations/tuner/__init__.py
Normal file
19
api/services/integrations/tuner/__init__.py
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from api.services.integrations.base import IntegrationPackageSpec
|
||||
from api.services.integrations.registry import register_package
|
||||
|
||||
from .completion import run_completion
|
||||
from .node import NODE
|
||||
from .runtime import create_runtime_sessions
|
||||
|
||||
PACKAGE = register_package(
|
||||
IntegrationPackageSpec(
|
||||
name="tuner",
|
||||
nodes=(NODE,),
|
||||
create_runtime_sessions=create_runtime_sessions,
|
||||
run_completion=run_completion,
|
||||
)
|
||||
)
|
||||
|
||||
__all__ = ["PACKAGE"]
|
||||
71
api/services/integrations/tuner/client.py
Normal file
71
api/services/integrations/tuner/client.py
Normal file
|
|
@ -0,0 +1,71 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
from loguru import logger
|
||||
from pydantic import BaseModel, field_validator
|
||||
|
||||
|
||||
class TunerDeliveryConfig(BaseModel):
|
||||
base_url: str
|
||||
api_key: str
|
||||
workspace_id: int
|
||||
agent_id: str
|
||||
|
||||
@field_validator("api_key", "agent_id")
|
||||
@classmethod
|
||||
def _must_not_be_empty(cls, value: str) -> str:
|
||||
if not value or not value.strip():
|
||||
raise ValueError("must not be empty")
|
||||
return value
|
||||
|
||||
@field_validator("workspace_id")
|
||||
@classmethod
|
||||
def _workspace_must_be_positive(cls, value: int) -> int:
|
||||
if value <= 0:
|
||||
raise ValueError("must be a positive integer")
|
||||
return value
|
||||
|
||||
|
||||
async def post_call(
|
||||
config: TunerDeliveryConfig,
|
||||
payload: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
url = (
|
||||
f"{config.base_url}/api/v1/public/call"
|
||||
f"?workspace_id={config.workspace_id}"
|
||||
f"&agent_remote_identifier={config.agent_id}"
|
||||
)
|
||||
headers = {"Authorization": f"Bearer {config.api_key}"}
|
||||
|
||||
logger.info(
|
||||
"[tuner] posting completed call {} to workspace {} / agent {}",
|
||||
payload.get("call_id"),
|
||||
config.workspace_id,
|
||||
config.agent_id,
|
||||
)
|
||||
|
||||
async with httpx.AsyncClient(timeout=10) as client:
|
||||
response = await client.post(url, json=payload, headers=headers)
|
||||
|
||||
if response.status_code == 409:
|
||||
logger.info("[tuner] call {} already exists in tuner", payload.get("call_id"))
|
||||
return {"status": "duplicate", "status_code": response.status_code}
|
||||
|
||||
if response.status_code >= 400:
|
||||
logger.error(
|
||||
"[tuner] POST failed for call {} with status {}: {}",
|
||||
payload.get("call_id"),
|
||||
response.status_code,
|
||||
response.text[:200],
|
||||
)
|
||||
|
||||
response.raise_for_status()
|
||||
|
||||
logger.info(
|
||||
"[tuner] POST succeeded for call {} with status {}",
|
||||
payload.get("call_id"),
|
||||
response.status_code,
|
||||
)
|
||||
return {"status": "delivered", "status_code": response.status_code}
|
||||
191
api/services/integrations/tuner/collector.py
Normal file
191
api/services/integrations/tuner/collector.py
Normal file
|
|
@ -0,0 +1,191 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from collections import deque
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Callable
|
||||
|
||||
from loguru import logger
|
||||
from pipecat.frames.frames import (
|
||||
BotStartedSpeakingFrame,
|
||||
BotStoppedSpeakingFrame,
|
||||
CancelFrame,
|
||||
EndFrame,
|
||||
FunctionCallInProgressFrame,
|
||||
FunctionCallResultFrame,
|
||||
MetricsFrame,
|
||||
StartFrame,
|
||||
UserStartedSpeakingFrame,
|
||||
UserStoppedSpeakingFrame,
|
||||
VADUserStoppedSpeakingFrame,
|
||||
)
|
||||
from pipecat.observers.base_observer import BaseObserver, FramePushed
|
||||
from pipecat.observers.turn_tracking_observer import TurnTrackingObserver
|
||||
from pipecat.observers.user_bot_latency_observer import UserBotLatencyObserver
|
||||
from pipecat.processors.frame_processor import FrameDirection
|
||||
from pipecat.utils.context.message_sanitization import strip_thought_ids_from_messages
|
||||
from tuner_pipecat_sdk.accumulator import CallAccumulator
|
||||
from tuner_pipecat_sdk.payload_builder import build_payload
|
||||
|
||||
from api.enums import WorkflowRunMode
|
||||
|
||||
TUNER_RECORDING_PLACEHOLDER = "pipecat://no-recording"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _PayloadConfig:
|
||||
call_id: str
|
||||
call_type: str
|
||||
recording_url: str
|
||||
asr_model: str
|
||||
llm_model: str
|
||||
tts_model: str
|
||||
sip_call_id: str | None = None
|
||||
sip_headers: dict[str, str] | None = None
|
||||
agent_version: int | None = None
|
||||
|
||||
|
||||
def mode_to_tuner_call_type(mode: str | None) -> str:
|
||||
if mode in {
|
||||
WorkflowRunMode.WEBRTC.value,
|
||||
WorkflowRunMode.SMALLWEBRTC.value,
|
||||
}:
|
||||
return "web_call"
|
||||
return "phone_call"
|
||||
|
||||
|
||||
class TunerCollector(BaseObserver):
|
||||
"""Collect runtime call metadata and build a deferred Tuner payload."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
workflow_run_id: int,
|
||||
call_type: str,
|
||||
asr_model: str = "",
|
||||
llm_model: str = "",
|
||||
tts_model: str = "",
|
||||
agent_version: int | None = None,
|
||||
max_frames: int = 500,
|
||||
) -> None:
|
||||
super().__init__()
|
||||
self._call_id = str(workflow_run_id)
|
||||
self._call_type = call_type
|
||||
self._asr_model = asr_model
|
||||
self._llm_model = llm_model
|
||||
self._tts_model = tts_model
|
||||
self._agent_version = agent_version
|
||||
self._acc = CallAccumulator()
|
||||
self._acc.call_start_abs_ns = time.time_ns()
|
||||
self._pipeline_start_rel_ns: int | None = None
|
||||
self._context_provider: Callable[[], list[dict[str, Any]]] | None = None
|
||||
self._processed_frames: set[int] = set()
|
||||
self._frame_history: deque[int] = deque(maxlen=max_frames)
|
||||
|
||||
def attach_context(self, provider: Callable[[], list[dict[str, Any]]]) -> None:
|
||||
self._context_provider = provider
|
||||
|
||||
def set_disconnection_reason(self, reason: str | None) -> None:
|
||||
if reason:
|
||||
self._acc.set_disconnection_reason(reason)
|
||||
|
||||
def attach_turn_tracking_observer(
|
||||
self, turn_tracker: TurnTrackingObserver | None
|
||||
) -> None:
|
||||
if turn_tracker is None:
|
||||
return
|
||||
|
||||
@turn_tracker.event_handler("on_turn_started")
|
||||
async def _on_turn_started(_tracker: Any, turn_number: int) -> None:
|
||||
self._acc.on_turn_started(turn_number, time.time_ns())
|
||||
|
||||
@turn_tracker.event_handler("on_turn_ended")
|
||||
async def _on_turn_ended(
|
||||
_tracker: Any, turn_number: int, _duration: float, was_interrupted: bool
|
||||
) -> None:
|
||||
self._acc.on_turn_ended(turn_number, was_interrupted)
|
||||
|
||||
def attach_latency_observer(
|
||||
self, latency_observer: UserBotLatencyObserver | None
|
||||
) -> None:
|
||||
if latency_observer is None:
|
||||
return
|
||||
|
||||
@latency_observer.event_handler("on_latency_measured")
|
||||
async def _on_latency_measured(_observer: Any, latency: float) -> None:
|
||||
self._acc.on_latency_measured(latency)
|
||||
|
||||
@latency_observer.event_handler("on_latency_breakdown")
|
||||
async def _on_latency_breakdown(_observer: Any, breakdown: Any) -> None:
|
||||
self._acc.on_latency_breakdown(breakdown)
|
||||
|
||||
async def on_push_frame(self, data: FramePushed):
|
||||
if data.direction != FrameDirection.DOWNSTREAM:
|
||||
return
|
||||
|
||||
if data.frame.id in self._processed_frames:
|
||||
return
|
||||
|
||||
self._processed_frames.add(data.frame.id)
|
||||
self._frame_history.append(data.frame.id)
|
||||
if len(self._processed_frames) > len(self._frame_history):
|
||||
self._processed_frames = set(self._frame_history)
|
||||
|
||||
frame = data.frame
|
||||
|
||||
# data.timestamp is a pipeline-relative clock (ns since pipeline start).
|
||||
# Convert to absolute ns so the accumulator's _rel_ms() works correctly.
|
||||
if self._pipeline_start_rel_ns is None:
|
||||
self._pipeline_start_rel_ns = data.timestamp
|
||||
timestamp_ns = self._acc.call_start_abs_ns + (
|
||||
data.timestamp - self._pipeline_start_rel_ns
|
||||
)
|
||||
|
||||
if isinstance(frame, StartFrame):
|
||||
self._acc.on_start(timestamp_ns)
|
||||
elif isinstance(frame, FunctionCallInProgressFrame):
|
||||
self._acc.on_function_call_in_progress(frame, timestamp_ns)
|
||||
elif isinstance(frame, FunctionCallResultFrame):
|
||||
self._acc.on_function_call_result(frame.tool_call_id, timestamp_ns)
|
||||
elif isinstance(frame, MetricsFrame):
|
||||
self._acc.on_metrics_frame(frame)
|
||||
elif isinstance(frame, UserStartedSpeakingFrame):
|
||||
self._acc.on_user_started_speaking(timestamp_ns)
|
||||
elif isinstance(frame, UserStoppedSpeakingFrame):
|
||||
self._acc.on_user_stopped_speaking(timestamp_ns)
|
||||
self._acc.on_user_turn_stopped(timestamp_ns)
|
||||
elif isinstance(frame, BotStartedSpeakingFrame):
|
||||
self._acc.on_bot_started_speaking(timestamp_ns)
|
||||
elif isinstance(frame, BotStoppedSpeakingFrame):
|
||||
self._acc.on_bot_stopped(timestamp_ns)
|
||||
elif isinstance(frame, VADUserStoppedSpeakingFrame):
|
||||
self._acc.on_vad_stopped(timestamp_ns)
|
||||
elif isinstance(frame, (CancelFrame, EndFrame)):
|
||||
self._acc.on_call_end(timestamp_ns)
|
||||
|
||||
def build_payload_snapshot(
|
||||
self,
|
||||
*,
|
||||
recording_url: str = TUNER_RECORDING_PLACEHOLDER,
|
||||
) -> dict[str, Any] | None:
|
||||
if self._context_provider is None:
|
||||
logger.warning(
|
||||
"[tuner] no context provider attached; skipping payload snapshot"
|
||||
)
|
||||
return None
|
||||
|
||||
transcript = strip_thought_ids_from_messages(list(self._context_provider()))
|
||||
payload = build_payload(
|
||||
self._acc,
|
||||
_PayloadConfig(
|
||||
call_id=self._call_id,
|
||||
call_type=self._call_type,
|
||||
recording_url=recording_url,
|
||||
asr_model=self._asr_model,
|
||||
llm_model=self._llm_model,
|
||||
tts_model=self._tts_model,
|
||||
agent_version=self._agent_version,
|
||||
),
|
||||
transcript,
|
||||
)
|
||||
return payload.to_dict()
|
||||
76
api/services/integrations/tuner/completion.py
Normal file
76
api/services/integrations/tuner/completion.py
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from api.constants import BACKEND_API_ENDPOINT, TUNER_BASE_URL
|
||||
from api.services.integrations.base import IntegrationCompletionContext
|
||||
|
||||
from .client import TunerDeliveryConfig, post_call
|
||||
from .collector import TUNER_RECORDING_PLACEHOLDER
|
||||
from .node import TunerNodeData
|
||||
|
||||
|
||||
def _build_recording_url(
|
||||
context: IntegrationCompletionContext,
|
||||
) -> str | None:
|
||||
workflow_run = context.workflow_run
|
||||
if context.public_token:
|
||||
base_url = f"{BACKEND_API_ENDPOINT}/api/v1/public/download/workflow/{context.public_token}"
|
||||
return f"{base_url}/recording" if workflow_run.recording_url else None
|
||||
return workflow_run.recording_url
|
||||
|
||||
|
||||
async def run_completion(
|
||||
nodes: list[dict[str, Any]],
|
||||
context: IntegrationCompletionContext,
|
||||
) -> dict[str, Any]:
|
||||
results: dict[str, Any] = {}
|
||||
payload_snapshot = (context.workflow_run.logs or {}).get("tuner_payload")
|
||||
recording_url = _build_recording_url(context) or TUNER_RECORDING_PLACEHOLDER
|
||||
|
||||
for node in nodes:
|
||||
node_id = node.get("id", "unknown")
|
||||
try:
|
||||
tuner_data = TunerNodeData.model_validate(node.get("data", {}))
|
||||
except Exception as exc:
|
||||
logger.warning(f"Tuner node #{node_id} failed validation, skipping: {exc}")
|
||||
results[f"tuner_{node_id}"] = {"error": "validation_failed"}
|
||||
continue
|
||||
|
||||
if not tuner_data.tuner_enabled:
|
||||
logger.debug(f"Tuner node '{tuner_data.name}' is disabled, skipping")
|
||||
continue
|
||||
|
||||
if not payload_snapshot:
|
||||
logger.warning(
|
||||
f"Tuner payload snapshot missing for node '{tuner_data.name}' (#{node_id})"
|
||||
)
|
||||
results[f"tuner_{node_id}"] = {"error": "missing_payload_snapshot"}
|
||||
continue
|
||||
|
||||
payload = copy.deepcopy(payload_snapshot)
|
||||
payload["recording_url"] = recording_url
|
||||
|
||||
try:
|
||||
config = TunerDeliveryConfig(
|
||||
base_url=TUNER_BASE_URL,
|
||||
api_key=tuner_data.tuner_api_key or "",
|
||||
workspace_id=tuner_data.tuner_workspace_id or 0,
|
||||
agent_id=tuner_data.tuner_agent_id or "",
|
||||
)
|
||||
delivery = await post_call(config, payload)
|
||||
results[f"tuner_{node_id}"] = {
|
||||
**delivery,
|
||||
"workspace_id": tuner_data.tuner_workspace_id,
|
||||
"agent_id": tuner_data.tuner_agent_id,
|
||||
"exported_at": datetime.now(UTC).isoformat(),
|
||||
}
|
||||
except Exception as exc:
|
||||
logger.error(f"Tuner export failed for node '{tuner_data.name}': {exc}")
|
||||
results[f"tuner_{node_id}"] = {"error": str(exc)}
|
||||
|
||||
return results
|
||||
139
api/services/integrations/tuner/node.py
Normal file
139
api/services/integrations/tuner/node.py
Normal file
|
|
@ -0,0 +1,139 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from pydantic import model_validator
|
||||
|
||||
from api.services.integrations.base import IntegrationNodeRegistration
|
||||
from api.services.workflow.node_data import BaseNodeData
|
||||
from api.services.workflow.node_specs._base import (
|
||||
GraphConstraints,
|
||||
NodeCategory,
|
||||
NodeExample,
|
||||
PropertyType,
|
||||
)
|
||||
from api.services.workflow.node_specs.model_spec import (
|
||||
build_spec,
|
||||
node_spec,
|
||||
spec_field,
|
||||
)
|
||||
|
||||
|
||||
@node_spec(
|
||||
name="tuner",
|
||||
display_name="Tuner",
|
||||
description="Export the completed call to Tuner for Agent Observability",
|
||||
llm_hint=(
|
||||
"Tuner is a post-call observability export. It does not participate in the "
|
||||
"conversation graph and should not be connected to other nodes."
|
||||
),
|
||||
category=NodeCategory.integration,
|
||||
icon="Activity",
|
||||
examples=[
|
||||
NodeExample(
|
||||
name="tuner_export",
|
||||
data={
|
||||
"name": "Primary Tuner Export",
|
||||
"tuner_enabled": True,
|
||||
"tuner_agent_id": "sales-bot-prod",
|
||||
"tuner_workspace_id": 42,
|
||||
"tuner_api_key": "tuner_live_xxxxxxxx",
|
||||
},
|
||||
)
|
||||
],
|
||||
graph_constraints=GraphConstraints(
|
||||
min_incoming=0,
|
||||
max_incoming=0,
|
||||
min_outgoing=0,
|
||||
max_outgoing=0,
|
||||
),
|
||||
property_order=(
|
||||
"name",
|
||||
"tuner_enabled",
|
||||
"tuner_agent_id",
|
||||
"tuner_workspace_id",
|
||||
"tuner_api_key",
|
||||
),
|
||||
field_overrides={
|
||||
"name": {
|
||||
"spec_default": "Tuner",
|
||||
"description": "Short identifier for this Tuner export configuration.",
|
||||
},
|
||||
"tuner_enabled": {
|
||||
"display_name": "Enabled",
|
||||
"description": "When false, Dograh skips exporting this call to Tuner.",
|
||||
},
|
||||
"tuner_agent_id": {
|
||||
"display_name": "Tuner Agent ID",
|
||||
"description": "The agent identifier registered in your Tuner workspace.",
|
||||
"required": True,
|
||||
},
|
||||
"tuner_workspace_id": {
|
||||
"display_name": "Tuner Workspace ID",
|
||||
"description": "Your numeric Tuner workspace ID.",
|
||||
"required": True,
|
||||
"min_value": 1,
|
||||
},
|
||||
"tuner_api_key": {
|
||||
"display_name": "Tuner API Key",
|
||||
"description": "Bearer token used when posting completed calls to Tuner.",
|
||||
"required": True,
|
||||
},
|
||||
},
|
||||
)
|
||||
class TunerNodeData(BaseNodeData):
|
||||
tuner_enabled: bool = spec_field(
|
||||
default=True,
|
||||
ui_type=PropertyType.boolean,
|
||||
display_name="Enabled",
|
||||
description="When false, Dograh skips exporting this call to Tuner.",
|
||||
)
|
||||
tuner_agent_id: str | None = spec_field(
|
||||
default=None,
|
||||
ui_type=PropertyType.string,
|
||||
display_name="Tuner Agent ID",
|
||||
description="The agent identifier registered in your Tuner workspace.",
|
||||
)
|
||||
tuner_workspace_id: int | None = spec_field(
|
||||
default=None,
|
||||
gt=0,
|
||||
ui_type=PropertyType.number,
|
||||
display_name="Tuner Workspace ID",
|
||||
description="Your numeric Tuner workspace ID.",
|
||||
)
|
||||
tuner_api_key: str | None = spec_field(
|
||||
default=None,
|
||||
ui_type=PropertyType.string,
|
||||
display_name="Tuner API Key",
|
||||
description="Bearer token used when posting completed calls to Tuner.",
|
||||
)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_enabled_config(self):
|
||||
if not self.tuner_enabled:
|
||||
return self
|
||||
|
||||
missing: list[str] = []
|
||||
if not self.tuner_agent_id or not self.tuner_agent_id.strip():
|
||||
missing.append("tuner_agent_id")
|
||||
if self.tuner_workspace_id is None:
|
||||
missing.append("tuner_workspace_id")
|
||||
if not self.tuner_api_key or not self.tuner_api_key.strip():
|
||||
missing.append("tuner_api_key")
|
||||
|
||||
if missing:
|
||||
fields = ", ".join(missing)
|
||||
raise ValueError(
|
||||
f"Tuner node is enabled but missing required fields: {fields}"
|
||||
)
|
||||
|
||||
return self
|
||||
|
||||
|
||||
SPEC = build_spec(TunerNodeData)
|
||||
|
||||
|
||||
NODE = IntegrationNodeRegistration(
|
||||
type_name="tuner",
|
||||
data_model=TunerNodeData,
|
||||
node_spec=SPEC,
|
||||
sensitive_fields=("tuner_api_key",),
|
||||
)
|
||||
101
api/services/integrations/tuner/runtime.py
Normal file
101
api/services/integrations/tuner/runtime.py
Normal file
|
|
@ -0,0 +1,101 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from api.services.configuration.registry import ServiceProviders
|
||||
from api.services.integrations.base import (
|
||||
IntegrationRuntimeContext,
|
||||
IntegrationRuntimeSession,
|
||||
)
|
||||
|
||||
from .collector import TunerCollector, mode_to_tuner_call_type
|
||||
|
||||
|
||||
def _format_model_label(provider: str | None, model: str | None) -> str:
|
||||
if provider and model:
|
||||
return f"{provider}/{model}"
|
||||
if model:
|
||||
return model
|
||||
return provider or ""
|
||||
|
||||
|
||||
def _resolve_model_labels(context: IntegrationRuntimeContext) -> tuple[str, str, str]:
|
||||
user_config = context.user_config
|
||||
|
||||
if context.is_realtime and user_config.realtime:
|
||||
realtime_provider = user_config.realtime.provider
|
||||
realtime_model = user_config.realtime.model
|
||||
llm_model = _format_model_label(realtime_provider, realtime_model)
|
||||
if realtime_provider in {
|
||||
ServiceProviders.GOOGLE_REALTIME.value,
|
||||
ServiceProviders.GOOGLE_VERTEX_REALTIME.value,
|
||||
ServiceProviders.OPENAI_REALTIME.value,
|
||||
}:
|
||||
return "", llm_model, ""
|
||||
return "", llm_model, ""
|
||||
|
||||
return (
|
||||
_format_model_label(
|
||||
getattr(user_config.stt, "provider", None),
|
||||
getattr(user_config.stt, "model", None),
|
||||
),
|
||||
_format_model_label(
|
||||
getattr(user_config.llm, "provider", None),
|
||||
getattr(user_config.llm, "model", None),
|
||||
),
|
||||
_format_model_label(
|
||||
getattr(user_config.tts, "provider", None),
|
||||
getattr(user_config.tts, "model", None),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
class TunerRuntimeSession(IntegrationRuntimeSession):
|
||||
name = "tuner"
|
||||
|
||||
def __init__(self, collector: TunerCollector) -> None:
|
||||
self._collector = collector
|
||||
|
||||
def attach(self, task: Any) -> None:
|
||||
self._collector.attach_turn_tracking_observer(task.turn_tracking_observer)
|
||||
self._collector.attach_latency_observer(task.user_bot_latency_observer)
|
||||
task.add_observer(self._collector)
|
||||
|
||||
async def on_call_finished(
|
||||
self,
|
||||
*,
|
||||
gathered_context: dict[str, Any],
|
||||
) -> dict[str, Any] | None:
|
||||
self._collector.set_disconnection_reason(
|
||||
gathered_context.get("call_disposition")
|
||||
)
|
||||
payload = self._collector.build_payload_snapshot()
|
||||
if payload is None:
|
||||
return None
|
||||
return {"tuner_payload": payload}
|
||||
|
||||
|
||||
def create_runtime_sessions(
|
||||
context: IntegrationRuntimeContext,
|
||||
) -> list[IntegrationRuntimeSession]:
|
||||
tuner_nodes = [
|
||||
node
|
||||
for node in context.workflow_graph.nodes.values()
|
||||
if node.node_type == "tuner" and getattr(node.data, "tuner_enabled", True)
|
||||
]
|
||||
if not tuner_nodes:
|
||||
return []
|
||||
|
||||
asr_model, llm_model, tts_model = _resolve_model_labels(context)
|
||||
|
||||
collector = TunerCollector(
|
||||
workflow_run_id=context.workflow_run_id,
|
||||
call_type=mode_to_tuner_call_type(context.workflow_run.mode),
|
||||
asr_model=asr_model,
|
||||
llm_model=llm_model,
|
||||
tts_model=tts_model,
|
||||
agent_version=getattr(context.run_definition, "version_number", None),
|
||||
)
|
||||
collector.attach_context(context.context_messages_provider)
|
||||
|
||||
return [TunerRuntimeSession(collector)]
|
||||
|
|
@ -1,3 +0,0 @@
|
|||
from .orchestrator import LoopTalkTestOrchestrator
|
||||
|
||||
__all__ = ["LoopTalkTestOrchestrator"]
|
||||
|
|
@ -1,220 +0,0 @@
|
|||
"""
|
||||
Audio streaming processor for LoopTalk real-time audio monitoring.
|
||||
|
||||
This processor captures audio from both actor and adversary agents and streams
|
||||
it to connected WebRTC clients for real-time monitoring.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
from typing import Dict, Set
|
||||
|
||||
from loguru import logger
|
||||
from pipecat.audio.utils import mix_audio
|
||||
from pipecat.frames.frames import (
|
||||
Frame,
|
||||
InputAudioRawFrame,
|
||||
OutputAudioRawFrame,
|
||||
)
|
||||
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
|
||||
|
||||
|
||||
class LoopTalkAudioStreamer(FrameProcessor):
|
||||
"""
|
||||
Processes audio frames from LoopTalk conversations and streams to WebRTC clients.
|
||||
|
||||
This processor sits in the pipeline and captures all audio frames, then
|
||||
forwards them to connected WebRTC clients for real-time monitoring.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
test_session_id: str,
|
||||
role: str, # "actor" or "adversary"
|
||||
**kwargs,
|
||||
):
|
||||
super().__init__(**kwargs)
|
||||
self._test_session_id = test_session_id
|
||||
self._role = role
|
||||
self._listeners: Set[asyncio.Queue] = set()
|
||||
self._sample_rate = 16000 # Default sample rate
|
||||
self._num_channels = 1
|
||||
|
||||
def add_listener(self, queue: asyncio.Queue):
|
||||
"""Add a listener queue for streaming audio."""
|
||||
self._listeners.add(queue)
|
||||
|
||||
def remove_listener(self, queue: asyncio.Queue):
|
||||
"""Remove a listener queue."""
|
||||
self._listeners.discard(queue)
|
||||
|
||||
async def process_frame(self, frame: Frame, direction: FrameDirection):
|
||||
"""Process audio frames and stream to listeners."""
|
||||
await super().process_frame(frame, direction)
|
||||
|
||||
# Capture both input and output audio
|
||||
if isinstance(frame, (InputAudioRawFrame, OutputAudioRawFrame)):
|
||||
# Extract audio data
|
||||
audio_data = frame.audio
|
||||
sample_rate = frame.sample_rate
|
||||
num_channels = frame.num_channels
|
||||
|
||||
# Store sample rate for reference
|
||||
if sample_rate:
|
||||
self._sample_rate = sample_rate
|
||||
if num_channels:
|
||||
self._num_channels = num_channels
|
||||
|
||||
# Stream to all listeners
|
||||
if self._listeners and audio_data:
|
||||
# Create a packet with metadata
|
||||
packet = {
|
||||
"test_session_id": self._test_session_id,
|
||||
"role": self._role,
|
||||
"audio": audio_data,
|
||||
"sample_rate": sample_rate,
|
||||
"num_channels": num_channels,
|
||||
"is_input": isinstance(frame, InputAudioRawFrame),
|
||||
}
|
||||
|
||||
# Send to all listeners without blocking
|
||||
for queue in list(self._listeners):
|
||||
try:
|
||||
queue.put_nowait(packet)
|
||||
except asyncio.QueueFull:
|
||||
logger.warning(
|
||||
f"Audio queue full for session {self._test_session_id}"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Error streaming audio: {e}")
|
||||
self._listeners.discard(queue)
|
||||
elif self._listeners and not audio_data:
|
||||
logger.warning(
|
||||
f"Audio streamer {self._role} received frame with no audio data"
|
||||
)
|
||||
elif audio_data and not self._listeners:
|
||||
# This is expected early in the session before WebSocket connects
|
||||
pass
|
||||
|
||||
# Always forward the frame
|
||||
await self.push_frame(frame, direction)
|
||||
|
||||
|
||||
class LoopTalkAudioMixer:
|
||||
"""
|
||||
Mixes audio from actor and adversary streams for combined playback.
|
||||
|
||||
This class manages the mixing of two audio streams (actor and adversary)
|
||||
to create a combined audio stream for monitoring.
|
||||
"""
|
||||
|
||||
def __init__(self, test_session_id: str):
|
||||
self._test_session_id = test_session_id
|
||||
self._actor_buffer = bytearray()
|
||||
self._adversary_buffer = bytearray()
|
||||
self._listeners: Set[asyncio.Queue] = set()
|
||||
self._sample_rate = 16000
|
||||
self._num_channels = 1
|
||||
self._buffer_size = 8000 # 0.5 seconds at 16kHz
|
||||
|
||||
def add_listener(self, queue: asyncio.Queue):
|
||||
"""Add a listener for mixed audio."""
|
||||
self._listeners.add(queue)
|
||||
|
||||
def remove_listener(self, queue: asyncio.Queue):
|
||||
"""Remove a listener."""
|
||||
self._listeners.discard(queue)
|
||||
|
||||
async def add_audio(
|
||||
self, role: str, audio_data: bytes, sample_rate: int, num_channels: int
|
||||
):
|
||||
"""Add audio data from actor or adversary."""
|
||||
if role == "actor":
|
||||
self._actor_buffer.extend(audio_data)
|
||||
elif role == "adversary":
|
||||
self._adversary_buffer.extend(audio_data)
|
||||
|
||||
# Update audio parameters
|
||||
self._sample_rate = sample_rate
|
||||
self._num_channels = num_channels
|
||||
|
||||
# Check if we have enough data to mix
|
||||
await self._check_and_mix()
|
||||
|
||||
async def _check_and_mix(self):
|
||||
"""Check buffers and mix audio when enough data is available."""
|
||||
# Mix when we have at least buffer_size in both buffers
|
||||
while (
|
||||
len(self._actor_buffer) >= self._buffer_size
|
||||
and len(self._adversary_buffer) >= self._buffer_size
|
||||
):
|
||||
# Extract chunks
|
||||
actor_chunk = bytes(self._actor_buffer[: self._buffer_size])
|
||||
adversary_chunk = bytes(self._adversary_buffer[: self._buffer_size])
|
||||
|
||||
# Remove from buffers
|
||||
del self._actor_buffer[: self._buffer_size]
|
||||
del self._adversary_buffer[: self._buffer_size]
|
||||
|
||||
# Mix audio
|
||||
mixed_audio = mix_audio(actor_chunk, adversary_chunk)
|
||||
|
||||
# Stream to listeners
|
||||
if self._listeners and mixed_audio:
|
||||
packet = {
|
||||
"test_session_id": self._test_session_id,
|
||||
"role": "mixed",
|
||||
"audio": mixed_audio,
|
||||
"sample_rate": self._sample_rate,
|
||||
"num_channels": self._num_channels,
|
||||
"is_input": False,
|
||||
}
|
||||
|
||||
for queue in list(self._listeners):
|
||||
try:
|
||||
queue.put_nowait(packet)
|
||||
except asyncio.QueueFull:
|
||||
logger.warning(
|
||||
f"Mixed audio queue full for session {self._test_session_id}"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Error streaming mixed audio: {e}")
|
||||
self._listeners.discard(queue)
|
||||
|
||||
|
||||
# Global registry for audio streamers and mixers
|
||||
_audio_streamers: Dict[str, Dict[str, LoopTalkAudioStreamer]] = {}
|
||||
_audio_mixers: Dict[str, LoopTalkAudioMixer] = {}
|
||||
|
||||
|
||||
def get_or_create_audio_streamer(
|
||||
test_session_id: str, role: str
|
||||
) -> LoopTalkAudioStreamer:
|
||||
"""Get or create an audio streamer for a test session and role."""
|
||||
if test_session_id not in _audio_streamers:
|
||||
_audio_streamers[test_session_id] = {}
|
||||
|
||||
if role not in _audio_streamers[test_session_id]:
|
||||
_audio_streamers[test_session_id][role] = LoopTalkAudioStreamer(
|
||||
test_session_id=test_session_id, role=role
|
||||
)
|
||||
|
||||
return _audio_streamers[test_session_id][role]
|
||||
|
||||
|
||||
def get_or_create_audio_mixer(test_session_id: str) -> LoopTalkAudioMixer:
|
||||
"""Get or create an audio mixer for a test session."""
|
||||
if test_session_id not in _audio_mixers:
|
||||
_audio_mixers[test_session_id] = LoopTalkAudioMixer(test_session_id)
|
||||
|
||||
return _audio_mixers[test_session_id]
|
||||
|
||||
|
||||
def cleanup_audio_streamers(test_session_id: str):
|
||||
"""Clean up audio streamers and mixers for a test session."""
|
||||
if test_session_id in _audio_streamers:
|
||||
del _audio_streamers[test_session_id]
|
||||
|
||||
if test_session_id in _audio_mixers:
|
||||
del _audio_mixers[test_session_id]
|
||||
|
||||
logger.info(f"Cleaned up audio streamers for test session {test_session_id}")
|
||||
|
|
@ -1 +0,0 @@
|
|||
"""Core modules for LoopTalk orchestration."""
|
||||
|
|
@ -1,170 +0,0 @@
|
|||
"""Pipeline building logic for LoopTalk agents."""
|
||||
|
||||
from typing import Any, Dict
|
||||
|
||||
from loguru import logger
|
||||
from pipecat.pipeline.pipeline import Pipeline
|
||||
from pipecat.processors.aggregators.llm_response_universal import (
|
||||
LLMContextAggregatorPair,
|
||||
)
|
||||
|
||||
from api.db.db_client import DBClient
|
||||
from api.services.looptalk.audio_streamer import get_or_create_audio_streamer
|
||||
from api.services.looptalk.internal_transport import InternalTransport
|
||||
from api.services.pipecat.audio_config import AudioConfig
|
||||
from api.services.pipecat.pipeline_builder import (
|
||||
create_pipeline_components,
|
||||
create_pipeline_task,
|
||||
)
|
||||
from api.services.pipecat.pipeline_engine_callbacks_processor import (
|
||||
PipelineEngineCallbacksProcessor,
|
||||
)
|
||||
from api.services.pipecat.service_factory import (
|
||||
create_llm_service,
|
||||
create_stt_service,
|
||||
create_tts_service,
|
||||
)
|
||||
from api.services.workflow.dto import ReactFlowDTO
|
||||
from api.services.workflow.pipecat_engine import PipecatEngine
|
||||
from api.services.workflow.workflow_graph import WorkflowGraph
|
||||
|
||||
|
||||
class LoopTalkPipelineBuilder:
|
||||
"""Builds pipelines for LoopTalk agents."""
|
||||
|
||||
def __init__(self, db_client: DBClient):
|
||||
"""Initialize the pipeline builder.
|
||||
|
||||
Args:
|
||||
db_client: Database client for fetching user configurations
|
||||
"""
|
||||
self.db_client = db_client
|
||||
|
||||
async def create_agent_pipeline(
|
||||
self,
|
||||
transport: InternalTransport,
|
||||
workflow: Any,
|
||||
test_session_id: int,
|
||||
agent_id: str,
|
||||
role: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""Create a pipeline for an agent (actor or adversary).
|
||||
|
||||
Args:
|
||||
transport: Internal transport for the agent
|
||||
workflow: Workflow model from database
|
||||
test_session_id: ID of the test session
|
||||
agent_id: Unique identifier for the agent
|
||||
role: Either "actor" or "adversary"
|
||||
|
||||
Returns:
|
||||
Dictionary containing pipeline task, engine, and components
|
||||
"""
|
||||
# Get user configuration from database
|
||||
user_config = await self.db_client.get_user_configurations(workflow.user_id)
|
||||
|
||||
# Create pipeline components
|
||||
audio_config = AudioConfig(
|
||||
transport_in_sample_rate=16000,
|
||||
transport_out_sample_rate=16000,
|
||||
vad_sample_rate=16000,
|
||||
pipeline_sample_rate=16000,
|
||||
)
|
||||
|
||||
# Use published definition for graph + configs
|
||||
released_def = workflow.released_definition
|
||||
wf_json = released_def.workflow_json
|
||||
wf_configs = released_def.workflow_configurations or {}
|
||||
|
||||
# Extract keyterms from workflow configurations
|
||||
keyterms = None
|
||||
if wf_configs and "dictionary" in wf_configs:
|
||||
dictionary = wf_configs["dictionary"]
|
||||
if dictionary and isinstance(dictionary, str):
|
||||
keyterms = [
|
||||
term.strip() for term in dictionary.split(",") if term.strip()
|
||||
]
|
||||
if keyterms:
|
||||
logger.info(f"Using {len(keyterms)} keyterms for STT: {keyterms}")
|
||||
|
||||
# Resolve model overrides from the version onto global user config
|
||||
from api.services.configuration.resolve import resolve_effective_config
|
||||
|
||||
model_overrides = wf_configs.get("model_overrides")
|
||||
user_config = resolve_effective_config(user_config, model_overrides)
|
||||
|
||||
# Create services
|
||||
stt = create_stt_service(user_config, audio_config, keyterms=keyterms)
|
||||
llm = create_llm_service(user_config)
|
||||
tts = create_tts_service(user_config, audio_config)
|
||||
|
||||
logger.debug(f"Created services for {role}: STT={stt}, LLM={llm}, TTS={tts}")
|
||||
|
||||
# Get workflow graph
|
||||
workflow_graph = WorkflowGraph(ReactFlowDTO.model_validate(wf_json))
|
||||
|
||||
# Create engine first (needed for create_pipeline_components)
|
||||
engine = PipecatEngine(
|
||||
llm=llm,
|
||||
workflow=workflow_graph,
|
||||
call_context_vars={},
|
||||
workflow_run_id=None, # LoopTalk doesn't have workflow runs
|
||||
)
|
||||
|
||||
# Create pipeline components with audio configuration and engine
|
||||
audio_buffer, transcript, context = create_pipeline_components(
|
||||
audio_config, engine
|
||||
)
|
||||
|
||||
# Set the context and audio_buffer after creation
|
||||
engine.set_context(context)
|
||||
|
||||
context_aggregator = LLMContextAggregatorPair(context)
|
||||
|
||||
# Create pipeline engine callback processor
|
||||
pipeline_engine_callback_processor = PipelineEngineCallbacksProcessor(
|
||||
max_call_duration_seconds=300,
|
||||
max_duration_end_task_callback=engine.create_max_duration_callback(),
|
||||
generation_started_callback=engine.create_generation_started_callback(),
|
||||
)
|
||||
|
||||
# Get aggregators
|
||||
user_context_aggregator = context_aggregator.user()
|
||||
assistant_context_aggregator = context_aggregator.assistant()
|
||||
|
||||
# Get audio streamer for real-time streaming
|
||||
audio_streamer = get_or_create_audio_streamer(str(test_session_id), role)
|
||||
|
||||
# Create pipeline with AudioBufferProcessor after transport.output()
|
||||
pipeline = Pipeline(
|
||||
[
|
||||
transport.input(),
|
||||
audio_streamer, # Stream audio to connected clients
|
||||
stt,
|
||||
transcript.user(),
|
||||
user_context_aggregator,
|
||||
llm,
|
||||
pipeline_engine_callback_processor,
|
||||
tts,
|
||||
transport.output(),
|
||||
audio_buffer, # AudioBufferProcessor - records both input and output audio
|
||||
transcript.assistant(),
|
||||
assistant_context_aggregator,
|
||||
]
|
||||
)
|
||||
|
||||
# Create pipeline task with unique conversation ID for tracing
|
||||
conversation_id = f"{test_session_id}-{role}-{agent_id}"
|
||||
task = create_pipeline_task(pipeline, conversation_id, audio_config)
|
||||
|
||||
# Set the task on the engine
|
||||
engine.set_task(task)
|
||||
|
||||
return {
|
||||
"task": task,
|
||||
"engine": engine,
|
||||
"audio_buffer": audio_buffer,
|
||||
"transcript": transcript,
|
||||
"assistant_context_aggregator": assistant_context_aggregator,
|
||||
"audio_streamer": audio_streamer,
|
||||
}
|
||||
|
|
@ -1,216 +0,0 @@
|
|||
"""Recording management for LoopTalk sessions."""
|
||||
|
||||
import wave
|
||||
from pathlib import Path
|
||||
from typing import Dict, Optional, Tuple
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from api.enums import StorageBackend
|
||||
from api.services.storage import storage_fs
|
||||
|
||||
|
||||
class RecordingManager:
|
||||
"""Manages audio recording and transcript files for LoopTalk sessions."""
|
||||
|
||||
def __init__(self, base_dir: Path):
|
||||
"""Initialize the recording manager.
|
||||
|
||||
Args:
|
||||
base_dir: Base directory for temporary recordings
|
||||
"""
|
||||
self.base_dir = base_dir
|
||||
self.base_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def get_recording_paths(self, test_session_id: int, role: str) -> Dict[str, Path]:
|
||||
"""Get file paths for recordings.
|
||||
|
||||
Args:
|
||||
test_session_id: ID of the test session
|
||||
role: Either "actor" or "adversary"
|
||||
|
||||
Returns:
|
||||
Dictionary with paths for audio, transcript, and temp audio files
|
||||
"""
|
||||
session_dir = self.base_dir / f"session_{test_session_id}"
|
||||
session_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
return {
|
||||
"audio": session_dir / f"{role}_audio.wav",
|
||||
"transcript": session_dir / f"{role}_transcript.txt",
|
||||
"temp_audio": session_dir / f"{role}_audio_temp.pcm",
|
||||
}
|
||||
|
||||
def convert_pcm_to_wav(
|
||||
self,
|
||||
test_session_id: int,
|
||||
role: str,
|
||||
sample_rate: int = 16000,
|
||||
num_channels: int = 1,
|
||||
) -> Optional[Path]:
|
||||
"""Convert PCM audio file to WAV format.
|
||||
|
||||
Args:
|
||||
test_session_id: ID of the test session
|
||||
role: Either "actor" or "adversary"
|
||||
sample_rate: Sample rate of the audio
|
||||
num_channels: Number of audio channels
|
||||
|
||||
Returns:
|
||||
Path to the WAV file if successful, None otherwise
|
||||
"""
|
||||
paths = self.get_recording_paths(test_session_id, role)
|
||||
|
||||
# Check if PCM file exists
|
||||
if not paths["temp_audio"].exists():
|
||||
logger.warning(f"No audio recorded for {role} in session {test_session_id}")
|
||||
return None
|
||||
|
||||
try:
|
||||
# Read PCM data
|
||||
with open(paths["temp_audio"], "rb") as f:
|
||||
pcm_data = f.read()
|
||||
|
||||
# Write WAV file
|
||||
with wave.open(str(paths["audio"]), "wb") as wav_file:
|
||||
wav_file.setnchannels(num_channels)
|
||||
wav_file.setsampwidth(2) # 16-bit audio
|
||||
wav_file.setframerate(sample_rate)
|
||||
wav_file.writeframes(pcm_data)
|
||||
|
||||
# Remove temporary PCM file
|
||||
paths["temp_audio"].unlink()
|
||||
|
||||
logger.info(
|
||||
f"Converted audio to WAV for {role} in session {test_session_id}: {paths['audio']}"
|
||||
)
|
||||
return paths["audio"]
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Failed to convert audio to WAV for {role} in session {test_session_id}: {e}"
|
||||
)
|
||||
return None
|
||||
|
||||
async def upload_recording_to_s3(
|
||||
self, test_session_id: int, role: str
|
||||
) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""Upload recording and transcript to S3.
|
||||
|
||||
Args:
|
||||
test_session_id: ID of the test session
|
||||
role: Either "actor" or "adversary"
|
||||
|
||||
Returns:
|
||||
Tuple of (audio_url, transcript_url) or (None, None) if failed
|
||||
"""
|
||||
paths = self.get_recording_paths(test_session_id, role)
|
||||
audio_url = None
|
||||
transcript_url = None
|
||||
|
||||
# Import here to avoid circular imports
|
||||
|
||||
current_backend = StorageBackend.get_current_backend()
|
||||
logger.info(
|
||||
f"LOOPTALK UPLOAD: Using {current_backend.label} (code: {current_backend.code}) for session {test_session_id}, role: {role}"
|
||||
)
|
||||
|
||||
# Upload audio if exists
|
||||
if paths["audio"].exists():
|
||||
audio_key = f"looptalk/recordings/{test_session_id}/{role}_audio.wav"
|
||||
try:
|
||||
success = await storage_fs.aupload_file(str(paths["audio"]), audio_key)
|
||||
if success:
|
||||
audio_url = audio_key
|
||||
logger.info(
|
||||
f"Uploaded {role} audio to {current_backend.label}: {audio_key}"
|
||||
)
|
||||
else:
|
||||
logger.error(
|
||||
f"Failed to upload {role} audio to {current_backend.label}"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Error uploading {role} audio to {current_backend.label}: {e}"
|
||||
)
|
||||
|
||||
# Upload transcript if exists
|
||||
if paths["transcript"].exists():
|
||||
transcript_key = (
|
||||
f"looptalk/transcripts/{test_session_id}/{role}_transcript.txt"
|
||||
)
|
||||
try:
|
||||
success = await storage_fs.aupload_file(
|
||||
str(paths["transcript"]), transcript_key
|
||||
)
|
||||
if success:
|
||||
transcript_url = transcript_key
|
||||
logger.info(
|
||||
f"Uploaded {role} transcript to {current_backend.label}: {transcript_key}"
|
||||
)
|
||||
else:
|
||||
logger.error(
|
||||
f"Failed to upload {role} transcript to {current_backend.label}"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Error uploading {role} transcript to {current_backend.label}: {e}"
|
||||
)
|
||||
|
||||
return audio_url, transcript_url
|
||||
|
||||
def cleanup_session_files(self, test_session_id: int):
|
||||
"""Clean up local files for a session.
|
||||
|
||||
Args:
|
||||
test_session_id: ID of the test session
|
||||
"""
|
||||
session_dir = self.base_dir / f"session_{test_session_id}"
|
||||
if session_dir.exists():
|
||||
try:
|
||||
import shutil
|
||||
|
||||
shutil.rmtree(session_dir)
|
||||
logger.debug(f"Cleaned up local files for session {test_session_id}")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to clean up session files: {e}")
|
||||
|
||||
def get_recording_info(self, test_session_id: int) -> Dict[str, any]:
|
||||
"""Get information about recordings for a test session.
|
||||
|
||||
Args:
|
||||
test_session_id: ID of the test session
|
||||
|
||||
Returns:
|
||||
Dictionary with recording information
|
||||
"""
|
||||
session_dir = self.base_dir / f"session_{test_session_id}"
|
||||
|
||||
info = {
|
||||
"test_session_id": test_session_id,
|
||||
"recording_dir": str(session_dir),
|
||||
"files": {},
|
||||
}
|
||||
|
||||
for role in ["actor", "adversary"]:
|
||||
paths = self.get_recording_paths(test_session_id, role)
|
||||
role_info = {}
|
||||
|
||||
# Check audio file
|
||||
if paths["audio"].exists():
|
||||
role_info["audio"] = {
|
||||
"path": str(paths["audio"]),
|
||||
"size_bytes": paths["audio"].stat().st_size,
|
||||
}
|
||||
|
||||
# Check transcript file
|
||||
if paths["transcript"].exists():
|
||||
role_info["transcript"] = {
|
||||
"path": str(paths["transcript"]),
|
||||
"size_bytes": paths["transcript"].stat().st_size,
|
||||
}
|
||||
|
||||
if role_info:
|
||||
info["files"][role] = role_info
|
||||
|
||||
return info
|
||||
|
|
@ -1,184 +0,0 @@
|
|||
"""Session management for LoopTalk test sessions."""
|
||||
|
||||
import asyncio
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from loguru import logger
|
||||
|
||||
|
||||
class SessionManager:
|
||||
"""Manages running LoopTalk test sessions."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize the session manager."""
|
||||
self._running_sessions: Dict[int, Dict[str, Any]] = {}
|
||||
self._disconnect_handlers: Dict[int, asyncio.Task] = {}
|
||||
|
||||
def add_session(self, test_session_id: int, session_info: Dict[str, Any]):
|
||||
"""Add a new session to the manager.
|
||||
|
||||
Args:
|
||||
test_session_id: ID of the test session
|
||||
session_info: Dictionary containing session information
|
||||
"""
|
||||
self._running_sessions[test_session_id] = session_info
|
||||
|
||||
def get_session(self, test_session_id: int) -> Optional[Dict[str, Any]]:
|
||||
"""Get session information.
|
||||
|
||||
Args:
|
||||
test_session_id: ID of the test session
|
||||
|
||||
Returns:
|
||||
Session information dictionary or None if not found
|
||||
"""
|
||||
return self._running_sessions.get(test_session_id)
|
||||
|
||||
def remove_session(self, test_session_id: int):
|
||||
"""Remove a session from the manager.
|
||||
|
||||
Args:
|
||||
test_session_id: ID of the test session
|
||||
"""
|
||||
if test_session_id in self._running_sessions:
|
||||
del self._running_sessions[test_session_id]
|
||||
|
||||
# Cancel any disconnect handler for this session
|
||||
if test_session_id in self._disconnect_handlers:
|
||||
handler = self._disconnect_handlers.pop(test_session_id)
|
||||
if not handler.done():
|
||||
handler.cancel()
|
||||
|
||||
def get_active_count(self) -> int:
|
||||
"""Get the number of currently active sessions."""
|
||||
return len(self._running_sessions)
|
||||
|
||||
def get_active_info(self) -> Dict[str, Any]:
|
||||
"""Get information about all active sessions."""
|
||||
return {
|
||||
"count": len(self._running_sessions),
|
||||
"sessions": [
|
||||
{
|
||||
"test_session_id": session_id,
|
||||
"conversation_id": info["conversation"].id,
|
||||
"start_time": info["start_time"],
|
||||
"duration_seconds": int(
|
||||
(datetime.now(UTC) - info["start_time"]).total_seconds()
|
||||
),
|
||||
}
|
||||
for session_id, info in self._running_sessions.items()
|
||||
],
|
||||
}
|
||||
|
||||
async def handle_agent_disconnect(
|
||||
self, test_session_id: int, disconnected_role: str, stop_callback: callable
|
||||
):
|
||||
"""Handle when one agent disconnects.
|
||||
|
||||
This will cancel the other agent as well to ensure clean shutdown.
|
||||
|
||||
Args:
|
||||
test_session_id: ID of the test session
|
||||
disconnected_role: Role that disconnected ("actor" or "adversary")
|
||||
stop_callback: Callback to stop the session
|
||||
"""
|
||||
logger.info(
|
||||
f"Handling {disconnected_role} disconnect for session {test_session_id}"
|
||||
)
|
||||
|
||||
# Check if we already have a disconnect handler running
|
||||
if test_session_id in self._disconnect_handlers:
|
||||
logger.debug(
|
||||
f"Disconnect handler already running for session {test_session_id}"
|
||||
)
|
||||
return
|
||||
|
||||
# Create a task to handle the disconnect
|
||||
async def _handle_disconnect():
|
||||
try:
|
||||
# Wait a short time to avoid race conditions
|
||||
await asyncio.sleep(0.5)
|
||||
|
||||
# Check if session still exists
|
||||
session_info = self.get_session(test_session_id)
|
||||
if not session_info:
|
||||
logger.debug(f"Session {test_session_id} already stopped")
|
||||
return
|
||||
|
||||
# Stop the session (which will cancel both agents)
|
||||
logger.info(
|
||||
f"Stopping session {test_session_id} due to {disconnected_role} disconnect"
|
||||
)
|
||||
await stop_callback(test_session_id)
|
||||
|
||||
except asyncio.CancelledError:
|
||||
logger.debug(
|
||||
f"Disconnect handler cancelled for session {test_session_id}"
|
||||
)
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Error handling disconnect for session {test_session_id}: {e}"
|
||||
)
|
||||
|
||||
# Store the task so we can cancel it if needed
|
||||
self._disconnect_handlers[test_session_id] = asyncio.create_task(
|
||||
_handle_disconnect()
|
||||
)
|
||||
|
||||
def update_audio_metadata(
|
||||
self,
|
||||
test_session_id: int,
|
||||
role: str,
|
||||
sample_rate: Optional[int] = None,
|
||||
num_channels: Optional[int] = None,
|
||||
):
|
||||
"""Update audio metadata for a role in a session.
|
||||
|
||||
Args:
|
||||
test_session_id: ID of the test session
|
||||
role: Either "actor" or "adversary"
|
||||
sample_rate: Sample rate of the audio
|
||||
num_channels: Number of audio channels
|
||||
"""
|
||||
if test_session_id not in self._running_sessions:
|
||||
return
|
||||
|
||||
if "audio_metadata" not in self._running_sessions[test_session_id]:
|
||||
self._running_sessions[test_session_id]["audio_metadata"] = {}
|
||||
|
||||
if role not in self._running_sessions[test_session_id]["audio_metadata"]:
|
||||
self._running_sessions[test_session_id]["audio_metadata"][role] = {}
|
||||
|
||||
metadata = self._running_sessions[test_session_id]["audio_metadata"][role]
|
||||
if sample_rate is not None:
|
||||
metadata["sample_rate"] = sample_rate
|
||||
if num_channels is not None:
|
||||
metadata["num_channels"] = num_channels
|
||||
|
||||
def get_audio_metadata(self, test_session_id: int, role: str) -> Dict[str, Any]:
|
||||
"""Get audio metadata for a role in a session.
|
||||
|
||||
Args:
|
||||
test_session_id: ID of the test session
|
||||
role: Either "actor" or "adversary"
|
||||
|
||||
Returns:
|
||||
Dictionary with sample_rate and num_channels
|
||||
"""
|
||||
default = {"sample_rate": 16000, "num_channels": 1}
|
||||
|
||||
if test_session_id not in self._running_sessions:
|
||||
return default
|
||||
|
||||
metadata = (
|
||||
self._running_sessions.get(test_session_id, {})
|
||||
.get("audio_metadata", {})
|
||||
.get(role, {})
|
||||
)
|
||||
|
||||
return {
|
||||
"sample_rate": metadata.get("sample_rate", 16000),
|
||||
"num_channels": metadata.get("num_channels", 1),
|
||||
}
|
||||
|
|
@ -1,75 +0,0 @@
|
|||
#
|
||||
# Copyright (c) 2024–2025, Daily
|
||||
#
|
||||
# SPDX-License-Identifier: BSD 2-Clause License
|
||||
#
|
||||
|
||||
"""Internal frame serializer for agent-to-agent communication."""
|
||||
|
||||
from loguru import logger
|
||||
from pipecat.frames.frames import (
|
||||
Frame,
|
||||
InputAudioRawFrame,
|
||||
OutputAudioRawFrame,
|
||||
)
|
||||
from pipecat.serializers.base_serializer import FrameSerializer
|
||||
|
||||
|
||||
class InternalFrameSerializer(FrameSerializer):
|
||||
"""Serializer for InternalTransport that filters frames between agents.
|
||||
|
||||
This serializer ensures only audio frames are passed between agents,
|
||||
preventing control frames from creating infinite loops.
|
||||
"""
|
||||
|
||||
async def serialize(self, frame: Frame) -> bytes | None:
|
||||
"""Only serialize audio frames for transmission between agents."""
|
||||
# Only pass audio frames between agents
|
||||
if isinstance(frame, OutputAudioRawFrame):
|
||||
# Use a fixed-size header to avoid parsing issues with binary data
|
||||
# Format: "AUDIO" (5 bytes) + sample_rate (4 bytes) + num_channels (2 bytes) + audio data
|
||||
header = b"AUDIO"
|
||||
sample_rate_bytes = frame.sample_rate.to_bytes(4, byteorder="big")
|
||||
num_channels_bytes = frame.num_channels.to_bytes(2, byteorder="big")
|
||||
|
||||
serialized = header + sample_rate_bytes + num_channels_bytes + frame.audio
|
||||
return serialized
|
||||
|
||||
# Don't pass control frames between agents
|
||||
return None
|
||||
|
||||
async def deserialize(self, data: bytes) -> Frame | None:
|
||||
"""Deserialize audio frames from partner agent."""
|
||||
if data.startswith(b"AUDIO"):
|
||||
try:
|
||||
# Fixed-size header parsing
|
||||
# Header: "AUDIO" (5 bytes) + sample_rate (4 bytes) + num_channels (2 bytes)
|
||||
if len(data) < 11: # Minimum size for header
|
||||
logger.error(
|
||||
f"InternalSerializer: Data too short for header: {len(data)} bytes"
|
||||
)
|
||||
return None
|
||||
|
||||
# Extract fixed-size fields
|
||||
# Skip header validation - we already checked startswith(b"AUDIO")
|
||||
sample_rate = int.from_bytes(data[5:9], byteorder="big")
|
||||
num_channels = int.from_bytes(data[9:11], byteorder="big")
|
||||
|
||||
# Extract audio data - everything after the header
|
||||
audio_data = data[11:]
|
||||
|
||||
# Check if audio data length is valid
|
||||
if len(audio_data) % 2 != 0:
|
||||
logger.warning(
|
||||
f"InternalSerializer: Audio data has odd length: {len(audio_data)}"
|
||||
)
|
||||
|
||||
# Convert to InputAudioRawFrame for the receiving agent
|
||||
return InputAudioRawFrame(
|
||||
audio=audio_data, num_channels=num_channels, sample_rate=sample_rate
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to deserialize audio frame: {e}")
|
||||
return None
|
||||
|
||||
return None
|
||||
|
|
@ -1,405 +0,0 @@
|
|||
#
|
||||
# Copyright (c) 2024–2025, Daily
|
||||
#
|
||||
# SPDX-License-Identifier: BSD 2-Clause License
|
||||
#
|
||||
|
||||
"""Internal transport for in-memory agent-to-agent communication."""
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from typing import Dict, Optional, Tuple
|
||||
|
||||
from loguru import logger
|
||||
from pipecat.frames.frames import (
|
||||
CancelFrame,
|
||||
EndFrame,
|
||||
InputAudioRawFrame,
|
||||
OutputAudioRawFrame,
|
||||
OutputDTMFFrame,
|
||||
OutputDTMFUrgentFrame,
|
||||
OutputImageRawFrame,
|
||||
StartFrame,
|
||||
StopFrame,
|
||||
)
|
||||
from pipecat.processors.frame_processor import FrameDirection
|
||||
from pipecat.transports.base_input import BaseInputTransport
|
||||
from pipecat.transports.base_output import BaseOutputTransport
|
||||
from pipecat.transports.base_transport import BaseTransport, TransportParams
|
||||
|
||||
from api.services.looptalk.internal_serializer import InternalFrameSerializer
|
||||
|
||||
|
||||
class InternalInputTransport(BaseInputTransport):
|
||||
"""Input side of internal transport for agent-to-agent communication."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
transport: Optional["InternalTransport"],
|
||||
params: TransportParams,
|
||||
**kwargs,
|
||||
):
|
||||
"""Initialize internal input transport.
|
||||
|
||||
Args:
|
||||
transport: The parent InternalTransport instance.
|
||||
params: Transport parameters for configuration.
|
||||
**kwargs: Additional keyword arguments including latency_seconds.
|
||||
"""
|
||||
# Extract latency configuration before passing to parent
|
||||
self._latency_seconds = kwargs.pop("latency_seconds", 0.0)
|
||||
|
||||
super().__init__(params, **kwargs)
|
||||
self._transport = transport
|
||||
self._queue: asyncio.Queue[bytes] = asyncio.Queue()
|
||||
self._partner: Optional["InternalOutputTransport"] = None
|
||||
self._running = False
|
||||
self._connected = False
|
||||
self._serializer = InternalFrameSerializer()
|
||||
# Queue for delayed packets (timestamp, data)
|
||||
self._delayed_queue: asyncio.Queue[Tuple[float, bytes]] = asyncio.Queue()
|
||||
self._latency_task: Optional[asyncio.Task] = None
|
||||
|
||||
def set_partner(self, partner: "InternalOutputTransport"):
|
||||
"""Connect this input transport to an output transport."""
|
||||
self._partner = partner
|
||||
|
||||
async def receive_data(self, data: bytes):
|
||||
"""Receive serialized data from the partner output transport."""
|
||||
# logger.debug("received data in input transport")
|
||||
if self._latency_seconds > 0:
|
||||
# Add to delayed queue with delivery timestamp
|
||||
delivery_time = time.monotonic() + self._latency_seconds
|
||||
await self._delayed_queue.put((delivery_time, data))
|
||||
else:
|
||||
# No latency, put directly in the main queue
|
||||
await self._queue.put(data)
|
||||
|
||||
async def start(self, frame: StartFrame):
|
||||
"""Start the input transport."""
|
||||
self._running = True
|
||||
await super().start(frame)
|
||||
await self._serializer.setup(frame)
|
||||
|
||||
# Set transport ready to initialize audio task for VAD processing
|
||||
await self.set_transport_ready(frame)
|
||||
|
||||
# Trigger on_client_connected event for InternalTransport (only once)
|
||||
if hasattr(self, "_transport") and self._transport and not self._connected:
|
||||
self._connected = True
|
||||
await self._transport._call_event_handler(
|
||||
"on_client_connected", self._transport
|
||||
)
|
||||
|
||||
# Start latency processor if latency is configured
|
||||
if self._latency_seconds > 0:
|
||||
self._latency_task = asyncio.create_task(self._latency_processor())
|
||||
|
||||
asyncio.create_task(self._run())
|
||||
|
||||
async def stop(self, frame: EndFrame | StopFrame | None = None):
|
||||
"""Stop the input transport."""
|
||||
self._running = False
|
||||
|
||||
# Stop latency processor
|
||||
if self._latency_task:
|
||||
self._latency_task.cancel()
|
||||
try:
|
||||
await self._latency_task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
self._latency_task = None
|
||||
|
||||
await super().stop(frame)
|
||||
|
||||
# Trigger on_client_disconnected event for InternalTransport
|
||||
if hasattr(self, "_transport") and self._transport:
|
||||
await self._transport._call_event_handler(
|
||||
"on_client_disconnected", self._transport
|
||||
)
|
||||
|
||||
async def _run(self):
|
||||
"""Main loop to process incoming data."""
|
||||
while self._running:
|
||||
try:
|
||||
data = await asyncio.wait_for(self._queue.get(), timeout=0.1)
|
||||
|
||||
# Deserialize the data
|
||||
frame = await self._serializer.deserialize(data)
|
||||
if frame:
|
||||
if isinstance(frame, InputAudioRawFrame):
|
||||
# Debug received audio
|
||||
try:
|
||||
import numpy as np
|
||||
|
||||
# Check if audio length is valid for int16
|
||||
if len(frame.audio) % 2 != 0:
|
||||
logger.error(
|
||||
f"InternalInput: Audio buffer has odd length: {len(frame.audio)}"
|
||||
)
|
||||
else:
|
||||
audio_array = np.frombuffer(frame.audio, dtype=np.int16)
|
||||
# logger.debug(f"InternalInput: Received audio - size: {len(frame.audio)} bytes, "
|
||||
# f"samples: {len(audio_array)}, min: {audio_array.min()}, max: {audio_array.max()}, "
|
||||
# f"sample_rate: {frame.sample_rate}")
|
||||
except Exception as e:
|
||||
logger.error(f"InternalInput: Error analyzing audio: {e}")
|
||||
|
||||
# Use the base class's audio processing which includes VAD
|
||||
await self.push_audio_frame(frame)
|
||||
else:
|
||||
# For non-audio frames, push directly
|
||||
await self.push_frame(frame, FrameDirection.DOWNSTREAM)
|
||||
|
||||
except asyncio.TimeoutError:
|
||||
continue
|
||||
except Exception as e:
|
||||
logger.error(f"Error in internal input transport: {e}")
|
||||
|
||||
async def _latency_processor(self):
|
||||
"""Process delayed packets and deliver them after the configured latency."""
|
||||
logger.info(
|
||||
f"InternalInput: Started latency processor with {self._latency_seconds}s delay"
|
||||
)
|
||||
|
||||
# Use a list to maintain order (we'll process in FIFO order)
|
||||
pending_packets = []
|
||||
|
||||
while self._running:
|
||||
try:
|
||||
# Get all new packets from the delayed queue (non-blocking)
|
||||
while True:
|
||||
try:
|
||||
packet = self._delayed_queue.get_nowait()
|
||||
pending_packets.append(packet)
|
||||
except asyncio.QueueEmpty:
|
||||
break
|
||||
|
||||
# Process packets that are ready
|
||||
current_time = time.monotonic()
|
||||
delivered = []
|
||||
|
||||
for i, (delivery_time, data) in enumerate(pending_packets):
|
||||
if current_time >= delivery_time:
|
||||
# Time to deliver this packet
|
||||
await self._queue.put(data)
|
||||
delivered.append(i)
|
||||
|
||||
# Remove delivered packets (in reverse order to maintain indices)
|
||||
for i in reversed(delivered):
|
||||
pending_packets.pop(i)
|
||||
|
||||
# Sleep briefly before next check
|
||||
await asyncio.sleep(0.005) # 5ms for more responsive delivery
|
||||
|
||||
except asyncio.CancelledError:
|
||||
# Deliver any remaining packets immediately on shutdown
|
||||
for _, data in pending_packets:
|
||||
await self._queue.put(data)
|
||||
break
|
||||
except Exception as e:
|
||||
logger.error(f"Error in latency processor: {e}")
|
||||
await asyncio.sleep(0.01)
|
||||
|
||||
logger.info("InternalInput: Stopped latency processor")
|
||||
|
||||
|
||||
class InternalOutputTransport(BaseOutputTransport):
|
||||
"""Output side of internal transport for agent-to-agent communication."""
|
||||
|
||||
def __init__(self, params: TransportParams, **kwargs):
|
||||
"""Initialize internal output transport.
|
||||
|
||||
Args:
|
||||
params: Transport parameters for configuration.
|
||||
**kwargs: Additional keyword arguments.
|
||||
"""
|
||||
super().__init__(params, **kwargs)
|
||||
self._partner: Optional[InternalInputTransport] = None
|
||||
self._serializer = InternalFrameSerializer()
|
||||
|
||||
# Audio timing synchronization (similar to WebsocketServerOutputTransport)
|
||||
# _send_interval is the time interval between audio chunks in seconds
|
||||
self._send_interval = 0
|
||||
self._next_send_time = 0
|
||||
|
||||
def set_partner(self, partner: InternalInputTransport):
|
||||
"""Connect this output transport to an input transport."""
|
||||
self._partner = partner
|
||||
|
||||
async def start(self, frame: StartFrame):
|
||||
"""Start the output transport."""
|
||||
await super().start(frame)
|
||||
await self._serializer.setup(frame)
|
||||
# Calculate the send interval based on audio chunk size (like WebsocketServerOutputTransport)
|
||||
self._send_interval = (
|
||||
self._params.audio_out_10ms_chunks * 10 / 1000
|
||||
) # Convert ms to seconds
|
||||
await self.set_transport_ready(frame)
|
||||
|
||||
async def write_audio_frame(self, frame: OutputAudioRawFrame):
|
||||
"""Write audio frame to partner through serializer with proper timing."""
|
||||
# Debug audio characteristics
|
||||
# import numpy as np
|
||||
# audio_array = np.frombuffer(frame.audio, dtype=np.int16)
|
||||
# logger.debug(f"InternalOutput: Sending audio - type: {type(frame).__name__}, size: {len(frame.audio)} bytes, "
|
||||
# f"samples: {len(audio_array)}, min: {audio_array.min()}, max: {audio_array.max()}, "
|
||||
# f"sample_rate: {frame.sample_rate}")
|
||||
|
||||
# Serialize and send the audio first
|
||||
data = await self._serializer.serialize(frame)
|
||||
if data and self._partner:
|
||||
await self._partner.receive_data(data)
|
||||
|
||||
# logger.debug(f"InternalOutput: Sent audio frame to partner")
|
||||
|
||||
# Then simulate audio playback timing (following WebsocketServerOutputTransport pattern)
|
||||
await self._write_audio_sleep()
|
||||
|
||||
async def write_video_frame(self, _frame: OutputImageRawFrame):
|
||||
"""Internal transport doesn't support video."""
|
||||
pass
|
||||
|
||||
async def write_dtmf(self, _frame: OutputDTMFFrame | OutputDTMFUrgentFrame):
|
||||
"""Internal transport doesn't support DTMF."""
|
||||
pass
|
||||
|
||||
async def stop(self, frame: EndFrame):
|
||||
"""Stop the output transport and reset timing."""
|
||||
await super().stop(frame)
|
||||
self._next_send_time = 0
|
||||
|
||||
async def cancel(self, frame: CancelFrame):
|
||||
"""Cancel the output transport and reset timing."""
|
||||
await super().cancel(frame)
|
||||
self._next_send_time = 0
|
||||
|
||||
async def _write_audio_sleep(self):
|
||||
"""Simulate audio playback timing (following WebsocketServerOutputTransport pattern)."""
|
||||
# Simulate a clock to ensure audio is sent at real-time pace
|
||||
current_time = time.monotonic()
|
||||
sleep_duration = max(0, self._next_send_time - current_time)
|
||||
await asyncio.sleep(sleep_duration)
|
||||
if sleep_duration == 0:
|
||||
self._next_send_time = time.monotonic() + self._send_interval
|
||||
else:
|
||||
self._next_send_time += self._send_interval
|
||||
|
||||
|
||||
class InternalTransport(BaseTransport):
|
||||
"""Internal transport for in-memory agent-to-agent communication."""
|
||||
|
||||
def __init__(self, params: TransportParams, **kwargs):
|
||||
"""Initialize internal transport.
|
||||
|
||||
Args:
|
||||
params: Transport parameters for configuration.
|
||||
**kwargs: Additional keyword arguments including latency_seconds.
|
||||
"""
|
||||
# Extract latency configuration before passing to parent
|
||||
self._latency_seconds = kwargs.pop("latency_seconds", 0.0)
|
||||
|
||||
super().__init__(**kwargs)
|
||||
self._params = params
|
||||
|
||||
# Create input and output transports
|
||||
self._input = InternalInputTransport(
|
||||
self,
|
||||
params,
|
||||
name=self._input_name or f"{self.name}#input",
|
||||
latency_seconds=self._latency_seconds,
|
||||
)
|
||||
self._output = InternalOutputTransport(
|
||||
params, name=self._output_name or f"{self.name}#output"
|
||||
)
|
||||
|
||||
# Register supported event handlers
|
||||
self._register_event_handler("on_client_connected")
|
||||
self._register_event_handler("on_client_disconnected")
|
||||
|
||||
def input(self) -> InternalInputTransport:
|
||||
"""Get the input transport."""
|
||||
return self._input
|
||||
|
||||
def output(self) -> InternalOutputTransport:
|
||||
"""Get the output transport."""
|
||||
return self._output
|
||||
|
||||
def connect_partner(self, partner: "InternalTransport"):
|
||||
"""Connect this transport to another internal transport."""
|
||||
# Connect output of this transport to input of partner
|
||||
self._output.set_partner(partner._input)
|
||||
# Connect output of partner to input of this transport
|
||||
partner._output.set_partner(self._input)
|
||||
|
||||
|
||||
class InternalTransportManager:
|
||||
"""Manages multiple internal transport pairs for load testing."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize internal transport manager."""
|
||||
self._transport_pairs: Dict[
|
||||
str, Tuple[InternalTransport, InternalTransport]
|
||||
] = {}
|
||||
|
||||
def create_transport_pair(
|
||||
self,
|
||||
test_session_id: str,
|
||||
actor_params: TransportParams,
|
||||
adversary_params: TransportParams,
|
||||
latency_seconds: float = 0.0,
|
||||
) -> Tuple[InternalTransport, InternalTransport]:
|
||||
"""Create a connected pair of internal transports.
|
||||
|
||||
Args:
|
||||
test_session_id: Unique identifier for the test session.
|
||||
actor_params: Transport parameters for the actor.
|
||||
adversary_params: Transport parameters for the adversary.
|
||||
latency_seconds: Simulated network latency in seconds (default: 0.0).
|
||||
|
||||
Returns:
|
||||
Tuple of (actor_transport, adversary_transport).
|
||||
"""
|
||||
# Create actor transport with latency
|
||||
actor_transport = InternalTransport(
|
||||
params=actor_params,
|
||||
name=f"actor-{test_session_id}",
|
||||
latency_seconds=latency_seconds,
|
||||
)
|
||||
|
||||
# Create adversary transport with latency
|
||||
adversary_transport = InternalTransport(
|
||||
params=adversary_params,
|
||||
name=f"adversary-{test_session_id}",
|
||||
latency_seconds=latency_seconds,
|
||||
)
|
||||
|
||||
# Connect them
|
||||
actor_transport.connect_partner(adversary_transport)
|
||||
|
||||
# Store the pair
|
||||
self._transport_pairs[test_session_id] = (actor_transport, adversary_transport)
|
||||
|
||||
logger.info(
|
||||
f"Created internal transport pair for test session: {test_session_id} with {latency_seconds}s latency"
|
||||
)
|
||||
|
||||
return actor_transport, adversary_transport
|
||||
|
||||
def get_transport_pair(
|
||||
self, test_session_id: str
|
||||
) -> Optional[Tuple[InternalTransport, InternalTransport]]:
|
||||
"""Get an existing transport pair."""
|
||||
return self._transport_pairs.get(test_session_id)
|
||||
|
||||
def remove_transport_pair(self, test_session_id: str):
|
||||
"""Remove a transport pair."""
|
||||
if test_session_id in self._transport_pairs:
|
||||
del self._transport_pairs[test_session_id]
|
||||
logger.info(
|
||||
f"Removed internal transport pair for test session: {test_session_id}"
|
||||
)
|
||||
|
||||
def get_active_test_count(self) -> int:
|
||||
"""Get the number of active test sessions."""
|
||||
return len(self._transport_pairs)
|
||||
|
|
@ -1,542 +0,0 @@
|
|||
import asyncio
|
||||
import os
|
||||
import uuid
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from loguru import logger
|
||||
from pipecat.pipeline.task import PipelineTask
|
||||
from pipecat.utils.run_context import set_current_run_id
|
||||
|
||||
from api.db.db_client import DBClient
|
||||
from api.services.looptalk.internal_transport import (
|
||||
InternalTransport,
|
||||
InternalTransportManager,
|
||||
)
|
||||
from api.services.pipecat.transport_setup import create_internal_transport
|
||||
|
||||
from .core.pipeline_builder import LoopTalkPipelineBuilder
|
||||
from .core.recording_manager import RecordingManager
|
||||
from .core.session_manager import SessionManager
|
||||
|
||||
|
||||
class LoopTalkTestOrchestrator:
|
||||
"""Orchestrates LoopTalk testing sessions with agent-to-agent conversations."""
|
||||
|
||||
def __init__(
|
||||
self, db_client: DBClient, network_latency_seconds: Optional[float] = None
|
||||
):
|
||||
self.db_client = db_client
|
||||
self.transport_manager = InternalTransportManager()
|
||||
self.session_manager = SessionManager()
|
||||
self.pipeline_builder = LoopTalkPipelineBuilder(db_client)
|
||||
self.recording_manager = RecordingManager(Path("/tmp/looptalk_recordings"))
|
||||
|
||||
# Default network latency (can be overridden per session)
|
||||
# Priority: constructor param > env var > default (100ms)
|
||||
if network_latency_seconds is not None:
|
||||
self._default_network_latency = network_latency_seconds
|
||||
else:
|
||||
env_latency = os.environ.get("LOOPTALK_NETWORK_LATENCY_MS")
|
||||
if env_latency:
|
||||
try:
|
||||
self._default_network_latency = (
|
||||
float(env_latency) / 1000.0
|
||||
) # Convert ms to seconds
|
||||
except ValueError:
|
||||
logger.warning(
|
||||
f"Invalid LOOPTALK_NETWORK_LATENCY_MS value: {env_latency}, using default 100ms"
|
||||
)
|
||||
self._default_network_latency = 0.1
|
||||
else:
|
||||
self._default_network_latency = 0.1 # 100ms default
|
||||
|
||||
async def start_test_session(
|
||||
self,
|
||||
test_session_id: int,
|
||||
organization_id: int,
|
||||
network_latency_seconds: Optional[float] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Start a LoopTalk test session."""
|
||||
|
||||
# Get test session details
|
||||
test_session = await self.db_client.get_test_session(
|
||||
test_session_id=test_session_id, organization_id=organization_id
|
||||
)
|
||||
|
||||
if not test_session:
|
||||
raise ValueError(f"Test session {test_session_id} not found")
|
||||
|
||||
if test_session.status != "pending":
|
||||
raise ValueError(f"Test session {test_session_id} is not in pending state")
|
||||
|
||||
try:
|
||||
# Update status to running
|
||||
await self.db_client.update_test_session_status(
|
||||
test_session_id=test_session_id, status="running"
|
||||
)
|
||||
|
||||
# Create conversation record
|
||||
conversation = await self.db_client.create_conversation(
|
||||
test_session_id=test_session_id
|
||||
)
|
||||
|
||||
# Create audio configuration for LoopTalk
|
||||
from api.services.pipecat.audio_config import AudioConfig
|
||||
|
||||
audio_config = AudioConfig(
|
||||
transport_in_sample_rate=16000,
|
||||
transport_out_sample_rate=16000,
|
||||
pipeline_sample_rate=16000,
|
||||
)
|
||||
|
||||
# Use provided latency or fall back to default
|
||||
latency = (
|
||||
network_latency_seconds
|
||||
if network_latency_seconds is not None
|
||||
else self._default_network_latency
|
||||
)
|
||||
logger.info(
|
||||
f"Using network latency of {latency}s for test session {test_session_id}"
|
||||
)
|
||||
|
||||
# Generate unique workflow run IDs for each agent
|
||||
actor_workflow_run_id = int(str(test_session_id) + "1")
|
||||
adversary_workflow_run_id = int(str(test_session_id) + "2")
|
||||
|
||||
# Create transports using the new method with turn analyzer
|
||||
actor_transport = create_internal_transport(
|
||||
workflow_run_id=actor_workflow_run_id,
|
||||
audio_config=audio_config,
|
||||
latency_seconds=latency,
|
||||
)
|
||||
adversary_transport = create_internal_transport(
|
||||
workflow_run_id=adversary_workflow_run_id,
|
||||
audio_config=audio_config,
|
||||
latency_seconds=latency,
|
||||
)
|
||||
|
||||
# Connect the transports
|
||||
actor_transport.connect_partner(adversary_transport)
|
||||
|
||||
# Store the transport pair in the manager
|
||||
self.transport_manager._transport_pairs[str(test_session_id)] = (
|
||||
actor_transport,
|
||||
adversary_transport,
|
||||
)
|
||||
|
||||
# Generate unique identifiers for actor and adversary
|
||||
actor_id = f"actor_{test_session_id}_{str(uuid.uuid4())[:8]}"
|
||||
adversary_id = f"adversary_{test_session_id}_{str(uuid.uuid4())[:8]}"
|
||||
|
||||
# Create pipelines for both agents
|
||||
actor_pipeline_info = await self.pipeline_builder.create_agent_pipeline(
|
||||
transport=actor_transport,
|
||||
workflow=test_session.actor_workflow,
|
||||
test_session_id=test_session_id,
|
||||
agent_id=actor_id,
|
||||
role="actor",
|
||||
)
|
||||
actor_pipeline_task = actor_pipeline_info["task"]
|
||||
|
||||
adversary_pipeline_info = await self.pipeline_builder.create_agent_pipeline(
|
||||
transport=adversary_transport,
|
||||
workflow=test_session.adversary_workflow,
|
||||
test_session_id=test_session_id,
|
||||
agent_id=adversary_id,
|
||||
role="adversary",
|
||||
)
|
||||
|
||||
adversary_pipeline_task = adversary_pipeline_info["task"]
|
||||
|
||||
# Register event handlers for both pipelines
|
||||
await self._register_transport_handlers(
|
||||
actor_transport, actor_pipeline_info, test_session_id, "actor"
|
||||
)
|
||||
await self._register_transport_handlers(
|
||||
adversary_transport,
|
||||
adversary_pipeline_info,
|
||||
test_session_id,
|
||||
"adversary",
|
||||
)
|
||||
|
||||
# Store session info
|
||||
session_info = {
|
||||
"test_session": test_session,
|
||||
"conversation": conversation,
|
||||
"actor_task": actor_pipeline_task,
|
||||
"adversary_task": adversary_pipeline_task,
|
||||
"actor_transport": actor_transport,
|
||||
"adversary_transport": adversary_transport,
|
||||
"start_time": datetime.now(UTC),
|
||||
}
|
||||
self.session_manager.add_session(test_session_id, session_info)
|
||||
|
||||
# Start both pipelines in background tasks
|
||||
from pipecat.pipeline.base_task import PipelineTaskParams
|
||||
|
||||
params = PipelineTaskParams(loop=asyncio.get_event_loop())
|
||||
|
||||
# Start the pipelines - this will trigger initialization through the normal pipeline start process
|
||||
# The workflow engines will be initialized when the pipeline starts
|
||||
|
||||
# Create conversation IDs for tracing
|
||||
actor_conversation_id = f"{test_session_id}-actor-{actor_id}"
|
||||
adversary_conversation_id = f"{test_session_id}-adversary-{adversary_id}"
|
||||
|
||||
# Create tasks but don't await them - they'll run in the background
|
||||
logger.debug(f"Running actor task with ID: {actor_id}")
|
||||
actor_task_future = asyncio.create_task(
|
||||
self._run_pipeline_with_context(
|
||||
actor_pipeline_task,
|
||||
params,
|
||||
actor_id,
|
||||
actor_conversation_id,
|
||||
"actor",
|
||||
)
|
||||
)
|
||||
|
||||
logger.debug(f"Running adversary task with ID: {adversary_id}")
|
||||
adversary_task_future = asyncio.create_task(
|
||||
self._run_pipeline_with_context(
|
||||
adversary_pipeline_task,
|
||||
params,
|
||||
adversary_id,
|
||||
adversary_conversation_id,
|
||||
"adversary",
|
||||
)
|
||||
)
|
||||
|
||||
# Store the futures so we can monitor them
|
||||
session_info["actor_task_future"] = actor_task_future
|
||||
session_info["adversary_task_future"] = adversary_task_future
|
||||
|
||||
logger.info(f"Started LoopTalk test session {test_session_id}")
|
||||
|
||||
return {
|
||||
"test_session_id": test_session_id,
|
||||
"conversation_id": conversation.id,
|
||||
"status": "running",
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to start test session {test_session_id}: {e}")
|
||||
await self.db_client.update_test_session_status(
|
||||
test_session_id=test_session_id, status="failed", error=str(e)
|
||||
)
|
||||
raise
|
||||
|
||||
async def _register_transport_handlers(
|
||||
self,
|
||||
transport: InternalTransport,
|
||||
pipeline_info: Dict[str, Any],
|
||||
test_session_id: int,
|
||||
role: str,
|
||||
):
|
||||
"""Register transport event handlers for a pipeline.
|
||||
|
||||
Args:
|
||||
transport: The transport to register handlers on
|
||||
pipeline_info: Dictionary containing pipeline components
|
||||
test_session_id: ID of the test session
|
||||
role: Either "actor" or "adversary"
|
||||
"""
|
||||
engine = pipeline_info["engine"]
|
||||
task = pipeline_info["task"]
|
||||
audio_buffer = pipeline_info["audio_buffer"]
|
||||
transcript = pipeline_info["transcript"]
|
||||
assistant_context_aggregator = pipeline_info["assistant_context_aggregator"]
|
||||
|
||||
# Register transport event handlers
|
||||
@transport.event_handler("on_client_connected")
|
||||
async def on_client_connected(transport, participant):
|
||||
logger.debug(f"LoopTalk {role} client connected - initializing workflow")
|
||||
# Start audio recording
|
||||
await audio_buffer.start_recording()
|
||||
await engine.initialize()
|
||||
|
||||
@transport.event_handler("on_client_disconnected")
|
||||
async def on_client_disconnected(transport, participant):
|
||||
logger.debug(f"LoopTalk {role} client disconnected")
|
||||
# Stop audio recording
|
||||
await audio_buffer.stop_recording()
|
||||
|
||||
# Handle disconnect propagation - stop the other agent too
|
||||
await self.session_manager.handle_agent_disconnect(
|
||||
test_session_id, role, self.stop_test_session
|
||||
)
|
||||
|
||||
await task.cancel()
|
||||
|
||||
# Register custom audio and transcript handlers for LoopTalk
|
||||
await self._register_looptalk_handlers(
|
||||
audio_buffer, transcript, test_session_id, role
|
||||
)
|
||||
|
||||
async def _register_looptalk_handlers(
|
||||
self, audio_buffer, transcript, test_session_id: int, role: str
|
||||
):
|
||||
"""Register LoopTalk-specific handlers for audio and transcript recording"""
|
||||
|
||||
paths = self.recording_manager.get_recording_paths(test_session_id, role)
|
||||
|
||||
# Store audio metadata for later WAV conversion
|
||||
audio_metadata = {"sample_rate": None, "num_channels": None}
|
||||
|
||||
# Audio handler - writes directly to PCM file
|
||||
@audio_buffer.event_handler("on_audio_data")
|
||||
async def on_audio_data(buffer, audio, sample_rate, num_channels):
|
||||
if not audio:
|
||||
return
|
||||
|
||||
# Store metadata on first write
|
||||
if audio_metadata["sample_rate"] is None:
|
||||
audio_metadata["sample_rate"] = sample_rate
|
||||
audio_metadata["num_channels"] = num_channels
|
||||
|
||||
# Append PCM data to temporary file
|
||||
try:
|
||||
with open(paths["temp_audio"], "ab") as f:
|
||||
f.write(audio)
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Failed to write audio for {role} in session {test_session_id}: {e}"
|
||||
)
|
||||
|
||||
# Transcript handler - writes directly to text file
|
||||
@transcript.event_handler("on_transcript_update")
|
||||
async def on_transcript_update(processor, frame):
|
||||
transcript_text = ""
|
||||
for msg in frame.messages:
|
||||
timestamp = f"[{msg.timestamp}] " if msg.timestamp else ""
|
||||
line = f"{timestamp}{msg.role}: {msg.content}\n"
|
||||
transcript_text += line
|
||||
|
||||
# Append transcript to file
|
||||
try:
|
||||
with open(paths["transcript"], "a") as f:
|
||||
f.write(transcript_text)
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Failed to write transcript for {role} in session {test_session_id}: {e}"
|
||||
)
|
||||
|
||||
# Store metadata in session info for later WAV conversion
|
||||
# Set default values if not yet captured
|
||||
if audio_metadata["sample_rate"] is None:
|
||||
audio_metadata["sample_rate"] = 16000 # Default sample rate
|
||||
audio_metadata["num_channels"] = 1 # Default channels
|
||||
|
||||
self.session_manager.update_audio_metadata(
|
||||
test_session_id,
|
||||
role,
|
||||
sample_rate=audio_metadata["sample_rate"],
|
||||
num_channels=audio_metadata["num_channels"],
|
||||
)
|
||||
|
||||
async def _run_pipeline_with_context(
|
||||
self,
|
||||
pipeline_task: PipelineTask,
|
||||
params,
|
||||
agent_id: str,
|
||||
conversation_id: str,
|
||||
role: str,
|
||||
):
|
||||
"""Run a pipeline task with the agent_id set in context"""
|
||||
set_current_run_id(agent_id)
|
||||
return await pipeline_task.run(params)
|
||||
|
||||
async def stop_test_session(self, test_session_id: int) -> Dict[str, Any]:
|
||||
"""Stop a running test session."""
|
||||
|
||||
session_info = self.session_manager.get_session(test_session_id)
|
||||
if not session_info:
|
||||
raise ValueError(f"Test session {test_session_id} is not running")
|
||||
|
||||
try:
|
||||
# Cancel both pipeline tasks
|
||||
await session_info["actor_task"].cancel()
|
||||
await session_info["adversary_task"].cancel()
|
||||
|
||||
# Also cancel the task futures if they exist
|
||||
if "actor_task_future" in session_info:
|
||||
session_info["actor_task_future"].cancel()
|
||||
if "adversary_task_future" in session_info:
|
||||
session_info["adversary_task_future"].cancel()
|
||||
|
||||
# Calculate duration
|
||||
duration_seconds = int(
|
||||
(datetime.now(UTC) - session_info["start_time"]).total_seconds()
|
||||
)
|
||||
|
||||
# Update conversation
|
||||
await self.db_client.update_conversation(
|
||||
conversation_id=session_info["conversation"].id,
|
||||
duration_seconds=duration_seconds,
|
||||
ended_at=datetime.now(UTC),
|
||||
)
|
||||
|
||||
# Update test session status
|
||||
await self.db_client.update_test_session_status(
|
||||
test_session_id=test_session_id,
|
||||
status="completed",
|
||||
results={
|
||||
"duration_seconds": duration_seconds,
|
||||
"conversation_id": session_info["conversation"].id,
|
||||
},
|
||||
)
|
||||
|
||||
# Finalize recordings for both actor and adversary
|
||||
# Convert PCM files to WAV
|
||||
actor_metadata = self.session_manager.get_audio_metadata(
|
||||
test_session_id, "actor"
|
||||
)
|
||||
adversary_metadata = self.session_manager.get_audio_metadata(
|
||||
test_session_id, "adversary"
|
||||
)
|
||||
|
||||
self.recording_manager.convert_pcm_to_wav(
|
||||
test_session_id,
|
||||
"actor",
|
||||
sample_rate=actor_metadata["sample_rate"],
|
||||
num_channels=actor_metadata["num_channels"],
|
||||
)
|
||||
self.recording_manager.convert_pcm_to_wav(
|
||||
test_session_id,
|
||||
"adversary",
|
||||
sample_rate=adversary_metadata["sample_rate"],
|
||||
num_channels=adversary_metadata["num_channels"],
|
||||
)
|
||||
|
||||
# Upload recordings to S3 (synchronously for load testing)
|
||||
(
|
||||
actor_audio_url,
|
||||
actor_transcript_url,
|
||||
) = await self.recording_manager.upload_recording_to_s3(
|
||||
test_session_id, "actor"
|
||||
)
|
||||
(
|
||||
adversary_audio_url,
|
||||
adversary_transcript_url,
|
||||
) = await self.recording_manager.upload_recording_to_s3(
|
||||
test_session_id, "adversary"
|
||||
)
|
||||
|
||||
# Update conversation with recording URLs
|
||||
await self.db_client.update_conversation(
|
||||
conversation_id=session_info["conversation"].id,
|
||||
actor_recording_url=actor_audio_url,
|
||||
adversary_recording_url=adversary_audio_url,
|
||||
transcript={
|
||||
"actor_transcript_url": actor_transcript_url,
|
||||
"adversary_transcript_url": adversary_transcript_url,
|
||||
},
|
||||
)
|
||||
|
||||
# Log recording locations
|
||||
logger.info(f"LoopTalk recordings uploaded to S3:")
|
||||
if actor_audio_url:
|
||||
logger.info(f" - Actor audio: {actor_audio_url}")
|
||||
if actor_transcript_url:
|
||||
logger.info(f" - Actor transcript: {actor_transcript_url}")
|
||||
if adversary_audio_url:
|
||||
logger.info(f" - Adversary audio: {adversary_audio_url}")
|
||||
if adversary_transcript_url:
|
||||
logger.info(f" - Adversary transcript: {adversary_transcript_url}")
|
||||
|
||||
# Clean up local files after successful upload
|
||||
self.recording_manager.cleanup_session_files(test_session_id)
|
||||
|
||||
# Clean up
|
||||
self.transport_manager.remove_transport_pair(str(test_session_id))
|
||||
self.session_manager.remove_session(test_session_id)
|
||||
|
||||
# Clean up audio streamers
|
||||
from api.services.looptalk.audio_streamer import cleanup_audio_streamers
|
||||
|
||||
cleanup_audio_streamers(str(test_session_id))
|
||||
|
||||
logger.info(f"Stopped LoopTalk test session {test_session_id}")
|
||||
|
||||
return {
|
||||
"test_session_id": test_session_id,
|
||||
"status": "completed",
|
||||
"duration_seconds": duration_seconds,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to stop test session {test_session_id}: {e}")
|
||||
await self.db_client.update_test_session_status(
|
||||
test_session_id=test_session_id, status="failed", error=str(e)
|
||||
)
|
||||
raise
|
||||
|
||||
async def start_load_test(
|
||||
self,
|
||||
organization_id: int,
|
||||
name_prefix: str,
|
||||
actor_workflow_id: int,
|
||||
adversary_workflow_id: int,
|
||||
config: Dict[str, Any],
|
||||
test_count: int,
|
||||
) -> Dict[str, Any]:
|
||||
"""Start a load test with multiple concurrent test sessions."""
|
||||
|
||||
# Validate test count
|
||||
if test_count < 1 or test_count > 10:
|
||||
raise ValueError("Test count must be between 1 and 10")
|
||||
|
||||
# Create test sessions
|
||||
test_sessions = await self.db_client.create_load_test_group(
|
||||
organization_id=organization_id,
|
||||
name_prefix=name_prefix,
|
||||
actor_workflow_id=actor_workflow_id,
|
||||
adversary_workflow_id=adversary_workflow_id,
|
||||
config=config,
|
||||
test_count=test_count,
|
||||
)
|
||||
|
||||
# Start all test sessions concurrently
|
||||
tasks = []
|
||||
for test_session in test_sessions:
|
||||
task = asyncio.create_task(
|
||||
self.start_test_session(
|
||||
test_session_id=test_session.id, organization_id=organization_id
|
||||
)
|
||||
)
|
||||
tasks.append(task)
|
||||
|
||||
# Wait for all to start
|
||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
|
||||
# Count successes and failures
|
||||
started = sum(1 for r in results if not isinstance(r, Exception))
|
||||
failed = sum(1 for r in results if isinstance(r, Exception))
|
||||
|
||||
load_test_group_id = test_sessions[0].load_test_group_id
|
||||
|
||||
logger.info(
|
||||
f"Started load test {load_test_group_id}: "
|
||||
f"{started} started, {failed} failed out of {test_count}"
|
||||
)
|
||||
|
||||
return {
|
||||
"load_test_group_id": load_test_group_id,
|
||||
"total": test_count,
|
||||
"started": started,
|
||||
"failed": failed,
|
||||
"test_session_ids": [ts.id for ts in test_sessions],
|
||||
}
|
||||
|
||||
def get_active_test_count(self) -> int:
|
||||
"""Get the number of currently active test sessions."""
|
||||
return self.session_manager.get_active_count()
|
||||
|
||||
def get_active_test_info(self) -> Dict[str, Any]:
|
||||
"""Get information about all active test sessions."""
|
||||
return self.session_manager.get_active_info()
|
||||
|
||||
def get_recording_info(self, test_session_id: int) -> Dict[str, Any]:
|
||||
"""Get information about recordings for a test session"""
|
||||
return self.recording_manager.get_recording_info(test_session_id)
|
||||
|
|
@ -5,8 +5,9 @@ from loguru import logger
|
|||
from api.db import db_client
|
||||
from api.enums import PostHogEvent, WorkflowRunState
|
||||
from api.services.campaign.circuit_breaker import circuit_breaker
|
||||
from api.services.integrations import IntegrationRuntimeSession
|
||||
from api.services.pipecat.audio_config import AudioConfig
|
||||
from api.services.pipecat.audio_playback import play_audio, play_audio_loop
|
||||
from api.services.pipecat.audio_playback import play_audio_loop
|
||||
from api.services.pipecat.in_memory_buffers import (
|
||||
InMemoryAudioBuffer,
|
||||
InMemoryLogsBuffer,
|
||||
|
|
@ -19,8 +20,6 @@ from api.tasks.arq import enqueue_job
|
|||
from api.tasks.function_names import FunctionNames
|
||||
from pipecat.frames.frames import (
|
||||
Frame,
|
||||
LLMContextFrame,
|
||||
TTSSpeakFrame,
|
||||
)
|
||||
from pipecat.pipeline.task import PipelineTask
|
||||
from pipecat.processors.audio.audio_buffer_processor import AudioBufferProcessor
|
||||
|
|
@ -68,8 +67,8 @@ def register_event_handlers(
|
|||
pipeline_metrics_aggregator: PipelineMetricsAggregator,
|
||||
audio_config=AudioConfig,
|
||||
pre_call_fetch_task: asyncio.Task | None = None,
|
||||
fetch_recording_audio=None,
|
||||
user_provider_id: str | None = None,
|
||||
integration_runtime_sessions: list[IntegrationRuntimeSession] | None = None,
|
||||
):
|
||||
"""Register all event handlers for transport and task events.
|
||||
|
||||
|
|
@ -97,20 +96,11 @@ def register_event_handlers(
|
|||
"initial_response_triggered": False,
|
||||
}
|
||||
|
||||
async def queue_initial_llm_context():
|
||||
# Queue LLMContextFrame after the VoicemailDetector since the detector
|
||||
# gates LLMContextFrames until voicemail detection completes. We also
|
||||
# don't want to trigger the Voicemail LLM with this initial frame.
|
||||
await engine.llm.queue_frame(LLMContextFrame(engine.context))
|
||||
|
||||
async def maybe_trigger_initial_response():
|
||||
"""Start the conversation after both pipeline_started and client_connected events.
|
||||
|
||||
If a pre-call fetch is in progress, plays a ringer while waiting for the
|
||||
response, then merges the result into the call context before proceeding.
|
||||
|
||||
If the start node has a greeting configured, play it directly via TTS.
|
||||
Otherwise, trigger an LLM generation for the opening message.
|
||||
"""
|
||||
if (
|
||||
ready_state["pipeline_started"]
|
||||
|
|
@ -165,46 +155,11 @@ def register_event_handlers(
|
|||
# Set the start node now (after pre-call fetch data is merged)
|
||||
# so that render_template() has the complete _call_context_vars.
|
||||
await engine.set_node(engine.workflow.start_node_id)
|
||||
|
||||
greeting_info = engine.get_start_greeting()
|
||||
if greeting_info:
|
||||
greeting_type, greeting_value = greeting_info
|
||||
if (
|
||||
greeting_type == "audio"
|
||||
and greeting_value
|
||||
and fetch_recording_audio
|
||||
):
|
||||
logger.debug(f"Playing audio greeting recording: {greeting_value}")
|
||||
result = await fetch_recording_audio(
|
||||
recording_pk=int(greeting_value)
|
||||
)
|
||||
if result:
|
||||
await play_audio(
|
||||
result.audio,
|
||||
sample_rate=audio_config.pipeline_sample_rate or 16000,
|
||||
queue_frame=transport.output().queue_frame,
|
||||
transcript=result.transcript,
|
||||
append_to_context=True,
|
||||
)
|
||||
else:
|
||||
logger.warning(
|
||||
f"Failed to fetch audio greeting {greeting_value}, "
|
||||
"falling back to LLM generation"
|
||||
)
|
||||
await queue_initial_llm_context()
|
||||
else:
|
||||
logger.debug("Playing text greeting via TTS")
|
||||
# append_to_context=True so the assistant aggregator commits
|
||||
# the greeting to the LLM context once TTS finishes; without
|
||||
# it the LLM would re-greet on its first generation.
|
||||
await task.queue_frame(
|
||||
TTSSpeakFrame(greeting_value, append_to_context=True)
|
||||
)
|
||||
else:
|
||||
logger.debug(
|
||||
"Both pipeline_started and client_connected received - triggering initial LLM generation"
|
||||
)
|
||||
await queue_initial_llm_context()
|
||||
await engine.queue_node_opening(
|
||||
node_id=engine.workflow.start_node_id,
|
||||
previous_node_id=None,
|
||||
generate_if_no_greeting=True,
|
||||
)
|
||||
|
||||
@transport.event_handler("on_client_connected")
|
||||
async def on_client_connected(_transport, _participant):
|
||||
|
|
@ -319,6 +274,20 @@ def register_event_handlers(
|
|||
)
|
||||
|
||||
# Clean up engine resources (including voicemail detector)
|
||||
integration_logs: dict[str, object] = {}
|
||||
for runtime_session in integration_runtime_sessions or []:
|
||||
try:
|
||||
session_logs = await runtime_session.on_call_finished(
|
||||
gathered_context=gathered_context
|
||||
)
|
||||
if session_logs:
|
||||
integration_logs.update(session_logs)
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Error finalizing integration runtime session '{runtime_session.name}': {e}",
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
await engine.cleanup()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
|
|
@ -368,14 +337,11 @@ def register_event_handlers(
|
|||
)
|
||||
)
|
||||
|
||||
# Save real-time feedback logs to workflow run
|
||||
logs_update: dict[str, object] = {}
|
||||
if not in_memory_logs_buffer.is_empty:
|
||||
try:
|
||||
feedback_events = in_memory_logs_buffer.get_events()
|
||||
await db_client.update_workflow_run(
|
||||
run_id=workflow_run_id,
|
||||
logs={"realtime_feedback_events": feedback_events},
|
||||
)
|
||||
logs_update["realtime_feedback_events"] = feedback_events
|
||||
logger.debug(
|
||||
f"Saved {len(feedback_events)} feedback events to workflow run logs"
|
||||
)
|
||||
|
|
@ -384,6 +350,17 @@ def register_event_handlers(
|
|||
else:
|
||||
logger.debug("Logs buffer is empty, skipping save")
|
||||
|
||||
logs_update.update(integration_logs)
|
||||
|
||||
if logs_update:
|
||||
try:
|
||||
await db_client.update_workflow_run(
|
||||
run_id=workflow_run_id,
|
||||
logs=logs_update,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Error saving workflow run logs: {e}", exc_info=True)
|
||||
|
||||
# Write buffers to temp files and enqueue combined processing task
|
||||
audio_temp_path = None
|
||||
transcript_temp_path = None
|
||||
|
|
|
|||
|
|
@ -6,6 +6,10 @@ from typing import List, Optional
|
|||
|
||||
from loguru import logger
|
||||
|
||||
from api.services.pipecat.realtime_feedback_events import (
|
||||
realtime_feedback_event_sort_key,
|
||||
stamp_realtime_feedback_event,
|
||||
)
|
||||
from api.utils.transcript import generate_transcript_text as _generate_transcript_text
|
||||
from pipecat.utils.enums import RealtimeFeedbackType
|
||||
|
||||
|
|
@ -98,16 +102,13 @@ class InMemoryLogsBuffer:
|
|||
|
||||
async def append(self, event: dict):
|
||||
"""Append a feedback event to the buffer with timestamp and current node."""
|
||||
# Add timestamp, turn tracking, and current node
|
||||
timestamped_event = {
|
||||
**event,
|
||||
"timestamp": datetime.now(UTC).isoformat(),
|
||||
"turn": self._turn_counter,
|
||||
}
|
||||
if self._current_node_id:
|
||||
timestamped_event["node_id"] = self._current_node_id
|
||||
if self._current_node_name:
|
||||
timestamped_event["node_name"] = self._current_node_name
|
||||
timestamped_event = stamp_realtime_feedback_event(
|
||||
event,
|
||||
timestamp=datetime.now(UTC).isoformat(),
|
||||
turn=self._turn_counter,
|
||||
node_id=self._current_node_id,
|
||||
node_name=self._current_node_name,
|
||||
)
|
||||
self._events.append(timestamped_event)
|
||||
logger.trace(
|
||||
f"Appended event {event.get('type')} to logs buffer for workflow {self._workflow_run_id}"
|
||||
|
|
@ -120,17 +121,12 @@ class InMemoryLogsBuffer:
|
|||
f"Incremented turn counter to {self._turn_counter} for workflow {self._workflow_run_id}"
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _event_sort_key(event: dict) -> str:
|
||||
payload_ts = event.get("payload", {}).get("timestamp")
|
||||
return payload_ts or event.get("timestamp", "")
|
||||
|
||||
def _sorted_events(self) -> List[dict]:
|
||||
# Stable sort by the realtime (payload) timestamp when available, falling
|
||||
# back to the buffer-append timestamp. Python's sort is stable, so events
|
||||
# sharing a key retain their original insertion order — this keeps
|
||||
# consecutive bot-text chunks of a single turn contiguous.
|
||||
return sorted(self._events, key=self._event_sort_key)
|
||||
return sorted(self._events, key=realtime_feedback_event_sort_key)
|
||||
|
||||
def get_events(self) -> List[dict]:
|
||||
"""Get all events for final storage, ordered by realtime timestamp."""
|
||||
|
|
|
|||
23
api/services/pipecat/minimax_tts.py
Normal file
23
api/services/pipecat/minimax_tts.py
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
"""MiniMax TTS wrapper that closes its aiohttp session in cleanup().
|
||||
|
||||
Pipecat's MiniMaxHttpTTSService leaves session disposal to the caller. Our
|
||||
factory creates a fresh session per service instance, so we own its close
|
||||
here to avoid leaking sockets/FDs on shutdown.
|
||||
"""
|
||||
|
||||
import aiohttp
|
||||
|
||||
from pipecat.services.minimax.tts import MiniMaxHttpTTSService
|
||||
|
||||
|
||||
class MiniMaxOwnedSessionTTSService(MiniMaxHttpTTSService):
|
||||
"""MiniMaxHttpTTSService variant that owns its aiohttp session lifecycle."""
|
||||
|
||||
def __init__(self, *args, aiohttp_session: aiohttp.ClientSession, **kwargs):
|
||||
super().__init__(*args, aiohttp_session=aiohttp_session, **kwargs)
|
||||
self._owned_session = aiohttp_session
|
||||
|
||||
async def cleanup(self):
|
||||
await super().cleanup()
|
||||
if not self._owned_session.closed:
|
||||
await self._owned_session.close()
|
||||
|
|
@ -152,8 +152,30 @@ def build_realtime_pipeline(
|
|||
return Pipeline(processors)
|
||||
|
||||
|
||||
def create_pipeline_task(pipeline, workflow_run_id, audio_config: AudioConfig = None):
|
||||
"""Create a pipeline task with appropriate parameters"""
|
||||
def create_pipeline_task(
|
||||
pipeline,
|
||||
workflow_run_id,
|
||||
audio_config: AudioConfig = None,
|
||||
*,
|
||||
conversation_parent_context=None,
|
||||
conversation_type: str = "voice",
|
||||
additional_span_attributes: dict | None = None,
|
||||
):
|
||||
"""Create a pipeline task with appropriate parameters.
|
||||
|
||||
Args:
|
||||
pipeline: The pipeline to run.
|
||||
workflow_run_id: Run id, used as the conversation id.
|
||||
audio_config: Optional audio configuration.
|
||||
conversation_parent_context: Optional OTEL context carrying a fixed
|
||||
trace id. When provided, the conversation span attaches to that
|
||||
trace instead of starting a new root trace (used by text chat to
|
||||
stitch every per-turn pipeline into one trace).
|
||||
conversation_type: ``conversation.type`` span attribute value.
|
||||
additional_span_attributes: Extra attributes set on the conversation
|
||||
span (e.g. ``langfuse.trace.name`` to name a stitched trace that
|
||||
has no real root span).
|
||||
"""
|
||||
# Set up pipeline params with audio configuration if provided
|
||||
pipeline_params = PipelineParams(
|
||||
enable_metrics=True,
|
||||
|
|
@ -178,6 +200,9 @@ def create_pipeline_task(pipeline, workflow_run_id, audio_config: AudioConfig =
|
|||
enable_tracing=True,
|
||||
enable_rtvi=False,
|
||||
conversation_id=f"{workflow_run_id}",
|
||||
conversation_parent_context=conversation_parent_context,
|
||||
conversation_type=conversation_type,
|
||||
additional_span_attributes=additional_span_attributes,
|
||||
)
|
||||
|
||||
# Check if turn logging is enabled
|
||||
|
|
|
|||
9
api/services/pipecat/realtime/__init__.py
Normal file
9
api/services/pipecat/realtime/__init__.py
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
"""Dograh-specific subclasses of pipecat realtime LLM services.
|
||||
|
||||
Each subclass wires Dograh engine integration quirks (user-mute gating,
|
||||
TTSSpeakFrame greeting trigger, node-transition handling, function-call
|
||||
deferral, etc.) onto the corresponding pipecat realtime service.
|
||||
|
||||
The pipecat fork's services stay close to upstream — Dograh behavior lives
|
||||
here.
|
||||
"""
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue