refactor: telephony integration

docs/integrations/telephony/custom.mdx

@@ -0,0 +1,184 @@
+---
+title: "Custom Telephony Provider"
+description: "Build your own telephony provider integration for Dograh AI"
+---
+
+## Overview
+
+Dograh AI's telephony abstraction layer allows you to integrate any telephony service by implementing the `TelephonyProvider` interface.
+
+## Provider Interface
+
+All telephony providers must implement this abstract base class:
+
+```python
+from abc import ABC, abstractmethod
+from typing import Any, Dict, List, Optional
+
+class TelephonyProvider(ABC):
+    """Abstract base class for telephony providers."""
+    
+    @abstractmethod
+    async def initiate_call(
+        self,
+        to_number: str,
+        webhook_url: str,
+        workflow_run_id: Optional[int] = None,
+        **kwargs: Any
+    ) -> Dict[str, Any]:
+        """Initiate an outbound call."""
+        pass
+    
+    @abstractmethod
+    async def get_call_status(self, call_id: str) -> Dict[str, Any]:
+        """Get current status of a call."""
+        pass
+    
+    @abstractmethod
+    async def get_available_phone_numbers(self) -> List[str]:
+        """Get list of available phone numbers."""
+        pass
+    
+    @abstractmethod
+    def validate_config(self) -> bool:
+        """Validate provider configuration."""
+        pass
+    
+    @abstractmethod
+    async def verify_webhook_signature(
+        self, url: str, params: Dict[str, Any], signature: str
+    ) -> bool:
+        """Verify webhook signature for security."""
+        pass
+    
+    @abstractmethod
+    async def get_webhook_response(
+        self, workflow_id: int, user_id: int, workflow_run_id: int
+    ) -> str:
+        """Generate initial webhook response."""
+        pass
+```
+
+## Implementation Guide
+
+### 1. Create Your Provider
+
+Create a new file in `api/services/telephony/providers/`:
+
+```python
+# api/services/telephony/providers/your_provider.py
+
+from typing import Any, Dict, List, Optional
+from api.services.telephony.base import TelephonyProvider
+
+class YourProvider(TelephonyProvider):
+    """Your custom telephony provider implementation."""
+    
+    def __init__(self, config: Dict[str, Any]):
+        """Initialize with configuration dictionary."""
+        # Extract your provider-specific configuration
+        self.api_key = config.get("api_key")
+        self.api_secret = config.get("api_secret")
+        self.from_number = config.get("from_numbers", [""])[0]
+    
+    def validate_config(self) -> bool:
+        """Check if all required configuration is present."""
+        return bool(self.api_key and self.api_secret and self.from_number)
+    
+    async def initiate_call(
+        self,
+        to_number: str,
+        webhook_url: str,
+        workflow_run_id: Optional[int] = None,
+        **kwargs: Any
+    ) -> Dict[str, Any]:
+        """Start an outbound call using your provider's API."""
+        # Implement your provider's call initiation logic
+        pass
+    
+    # Implement other required methods...
+```
+
+### 2. Register in Factory
+
+Update `api/services/telephony/factory.py` to include your provider:
+
+```python
+from api.services.telephony.providers.your_provider import YourProvider
+
+async def get_telephony_provider(
+    organization_id: Optional[int] = None
+) -> TelephonyProvider:
+    """Factory function to get appropriate telephony provider."""
+    
+    config = await load_telephony_config(organization_id)
+    provider_type = config.get("provider", "twilio").lower()
+    
+    if provider_type == "twilio":
+        return TwilioProvider(config)
+    elif provider_type == "your_provider":
+        return YourProvider(config)
+    else:
+        raise ValueError(f"Unknown telephony provider: {provider_type}")
+```
+
+### 3. Add Configuration Support
+
+For OSS deployment (environment variables):
+
+```bash
+# .env
+TELEPHONY_PROVIDER=your_provider
+YOUR_PROVIDER_API_KEY=your_api_key
+YOUR_PROVIDER_API_SECRET=your_api_secret
+YOUR_PROVIDER_FROM_NUMBER=+1234567890
+```
+
+Update the configuration loader in `factory.py`:
+
+```python
+if provider == "your_provider":
+    return {
+        "provider": "your_provider",
+        "api_key": os.getenv("YOUR_PROVIDER_API_KEY"),
+        "api_secret": os.getenv("YOUR_PROVIDER_API_SECRET"),
+        "from_numbers": [os.getenv("YOUR_PROVIDER_FROM_NUMBER")]
+    }
+```
+
+## Audio Format Considerations
+
+Different providers use different audio formats. Twilio uses MULAW at 8000 Hz encoded in Base64. Your provider may differ, so ensure proper audio format conversion in your WebSocket handler.
+
+## Testing
+
+Create unit tests for your provider:
+
+```python
+# tests/test_your_provider.py
+
+import pytest
+from api.services.telephony.providers.your_provider import YourProvider
+
+@pytest.mark.asyncio
+async def test_validate_config():
+    config = {
+        "api_key": "test_key",
+        "api_secret": "test_secret",
+        "from_numbers": ["+1234567890"]
+    }
+    provider = YourProvider(config)
+    assert provider.validate_config() is True
+```
+
+## Best Practices
+
+1. **Error Handling**: Implement robust error handling with meaningful messages
+2. **Logging**: Use `loguru.logger` for consistent logging
+3. **Async Operations**: All I/O operations should be async
+4. **Configuration Validation**: Validate config on initialization
+5. **Security**: Always verify webhook signatures
+
+## Reference Implementation
+
+See the Twilio provider implementation at `api/services/telephony/providers/twilio_provider.py` for a complete example.

docs/integrations/telephony/overview.mdx

@@ -0,0 +1,115 @@
+---
+title: "Telephony Integration"
+description: "Connect voice agents with telephony providers for inbound and outbound calls"
+---
+
+## Overview
+
+Dograh AI's telephony integration system provides a unified interface for connecting with various telephony providers. This abstraction layer allows you to easily switch between providers without changing your application logic.
+
+## Supported Providers
+
+<CardGroup cols={2}>
+  <Card title="Twilio" href="/integrations/telephony/twilio">
+    Industry-leading cloud communications platform with global reach
+  </Card>
+  {/* Additional providers can be added in the future by implementing the TelephonyProvider interface */}
+  <Card title="Custom Provider" href="/integrations/telephony/custom">
+    Build your own telephony provider integration
+  </Card>
+</CardGroup>
+
+## Architecture
+
+The telephony integration system uses a provider abstraction pattern that ensures consistency across different services:
+
+```python
+# All providers implement this interface
+class TelephonyProvider:
+    async def initiate_call(to_number, webhook_url, ...)
+    async def get_call_status(call_id)
+    async def verify_webhook_signature(url, params, signature)
+    # ... more methods
+```
+
+## Configuration Methods
+
+### OSS Deployment (Environment Variables)
+
+For self-hosted deployments, configure your telephony provider using environment variables:
+
+```bash
+# .env file
+TELEPHONY_PROVIDER=twilio  # Required to specify which provider to use
+TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+TWILIO_AUTH_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+TWILIO_FROM_NUMBER=+1234567890
+```
+
+### SaaS Deployment (Database Configuration)
+
+For cloud deployments, configure providers through the web interface:
+
+1. Navigate to **Settings** → **Integrations** → **Telephony**
+2. Select your provider
+3. Enter credentials
+4. Test connection
+
+## Common Features
+
+The telephony integration in Dograh AI supports:
+
+- **Outbound Calls**: Initiate calls to any phone number
+- **Call Status Tracking**: Monitor call lifecycle events (initiated, ringing, answered, completed, failed)
+- **WebSocket Streaming**: Real-time audio streaming for voice agents
+- **Webhook Authentication**: Secure webhook signature verification
+
+## API Endpoints
+
+The telephony system exposes these unified endpoints:
+
+| Endpoint | Method | Description |
+|----------|---------|-------------|
+| `/api/v1/telephony/initiate-call` | POST | Start an outbound call |
+| `/api/v1/telephony/status-callback/{id}` | POST | Receive call status updates |
+| `/api/v1/telephony/twiml` | POST | Handle initial webhook |
+| `/api/v1/telephony/ws/{id}` | WebSocket | Real-time audio streaming |
+
+## Implementation Status
+
+- **Twilio**: ✅ Fully implemented and tested
+- **Custom Providers**: The abstraction layer supports adding new providers by implementing the `TelephonyProvider` interface
+- **API Endpoints**: All telephony operations use the unified `/api/v1/telephony/*` endpoints:
+  - `/api/v1/telephony/initiate-call` - Start outbound calls
+  - `/api/v1/telephony/status-callback/{id}` - Receive call status updates
+  - `/api/v1/telephony/ws/{workflow_id}/{user_id}/{workflow_run_id}` - WebSocket for audio streaming
+
+## Troubleshooting
+
+<AccordionGroup>
+  <Accordion title="Calls not connecting">
+    - Verify credentials are correctly configured
+    - Check phone number format (must include country code)
+    - Ensure webhook URLs are publicly accessible
+    - Review provider-specific error logs
+  </Accordion>
+  
+  <Accordion title="Audio quality issues">
+    - Check network bandwidth and latency
+    - Verify audio codec compatibility
+    - Review WebSocket connection stability
+    - Ensure proper audio format (MULAW for Twilio)
+  </Accordion>
+  
+  <Accordion title="Webhook signature validation failing">
+    - Confirm auth tokens match between provider and configuration
+    - Verify webhook URL matches exactly (including parameters)
+    - Check for proxy or load balancer modifications
+  </Accordion>
+</AccordionGroup>
+
+## Next Steps
+
+- [Set up your first telephony provider](/integrations/telephony/twilio)
+- [Build a custom provider integration](/integrations/telephony/custom)
+- [Configure webhooks and callbacks](/integrations/telephony/webhooks)

docs/integrations/telephony/twilio.mdx

@@ -0,0 +1,127 @@
+---
+title: "Twilio Integration"
+description: "Configure Twilio for voice communication in Dograh AI"
+---
+
+## Overview
+
+Twilio is a cloud communications platform that enables voice calling, messaging, and video capabilities. Dograh AI's Twilio integration provides seamless connectivity for your voice agents.
+
+## Prerequisites
+
+Before setting up Twilio integration, you'll need:
+
+- A [Twilio account](https://www.twilio.com/try-twilio)
+- Account SID and Auth Token from your Twilio Console
+- At least one Twilio phone number
+- Dograh AI instance running and accessible
+
+## Video Tutorial
+
+Watch this step-by-step guide to set up Twilio with Dograh AI:
+
+<iframe
+  className="w-full aspect-video rounded-xl"
+  src="https://www.tella.tv/video/cmgbvzkrt00jk0clacu16blm3/embed?b=1&title=1&a=1&loop=0&t=0&muted=0&wt=1"
+  title="Dograh Twilio Setup"
+  allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
+  allowFullScreen
+></iframe>
+
+## Configuration
+
+### Step 1: Get Twilio Credentials
+
+1. Log in to your [Twilio Console](https://console.twilio.com/)
+2. Find your **Account SID** and **Auth Token** on the dashboard
+3. Navigate to **Phone Numbers** → **Manage** → **Active Numbers**
+4. Copy your phone number(s)
+
+### Step 2: Configure in Dograh AI
+
+<Tabs>
+  <Tab title="Web Interface (SaaS)">
+    1. Navigate to **Settings** → **Integrations** → **Telephony**
+    2. Select **Twilio** as your provider
+    3. Enter your credentials:
+       - Account SID
+       - Auth Token
+       - Phone Numbers (comma-separated if multiple)
+    4. Click **Test Connection**
+    5. Save configuration
+  </Tab>
+  
+  <Tab title="Environment Variables (OSS)">
+    Add these variables to your `.env` file:
+    
+    ```bash
+    # Telephony Configuration
+    TELEPHONY_PROVIDER=twilio  # Specifies Twilio as the telephony provider
+    TWILIO_ACCOUNT_SID="ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
+    TWILIO_AUTH_TOKEN="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
+    TWILIO_FROM_NUMBER="+1234567890"
+    # For multiple numbers, use comma separation:
+    # TWILIO_FROM_NUMBER="+1234567890,+0987654321"
+    ```
+    
+    Restart your Dograh AI services:
+    ```bash
+    docker-compose restart api
+    ```
+  </Tab>
+</Tabs>
+
+### Step 3: Test Your Configuration
+
+1. Create a test workflow
+2. Click "Test Call" to verify connection
+3. Check call logs for successful connection
+
+## How It Works
+
+### Outbound Calling
+When you initiate a call through Dograh AI:
+1. The system selects a phone number from your configured pool
+2. Twilio places the call to your recipient
+3. Once connected, audio streams through WebSocket for real-time voice interaction
+4. Call status updates are tracked throughout the lifecycle
+
+
+## Campaign Features
+
+When using Twilio with campaigns:
+- **Rate Limiting**: Enforced per organization to prevent overwhelming
+- **Automatic Retry**: Failed calls (busy/no-answer) are retried automatically
+- **Concurrent Call Management**: System manages call slots to optimize throughput
+
+## Troubleshooting
+
+<AccordionGroup>
+  <Accordion title="Invalid phone number error">
+    Ensure phone numbers include country code in E.164 format: `+1234567890`
+  </Accordion>
+  
+  <Accordion title="Authentication failed">
+    - Verify Account SID and Auth Token are correct
+    - Check for extra spaces in credentials
+    - Ensure credentials haven't been regenerated in Twilio Console
+  </Accordion>
+  
+  <Accordion title="Webhook signature validation failing">
+    - Confirm your Auth Token matches exactly
+    - Verify the webhook URL matches what Twilio sends
+    - Check if you're behind a proxy that modifies requests
+  </Accordion>
+  
+  <Accordion title="No audio on calls">
+    - Verify WebSocket connection is established
+    - Check firewall rules for WebSocket traffic
+    - Ensure audio pipeline is configured correctly
+  </Accordion>
+</AccordionGroup>
+
+## Best Practices
+
+- Store credentials securely in environment variables (OSS) or database (SaaS)
+- Test your configuration with a single call before running campaigns
+- Monitor Twilio Console for usage and billing

docs/integrations/telephony/webhooks.mdx

@@ -0,0 +1,93 @@
+---
+title: "Webhooks and Callbacks"
+description: "How Dograh AI handles telephony webhooks and audio streaming"
+---
+
+## Overview
+
+Dograh AI uses webhooks to communicate with telephony providers for call events and audio streaming. Webhooks are automatically configured when you initiate calls.
+
+## Webhook Types
+
+### 1. Initial Call Webhook
+
+When a call is initiated, the telephony provider requests instructions.
+
+**Endpoint**: `/api/v1/telephony/twiml`
+
+**Purpose**: Returns provider-specific instructions (TwiML for Twilio)
+
+**Example Response**:
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<Response>
+    <Connect>
+        <Stream url="wss://your-domain/api/v1/telephony/ws/123/456/789" />
+    </Connect>
+</Response>
+```
+
+### 2. Status Callback
+
+Receives call lifecycle events.
+
+**Endpoint**: `/api/v1/telephony/status-callback/{workflow_run_id}`
+
+**Events Tracked**:
+- `initiated` - Call request received
+- `ringing` - Call is ringing
+- `answered` - Call was answered
+- `completed` - Call ended normally
+- `busy` - Line was busy
+- `no-answer` - Call not answered
+- `failed` - Call failed
+
+### 3. WebSocket Audio Stream
+
+Real-time audio streaming for voice interaction.
+
+**Endpoint**: `/api/v1/telephony/ws/{workflow_id}/{user_id}/{workflow_run_id}`
+
+## How It Works
+
+Dograh AI automatically:
+1. Constructs webhook URLs based on your deployment
+2. Passes them to the telephony provider when initiating calls
+3. Verifies webhook signatures for security
+4. Processes status updates to track call lifecycle
+5. Manages WebSocket connections for audio streaming
+
+## Local Development
+
+For local development, use the built-in Cloudflare tunnel:
+
+```yaml
+# docker-compose.yml includes:
+cloudflared:
+  image: cloudflare/cloudflared:latest
+  command: tunnel --no-autoupdate --url http://api:8000
+```
+
+The tunnel URL is automatically detected and used for webhooks.
+
+## Troubleshooting
+
+<AccordionGroup>
+  <Accordion title="Webhook URL not accessible">
+    - Verify your domain/tunnel URL is publicly accessible
+    - Check firewall rules allow incoming HTTPS traffic
+    - Test with `curl` from external network
+  </Accordion>
+  
+  <Accordion title="WebSocket connection dropping">
+    - Check WebSocket upgrade headers are preserved
+    - Verify no timeout on load balancer/proxy
+    - Monitor for memory/CPU constraints
+  </Accordion>
+  
+  <Accordion title="Status callbacks not received">
+    - Verify workflow_run_id is included in URL
+    - Check provider console for webhook errors
+    - Review webhook retry logs
+  </Accordion>
+</AccordionGroup>