refactor: telephony integration

docs/docs.json

@@ -38,11 +38,25 @@
               "features/campaigns",
               "features/looptalk"
             ]
+          }
+        ]
+      },
+      {
+        "tab": "Integrations",
+        "groups": [
+          {
+            "group": "Overview",
+            "pages": [
+              "integrations/overview"
+            ]
           },
           {
-            "group": "Telephony Integrations",
+            "group": "Telephony",
             "pages": [
-              "telephony/twilio"
+              "integrations/telephony/overview",
+              "integrations/telephony/twilio",
+              "integrations/telephony/webhooks",
+              "integrations/telephony/custom"
             ]
           }
         ]

docs/integrations/overview.mdx

@@ -0,0 +1,50 @@
+---
+title: "Integrations Overview"
+description: "Connect Dograh AI with external services and platforms"
+---
+
+## Introduction
+
+Dograh AI provides a flexible integration architecture that allows you to connect with various external services and platforms. Our integration system is designed with modularity and extensibility in mind, making it easy to add new providers while maintaining backward compatibility.
+
+## Integration Categories
+
+### Telephony Providers
+Connect your voice agents with telephony services to make and receive calls.
+
+<Card title="Telephony Providers" href="/integrations/telephony/overview">
+  Configure telephony providers like Twilio, Vonage, and Plivo for voice communication
+</Card>
+
+### Future Integration Categories
+
+The integration architecture is designed to support additional categories in the future, such as storage services, analytics platforms, and CRM systems.
+
+## Architecture Principles
+
+Our integration system follows these core principles:
+
+- **Provider Abstraction**: All integrations implement a common interface, making it easy to switch between providers
+- **Configuration Flexibility**: Support for both environment-based (OSS) and database-based (SaaS) configuration
+- **Backward Compatibility**: New integrations don't break existing implementations
+- **Secure by Default**: All credentials are encrypted and never exposed in logs or UI
+
+## Getting Started
+
+1. Choose the integration category you need
+2. Follow the provider-specific setup guide
+3. Configure credentials through the UI or environment variables
+4. Test your integration with the provided verification tools
+
+## Best Practices
+
+- Store credentials securely using environment variables (OSS) or database configuration (SaaS)
+- Test integrations in a development environment before production deployment
+- Use the provider abstraction to maintain clean separation between business logic and provider specifics
+- Monitor integration health through application logs
+
+## Need Help?
+
+- Check provider-specific documentation for detailed setup instructions
+- Visit our [GitHub Issues](https://github.com/dograh-hq/dograh/issues) for community support
+- Join our [Slack community](https://join.slack.com/t/dograh-ai/shared_invite/zt-2u29h3bkm-RrkJ2f2B5lvTVZo0ZQ1MMA) for assistance

docs/integrations/telephony/custom.mdx

@@ -0,0 +1,184 @@
+---
+title: "Custom Telephony Provider"
+description: "Build your own telephony provider integration for Dograh AI"
+---
+
+## Overview
+
+Dograh AI's telephony abstraction layer allows you to integrate any telephony service by implementing the `TelephonyProvider` interface.
+
+## Provider Interface
+
+All telephony providers must implement this abstract base class:
+
+```python
+from abc import ABC, abstractmethod
+from typing import Any, Dict, List, Optional
+
+class TelephonyProvider(ABC):
+    """Abstract base class for telephony providers."""
+    
+    @abstractmethod
+    async def initiate_call(
+        self,
+        to_number: str,
+        webhook_url: str,
+        workflow_run_id: Optional[int] = None,
+        **kwargs: Any
+    ) -> Dict[str, Any]:
+        """Initiate an outbound call."""
+        pass
+    
+    @abstractmethod
+    async def get_call_status(self, call_id: str) -> Dict[str, Any]:
+        """Get current status of a call."""
+        pass
+    
+    @abstractmethod
+    async def get_available_phone_numbers(self) -> List[str]:
+        """Get list of available phone numbers."""
+        pass
+    
+    @abstractmethod
+    def validate_config(self) -> bool:
+        """Validate provider configuration."""
+        pass
+    
+    @abstractmethod
+    async def verify_webhook_signature(
+        self, url: str, params: Dict[str, Any], signature: str
+    ) -> bool:
+        """Verify webhook signature for security."""
+        pass
+    
+    @abstractmethod
+    async def get_webhook_response(
+        self, workflow_id: int, user_id: int, workflow_run_id: int
+    ) -> str:
+        """Generate initial webhook response."""
+        pass
+```
+
+## Implementation Guide
+
+### 1. Create Your Provider
+
+Create a new file in `api/services/telephony/providers/`:
+
+```python
+# api/services/telephony/providers/your_provider.py
+
+from typing import Any, Dict, List, Optional
+from api.services.telephony.base import TelephonyProvider
+
+class YourProvider(TelephonyProvider):
+    """Your custom telephony provider implementation."""
+    
+    def __init__(self, config: Dict[str, Any]):
+        """Initialize with configuration dictionary."""
+        # Extract your provider-specific configuration
+        self.api_key = config.get("api_key")
+        self.api_secret = config.get("api_secret")
+        self.from_number = config.get("from_numbers", [""])[0]
+    
+    def validate_config(self) -> bool:
+        """Check if all required configuration is present."""
+        return bool(self.api_key and self.api_secret and self.from_number)
+    
+    async def initiate_call(
+        self,
+        to_number: str,
+        webhook_url: str,
+        workflow_run_id: Optional[int] = None,
+        **kwargs: Any
+    ) -> Dict[str, Any]:
+        """Start an outbound call using your provider's API."""
+        # Implement your provider's call initiation logic
+        pass
+    
+    # Implement other required methods...
+```
+
+### 2. Register in Factory
+
+Update `api/services/telephony/factory.py` to include your provider:
+
+```python
+from api.services.telephony.providers.your_provider import YourProvider
+
+async def get_telephony_provider(
+    organization_id: Optional[int] = None
+) -> TelephonyProvider:
+    """Factory function to get appropriate telephony provider."""
+    
+    config = await load_telephony_config(organization_id)
+    provider_type = config.get("provider", "twilio").lower()
+    
+    if provider_type == "twilio":
+        return TwilioProvider(config)
+    elif provider_type == "your_provider":
+        return YourProvider(config)
+    else:
+        raise ValueError(f"Unknown telephony provider: {provider_type}")
+```
+
+### 3. Add Configuration Support
+
+For OSS deployment (environment variables):
+
+```bash
+# .env
+TELEPHONY_PROVIDER=your_provider
+YOUR_PROVIDER_API_KEY=your_api_key
+YOUR_PROVIDER_API_SECRET=your_api_secret
+YOUR_PROVIDER_FROM_NUMBER=+1234567890
+```
+
+Update the configuration loader in `factory.py`:
+
+```python
+if provider == "your_provider":
+    return {
+        "provider": "your_provider",
+        "api_key": os.getenv("YOUR_PROVIDER_API_KEY"),
+        "api_secret": os.getenv("YOUR_PROVIDER_API_SECRET"),
+        "from_numbers": [os.getenv("YOUR_PROVIDER_FROM_NUMBER")]
+    }
+```
+
+## Audio Format Considerations
+
+Different providers use different audio formats. Twilio uses MULAW at 8000 Hz encoded in Base64. Your provider may differ, so ensure proper audio format conversion in your WebSocket handler.
+
+## Testing
+
+Create unit tests for your provider:
+
+```python
+# tests/test_your_provider.py
+
+import pytest
+from api.services.telephony.providers.your_provider import YourProvider
+
+@pytest.mark.asyncio
+async def test_validate_config():
+    config = {
+        "api_key": "test_key",
+        "api_secret": "test_secret",
+        "from_numbers": ["+1234567890"]
+    }
+    provider = YourProvider(config)
+    assert provider.validate_config() is True
+```
+
+## Best Practices
+
+1. **Error Handling**: Implement robust error handling with meaningful messages
+2. **Logging**: Use `loguru.logger` for consistent logging
+3. **Async Operations**: All I/O operations should be async
+4. **Configuration Validation**: Validate config on initialization
+5. **Security**: Always verify webhook signatures
+
+## Reference Implementation
+
+See the Twilio provider implementation at `api/services/telephony/providers/twilio_provider.py` for a complete example.

docs/integrations/telephony/overview.mdx

@@ -0,0 +1,115 @@
+---
+title: "Telephony Integration"
+description: "Connect voice agents with telephony providers for inbound and outbound calls"
+---
+
+## Overview
+
+Dograh AI's telephony integration system provides a unified interface for connecting with various telephony providers. This abstraction layer allows you to easily switch between providers without changing your application logic.
+
+## Supported Providers
+
+<CardGroup cols={2}>
+  <Card title="Twilio" href="/integrations/telephony/twilio">
+    Industry-leading cloud communications platform with global reach
+  </Card>
+  {/* Additional providers can be added in the future by implementing the TelephonyProvider interface */}
+  <Card title="Custom Provider" href="/integrations/telephony/custom">
+    Build your own telephony provider integration
+  </Card>
+</CardGroup>
+
+## Architecture
+
+The telephony integration system uses a provider abstraction pattern that ensures consistency across different services:
+
+```python
+# All providers implement this interface
+class TelephonyProvider:
+    async def initiate_call(to_number, webhook_url, ...)
+    async def get_call_status(call_id)
+    async def verify_webhook_signature(url, params, signature)
+    # ... more methods
+```
+
+## Configuration Methods
+
+### OSS Deployment (Environment Variables)
+
+For self-hosted deployments, configure your telephony provider using environment variables:
+
+```bash
+# .env file
+TELEPHONY_PROVIDER=twilio  # Required to specify which provider to use
+TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+TWILIO_AUTH_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+TWILIO_FROM_NUMBER=+1234567890
+```
+
+### SaaS Deployment (Database Configuration)
+
+For cloud deployments, configure providers through the web interface:
+
+1. Navigate to **Settings** → **Integrations** → **Telephony**
+2. Select your provider
+3. Enter credentials
+4. Test connection
+
+## Common Features
+
+The telephony integration in Dograh AI supports:
+
+- **Outbound Calls**: Initiate calls to any phone number
+- **Call Status Tracking**: Monitor call lifecycle events (initiated, ringing, answered, completed, failed)
+- **WebSocket Streaming**: Real-time audio streaming for voice agents
+- **Webhook Authentication**: Secure webhook signature verification
+
+## API Endpoints
+
+The telephony system exposes these unified endpoints:
+
+| Endpoint | Method | Description |
+|----------|---------|-------------|
+| `/api/v1/telephony/initiate-call` | POST | Start an outbound call |
+| `/api/v1/telephony/status-callback/{id}` | POST | Receive call status updates |
+| `/api/v1/telephony/twiml` | POST | Handle initial webhook |
+| `/api/v1/telephony/ws/{id}` | WebSocket | Real-time audio streaming |
+
+## Implementation Status
+
+- **Twilio**: ✅ Fully implemented and tested
+- **Custom Providers**: The abstraction layer supports adding new providers by implementing the `TelephonyProvider` interface
+- **API Endpoints**: All telephony operations use the unified `/api/v1/telephony/*` endpoints:
+  - `/api/v1/telephony/initiate-call` - Start outbound calls
+  - `/api/v1/telephony/status-callback/{id}` - Receive call status updates
+  - `/api/v1/telephony/ws/{workflow_id}/{user_id}/{workflow_run_id}` - WebSocket for audio streaming
+
+## Troubleshooting
+
+<AccordionGroup>
+  <Accordion title="Calls not connecting">
+    - Verify credentials are correctly configured
+    - Check phone number format (must include country code)
+    - Ensure webhook URLs are publicly accessible
+    - Review provider-specific error logs
+  </Accordion>
+  
+  <Accordion title="Audio quality issues">
+    - Check network bandwidth and latency
+    - Verify audio codec compatibility
+    - Review WebSocket connection stability
+    - Ensure proper audio format (MULAW for Twilio)
+  </Accordion>
+  
+  <Accordion title="Webhook signature validation failing">
+    - Confirm auth tokens match between provider and configuration
+    - Verify webhook URL matches exactly (including parameters)
+    - Check for proxy or load balancer modifications
+  </Accordion>
+</AccordionGroup>
+
+## Next Steps
+
+- [Set up your first telephony provider](/integrations/telephony/twilio)
+- [Build a custom provider integration](/integrations/telephony/custom)
+- [Configure webhooks and callbacks](/integrations/telephony/webhooks)

docs/integrations/telephony/twilio.mdx

@@ -0,0 +1,127 @@
+---
+title: "Twilio Integration"
+description: "Configure Twilio for voice communication in Dograh AI"
+---
+
+## Overview
+
+Twilio is a cloud communications platform that enables voice calling, messaging, and video capabilities. Dograh AI's Twilio integration provides seamless connectivity for your voice agents.
+
+## Prerequisites
+
+Before setting up Twilio integration, you'll need:
+
+- A [Twilio account](https://www.twilio.com/try-twilio)
+- Account SID and Auth Token from your Twilio Console
+- At least one Twilio phone number
+- Dograh AI instance running and accessible
+
+## Video Tutorial
+
+Watch this step-by-step guide to set up Twilio with Dograh AI:
+
+<iframe
+  className="w-full aspect-video rounded-xl"
+  src="https://www.tella.tv/video/cmgbvzkrt00jk0clacu16blm3/embed?b=1&title=1&a=1&loop=0&t=0&muted=0&wt=1"
+  title="Dograh Twilio Setup"
+  allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
+  allowFullScreen
+></iframe>
+
+## Configuration
+
+### Step 1: Get Twilio Credentials
+
+1. Log in to your [Twilio Console](https://console.twilio.com/)
+2. Find your **Account SID** and **Auth Token** on the dashboard
+3. Navigate to **Phone Numbers** → **Manage** → **Active Numbers**
+4. Copy your phone number(s)
+
+### Step 2: Configure in Dograh AI
+
+<Tabs>
+  <Tab title="Web Interface (SaaS)">
+    1. Navigate to **Settings** → **Integrations** → **Telephony**
+    2. Select **Twilio** as your provider
+    3. Enter your credentials:
+       - Account SID
+       - Auth Token
+       - Phone Numbers (comma-separated if multiple)
+    4. Click **Test Connection**
+    5. Save configuration
+  </Tab>
+  
+  <Tab title="Environment Variables (OSS)">
+    Add these variables to your `.env` file:
+    
+    ```bash
+    # Telephony Configuration
+    TELEPHONY_PROVIDER=twilio  # Specifies Twilio as the telephony provider
+    TWILIO_ACCOUNT_SID="ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
+    TWILIO_AUTH_TOKEN="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
+    TWILIO_FROM_NUMBER="+1234567890"
+    # For multiple numbers, use comma separation:
+    # TWILIO_FROM_NUMBER="+1234567890,+0987654321"
+    ```
+    
+    Restart your Dograh AI services:
+    ```bash
+    docker-compose restart api
+    ```
+  </Tab>
+</Tabs>
+
+### Step 3: Test Your Configuration
+
+1. Create a test workflow
+2. Click "Test Call" to verify connection
+3. Check call logs for successful connection
+
+## How It Works
+
+### Outbound Calling
+When you initiate a call through Dograh AI:
+1. The system selects a phone number from your configured pool
+2. Twilio places the call to your recipient
+3. Once connected, audio streams through WebSocket for real-time voice interaction
+4. Call status updates are tracked throughout the lifecycle
+
+
+## Campaign Features
+
+When using Twilio with campaigns:
+- **Rate Limiting**: Enforced per organization to prevent overwhelming
+- **Automatic Retry**: Failed calls (busy/no-answer) are retried automatically
+- **Concurrent Call Management**: System manages call slots to optimize throughput
+
+## Troubleshooting
+
+<AccordionGroup>
+  <Accordion title="Invalid phone number error">
+    Ensure phone numbers include country code in E.164 format: `+1234567890`
+  </Accordion>
+  
+  <Accordion title="Authentication failed">
+    - Verify Account SID and Auth Token are correct
+    - Check for extra spaces in credentials
+    - Ensure credentials haven't been regenerated in Twilio Console
+  </Accordion>
+  
+  <Accordion title="Webhook signature validation failing">
+    - Confirm your Auth Token matches exactly
+    - Verify the webhook URL matches what Twilio sends
+    - Check if you're behind a proxy that modifies requests
+  </Accordion>
+  
+  <Accordion title="No audio on calls">
+    - Verify WebSocket connection is established
+    - Check firewall rules for WebSocket traffic
+    - Ensure audio pipeline is configured correctly
+  </Accordion>
+</AccordionGroup>
+
+## Best Practices
+
+- Store credentials securely in environment variables (OSS) or database (SaaS)
+- Test your configuration with a single call before running campaigns
+- Monitor Twilio Console for usage and billing

docs/integrations/telephony/webhooks.mdx

@@ -0,0 +1,93 @@
+---
+title: "Webhooks and Callbacks"
+description: "How Dograh AI handles telephony webhooks and audio streaming"
+---
+
+## Overview
+
+Dograh AI uses webhooks to communicate with telephony providers for call events and audio streaming. Webhooks are automatically configured when you initiate calls.
+
+## Webhook Types
+
+### 1. Initial Call Webhook
+
+When a call is initiated, the telephony provider requests instructions.
+
+**Endpoint**: `/api/v1/telephony/twiml`
+
+**Purpose**: Returns provider-specific instructions (TwiML for Twilio)
+
+**Example Response**:
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<Response>
+    <Connect>
+        <Stream url="wss://your-domain/api/v1/telephony/ws/123/456/789" />
+    </Connect>
+</Response>
+```
+
+### 2. Status Callback
+
+Receives call lifecycle events.
+
+**Endpoint**: `/api/v1/telephony/status-callback/{workflow_run_id}`
+
+**Events Tracked**:
+- `initiated` - Call request received
+- `ringing` - Call is ringing
+- `answered` - Call was answered
+- `completed` - Call ended normally
+- `busy` - Line was busy
+- `no-answer` - Call not answered
+- `failed` - Call failed
+
+### 3. WebSocket Audio Stream
+
+Real-time audio streaming for voice interaction.
+
+**Endpoint**: `/api/v1/telephony/ws/{workflow_id}/{user_id}/{workflow_run_id}`
+
+## How It Works
+
+Dograh AI automatically:
+1. Constructs webhook URLs based on your deployment
+2. Passes them to the telephony provider when initiating calls
+3. Verifies webhook signatures for security
+4. Processes status updates to track call lifecycle
+5. Manages WebSocket connections for audio streaming
+
+## Local Development
+
+For local development, use the built-in Cloudflare tunnel:
+
+```yaml
+# docker-compose.yml includes:
+cloudflared:
+  image: cloudflare/cloudflared:latest
+  command: tunnel --no-autoupdate --url http://api:8000
+```
+
+The tunnel URL is automatically detected and used for webhooks.
+
+## Troubleshooting
+
+<AccordionGroup>
+  <Accordion title="Webhook URL not accessible">
+    - Verify your domain/tunnel URL is publicly accessible
+    - Check firewall rules allow incoming HTTPS traffic
+    - Test with `curl` from external network
+  </Accordion>
+  
+  <Accordion title="WebSocket connection dropping">
+    - Check WebSocket upgrade headers are preserved
+    - Verify no timeout on load balancer/proxy
+    - Monitor for memory/CPU constraints
+  </Accordion>
+  
+  <Accordion title="Status callbacks not received">
+    - Verify workflow_run_id is included in URL
+    - Check provider console for webhook errors
+    - Review webhook retry logs
+  </Accordion>
+</AccordionGroup>