trustgraph/docs/tech-specs/flow-service-queue-lifecycle.md

---
layout: default
title: "Flow Service Separation and Queue Lifecycle Management"
parent: "Tech Specs"
---

# Flow Service Separation and Queue Lifecycle Management

## Overview

This specification describes the separation of the flow service from the
config service into an independent service, and the addition of explicit
queue lifecycle management to the pub/sub backend abstraction.

Every queue in the system has an explicit owner responsible for its
creation and deletion:

- **Flow and blueprint queues** — owned by the flow service
- **System queues** (config, librarian, knowledge, etc.) — owned by
  the services themselves

Consumers never create queues. They connect to queues that already
exist.

This addresses a fundamental problem across broker backends: without an
authoritative lifecycle owner, queues are created as a side effect of
consumer connections and never explicitly deleted. In RabbitMQ, this
leads to orphaned durable queues. In Pulsar, persistent topics and
subscriptions survive consumer crashes. In Kafka, topics persist
indefinitely. The solution is the same for all backends: explicit
lifecycle management through the `PubSubBackend` protocol.

---

## Background

### Current Architecture

The flow service (`FlowConfig`) and config service (`Configuration`)
are co-located in a single process: `trustgraph-flow/config/service`.
They share a `Processor` class that inherits from `AsyncProcessor` and
manages both config and flow request/response queues. `FlowConfig`
receives a direct reference to the `Configuration` object, giving it
backdoor access to `inc_version()` and `push()` — methods that bypass
the config service's own pub/sub interface.

The flow service manages flow lifecycle (start/stop) by manipulating
config state — active-flow entries, flow records, blueprint lookups —
but takes no active part in broker queue management. Queues are created
implicitly when the first consumer connects and are never explicitly
deleted.

### The Queue Lifecycle Problem

Queues are currently created as a side effect of consumer connections
(in `_connect()` for RabbitMQ, in `subscribe()` for Pulsar). No single
component owns queue lifecycle, leading to two failure modes:

- **Orphaned queues**: When a flow is stopped, consumers shut down but
  their queues remain — along with any messages in them. In RabbitMQ,
  durable queues persist indefinitely. In Pulsar, persistent topics and
  their subscriptions survive consumer disconnection unless
  `unsubscribe()` is explicitly called — which doesn't happen on crash
  or error paths.
- **Premature deletion**: If consumers attempt to delete queues on
  exit, error-path shutdowns destroy queues that other consumers or the
  system still need.

Neither strategy is reliable because neither the consumer nor the
broker knows whether a queue should exist — only the flow manager
knows that.

### Why Separation

The flow service currently piggybacks on the config service process.
Adding broker queue management to the flow service introduces operations
that may have significant latency — RabbitMQ queue operations are
generally fast, but Kafka topic creation can involve partition
assignment, replication, and leader election delays.

The config service is on the critical path for every service in the
system — all services read configuration on startup and respond to
config pushes. Blocking the config service's request/response loop
while waiting for broker operations risks cascading latency across the
entire deployment.

Separating the flow service into its own process gives it an
independent latency budget. A slow `start-flow` (waiting for queue
creation across multiple brokers) does not affect config reads.
Additionally, the flow service's direct access to the `Configuration`
object is a coupling that masks what should be a clean client
relationship — the flow service only needs to read and write config
entries, which is exactly what the existing config client provides.

---

## Design

### Queue Ownership Model

Every queue in the system has exactly one owner responsible for its
creation and deletion:

| Queue type | Owner | Created when | Deleted when |
|---|---|---|---|
| Flow queues | Flow service | `start-flow` | `stop-flow` |
| Blueprint queues | Flow service | `start-flow` (idempotent) | Never (shared across flow instances) |
| System queues (config, librarian, knowledge, etc.) | Each service | Service startup | Never (system lifetime) |

Consumers never create queues. The consumer's `_connect()` method
connects to a queue that must already exist — it does not declare or
create it.

### Flow Service as Independent Service

The flow service becomes its own `Processor(AsyncProcessor)` in a
separate module and process. It:

- Listens on the existing flow request/response queues (already distinct
  from config queues — no consumer migration needed).
- Uses the async `ConfigClient` (extending `RequestResponse`) to
  read/write config state (blueprints, active-flow entries, flow
  records). Config pushes are triggered automatically by config
  writes — the backdoor `inc_version()` and `push()` calls are no
  longer needed.
- Has direct access to the pub/sub backend (inherited from
  `AsyncProcessor`) for queue lifecycle operations.

The config service (`trustgraph-flow/config/service`) is simplified:
the flow consumer, flow producer, and `FlowConfig` class are removed.
It returns to being purely a config service.

### Queue Lifecycle in the Pub/Sub Backend

The `PubSubBackend` protocol gains queue management methods. All new
methods are async — backends that use blocking I/O (e.g., pika for
RabbitMQ) handle threading internally.

```
PubSubBackend:
    create_producer(...)              # existing
    create_consumer(...)              # existing
    close()                           # existing
    async create_queue(topic, subscription)    # new
    async delete_queue(topic, subscription)    # new
    async queue_exists(topic, subscription)    # new
    async ensure_queue(topic, subscription)    # new
```

- `create_queue` — create a queue. Idempotent if queue already exists
  with the same properties. Fails if properties mismatch.
- `delete_queue` — delete a queue and its messages. Idempotent if
  queue does not exist.
- `queue_exists` — check whether a queue exists. Returns bool.
- `ensure_queue` — create-if-not-exists convenience wrapper.

The `topic` and `subscription` parameters together identify the queue,
mirroring `create_consumer` where the queue name is derived from both.

Backend implementations:

- **RabbitMQ**: `queue_declare`, `queue_delete`, and
  `queue_declare(passive=True)` via pika. Blocking calls wrapped in
  `asyncio.to_thread` inside the backend. Queue name derived using the
  existing `_parse_queue_id` logic.
- **Pulsar**: REST calls to the Pulsar admin API (port 8080).
  Create/delete persistent topics, delete subscriptions. Requires admin
  URL as additional configuration alongside the broker URL.
- **Kafka** (future): `AdminClient.create_topics()` and
  `AdminClient.delete_topics()` from the `confluent-kafka` library.
  Uses the same bootstrap servers as the broker connection.

### Flow Start: Queue Creation

When `handle_start_flow` processes a flow start request, after
resolving parameters and computing the template-substituted topic
identifiers, it creates queues before writing config state.

Queues are created for both `cls["blueprint"]` and `cls["flow"]`
entries. Blueprint queue creation is idempotent — multiple flows
creating the same blueprint queue is safe.

The flow start request returns only after queues are confirmed ready.
This gives callers a hard guarantee: when `start-flow` succeeds, the
data path is fully wired.

### Flow Stop: Two-Phase Shutdown

Stopping a flow is a two-phase transaction with a retry window between
them.

**Phase 1 — Signal processors to shut down:**

1. Set the flow record's status to `"stopping"`. This marks the flow
   as in-progress so that if the flow service crashes mid-stop, it can
   identify and resume incomplete shutdowns on restart.
2. Remove the flow's variants from each processor's `active-flow`
   config entries.
3. Config push fires automatically. Each `FlowProcessor` receives the
   update, compares wanted vs current flows, and calls `stop_flow` on
   flows no longer wanted — closing consumers and producers.

**Phase 2 — Delete queues with retries, then finalise:**

1. Retry queue deletion with delays, giving processors time to react
   to the config change and disconnect. Queue deletion is idempotent —
   if a queue was already removed by a previous attempt or was never
   created, the operation succeeds silently. Only `cls["flow"]` entries
   (per-flow-instance queues) are deleted — `cls["blueprint"]` entries
   are shared infrastructure and are not touched.
2. Delete the `flow` record from config.

The flow service retries persistently. A queue that cannot be deleted
after retries is logged as an error but does not block the stop
transaction from completing — a leaked queue is less harmful than a
flow that cannot be stopped.

**Crash recovery:** On startup, the flow service scans for flow
records with `"status": "stopping"`. These represent incomplete
shutdowns from a previous run. For each, it resumes from the
appropriate point — if active-flow entries are already cleared, it
proceeds directly to phase 2 (queue deletion and flow record cleanup).

### System Service Queues

System services (config, librarian, knowledge, etc.) are not managed
by the flow service. Each service calls `ensure_queue` for its own
queues during startup. These queues persist for the lifetime of the
system and are never explicitly deleted.

### Consumer Connection

Consumers never create queues. The consumer connects to a queue that
must already exist — created either by the flow service (for flow and
blueprint queues) or by the service itself (for system queues).

For RabbitMQ, this means `_connect()` no longer calls `queue_declare`.
It connects to a queue by name and fails if the queue does not exist.

For Pulsar, `subscribe()` inherently creates a subscription. This is
how Pulsar works and does not conflict with the lifecycle model —
Pulsar's broker manages subscription state, and the flow service uses
the admin API for explicit cleanup.

---

## Operational Impact

### Deployment

The flow service is a new container/process alongside the existing
config service. It requires:

- Access to the message broker (same credentials as other services —
  inherited from `AsyncProcessor` via standard CLI args).
- Access to the config service via pub/sub (config request/response
  queues — same as any other service that reads config).
- For Pulsar: the admin API URL (separate from the broker URL).

It does not require direct Cassandra access.

### Backward Compatibility

- The flow request/response queue interface is unchanged — API gateway
  and CLI tools that send flow requests continue to work without
  modification.
- The config service loses its flow handling capability, so both
  services must be deployed together. This is a breaking change in
  deployment topology but not in API.
- Queue deletion on flow stop is new behaviour. Existing deployments
  that rely on queues persisting after flow stop (e.g. for post-mortem
  message inspection) would need to drain queues before stopping flows.

---

## Assumptions

- **The flow service is the sole writer of flow configuration.** The
  two-phase stop transaction relies on the flow record's `"stopping"`
  status being authoritative — no other service or process modifies
  flow records, active-flow entries, or flow blueprints. This is true
  today (only `FlowConfig` writes to these config keys) and must remain
  true after separation. The config service provides the storage, but
  the flow service owns the semantics.

---

## Design Decisions

| Decision | Resolution | Rationale |
|---|---|---|
| Queue ownership | Every queue has exactly one explicit owner | Eliminates implicit creation, makes lifecycle auditable |
| Queue deletion strategy | Retry persistently, don't block stop | A leaked queue is less harmful than a flow stuck in stopping state |
| Purge without delete | Not needed | Flows are fully dynamic — stop and restart recreates everything |
| Blueprint-level queues | Created on flow start (idempotent), never deleted | Shared across flow instances; creation is safe, deletion would break other flows |
| Flow stop atomicity | Two-phase with `"stopping"` state | Allows crash recovery; flow service can resume incomplete shutdowns |
| Backend protocol methods | All async | Backends hide blocking I/O internally; callers never deal with threading |
| Pulsar lifecycle | REST admin API, not no-op | Persistent topics and subscriptions survive crashes; explicit cleanup needed |
| Consumer queue creation | Consumers never create queues | Single ownership; `_connect()` connects to existing queues only |