chore: worktree dev setup (#484)

* chore: auto-assign per-worktree backend port via VS Code folderOpen task

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: remove .conductor dev setup (moved to native git worktrees)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Abhishek 2026-06-30 15:40:29 +05:30 committed by GitHub
parent 6937e01b49
commit 982030d26e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
11 changed files with 108 additions and 386 deletions

View file

@ -1,101 +0,0 @@
# Conductor workspace setup
This directory makes [Conductor](https://conductor.build) workspaces (git
worktrees) self-contained: each one installs its own deps and runs its own
backend + UI on a dedicated port range, so you can develop several branches at
once without collisions.
## How it works
Conductor gives every workspace a block of **10 ports** starting at
`$CONDUCTOR_PORT`. We use:
| Port | Service | Script |
| --------------- | ------------------ | ----------------------- |
| `CONDUCTOR_PORT` | UI (Next.js) | `run-ui.sh` |
| `CONDUCTOR_PORT + 1` | Backend (uvicorn) | `run-backend.sh` |
| `+2 .. +9` | reserved | — |
The UI sits on the base port so Conductor's **Open** button (`preview_urls`)
lands on it directly — preview templates only substitute `$CONDUCTOR_PORT`, with
no `+1` arithmetic. The UI is wired to its own workspace's backend
(`NEXT_PUBLIC_BACKEND_URL`, `BACKEND_URL`) and the backend's CORS / `UI_APP_URL`
point back at its own UI — all derived from `$CONDUCTOR_PORT`, so no two
workspaces interfere.
### Shared, NOT per-workspace
Postgres, Redis, and MinIO run as a **single shared Docker stack** (compose
project `dograh`). `setup.sh` brings it up idempotently with
`COMPOSE_PROJECT_NAME=dograh` so every workspace reuses the same one instead of
fighting over the fixed 5432/6379/9000 ports. All workspaces therefore share one
database — fine for app servers, but it means the **Arq worker should run in only
one workspace** (`run-worker.sh`), since a single worker drains the shared queue
for everyone.
## The Run menu
> **Important:** a Conductor workspace runs **one run script at a time**.
> `run_mode = "concurrent"` lets *different workspaces* run simultaneously — it
> does **not** let you click two run buttons in the *same* workspace. That's why
> **dev** starts the UI and backend together instead of relying on two buttons.
The Run dropdown offers:
- **dev** (default) — UI (`$CONDUCTOR_PORT`) **and** backend (`$CONDUCTOR_PORT+1`)
together, via `concurrently`. **Use this day to day.**
- **ui** — Next.js only, for debugging the frontend alone.
- **backend** — uvicorn only, for debugging the API alone.
- **worker** — Arq worker; start in just one workspace.
Hit **dev** and both servers come up; click Conductor's **Open** button to launch
the UI. Start **worker** once (e.g. in your primary workspace) when you need
background jobs — the same way you used to run a single arq worker in VS Code.
## Creating a new workspace
In Conductor: **New Workspace** → pick a branch. `setup.sh` then runs
automatically and:
1. copies the gitignored env files from your main checkout (see
[Environment files](#environment-files)),
2. checks out the `pipecat` submodule,
3. builds `venv` (Python 3.13) + installs backend/pipecat deps,
4. `npm install` for the UI,
5. ensures the shared Docker stack is up,
6. runs `alembic upgrade head`.
The first setup is slow (deps); afterwards the Run buttons are instant.
## Environment files
The app's env files hold real secrets, so they're **gitignored** and never
committed — a fresh worktree won't have them. `setup.sh` copies them from your
main checkout (`$CONDUCTOR_ROOT_PATH`) into each new workspace:
| File | Used by |
| ----------------------------- | -------------------------------- |
| `api/.env` | backend (DB/Redis URLs, secrets) |
| `api/.env.test` | backend test runs |
| `ui/.env` | UI (backend URL, public config) |
| `ui/.env.local` | UI secrets (Stack/PostHog/etc.) |
| `ui/.env.sentry-build-plugin` | UI Sentry source-map upload |
The copy is idempotent (only fills in what's missing), so re-running setup won't
clobber a workspace-local edit. **Add a new env file?** List it in the loop near
the top of `setup.sh`.
## Files
| File | Purpose |
| --------------------- | -------------------------------------------------- |
| `settings.toml` | Conductor config: setup + run scripts, preview_urls |
| `setup.sh` | One-time workspace bootstrap |
| `run-dev.sh` | Default run: UI + backend together (`concurrently`) |
| `run-ui.sh` | Foreground Next.js on `$CONDUCTOR_PORT` |
| `run-backend.sh` | Foreground uvicorn on `$CONDUCTOR_PORT + 1` |
| `run-worker.sh` | Foreground Arq worker (run in one workspace only) |
> Need machine-local tweaks (e.g. a different port base or skipping the worker)?
> Put them in `.conductor/settings.local.toml`, which is personal and not
> committed.

View file

@ -1,30 +0,0 @@
#!/usr/bin/env bash
# Conductor run script — Backend (FastAPI/uvicorn) for THIS workspace.
#
# Binds $CONDUCTOR_PORT + 1 and points CORS / UI_APP_URL at this workspace's UI
# (which runs on $CONDUCTOR_PORT). Runs in the FOREGROUND with exec (no &) so
# Conductor's SIGHUP cleanly tears it down.
set -euo pipefail
cd "${CONDUCTOR_WORKSPACE_PATH:-$PWD}"
UI_PORT="${CONDUCTOR_PORT:-8000}"
BACKEND_PORT="$((UI_PORT + 1))"
# Load the workspace's api/.env, then override the port-specific bits. The
# backend reads config via os.getenv, and these exports happen after the source,
# so they win over the values copied from the main checkout (which assume 8000/3000).
if [[ -f api/.env ]]; then set -a; # shellcheck disable=SC1091
source api/.env; set +a; fi
export FASTAPI_PORT="$BACKEND_PORT"
export UI_APP_URL="http://localhost:${UI_PORT}"
export CORS_ALLOWED_ORIGINS="http://localhost:${UI_PORT},http://127.0.0.1:${UI_PORT}"
if [[ ! -d venv ]]; then
echo "ERROR: venv missing. Re-run workspace setup (.conductor/setup.sh)." >&2
exit 1
fi
# shellcheck disable=SC1091
source venv/bin/activate
echo "[backend] workspace=${CONDUCTOR_WORKSPACE_NAME:-?} port=${BACKEND_PORT} ui=${UI_PORT}"
exec uvicorn api.app:app --host 0.0.0.0 --port "$BACKEND_PORT" --reload --reload-dir api

View file

@ -1,59 +0,0 @@
#!/usr/bin/env bash
# Conductor run script — full dev stack (UI + backend) for THIS workspace.
#
# A Conductor workspace runs ONE run script at a time (run_mode governs
# concurrency ACROSS workspaces, not multiple run buttons within one). So to get
# the UI AND the backend up together, we launch both with `concurrently`.
#
# Conductor stops a run with SIGHUP (then SIGKILL after 200ms). Neither npx nor
# concurrently reliably tears down their child tree on SIGHUP, so we supervise:
# trap the signal and recursively kill the whole tree ourselves. We do NOT exec,
# so this shell stays alive to handle the trap.
#
# Ports: UI on $CONDUCTOR_PORT, backend on $CONDUCTOR_PORT + 1.
set -uo pipefail
cd "${CONDUCTOR_WORKSPACE_PATH:-$PWD}"
UI_PORT="${CONDUCTOR_PORT:-8000}"
BACKEND_PORT="$((UI_PORT + 1))"
# Ensure node/npx is on PATH (load nvm + honor .nvmrc if needed).
if ! command -v npx >/dev/null 2>&1; then
export NVM_DIR="${NVM_DIR:-$HOME/.nvm}"
# shellcheck disable=SC1091
[[ -s "$NVM_DIR/nvm.sh" ]] && . "$NVM_DIR/nvm.sh"
command -v nvm >/dev/null 2>&1 && nvm use >/dev/null 2>&1 || true
fi
# Prefer a locally-installed concurrently (fast); fall back to npx, which fetches
# it once then caches — so this also works in workspaces created before it existed.
if [[ -x ui/node_modules/.bin/concurrently ]]; then
RUNNER=(ui/node_modules/.bin/concurrently)
else
RUNNER=(npx --yes concurrently)
fi
# Recursively SIGTERM a process and every descendant (children first). npm/npx/
# next don't reliably forward signals, so we signal each PID in the tree directly.
kill_tree() {
local pid=$1 child
for child in $(pgrep -P "$pid" 2>/dev/null); do kill_tree "$child"; done
kill -TERM "$pid" 2>/dev/null || true
}
shutdown() {
trap - HUP INT TERM EXIT
[[ -n "${CHILD:-}" ]] && kill_tree "$CHILD"
exit 0
}
trap shutdown HUP INT TERM EXIT
echo "[dev] workspace=${CONDUCTOR_WORKSPACE_NAME:-?} ui=:${UI_PORT} backend=:${BACKEND_PORT}"
"${RUNNER[@]}" \
--names "ui,backend" \
--prefix-colors "magenta,cyan" \
--kill-others \
"bash .conductor/run-ui.sh" \
"bash .conductor/run-backend.sh" &
CHILD=$!
wait "$CHILD"

View file

@ -1,29 +0,0 @@
#!/usr/bin/env bash
# Conductor run script — UI (Next.js) for THIS workspace.
#
# Binds $CONDUCTOR_PORT (so Conductor's Open button / preview_urls land here) and
# talks to this workspace's backend on $CONDUCTOR_PORT + 1. Foreground exec (no &)
# so Conductor can stop it cleanly.
set -euo pipefail
cd "${CONDUCTOR_WORKSPACE_PATH:-$PWD}"
UI_PORT="${CONDUCTOR_PORT:-8000}"
BACKEND_PORT="$((UI_PORT + 1))"
BACKEND="http://localhost:${BACKEND_PORT}"
# Ensure node is on PATH (load nvm + honor .nvmrc if needed).
if ! command -v node >/dev/null 2>&1; then
export NVM_DIR="${NVM_DIR:-$HOME/.nvm}"
# shellcheck disable=SC1091
[[ -s "$NVM_DIR/nvm.sh" ]] && . "$NVM_DIR/nvm.sh"
command -v nvm >/dev/null 2>&1 && nvm use >/dev/null 2>&1 || true
fi
# Shell env overrides .env files in Next.js, so this points the UI at the right
# backend for this workspace.
export BACKEND_URL="$BACKEND"
export NEXT_PUBLIC_BACKEND_URL="$BACKEND"
cd ui
echo "[ui] workspace=${CONDUCTOR_WORKSPACE_NAME:-?} port=${UI_PORT} backend=${BACKEND}"
exec npm run dev -- --port "$UI_PORT"

View file

@ -1,21 +0,0 @@
#!/usr/bin/env bash
# Conductor run script — Arq background worker.
#
# Run this in ONE workspace only. Every workspace shares the same Redis/Postgres,
# so a single arq worker drains the task queue for all of them — multiple workers
# would just fight over the same jobs. Foreground exec so Conductor stops it cleanly.
set -euo pipefail
cd "${CONDUCTOR_WORKSPACE_PATH:-$PWD}"
if [[ -f api/.env ]]; then set -a; # shellcheck disable=SC1091
source api/.env; set +a; fi
if [[ ! -d venv ]]; then
echo "ERROR: venv missing. Re-run workspace setup (.conductor/setup.sh)." >&2
exit 1
fi
# shellcheck disable=SC1091
source venv/bin/activate
echo "[worker] arq worker on shared Redis — workspace=${CONDUCTOR_WORKSPACE_NAME:-?}"
exec python -m arq api.tasks.arq.WorkerSettings --custom-log-dict api.tasks.arq.LOG_CONFIG

View file

@ -1,54 +0,0 @@
# Conductor project config — shared, committed so every contributor (and every
# new workspace) gets the same per-worktree dev workflow.
#
# Each Conductor workspace is its own git worktree and gets its own 10-port range
# starting at $CONDUCTOR_PORT. We use:
# $CONDUCTOR_PORT -> ui (Next.js) <- Conductor's Open button lands here
# $CONDUCTOR_PORT + 1 -> backend (FastAPI/uvicorn)
# +2 .. +9 -> reserved for future per-workspace services
#
# Postgres/Redis/MinIO are a single SHARED Docker stack (project name "dograh"),
# not per-workspace — see .conductor/setup.sh and .conductor/README.md.
"$schema" = "https://conductor.build/schemas/settings.repo.schema.json"
# Conductor's Open button. The UI runs on $CONDUCTOR_PORT, so this opens it.
# (Preview URL templates only substitute $CONDUCTOR_PORT — no "+1" arithmetic —
# which is why the UI, the thing you actually open, sits on the base port.)
[[preview_urls]]
name = "UI"
url = "http://localhost:$CONDUCTOR_PORT"
[scripts]
# Runs once when a workspace is created: copies gitignored env files, inits the
# pipecat submodule, builds the venv + node_modules, ensures the shared Docker
# stack, and runs migrations.
setup = "bash .conductor/setup.sh"
# concurrent => multiple WORKSPACES can run their dev servers at the same time
# (each on its own CONDUCTOR_PORT range). NOTE: within a single workspace,
# Conductor runs ONE run script at a time — that's why the default "dev" script
# below starts the UI and backend together rather than relying on two buttons.
run_mode = "concurrent"
# Full dev stack: UI ($CONDUCTOR_PORT) + backend ($CONDUCTOR_PORT+1) together.
# Default Run button — this is the one to use day to day.
[scripts.run.dev]
command = "bash .conductor/run-dev.sh"
icon = "play"
default = true
# UI only (Next.js) on $CONDUCTOR_PORT — handy for debugging the frontend alone.
[scripts.run.ui]
command = "bash .conductor/run-ui.sh"
icon = "play"
# Backend only (FastAPI/uvicorn) on $CONDUCTOR_PORT + 1 — debugging the API alone.
[scripts.run.backend]
command = "bash .conductor/run-backend.sh"
icon = "server"
# Arq background worker. Start in ONE workspace only — all workspaces share the
# same Redis/Postgres, so a single worker drains the queue for everyone.
[scripts.run.worker]
command = "bash .conductor/run-worker.sh"
icon = "server"

View file

@ -1,91 +0,0 @@
#!/usr/bin/env bash
# Conductor setup script — runs ONCE when a workspace (git worktree) is created.
#
# A fresh worktree only has git-tracked files, so this recreates the rest: the
# gitignored env files, the pipecat submodule checkout, a Python venv,
# ui/node_modules, the shared local Docker stack, and the DB schema.
#
# Conductor injects: CONDUCTOR_ROOT_PATH (main checkout), CONDUCTOR_WORKSPACE_PATH,
# CONDUCTOR_WORKSPACE_NAME, CONDUCTOR_PORT (first of 10), CONDUCTOR_IS_LOCAL.
set -euo pipefail
ROOT="${CONDUCTOR_ROOT_PATH:-}"
WS="${CONDUCTOR_WORKSPACE_PATH:-$PWD}"
cd "$WS"
log() { printf '\n\033[1;36m[conductor-setup]\033[0m %s\n' "$*"; }
# 1) Copy the gitignored env files from the main checkout. These hold real
# secrets, so they're never committed — a fresh worktree won't have them. We copy
# only what's missing (idempotent), so re-running setup won't clobber local edits.
# (See README "Environment files" for the canonical list.)
if [[ -n "$ROOT" && "$ROOT" != "$WS" ]]; then
log "Copying gitignored env files from $ROOT"
for f in api/.env api/.env.test ui/.env ui/.env.local ui/.env.sentry-build-plugin; do
if [[ ! -f "$f" && -f "$ROOT/$f" ]]; then
mkdir -p "$(dirname "$f")"
cp "$ROOT/$f" "$f"
echo " copied $f"
fi
done
fi
# 2) pipecat submodule — REQUIRED here, NOT redundant with step 3.
# A fresh git worktree has an empty pipecat/. setup_requirements.sh --dev
# (step 3) deliberately SKIPS `git submodule update` (it assumes CI already
# checked out submodules) but still runs `uv pip install -e ./pipecat`, which
# fails unless the checkout is already on disk. So we populate it first.
log "Initializing git submodules (pipecat)"
git submodule update --init --recursive
# 3) Python venv with 3.13 + backend/pipecat deps ----------------------------
# Bare `python3` may be 3.14 here; setup_requirements.sh requires 3.12/3.13.
PY313="$(command -v python3.13 || true)"
if [[ -z "$PY313" ]]; then
echo "ERROR: python3.13 not found. Install it (e.g. brew install python@3.13)." >&2
exit 1
fi
if [[ ! -d venv ]]; then
log "Creating venv with $PY313 ($("$PY313" --version))"
"$PY313" -m venv venv
fi
# shellcheck disable=SC1091
source venv/bin/activate
log "Installing backend + pipecat deps (uv) — this is the slow step"
bash scripts/setup_requirements.sh --dev
# 4) UI deps -----------------------------------------------------------------
ensure_node() {
if ! command -v node >/dev/null 2>&1; then
export NVM_DIR="${NVM_DIR:-$HOME/.nvm}"
# shellcheck disable=SC1091
[[ -s "$NVM_DIR/nvm.sh" ]] && . "$NVM_DIR/nvm.sh"
command -v nvm >/dev/null 2>&1 && nvm use >/dev/null 2>&1 || true
fi
}
ensure_node
log "Installing UI deps (npm install)"
( cd ui && npm install )
# 5) Shared local Docker stack (idempotent, pinned project name) -------------
# All workspaces share ONE stack named "dograh" so they never collide on the
# fixed Postgres/Redis/MinIO ports. If it's already up this is a no-op.
if command -v docker >/dev/null 2>&1; then
log "Ensuring shared Docker stack (COMPOSE_PROJECT_NAME=dograh)"
COMPOSE_PROJECT_NAME=dograh docker compose -f docker-compose-local.yaml up -d \
|| echo " (warning: could not start docker stack; start it manually)"
else
echo " (docker not found; start Postgres/Redis/MinIO yourself)"
fi
# 6) DB migrations (best-effort; shared DB, alembic is idempotent) -----------
if [[ -f api/.env ]]; then
log "Running DB migrations (alembic upgrade head)"
set -a; # shellcheck disable=SC1091
source api/.env; set +a
alembic -c api/alembic.ini upgrade head || echo " (warning: migrations skipped/failed — is the DB up?)"
fi
PORT="${CONDUCTOR_PORT:-8000}"
log "Setup complete for '${CONDUCTOR_WORKSPACE_NAME:-?}'."
echo " Run menu: backend -> :${PORT} ui -> :$((PORT + 1)) worker -> shared arq"

2
.vscode/launch.json vendored
View file

@ -23,6 +23,8 @@
"api.app:app",
"--reload",
"--host", "0.0.0.0"
// Port comes from UVICORN_PORT in api/.env (per-worktree);
// unset -> uvicorn's default 8000. See scripts/worktree-assign-port.sh.
],
"cwd": "${workspaceFolder}",
"envFile": "${workspaceFolder}/api/.env",

View file

@ -1,3 +1,9 @@
{
"python.defaultInterpreterPath": "${workspaceFolder}/venv/bin/python"
"python.defaultInterpreterPath": "${workspaceFolder}/venv/bin/python",
"git.detectWorktrees": true,
"git.worktreeIncludeFiles": [
"api/.env",
"api/.env.test",
"ui/.env.local"
]
}

23
.vscode/tasks.json vendored Normal file
View file

@ -0,0 +1,23 @@
{
// Auto-runs when a worktree folder is opened. The first time, VS Code asks
// to "Allow Automatic Tasks in Folder" (or run it via the command palette:
// "Tasks: Allow Automatic Tasks in Folder"). Assigns this worktree a unique
// backend port and points the UI env at it see scripts/worktree-assign-port.sh.
"version": "2.0.0",
"tasks": [
{
"label": "Assign worktree port",
"type": "shell",
"command": "${workspaceFolder}/scripts/worktree-assign-port.sh",
"presentation": {
"reveal": "silent",
"panel": "shared",
"close": true
},
"runOptions": {
"runOn": "folderOpen"
},
"problemMatcher": []
}
]
}

76
scripts/worktree-assign-port.sh Executable file
View file

@ -0,0 +1,76 @@
#!/usr/bin/env bash
# Assign a unique backend port to this git worktree and rewrite the env files
# that depend on it. Runs automatically as a VS Code "folderOpen" task (see
# .vscode/tasks.json), so it executes once per worktree when you open it.
#
# Scheme:
# - The MAIN worktree is left untouched (backend stays on uvicorn's default 8000).
# - Each linked worktree gets the next free backend port: 8001, 8002, ...
# - api/.env : UVICORN_PORT -> the assigned backend port
# - ui/.env.local : BACKEND_URL -> http://localhost:<port>
# NEXT_PUBLIC_BACKEND_URL -> http://localhost:<port>
#
# CORS is intentionally NOT touched: local dev runs DEPLOYMENT_MODE="oss", where
# the API forces allow_origins=["*"] and ignores CORS_ALLOWED_ORIGINS entirely.
#
# Idempotent: re-running keeps an already-assigned, non-colliding port. The UI
# dev server is left alone — `npm run dev` auto-selects a free port (3000, 3001…).
set -euo pipefail
ROOT="$(git rev-parse --show-toplevel)"
MAIN="$(git worktree list --porcelain | sed -n '1s/^worktree //p')"
[ "$ROOT" = "$MAIN" ] && { echo "[worktree] main worktree -> backend 8000 (untouched)"; exit 0; }
AENV="$ROOT/api/.env"
UENV="$ROOT/ui/.env.local"
[ -f "$AENV" ] || { echo "[worktree] no api/.env yet; skipping"; exit 0; }
# Echo the UVICORN_PORT value from an env file (empty if unset/missing).
port_of() { { grep -E '^[[:space:]]*UVICORN_PORT=' "$1" 2>/dev/null | tail -1 | sed -E 's/^[^=]*=//; s/[[:space:]]//g'; } || true; }
# Ports already in use by OTHER worktrees (main implicitly uses 8000).
used=(8000)
while IFS= read -r line; do
case "$line" in
"worktree "*)
wt="${line#worktree }"
[ "$wt" = "$ROOT" ] && continue
p="$(port_of "$wt/api/.env")"
[ -n "$p" ] && used+=("$p")
;;
esac
done < <(git worktree list --porcelain)
mine="$(port_of "$AENV")"
# Keep my port if it's set and not claimed by another worktree; else take max+1.
reassign=1
if [ -n "$mine" ]; then
reassign=0
for u in "${used[@]}"; do [ "$u" = "$mine" ] && reassign=1; done
fi
if [ "$reassign" -eq 1 ]; then
max=0
for u in "${used[@]}"; do [ "$u" -gt "$max" ] && max="$u"; done
B=$((max + 1))
else
B="$mine"
fi
# Insert or update KEY=VALUE in an env file, preserving everything else.
upsert() {
local key="$1" val="$2" file="$3"
if grep -qE "^[[:space:]]*${key}=" "$file"; then
sed -i.bak -E "s|^[[:space:]]*${key}=.*|${key}=${val}|" "$file" && rm -f "$file.bak"
else
printf '\n%s=%s\n' "$key" "$val" >> "$file"
fi
}
upsert UVICORN_PORT "$B" "$AENV"
if [ -f "$UENV" ]; then
upsert BACKEND_URL "http://localhost:$B" "$UENV"
upsert NEXT_PUBLIC_BACKEND_URL "http://localhost:$B" "$UENV"
fi
echo "[worktree] $(basename "$ROOT"): backend=$B (UI auto-port via 'npm run dev')"