Commit graph

384 commits

Author SHA1 Message Date
CREDO23
f541114544 feat(index-cache): add cached embedding set table and repository 2026-06-12 16:48:01 +02:00
CREDO23
d898716cf4 feat(migration): add etl_cache_parses table 2026-06-12 11:23:50 +02:00
DESKTOP-RTLN3BA\$punk
cff721aa42 feat(migration): evolve podcast lifecycle by detaching from zero_publication and updating column handling 2026-06-11 16:17:14 -07:00
DESKTOP-RTLN3BA\$punk
05190da0a9 chore: linting 2026-06-11 15:31:43 -07:00
CREDO23
41f4a58663 Merge remote-tracking branch 'upstream/dev' into improvement-podcast-graph
# Conflicts:
#	surfsense_backend/app/tasks/celery_tasks/podcast_tasks.py
2026-06-11 23:14:49 +02:00
DESKTOP-RTLN3BA\$punk
65e511f77b feat: enhance credit management and user experience
- Updated database queries to check for column existence with schema context.
- Modified credit purchase quantity limits to allow up to 10,000 credits.
- Improved user interface for credit purchases, enabling custom amounts and clamping input values.
- Adjusted FAQ content to clarify credit purchasing process.
2026-06-10 22:52:27 -07:00
DESKTOP-RTLN3BA\$punk
a7407502d3 feat(refactor): refactor payment system to implement unified credit wallet.
- Updated environment variables and - configurations for credit purchases via Stripe, replacing legacy page pack system.
- Introduced auto-reload feature for credit top-ups and modified database models to track credit transactions.
- Updated notification system to handle insufficient credits and auto-reload failures.
- Adjusted API routes and schemas to reflect changes in credit management.
2026-06-10 16:49:03 -07:00
CREDO23
467bcd4f7b feat(podcasts): add zero publication migration 2026-06-10 18:44:12 +02:00
CREDO23
63f5f12834 feat(podcasts): add lifecycle migration 2026-06-10 18:44:12 +02:00
DESKTOP-RTLN3BA\$punk
ce952d2ad1 chore: linting 2026-06-09 00:42:26 -07:00
Anish Sarkar
4e00f24a03 feat(docker): add ZERO_AUTO_RESET configuration for improved replication safety
- Introduced the ZERO_AUTO_RESET environment variable to enable automatic reset of the SQLite replica in case of replication halts.
- Updated Docker Compose files to include ZERO_AUTO_RESET in service configurations.
- Enhanced documentation to clarify the purpose and usage of the new variable.
2026-06-06 14:21:14 +05:30
Anish Sarkar
290a9539ef feat(db): Remove document summary LLM schema 2026-06-04 00:48:53 +05:30
Anish Sarkar
c75a080997 Merge remote-tracking branch 'upstream/dev' into feat/ui-fixes 2026-06-03 21:53:35 +05:30
CREDO23
0259c53f50 fix(db): restore automation_runs to zero_publication 2026-06-02 16:56:01 +02:00
Anish Sarkar
e15df9d949 feat(migration): add automation_runs table to gateway migration 2026-06-02 20:13:31 +05:30
CREDO23
537b6e3b33 feat(db): add document_files migration 2026-06-02 16:10:50 +02:00
Anish Sarkar
375056858f chore: update migration number 2026-06-02 01:27:34 +05:30
Anish Sarkar
8f8abe6f58 chore: renamed alembic migrations 2026-06-02 01:24:50 +05:30
Anish Sarkar
2e64d5d3de Merge remote-tracking branch 'upstream/dev' into feat/whatsapp-gateway-integration 2026-06-02 01:22:32 +05:30
Anish Sarkar
1f83898a87 feat(gateway): renamed the gateway related alembic migrations 2026-06-02 01:19:28 +05:30
CREDO23
ec2b57bfb3 feat(db): add automation_runs to zero_publication (thin column list)
Resyncs zero_publication via the canonical ALTER PUBLICATION ... SET
TABLE pattern (mirrors migration 143) to publish a thin live view of
automation_runs for dashboard real-time updates. Heavy JSONB fields
(definition_snapshot, inputs, output, artifacts, error) stay on REST.
2026-06-01 21:02:16 +02:00
Anish Sarkar
e3de7c4667 Merge remote-tracking branch 'upstream/dev' into feat/whatsapp-gateway-integration 2026-06-02 00:29:32 +05:30
CREDO23
b2b70cfa3e fix: make migration 144 idempotent 2026-06-01 20:11:25 +02:00
Anish Sarkar
4b8ca29f9e feat(gateway): add Discord external chat platform 2026-06-01 20:58:17 +05:30
Anish Sarkar
b0b0f3517b feat(gateway): add Slack external chat platform 2026-06-01 12:35:52 +05:30
CREDO23
356400ae2a feat(automations): migrate automation_trigger_type enum to add event 2026-05-29 17:48:39 +02:00
DESKTOP-RTLN3BA\$punk
40ca9e6ed2 refactor: remove search_surfsense_docs tool and related references
- Deleted the `search_surfsense_docs` tool and its associated files, streamlining the agent's toolset.
- Updated various components and prompts to remove references to the now-removed tool, ensuring consistency across the codebase.
- Adjusted documentation to direct users to the SurfSense documentation link for product-related queries instead.
2026-05-28 22:35:14 -07:00
DESKTOP-RTLN3BA\$punk
94e834134f chore: linting 2026-05-28 19:21:29 -07:00
Anish Sarkar
3faaa25af6 refactor(database): update default source values to 'surfsense' for chat threads and messages 2026-05-28 13:11:05 +05:30
Anish Sarkar
a57b741d5e refactor(gateway): rename persistence models to external chat 2026-05-28 04:37:27 +05:30
CREDO23
27ab367a13 feat(automations): static_inputs on triggers + vertical-slice api/services 2026-05-27 21:21:43 +02:00
Anish Sarkar
81cf63ac96 feat(gateway): add messaging gateway persistence schema 2026-05-27 23:34:46 +05:30
CREDO23
f08b316441 add next_fire_at to automation_triggers and croniter dep 2026-05-27 17:55:58 +02:00
CREDO23
f646b5cbab feat(rbac): backfill automations permissions on existing roles 2026-05-27 15:37:25 +02:00
CREDO23
fe32cd35ed refactor(automation): rename trigger config column to params 2026-05-27 13:29:18 +02:00
CREDO23
35117a952d refactor(automation): drop agent_session_id from AutomationRun
A run can contain zero, one, or N agent_task steps. A single
agent_session_id at the run level holds at most one of them, so the
column is the wrong shape for the data.

Per-step session ids (LangGraph thread/checkpoint reference for an
agent_task step) live inside step_results[i] alongside the rest of
the per-step bag (status, timings, output). Each agent step records
its own; non-agent steps record nothing. Run-level "primary session"
is a UI concern, not a schema concern.

Trade-off: trace -> run reverse lookup is now a JSONB query, not an
index hit. Usually traversal goes run -> trace; if the reverse
becomes hot we add a GIN index on step_results or a generated
column — both additive.

Changes:
- AutomationRun: drop the agent_session_id column; module docstring
  notes where per-step session ids now live.
- Migration 144: drop the column from the CREATE TABLE; downgrade
  unchanged.

Safe to edit migration 144 in place (vs. add 145 with ALTER ... DROP):
this branch has not shipped and the table has never existed in any
deployed database.
2026-05-27 11:41:32 +02:00
CREDO23
d9183464d9 feat(automation): add Alembic migration for the three automation tables
Migration 144 -> 143. Matches the SQLAlchemy models added in commit 7
and the v1 data model in automation-design-plan.md §9.

Up:
  - CREATE TYPE automation_status / automation_trigger_type /
    automation_run_status (PostgreSQL ENUMs created first because the
    tables reference them).
  - CREATE TABLE automations with FK to searchspaces (CASCADE) and
    user (SET NULL); five indexes matching the SQLAlchemy model.
  - CREATE TABLE automation_triggers with FK to automations
    (CASCADE); four indexes.
  - CREATE TABLE automation_runs with FK to automations (CASCADE) and
    automation_triggers (SET NULL — null trigger_id == manual via UI);
    four indexes.

Down: drops every index, table, and ENUM in reverse-dependency order
so the migration is reversible without ON DELETE side effects.

Verified: `alembic history` resolves 143 -> 144 (head) cleanly.

domain_events (Phase 3) and mcp_connections / mcp_tools (Phase 4) ship
in their own migrations when the consuming feature lands; this
migration only covers the three v1 tables.
2026-05-26 22:44:33 +02:00
Anish Sarkar
af1d2fa430 Merge remote-tracking branch 'upstream/dev' into fix/zero-cache-stale-replica-1355 2026-05-16 19:30:09 +05:30
Anish Sarkar
c43bfdb1d9 chore(migration): update migration files to enforce new publication mutation pattern 2026-05-12 17:31:49 +05:30
Anish Sarkar
d923d34e38 feat(migration): add migration 143 to force zero-cache resync after Zero upgrade 2026-05-12 16:43:50 +05:30
Anish Sarkar
2c8828f60c fix: ensure idempotency in alembic migrations by checking for existing columns and indexes before creation 2026-05-10 22:45:26 +05:30
Anish Sarkar
319923fb40 fix: add checks for existing tables and indexes before creating them in alembic migrations for idempotency 2026-05-10 22:40:29 +05:30
Anish Sarkar
292b4d70ac chore: enhance E2E tests workflow by adding caching for Next.js build and updating test command 2026-05-10 22:21:06 +05:30
DESKTOP-RTLN3BA\$punk
19b6e0a025 feat: moved chat persistance to Server Side 2026-05-04 03:06:15 -07:00
DESKTOP-RTLN3BA\$punk
ae9d36d77f feat: unified credits and its cost calculations 2026-05-02 14:34:23 -07:00
Anish Sarkar
cd25175b84 chore: ran linting 2026-05-02 03:36:13 +05:30
Anish Sarkar
e9d964514b feat(alembic): add user table to zero_publication for selective replication of usage metrics 2026-05-02 03:31:03 +05:30
Anish Sarkar
421a4d7d08 refactor(auto_model_pin): simplify thread-level pinning by removing unused fields and indexes 2026-05-01 19:32:42 +05:30
DESKTOP-RTLN3BA\$punk
c644f02d05 chore: linting
Some checks are pending
Build and Push Docker Images / tag_release (push) Waiting to run
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (backend, surfsense-backend) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (web, surfsense-web) (push) Blocked by required conditions
2026-04-30 18:42:38 -07:00
Anish Sarkar
2a01711bc9 feat(chat): expand error handling for chat operations by introducing a passthrough code set, improving response management and user feedback 2026-04-30 16:20:14 +05:30