Commit graph

453 commits

Author SHA1 Message Date
CREDO23
99cf212c31 test: fix auth-mode mismatch and stale QuotaInsufficientError kwargs
Pin AUTH_TYPE=LOCAL (and REGISTRATION_ENABLED=TRUE) in the test bootstrap so
the email/password auth routers mount during integration tests regardless of a
developer's .env=GOOGLE; without this the upload tests 404 on registration.
Also update three tests to the current QuotaInsufficientError signature
(balance_micros) after used_micros/limit_micros were removed.
2026-06-12 12:19:49 +02:00
CREDO23
d5e0280097 test(etl-cache): cover two-phase eviction task on real infra 2026-06-12 11:54:36 +02:00
CREDO23
1460173dad test(etl-cache): cover extract_with_cache end-to-end 2026-06-12 11:50:57 +02:00
CREDO23
c49a0f1233 test(etl-cache): cover store, service, and repository on real infra 2026-06-12 11:50:57 +02:00
CREDO23
3dec3231d0 test(etl-cache): cover over-budget eviction selection 2026-06-12 11:50:52 +02:00
CREDO23
a3e7047c35 test(etl-cache): cover cacheability gate rules 2026-06-12 11:50:52 +02:00
CREDO23
dddacbe762 test(etl-cache): cover content-addressing dedup and key shape 2026-06-12 11:50:52 +02:00
DESKTOP-RTLN3BA\$punk
05190da0a9 chore: linting 2026-06-11 15:31:43 -07:00
CREDO23
41f4a58663 Merge remote-tracking branch 'upstream/dev' into improvement-podcast-graph
# Conflicts:
#	surfsense_backend/app/tasks/celery_tasks/podcast_tasks.py
2026-06-11 23:14:49 +02:00
CREDO23
ca9b157676 fix(podcasts): keep legacy episodes readable and guard regenerate 2026-06-11 12:43:07 +02:00
CREDO23
aa7f14d94f feat(podcasts): add revert-regeneration and surface cancel on the live card 2026-06-11 12:31:42 +02:00
CREDO23
f0fc660d70 feat(podcasts): constrain monologue briefs to a single speaker 2026-06-11 11:56:57 +02:00
CREDO23
eb56acc407 refactor(podcasts): regenerate via brief gate, render brief inline in chat 2026-06-11 11:45:17 +02:00
CREDO23
11a6b178a0 refactor(podcasts): drop transcript gate, add regenerate-from-ready and voice previews 2026-06-11 10:42:13 +02:00
CREDO23
c84525897b test(podcasts): relocate stateful tests to integration
Move the lifecycle service, Celery task bodies, and mark_failed coverage out of
DB-faking unit tests and into integration tests against a real Postgres, faking
only true externals (broker, object store, TTS, ffmpeg, billing, LLM). Add HTTP
slices for cancel, voices, scoping, and public-chat streaming. The unit tier is
now fake-free pure logic with no session doubles.
2026-06-11 06:27:00 +02:00
DESKTOP-RTLN3BA\$punk
a7407502d3 feat(refactor): refactor payment system to implement unified credit wallet.
- Updated environment variables and - configurations for credit purchases via Stripe, replacing legacy page pack system.
- Introduced auto-reload feature for credit top-ups and modified database models to track credit transactions.
- Updated notification system to handle insufficient credits and auto-reload failures.
- Adjusted API routes and schemas to reflect changes in credit management.
2026-06-10 16:49:03 -07:00
CREDO23
8f38737ad9 test(podcasts): retarget celery and observability tests to new tasks 2026-06-10 21:45:04 +02:00
CREDO23
aa7aa81c16 refactor(podcasts): drop language detection from brief 2026-06-10 20:51:38 +02:00
CREDO23
15e44616f3 test(podcasts): cover drafting billing gate 2026-06-10 18:44:26 +02:00
CREDO23
0bed4a0d38 test(podcasts): cover failure recording 2026-06-10 18:44:25 +02:00
CREDO23
0c7987cd9e test(podcasts): cover api read model 2026-06-10 18:44:25 +02:00
CREDO23
fa7ab8a06d test(podcasts): cover renderer validation 2026-06-10 18:44:25 +02:00
CREDO23
36c201f9e2 test(podcasts): cover structured json parsing 2026-06-10 18:44:25 +02:00
CREDO23
0c92ee963e test(podcasts): cover voice catalog 2026-06-10 18:44:25 +02:00
CREDO23
e926990d8e test(podcasts): cover language and voice resolution 2026-06-10 18:44:25 +02:00
CREDO23
aaa9f01087 test(podcasts): cover brief and transcript contracts 2026-06-10 18:44:25 +02:00
CREDO23
9d8e4e4f9d test(podcasts): cover lifecycle state machine 2026-06-10 18:44:25 +02:00
CREDO23
f61e8af8c0 test(podcasts): add shared test fixtures 2026-06-10 18:44:25 +02:00
CREDO23
59c1cf14c7 test(indexers): cover mark_connector_documents_failed behavior 2026-06-10 00:11:00 +02:00
CREDO23
77544ab768 test(google-drive): assert stuck pending/processing docs retry 2026-06-10 00:11:00 +02:00
CREDO23
9f76daec8f test(indexers): update download mock return shape 2026-06-09 23:39:25 +02:00
CREDO23
bdd3728c5b test(dropbox): update download failure return shape 2026-06-09 23:39:25 +02:00
CREDO23
b5aa41beb6 test(onedrive): update download failure return shape 2026-06-09 23:39:25 +02:00
CREDO23
5f59ad3ad3 test(google-drive): update download failure return shape 2026-06-09 23:39:25 +02:00
DESKTOP-RTLN3BA\$punk
ce952d2ad1 chore: linting 2026-06-09 00:42:26 -07:00
CREDO23
53a3920a82 fix(e2e): load .env after harness env defaults 2026-06-05 19:24:26 +02:00
CREDO23
8bdfd00a15 Merge upstream/dev 2026-06-05 19:18:12 +02:00
CREDO23
52ff304d64 fix(e2e): delegate connector work via task in fake LLM 2026-06-05 18:49:57 +02:00
CREDO23
bfadde93b7 fix(e2e): call .unique() when minting test token
The User mapper eager-loads the oauth_accounts collection via joined load
under AUTH_TYPE=GOOGLE, so the mint endpoint's query must call .unique()
before scalar_one_or_none() to avoid InvalidRequestError (500).
2026-06-05 18:17:11 +02:00
CREDO23
88fe213176 refactor(agents): extract subagent-invocation contract to subagents/shared
The knowledge_base subagent imported subagent_invoke_config + EXCLUDED_STATE_KEYS
from main_agent's checkpointed_subagent_middleware -- a subagent reaching into
main-agent internals. Both symbols (plus the recursion-limit constant they need)
are a subagent-invocation contract shared by the orchestrator's task middleware
and any nested-invoking subagent. Move them to subagents/shared/invocation.py;
config.py keeps the HITL resume side-channel and constants.py keeps the
main-agent tuning knobs. All consumers (task_tool, kb tool, tests) repointed.
2026-06-05 14:18:44 +02:00
CREDO23
0081b627e9 refactor(agents): move kb_persistence middleware into main_agent (owner)
The KB-persistence impl lived in shared/middleware/ but no subagent uses it --
consumers are the main_agent builder and the boundary event loop. Colocate with
its owner using the folder-per-middleware shape; __init__ re-exports the public
surface. Tests that reached module internals now alias the .middleware submodule.

  main_agent/middleware/kb_persistence.py -> kb_persistence/builder.py
  shared/middleware/kb_persistence.py     -> kb_persistence/middleware.py
2026-06-05 14:11:55 +02:00
CREDO23
a7a642fedc refactor(agents): move busy_mutex middleware into main_agent (owner)
The busy-mutex impl (BusyMutexMiddleware + cancel/turn-lifecycle primitives)
lived in shared/middleware/ but no subagent uses it -- consumers are the
main_agent builder and the boundary (turn lifecycle). Colocate with its owner
using the folder-per-middleware shape; __init__ re-exports the public surface so
boundary import sites only change package path:

  main_agent/middleware/busy_mutex.py    -> busy_mutex/builder.py
  shared/middleware/busy_mutex.py        -> busy_mutex/middleware.py
2026-06-05 14:08:45 +02:00
CREDO23
84b775c0ac refactor(agents): unify permissions into one vertical-slice package
Per-file verification of the slice-3 candidates showed receipts/ and
date_filters.py are shared contracts (consumed by shared/state + shared
middleware + subagents), so they correctly stay put.

permissions was the real misfit: the rule *model* lived at shared/permissions.py
while its enforcement lived at shared/middleware/permissions/. Unify them into a
single self-contained subsystem:

  shared/permissions.py                 -> shared/permissions/model.py
  shared/middleware/permissions/{deny,ask,middleware}
                                        -> shared/permissions/{deny,ask,middleware}

The package __init__ re-exports the model API + build_permission_mw, so the 32
external model consumers keep importing `from ...shared.permissions import Rule`
unchanged; only the 8 internal files redirect to `.model` (cycle-safe, model
loaded before middleware).
2026-06-05 13:29:48 +02:00
CREDO23
f2a61bc0ef refactor(agents): consolidate chat runtime infra under chat/runtime
Move the lower-level runtime/infra modules out of multi_agent_chat/shared/
(they were never used by subagents, so they failed the shared-by-all-siblings
rule) and unify them with the already-relocated checkpointer:

  agents/runtime/                      -> agents/chat/runtime/
  mac/shared/errors.py                 -> chat/runtime/errors.py
  mac/shared/llm_config.py             -> chat/runtime/llm_config.py
  mac/shared/prompt_caching.py         -> chat/runtime/prompt_caching.py
  mac/shared/mention_resolver.py       -> chat/runtime/mention_resolver.py
  mac/shared/path_resolver.py          -> chat/runtime/path_resolver.py

These sit below the agent packages: the boundary + agent factory + shared
middleware depend on them, and they import no agent code (acyclic).
2026-06-05 13:19:24 +02:00
CREDO23
24b62a63b4 refactor(agents): introduce chat/ category; dissolve top-level agents/shared
Recursive shared-folder rule: a shared/ must be shared by ALL siblings at its
level. The kernel (context, compaction, retry_after, web_search) was shared by
only 2 of the agents -- anonymous_chat + multi_agent_chat -- never by podcaster
or video_presentation. Those 2 are the "chat" category, so their shared code
belongs in that category's shared/, not the top-level one.

  app/agents/anonymous_chat/   -> app/agents/chat/anonymous_chat/
  app/agents/multi_agent_chat/ -> app/agents/chat/multi_agent_chat/
  app/agents/shared/           -> app/agents/chat/shared/   (anon<->mac kernel)

Top-level app/agents/shared/ is gone: nothing was shared across all three
categories (chat / podcaster / video_presentation).

~289 import sites rewritten (app.agents.{anonymous_chat,multi_agent_chat,shared}
-> app.agents.chat.*); all moves are git renames (history preserved).
app/agents/ now: chat/, podcaster/, video_presentation/, runtime/.
2026-06-05 12:54:02 +02:00
CREDO23
d59bb2b5aa refactor(agents): evict mac-only tools/middleware from shared kernel
These were never shared with anonymous_chat (nor podcaster/video_presentation)
-- only multi_agent_chat (subagents/main agent) and the boundary use them:

  shared/tools/mcp/             -> multi_agent_chat/shared/tools/mcp/
  shared/tools/hitl.py          -> multi_agent_chat/shared/tools/hitl.py
  shared/tools/catalog.py       -> multi_agent_chat/shared/tools/catalog.py
  shared/middleware/dedup_tool_calls.py
                                -> multi_agent_chat/shared/middleware/dedup_tool_calls.py

app/agents/shared/ now holds only the genuine anon<->mac kernel:
context, middleware/{compaction,retry_after}, tools/web_search.
2026-06-05 12:50:46 +02:00
CREDO23
b7ea829371 refactor(agents): relocate boundary-only infra out of shared/
Neither module is imported by any sibling agent package, so neither belongs in
the cross-agent shared kernel:

- checkpointer.py -> app/agents/runtime/checkpointer.py
  LangGraph Postgres checkpoint saver. It's cross-agent *runtime infra* wired by
  the boundary (app lifespan + anonymous_chat & multi_agent_chat flows), not
  agent code. New app/agents/runtime/ layer holds boundary-wired agent infra.

- shared/system_prompt.py + shared/prompts/ -> app/prompts/
  The legacy single-agent prompt composer. The live agents don't use it
  (main_agent has its own system_prompt/ builder; anonymous_chat builds inline);
  its only consumer is new_llm_config_routes for displaying default instructions.
  Moved to the existing non-agent prompt domain:
    system_prompt.py        -> app/prompts/default_system_instructions.py
    prompts/                 -> app/prompts/system_prompt_composer/

app/agents/shared/ now contains only genuinely cross-agent code: context,
middleware/{compaction,retry_after,dedup_tool_calls}, tools/.

NOTE: get_default_system_instructions() (LLM-config UI) composes from the legacy
library, which differs from what the live agents actually run -- pre-existing
latent staleness, not changed here.
2026-06-05 12:36:44 +02:00
CREDO23
82c5dc5b02 refactor(agents): move mac-only modules out of the cross-agent shared kernel
app/agents/shared/ is a sibling of anonymous_chat/podcaster/multi_agent_chat/
video_presentation, so it should only hold code shared across 2+ of those
agents. In practice podcaster and video_presentation import nothing from it,
and anonymous_chat needs only context + compaction + retry_after + web_search.
Everything else was multi_agent_chat-only (the boundary just passes through).

Move the multi_agent_chat-only cluster into multi_agent_chat/shared/ (files
moved verbatim via git rename; ~116 import sites rewritten):

  errors, feature_flags, filesystem_selection, path_resolver, prompt_caching,
  sandbox, llm_config, mention_resolver
  middleware/busy_mutex, middleware/kb_persistence

busy_mutex/llm_config/mention_resolver are boundary-only but import the moved
modules, so they were folded in to avoid a backwards shared -> multi_agent_chat
dependency. main_agent builders now import the impls directly; the shared
middleware barrel keeps only the genuinely-shared compaction + retry_after.

Also delete the dead leftover shared/plugins and shared/skills dirs (live
copies already live under main_agent/).

Remaining in app/agents/shared/: context, system_prompt(+prompts), checkpointer,
middleware/{compaction,retry_after,dedup_tool_calls}, tools/. checkpointer and
system_prompt are boundary-only infra pending a dedicated home decision.
2026-06-05 12:30:15 +02:00
CREDO23
c0c4f57f5d refactor(agents): delete dead PermissionMiddleware twin in shared kernel
app/agents/shared/middleware/permission.py was an older, monolithic
PermissionMiddleware superseded by the modular permissions/ package under
multi_agent_chat/shared/middleware/ (core + evaluation + ask/ + factory).
Production wires only the package (main_agent stack + every subagent
builder); the kernel file was reachable only through the shared barrel
re-export (itself unused) and two tests pinned to its dead internals
(_raise_interrupt, _normalize_permission_decision, old after_model shape).

- delete app/agents/shared/middleware/permission.py
- drop PermissionMiddleware from the shared middleware barrel
- delete test_permission_middleware.py (covered the dead impl only; live
  behavior is covered by tests/.../middleware/shared/permissions/*)
- test_desktop_safety_rules.py: keep the ruleset-level regression tests,
  drop the dead import + TestPermissionMiddlewareIntegration class
2026-06-05 12:10:08 +02:00
CREDO23
8ae190a11d refactor(agents): move MAC middleware impls out of shared kernel
knowledge_search, memory_injection and scoped_model_fallback no longer
belong in the cross-agent kernel (app/agents/shared/middleware): they are
consumed only inside multi_agent_chat. Relocate each impl next to the
builder that uses it:

- knowledge_search.py -> multi_agent_chat/shared/middleware/ (genuinely
  shared: its _render_priority_message feeds kb_context_projection, used by
  both the main agent and the KB subagent)
- memory_injection.py -> multi_agent_chat/shared/middleware/ (beside its
  memory.py builder)
- scoped_model_fallback.py -> multi_agent_chat/shared/middleware/resilience/
  (beside fallback.py/bundle.py)

Impls moved verbatim (git rename). Builders/consumers now import the local
sibling; main_agent knowledge_priority imports the new shared path; shared
middleware barrel trimmed.

Tests: repoint imports; convert the knowledge_search monkeypatch targets
from brittle dotted-string form to object-based patching (monkeypatch.setattr
on the imported module), which is robust to import ordering. No behavior
change.
2026-06-05 12:04:31 +02:00