apunkt/SurfSense
1
0
Fork 0
mirror of https://github.com/MODSetter/SurfSense.git synced 2026-04-30 03:16:25 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

feat: add processing mode support for document uploads and ETL pipeline, improded error handling ux
Some checks are pending
Build and Push Docker Images / tag_release (push) Waiting to run
Details
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64) (push) Blocked by required conditions
Details
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64) (push) Blocked by required conditions
Details
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64) (push) Blocked by required conditions
Details
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64) (push) Blocked by required conditions
Details
Build and Push Docker Images / create_manifest (backend, surfsense-backend) (push) Blocked by required conditions
Details
Build and Push Docker Images / create_manifest (web, surfsense-web) (push) Blocked by required conditions
Details

Browse source 
- Introduced a `ProcessingMode` enum to differentiate between basic and premium processing modes.
- Updated `EtlRequest` to include a `processing_mode` field, defaulting to basic.
- Enhanced ETL pipeline services to utilize the selected processing mode for Azure Document Intelligence and LlamaCloud parsing.
- Modified various routes and services to handle processing mode, affecting document upload and indexing tasks.
- Improved error handling and logging to include processing mode details.
- Added tests to validate processing mode functionality and its impact on ETL operations.
This commit is contained in:
DESKTOP-RTLN3BA\$punk 2026-04-14 21:26:00 -07:00
parent b659f41bab
commit 656e061f84
104 changed files with 1900 additions and 909 deletions
Download patch file Download diff file Expand all files Collapse all files

5
surfsense_web/lib/folder-sync-upload.ts
View file

@ -17,6 +17,7 @@ export interface FolderSyncParams {
excludePatterns: string[];
fileExtensions: string[];
enableSummary: boolean;
processingMode?: "basic" | "premium";
rootFolderId?: number | null;
onProgress?: (progress: FolderSyncProgress) => void;
signal?: AbortSignal;
@ -62,6 +63,7 @@ async function uploadBatchesWithConcurrency(
searchSpaceId: number;
rootFolderId: number | null;
enableSummary: boolean;
processingMode?: "basic" | "premium";
signal?: AbortSignal;
onBatchComplete?: (filesInBatch: number) => void;
}
@ -99,6 +101,7 @@ async function uploadBatchesWithConcurrency(
relative_paths: batch.map((e) => e.relativePath),
root_folder_id: resolvedRootFolderId,
enable_summary: params.enableSummary,
processing_mode: params.processingMode,
},
params.signal
);
@ -145,6 +148,7 @@ export async function uploadFolderScan(params: FolderSyncParams): Promise<number
excludePatterns,
fileExtensions,
enableSummary,
processingMode,
signal,
} = params;
let rootFolderId = params.rootFolderId ?? null;
@ -190,6 +194,7 @@ export async function uploadFolderScan(params: FolderSyncParams): Promise<number
searchSpaceId,
rootFolderId: rootFolderId ?? null,
enableSummary,
processingMode,
signal,
onBatchComplete: (count) => {
uploaded += count;