feat: add processing mode support for document uploads and ETL pipeline, improded error handling ux
Some checks are pending
Build and Push Docker Images / tag_release (push) Waiting to run
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-24.04-arm, linux/arm64, arm64) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_backend, ./surfsense_backend/Dockerfile, backend, surfsense-backend, ubuntu-latest, linux/amd64, amd64) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-24.04-arm, linux/arm64, arm64) (push) Blocked by required conditions
Build and Push Docker Images / build (./surfsense_web, ./surfsense_web/Dockerfile, web, surfsense-web, ubuntu-latest, linux/amd64, amd64) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (backend, surfsense-backend) (push) Blocked by required conditions
Build and Push Docker Images / create_manifest (web, surfsense-web) (push) Blocked by required conditions

- Introduced a `ProcessingMode` enum to differentiate between basic and premium processing modes.
- Updated `EtlRequest` to include a `processing_mode` field, defaulting to basic.
- Enhanced ETL pipeline services to utilize the selected processing mode for Azure Document Intelligence and LlamaCloud parsing.
- Modified various routes and services to handle processing mode, affecting document upload and indexing tasks.
- Improved error handling and logging to include processing mode details.
- Added tests to validate processing mode functionality and its impact on ETL operations.
This commit is contained in:
DESKTOP-RTLN3BA\$punk 2026-04-14 21:26:00 -07:00
parent b659f41bab
commit 656e061f84
104 changed files with 1900 additions and 909 deletions

View file

@ -145,18 +145,14 @@ export function GeneralSettingsManager({ searchSpaceId }: GeneralSettingsManager
<form onSubmit={onSubmit} className="space-y-6">
<div className="flex flex-col gap-6">
<div className="space-y-2">
<Label htmlFor="search-space-name">
{t("general_name_label")}
</Label>
<Label htmlFor="search-space-name">{t("general_name_label")}</Label>
<Input
id="search-space-name"
placeholder={t("general_name_placeholder")}
value={name}
onChange={(e) => setName(e.target.value)}
/>
<p className="text-xs text-muted-foreground">
{t("general_name_description")}
</p>
<p className="text-xs text-muted-foreground">{t("general_name_description")}</p>
</div>
<div className="space-y-2">
@ -170,9 +166,7 @@ export function GeneralSettingsManager({ searchSpaceId }: GeneralSettingsManager
value={description}
onChange={(e) => setDescription(e.target.value)}
/>
<p className="text-xs text-muted-foreground">
{t("general_description_description")}
</p>
<p className="text-xs text-muted-foreground">{t("general_description_description")}</p>
</div>
</div>
@ -189,28 +183,28 @@ export function GeneralSettingsManager({ searchSpaceId }: GeneralSettingsManager
</div>
</form>
<div className="border-t pt-6 flex flex-col gap-3 md:flex-row md:items-center md:justify-between">
<div className="space-y-1">
<Label>Export knowledge base</Label>
<p className="text-xs text-muted-foreground">
Download all documents in this search space as a ZIP of markdown files.
</p>
<div className="border-t pt-6 flex flex-col gap-3 md:flex-row md:items-center md:justify-between">
<div className="space-y-1">
<Label>Export knowledge base</Label>
<p className="text-xs text-muted-foreground">
Download all documents in this search space as a ZIP of markdown files.
</p>
</div>
<Button
type="button"
variant="secondary"
size="sm"
disabled={isExporting}
onClick={handleExportKB}
className="relative w-fit shrink-0"
>
<span className={isExporting ? "opacity-0" : ""}>
<FolderArchive className="h-3 w-3 opacity-60" />
</span>
<span className={isExporting ? "opacity-0" : ""}>Export</span>
{isExporting && <Spinner size="sm" className="absolute" />}
</Button>
</div>
<Button
type="button"
variant="secondary"
size="sm"
disabled={isExporting}
onClick={handleExportKB}
className="relative w-fit shrink-0"
>
<span className={isExporting ? "opacity-0" : ""}>
<FolderArchive className="h-3 w-3 opacity-60" />
</span>
<span className={isExporting ? "opacity-0" : ""}>Export</span>
{isExporting && <Spinner size="sm" className="absolute" />}
</Button>
</div>
</div>
);
}