Spensara Spensara Knowledge Base & Documentation Repository
System Overview Brink & Borg Master Index (722 Docs) Spensara Platform Project Tracker

Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)
Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Categories

Branches (328)

Branch-specific documentation across extract-raw-xml, feature-fleet-import, kds_cdc_improvments, life-cycle, and tenat-admin.
branches/feature-fleet-import/implementation-specs/specs/ImportFromS3ToRedshift/TASK6.md 5,832 bytes · Updated Oct 23, 2026 10:44 AM EDT
Raw File

TASK 6: Add Admin Endpoints, TickerQ Functions, And Module Wiring

Summary

Expose the Fleet import through an admin-only asynchronous API, reserve the source before enqueueing, and run/resume it through dedicated TickerQ functions.

Dependencies

Files To Create

Recommended locations:

  • backend/Kds.Ai.ApiService/Endpoints/FleetS3ImportEndpoints.cs
  • backend/Kds.Ai.ApiService/Jobs/FleetS3ImportJobs.cs
  • backend/Kds.Ai.ApiService/Services/FleetS3ImportRequestService.cs
  • API request/result/problem models if not nested in the endpoint;
  • backend/Kds.Ai.Integrations/FleetS3/DependencyInjectionExtensions.cs or a focused module method in current integrations DI.

Files To Modify

  • backend/Kds.Ai.ApiService/Program.cs
  • backend/Kds.Ai.ApiService/DependencyInjectionExtensions.cs
  • backend/Kds.Ai.Integrations/DependencyInjectionExtensions.cs only for shared registration entry points;
  • test AppFactory/fake ticker registration where required.

Import Endpoint

text
POST /etl/fleet-s3/import

Request:

json
{
  "customerId": "00000000-0000-0000-0000-000000000000",
  "tenantId": "00000000-0000-0000-0000-000000000000",
  "sourceFolder": "front-counter",
  "overrideExisting": true,
  "requestedBy": null
}

overrideExisting defaults to true when omitted. requestedBy defaults to the authenticated email and must be bounded/sanitized before diagnostics persistence.

Endpoint/service processing order:

  1. validate source folder syntax and normalize it;
  2. load customer; return 404 if absent;
  3. compare supplied tenant UUID with customer.TenantId; return validation problem on mismatch;
  4. load warehouse mapping and require IsActive plus ProvisioningState.Succeeded;
  5. reserve a Task 1 Fleet batch with initial details;
  6. enqueue RunManualFleetS3Import carrying only batch ID and immutable normalized request context;
  7. if enqueue fails, mark the reserved batch failed so the source scope is released;
  8. return 202 Accepted with both TickerQ job ID and import batch ID.

Do not list S3 or open Redshift from the HTTP request.

Recommended status outcomes:

  • 202: reserved and enqueued;
  • 400: malformed folder/request or customer/tenant mismatch;
  • 401/403: existing auth middleware/policy;
  • 404: customer missing;
  • 409: warehouse inactive/not provisioned or source import already running;
  • 500: reservation/enqueue infrastructure failure, with bounded server logs.

Publication Retry Endpoint

text
POST /etl/fleet-s3/import/{batchId}/retry-publication

This admin-only route:

  • loads and validates a Fleet batch owned by the supplied customer/tenant context or derives immutable identity from the batch;
  • accepts only failed publication state with redshiftCommitted = true;
  • atomically reacquires the source reservation;
  • enqueues the same import function in publication-only resume mode;
  • releases/fails the reservation if enqueue fails;
  • returns 202 with job and batch IDs.

It must not create a new import batch or permit retry of pre-commit validation failures as publication-only work.

TickerQ Functions

Add functions equivalent to:

  • RunManualFleetS3Import -> call orchestrator for the reserved batch;
  • CleanupFleetS3ImportWorkspace -> execute idempotent retained-workspace cleanup.

Rules:

  • do not use function-wide SkipIfAlreadyRunning, because different tenants/folders may safely run independently;
  • Task 1's durable source reservation is the concurrency authority;
  • duplicate delivery of the same batch is handled idempotently by Task 5;
  • every delivery must atomically acquire and renew Task 1's per-batch execution lease; a concurrent holder causes a no-op/conflict outcome;
  • Ticker input contains no credentials, bucket override, SQL, or target table names;
  • interrupted deliveries may resume the same still-Running batch after lease expiry;
  • handled failures are durably terminal for that delivery: later automatic deliveries inspect batch state and perform no work; pre-commit recovery uses a new import, while publication recovery uses the explicit retry endpoint.

DI And Configuration

Add one AddFleetS3ImportModule() registration path that:

  • binds/validates Fleet options;
  • registers Fleet storage, Redshift executor, publisher, orchestrator, and cleanup services with appropriate lifetimes;
  • reuses existing IAmazonS3, tenant warehouse, Redshift auth, and ingestion repository registrations;
  • does not replace the Brink ICuratedExportCatalog registration;
  • is invoked explicitly from Program.cs beside existing ETL modules.

Map MapFleetS3ImportEndpoints() beside MapBrinkPosEtlEndpoints().

Testing

Add API tests modeled on BrinkPosManualEtlEndpointTests for:

  • admin success returns 202, job ID, and batch ID;
  • omitted override serializes as true in ticker input/details;
  • requested-by email fallback;
  • unauthenticated and non-admin rejection;
  • customer not found;
  • tenant mismatch;
  • unsafe source folder variants;
  • inactive and non-succeeded warehouse mapping;
  • duplicate source reservation conflict;
  • different folder allowed;
  • enqueue failure marks batch failed/releases reservation;
  • publication retry eligibility and conflict rules;
  • ticker function calls orchestrator with the exact reserved batch ID.
  • duplicate delivery cannot obtain two live execution leases.

Acceptance Criteria

  • identity and destination are resolved before enqueue;
  • two API nodes cannot enqueue two active imports for the same tenant/source reservation;
  • the HTTP process performs no bulk data work;
  • callers cannot control storage/database/SQL targets;
  • publication retry is explicit and cannot overwrite a newer batch;
  • endpoint OpenAPI metadata documents response/problem shapes.