Spensara Spensara Knowledge Base & Documentation Repository
System Overview Brink & Borg Master Index (722 Docs) Spensara Platform Project Tracker

Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)
Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Categories

Branches (328)

Branch-specific documentation across extract-raw-xml, feature-fleet-import, kds_cdc_improvments, life-cycle, and tenat-admin.
branches/feature-fleet-import/implementation-specs/specs/ImportFromS3ToRedshift/TASK2.md 7,517 bytes · Updated Oct 23, 2026 10:44 AM EDT
Raw File

TASK 2: Add Immutable Fleet S3 Snapshot And Date Manifests

Implementation Status

Completed: 2026-07-22

  • Added paginated S3 listing, object metadata lookup, and ETag/version-conditional server-side copy primitives to the Warehouse S3 client.
  • Added exact-case Fleet source discovery, strict partition/date/size validation, bounded immutable snapshot copying, deterministic collision-safe keys, inventories, per-date mandatory Redshift manifests, and a manifest index.
  • Added validated complete-snapshot retries and exact batch-prefix cleanup/rebuild for partial work without downloading source payloads.
  • Added the Warehouse storage abstraction, Fleet integration adapter, and dependency injection registrations; corrected canonical source-folder handling to preserve the frozen exact-case contract.
  • Added focused fake-S3 coverage for pagination, filtering, invalid input, source mutation, bounded concurrency/cancellation, deterministic manifests, no source downloads, idempotent retry, and partial recovery.
  • Verification: Kds.Ai.Data.Tests passed 123 runnable tests with 8 opt-in live tests skipped; Kds.Ai.Integrations.Tests passed 159 runnable tests with 11 opt-in live tests skipped. The repository-wide warnings-as-errors build passes with NuGet audit disabled.
  • The standard repository-wide warnings-as-errors build remains blocked outside TASK 2 by existing NU1903 findings for Microsoft.OpenApi 2.0.0 in Kds.Ai.ApiService and its test project.

Summary

Create a batch-owned immutable source workspace by listing eligible customer objects, conditionally copying them server-side, and writing an inventory plus one Redshift COPY manifest per path business date.

Dependencies

Current Codebase Findings

  • AmazonS3ObjectStoreClient already owns AWS SDK S3 access and paginated delete behavior.
  • IS3ObjectStoreClient does not currently expose listing metadata, conditional copy, or object existence/listing for staged publication.
  • S3EtlStorage is appropriate infrastructure precedent, but its existing raw/load/curated contracts should not be overloaded with Fleet source semantics.
  • a Redshift manifest contains object URLs but not a durable copy of object contents.
  • JSON COPY into one SUPER column does not expose each row's source key.

Files To Create

Recommended locations:

  • backend/Kds.Ai.Warehouse/Abstractions/IFleetS3ImportStorage.cs
  • backend/Kds.Ai.Warehouse/Models/FleetS3/ for storage-neutral requests/results
  • backend/Kds.Ai.Warehouse/Services/Storage/FleetS3ImportStorage.cs
  • backend/Kds.Ai.Warehouse/Services/Storage/FleetS3ImportKeyBuilder.cs
  • backend/Kds.Ai.Warehouse/Configuration/FleetS3ImportStorageOptions.cs
  • focused tests beside S3EtlStorageTests.

Files To Modify

  • backend/Kds.Ai.Warehouse/Services/Storage/IS3ObjectStoreClient.cs
  • backend/Kds.Ai.Warehouse/Services/Storage/AmazonS3ObjectStoreClient.cs
  • backend/Kds.Ai.Warehouse/DependencyInjectionExtensions.cs
  • existing fake S3 client test infrastructure.

Add only generally useful low-level object-store primitives to IS3ObjectStoreClient. Keep Fleet path filtering and manifests in FleetS3ImportStorage.

Eligible Source Objects

List with paginated ListObjectsV2 under exactly:

text
fleet/{canonicalSourceFolder}/

An eligible key must match the entire relative shape:

text
year=YYYY/month=MM/day=DD/{fileName}.json

Rules:

  • no extra directory levels;
  • .json comparison is ordinal-ignore-case;
  • year/month/day must form a valid DateOnly;
  • zero-byte objects are invalid;
  • objects at or above 4 MiB are invalid;
  • directory markers and non-JSON files are ignored and counted in diagnostics;
  • malformed JSON-looking partition paths fail the snapshot instead of being silently ignored;
  • zero eligible objects fails before any curated operation.

Immutable Snapshot Algorithm

  1. Complete the paginated source listing and record key, ETag, optional version ID, size, last-modified time, and parsed path date.
  2. Reject duplicate keys, invalid date paths, invalid sizes, and empty inventory.
  3. Sort deterministically by path date then source key.
  4. Copy every eligible object to a batch-owned workspace using server-side CopyObject, with the listed ETag/version as a source precondition.
  5. Use bounded concurrency from options; cancellation or one failed copy fails the snapshot.
  6. If a source object changed after listing, its conditional copy fails the entire batch. Never silently copy the newer bytes.
  7. Write a compact inventory JSON containing source/snapshot identity and checksums/ETags.
  8. Write one Redshift manifest per path business date, referencing only batch-owned snapshot objects and setting every entry mandatory: true.
  9. Write a manifest-index JSON containing inventory URI, date-to-manifest mappings, counts, bytes, and min/max dates.
  10. Return only the bounded snapshot summary and index URI to the orchestrator.

No source payload bytes pass through application memory. Inventory and manifest JSON may be generated in memory because they contain metadata only.

Workspace Layout

Use a deterministic platform-owned layout outside curated/, conceptually:

text
load/company_id={customerId}/tenant_id={tenantId}/entity=fleet_s3_import/
  batch_id={batchId}/source_id={sourceFolder}/
    objects/dt={yyyy-MM-dd}/{stableObjectName}.json
    manifests/dt={yyyy-MM-dd}.manifest.json
    inventory.json
    manifest-index.json

stableObjectName must prevent collisions without leaking unsafe key text; a hash of the full source key plus sanitized filename is acceptable.

The storage service, not job code, owns this layout. This key builder owns only snapshot, manifest, inventory, and staging workspace keys. Task 4 owns the already-frozen active Athena key contract.

Manifest Contract

Each manifest uses the Redshift mandatory-entry shape:

json
{
  "entries": [
    { "url": "s3://bucket/snapshot-key.json", "mandatory": true }
  ]
}

Do not include source objects directly once snapshotting is complete.

Idempotency And Retry

  • Creating an already complete snapshot for the same batch returns the existing validated manifest index.
  • Partial work from a prior attempt is deleted and rebuilt, or resumed only after validating every expected snapshot object and manifest. Do not trust object presence alone.
  • A new import batch always gets a new workspace.
  • Snapshot data is retained through post-commit publication retry.

Testing

Add fake-S3 tests for:

  • pagination over more than one page;
  • exact path filtering and valid date parsing;
  • case-sensitive source-folder prefix behavior;
  • empty, malformed, zero-byte, and oversized inputs;
  • conditional copy with listed ETag/version;
  • source mutation during copy;
  • bounded copy concurrency/cancellation;
  • deterministic ordering and collision-safe snapshot keys;
  • one manifest per populated date with mandatory entries;
  • no payload download calls;
  • idempotent complete retry and partial-work recovery;
  • workspace deletion limited to the exact batch prefix.

Acceptance Criteria

  • the customer may change the source folder after snapshot completion without changing the current import;
  • every loaded object is a batch-owned immutable snapshot object;
  • path business date remains associated with each manifest;
  • 36,500+ object metadata entries are supported without writing them into batch details;
  • no source object is downloaded by the application.