# TASK 2: Add Immutable Fleet S3 Snapshot And Date Manifests

## Implementation Status

**Completed: 2026-07-22**

- Added paginated S3 listing, object metadata lookup, and ETag/version-conditional server-side copy primitives to the Warehouse S3 client.
- Added exact-case Fleet source discovery, strict partition/date/size validation, bounded immutable snapshot copying, deterministic collision-safe keys, inventories, per-date mandatory Redshift manifests, and a manifest index.
- Added validated complete-snapshot retries and exact batch-prefix cleanup/rebuild for partial work without downloading source payloads.
- Added the Warehouse storage abstraction, Fleet integration adapter, and dependency injection registrations; corrected canonical source-folder handling to preserve the frozen exact-case contract.
- Added focused fake-S3 coverage for pagination, filtering, invalid input, source mutation, bounded concurrency/cancellation, deterministic manifests, no source downloads, idempotent retry, and partial recovery.
- Verification: `Kds.Ai.Data.Tests` passed 123 runnable tests with 8 opt-in live tests skipped; `Kds.Ai.Integrations.Tests` passed 159 runnable tests with 11 opt-in live tests skipped. The repository-wide warnings-as-errors build passes with NuGet audit disabled.
- The standard repository-wide warnings-as-errors build remains blocked outside TASK 2 by existing NU1903 findings for `Microsoft.OpenApi` 2.0.0 in `Kds.Ai.ApiService` and its test project.

## Summary

Create a batch-owned immutable source workspace by listing eligible customer objects, conditionally copying them server-side, and writing an inventory plus one Redshift COPY manifest per path business date.

## Dependencies

- [TASK 0](./TASK0.md)

## Current Codebase Findings

- `AmazonS3ObjectStoreClient` already owns AWS SDK S3 access and paginated delete behavior.
- `IS3ObjectStoreClient` does not currently expose listing metadata, conditional copy, or object existence/listing for staged publication.
- `S3EtlStorage` is appropriate infrastructure precedent, but its existing raw/load/curated contracts should not be overloaded with Fleet source semantics.
- a Redshift manifest contains object URLs but not a durable copy of object contents.
- JSON `COPY` into one `SUPER` column does not expose each row's source key.

## Files To Create

Recommended locations:

- `backend/Kds.Ai.Warehouse/Abstractions/IFleetS3ImportStorage.cs`
- `backend/Kds.Ai.Warehouse/Models/FleetS3/` for storage-neutral requests/results
- `backend/Kds.Ai.Warehouse/Services/Storage/FleetS3ImportStorage.cs`
- `backend/Kds.Ai.Warehouse/Services/Storage/FleetS3ImportKeyBuilder.cs`
- `backend/Kds.Ai.Warehouse/Configuration/FleetS3ImportStorageOptions.cs`
- focused tests beside `S3EtlStorageTests`.

## Files To Modify

- `backend/Kds.Ai.Warehouse/Services/Storage/IS3ObjectStoreClient.cs`
- `backend/Kds.Ai.Warehouse/Services/Storage/AmazonS3ObjectStoreClient.cs`
- `backend/Kds.Ai.Warehouse/DependencyInjectionExtensions.cs`
- existing fake S3 client test infrastructure.

Add only generally useful low-level object-store primitives to `IS3ObjectStoreClient`. Keep Fleet path filtering and manifests in `FleetS3ImportStorage`.

## Eligible Source Objects

List with paginated `ListObjectsV2` under exactly:

```text
fleet/{canonicalSourceFolder}/
```

An eligible key must match the entire relative shape:

```text
year=YYYY/month=MM/day=DD/{fileName}.json
```

Rules:

- no extra directory levels;
- `.json` comparison is ordinal-ignore-case;
- year/month/day must form a valid `DateOnly`;
- zero-byte objects are invalid;
- objects at or above 4 MiB are invalid;
- directory markers and non-JSON files are ignored and counted in diagnostics;
- malformed JSON-looking partition paths fail the snapshot instead of being silently ignored;
- zero eligible objects fails before any curated operation.

## Immutable Snapshot Algorithm

1. Complete the paginated source listing and record key, ETag, optional version ID, size, last-modified time, and parsed path date.
2. Reject duplicate keys, invalid date paths, invalid sizes, and empty inventory.
3. Sort deterministically by path date then source key.
4. Copy every eligible object to a batch-owned workspace using server-side `CopyObject`, with the listed ETag/version as a source precondition.
5. Use bounded concurrency from options; cancellation or one failed copy fails the snapshot.
6. If a source object changed after listing, its conditional copy fails the entire batch. Never silently copy the newer bytes.
7. Write a compact inventory JSON containing source/snapshot identity and checksums/ETags.
8. Write one Redshift manifest per path business date, referencing only batch-owned snapshot objects and setting every entry `mandatory: true`.
9. Write a manifest-index JSON containing inventory URI, date-to-manifest mappings, counts, bytes, and min/max dates.
10. Return only the bounded snapshot summary and index URI to the orchestrator.

No source payload bytes pass through application memory. Inventory and manifest JSON may be generated in memory because they contain metadata only.

## Workspace Layout

Use a deterministic platform-owned layout outside `curated/`, conceptually:

```text
load/company_id={customerId}/tenant_id={tenantId}/entity=fleet_s3_import/
  batch_id={batchId}/source_id={sourceFolder}/
    objects/dt={yyyy-MM-dd}/{stableObjectName}.json
    manifests/dt={yyyy-MM-dd}.manifest.json
    inventory.json
    manifest-index.json
```

`stableObjectName` must prevent collisions without leaking unsafe key text; a hash of the full source key plus sanitized filename is acceptable.

The storage service, not job code, owns this layout.
This key builder owns only snapshot, manifest, inventory, and staging workspace keys. Task 4 owns the already-frozen active Athena key contract.

## Manifest Contract

Each manifest uses the Redshift mandatory-entry shape:

```json
{
  "entries": [
    { "url": "s3://bucket/snapshot-key.json", "mandatory": true }
  ]
}
```

Do not include source objects directly once snapshotting is complete.

## Idempotency And Retry

- Creating an already complete snapshot for the same batch returns the existing validated manifest index.
- Partial work from a prior attempt is deleted and rebuilt, or resumed only after validating every expected snapshot object and manifest. Do not trust object presence alone.
- A new import batch always gets a new workspace.
- Snapshot data is retained through post-commit publication retry.

## Testing

Add fake-S3 tests for:

- pagination over more than one page;
- exact path filtering and valid date parsing;
- case-sensitive source-folder prefix behavior;
- empty, malformed, zero-byte, and oversized inputs;
- conditional copy with listed ETag/version;
- source mutation during copy;
- bounded copy concurrency/cancellation;
- deterministic ordering and collision-safe snapshot keys;
- one manifest per populated date with mandatory entries;
- no payload download calls;
- idempotent complete retry and partial-work recovery;
- workspace deletion limited to the exact batch prefix.

## Acceptance Criteria

- the customer may change the source folder after snapshot completion without changing the current import;
- every loaded object is a batch-owned immutable snapshot object;
- path business date remains associated with each manifest;
- 36,500+ object metadata entries are supported without writing them into batch details;
- no source object is downloaded by the application.