# TASK 7: Add End-To-End, Live, Performance, And Removal Validation

## Summary

Close the Fleet temporary import with cross-component regression coverage, opt-in live Redshift/S3/Athena verification, expected-volume testing, and an operator removal runbook.

## Dependencies

- [TASK 2](./TASK2.md)
- [TASK 3](./TASK3.md)
- [TASK 4](./TASK4.md)
- [TASK 5](./TASK5.md)
- [TASK 6](./TASK6.md)

## Test Locations

Use existing TUnit projects:

- API/auth/reservation: `backend/tests/Kds.Ai.ApiService.Tests/`
- operational repository and S3 storage: `backend/tests/Kds.Ai.Data.Tests/`
- Fleet orchestration, SQL, export, and live Redshift: `backend/tests/Kds.Ai.Integrations.Tests/FleetS3/`

Follow existing `KDSAI_RUN_LIVE_TESTS=1` gating and skip when required AWS/Redshift/Athena configuration is absent.

The live Athena harness belongs in `backend/tests/Kds.Ai.Integrations.Tests/Kds.Ai.Integrations.Tests.csproj`. Add a test-only `AWSSDK.Athena` dependency (or use an already approved shared AWS test client if one exists by implementation time), bind workgroup/database/table/query-results S3 settings from environment configuration, poll query execution to a bounded terminal state, and delete only disposable test output. No Athena SDK dependency is required by production code.

## Deterministic Integration Scenarios

Cover at minimum:

1. sample object maps to one order and four items with exact documented values;
2. multi-day paginated source creates date manifests and imports every object;
3. omitted override replaces because default is true;
4. `overrideExisting = false` permits first import and rejects replacement unchanged;
5. override rerun reflects added, changed, and removed source objects;
6. duplicate transaction IDs fail before deletion;
7. malformed schema/type/date/terminal/length/decimal/ordinal data fails all-or-nothing;
8. empty eligible source never deletes existing data;
9. source mutation during conditional snapshot copy fails;
10. injected Redshift failure after delete rolls back both targets;
11. exact tenant/source isolation preserves Brink, KDS, another folder, and another tenant;
12. same-source concurrent request conflicts while independent sources proceed;
13. crash after Redshift commit reconciles by batch lineage;
14. crash after replacement-intent persistence but before commit safely restarts and preserves old-only affected dates;
15. publication failure retries without another Redshift delete/insert;
16. verbose UNLOAD manifest count mismatch blocks activation;
17. old affected dates are removed from active Athena output;
18. newer import supersedes an old publication retry;
19. publication retry expiry requires a repairing full import;
20. cleanup removes only retained batch workspaces.

## Live Redshift/S3 Test

With isolated tenant/test prefixes:

- upload sample and generated multi-date fixtures to a disposable Fleet folder;
- run the real source snapshot service and verify source objects were not downloaded;
- execute real Redshift SUPER COPY and transformation;
- query both curated targets and compare every mapped column/count;
- rerun replacement and verify exact source isolation;
- force a pre-commit SQL failure and verify prior rows remain;
- run real UNLOAD, staged-object activation, and removed-date cleanup;
- always clean test source, snapshot, staging, active Fleet prefixes, and curated rows in teardown.

Do not point destructive tests at a customer folder or a shared source discriminator.

## Live Athena Test

Use a configured disposable Athena workgroup/database/table and assert:

- both entities are queryable from the stable Fleet path;
- `tenant_id`, `tenant_name`, `_etl_source_type`, `_etl_batch_id`, `business_date`, and `currency_code` are present/correct;
- Athena counts equal Redshift counts per entity/date/source;
- replacement does not double counts;
- removed dates disappear;
- another source folder remains visible;
- the deployed table/partition projection traverses the fixed source-scoped descendant layout.

If the deployed catalog cannot discover the approved key ordering, treat that as a release blocker and update the Glue table location/projection to scan the fixed entity/date root. Do not silently publish unqueryable Parquet and do not change the key contract at this late phase.

## Expected-Volume Test

Use fake S3 metadata for at least 36,500 objects across 365 dates and verify:

- listing pagination remains bounded;
- inventory/manifest generation is deterministic;
- one date manifest/COPY is planned per populated date, not one command per object;
- details JSON remains bounded and references the external manifest index;
- copy concurrency never exceeds configuration;
- no full source payload collection/deserialization occurs.

Record representative source snapshot duration, Redshift COPY/validation duration, and publication duration in test/runbook output. Do not create a brittle wall-clock unit-test threshold; define an operational timeout based on measured live behavior.

## Regression Verification

Run:

```text
dotnet test --project backend/tests/Kds.Ai.Data.Tests/Kds.Ai.Data.Tests.csproj
dotnet test --project backend/tests/Kds.Ai.Integrations.Tests/Kds.Ai.Integrations.Tests.csproj
dotnet test --project backend/tests/Kds.Ai.ApiService.Tests/Kds.Ai.ApiService.Tests/Kds.Ai.ApiService.Tests.csproj
dotnet format Kds.Ai.slnx --include <modified C# files>
dotnet build Kds.Ai.slnx -p:TreatWarningsAsErrors=true
```

Run commands sequentially to avoid shared build-output locks. Execute live tests at the end with the required environment gate.

Regression assertions must prove:

- `RedshiftCopyCommandFactory` remains Parquet-only;
- Brink/KDS target and export catalogs are unchanged;
- existing date-scoped ingestion batch lifecycle/current behavior remains green;
- no migration appears under tenant curated Redshift bootstrap SQL.

## Documentation And Runbook

Update relevant ETL/operator documentation with:

- endpoint and publication-retry usage;
- server configuration and required IAM permissions;
- source upload/snapshot consistency rule;
- batch phase meanings and failure categories;
- how to identify imported Redshift/Athena rows;
- how to inspect manifest index without exposing payloads;
- safe retry decision tree;
- retained-workspace cleanup behavior;
- expected 36,500+ object/year operating envelope;
- promotion trigger to standard ETL.

Document removal commands/procedure using exact boundaries only:

```text
tenant_name = {tenantId}
AND _etl_source_type = fleet_s3:{sourceFolder}
```

Remove items before orders, remove only the exact active Athena source prefixes, and retain operational audit history.

## Release Checklist

- [ ] all deterministic scenarios pass
- [ ] live S3 snapshot and Redshift import pass
- [ ] live Athena parity/replacement pass
- [ ] expected-volume test passes without unbounded details/payload memory
- [ ] IAM least-privilege permissions are verified
- [ ] no curated DDL is changed
- [ ] source-specific removal procedure is reviewed
- [ ] temporary-feature ownership and sunset trigger are documented

## Acceptance Criteria

- every acceptance criterion in `proposed-solution.md` has an automated or explicit live test;
- sample and multi-day imports are queryable in both Redshift and Athena;
- failure/retry/concurrency behavior is proven at system boundaries;
- the implementation supports expected yearly volume without per-file downloads/inserts;
- operators can safely retry, diagnose, clean up, and remove one Fleet source slice.