Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)

Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Branches (328)

Branch-specific documentation across extract-raw-xml, feature-fleet-import, kds_cdc_improvments, life-cycle, and tenat-admin.

branches/feature-fleet-import/implementation-specs/specs/ImportFromS3ToRedshift/TASK7.md 7,551 bytes · Updated Oct 23, 2026 10:44 AM EDT

Raw File

TASK 7: Add End-To-End, Live, Performance, And Removal Validation

Summary

Close the Fleet temporary import with cross-component regression coverage, opt-in live Redshift/S3/Athena verification, expected-volume testing, and an operator removal runbook.

Dependencies

Test Locations

Use existing TUnit projects:

API/auth/reservation: backend/tests/Kds.Ai.ApiService.Tests/
operational repository and S3 storage: backend/tests/Kds.Ai.Data.Tests/
Fleet orchestration, SQL, export, and live Redshift: backend/tests/Kds.Ai.Integrations.Tests/FleetS3/

Follow existing KDSAI_RUN_LIVE_TESTS=1 gating and skip when required AWS/Redshift/Athena configuration is absent.

The live Athena harness belongs in backend/tests/Kds.Ai.Integrations.Tests/Kds.Ai.Integrations.Tests.csproj. Add a test-only AWSSDK.Athena dependency (or use an already approved shared AWS test client if one exists by implementation time), bind workgroup/database/table/query-results S3 settings from environment configuration, poll query execution to a bounded terminal state, and delete only disposable test output. No Athena SDK dependency is required by production code.

Deterministic Integration Scenarios

Cover at minimum:

sample object maps to one order and four items with exact documented values;
multi-day paginated source creates date manifests and imports every object;
omitted override replaces because default is true;
overrideExisting = false permits first import and rejects replacement unchanged;
override rerun reflects added, changed, and removed source objects;
duplicate transaction IDs fail before deletion;
malformed schema/type/date/terminal/length/decimal/ordinal data fails all-or-nothing;
empty eligible source never deletes existing data;
source mutation during conditional snapshot copy fails;
injected Redshift failure after delete rolls back both targets;
exact tenant/source isolation preserves Brink, KDS, another folder, and another tenant;
same-source concurrent request conflicts while independent sources proceed;
crash after Redshift commit reconciles by batch lineage;
crash after replacement-intent persistence but before commit safely restarts and preserves old-only affected dates;
publication failure retries without another Redshift delete/insert;
verbose UNLOAD manifest count mismatch blocks activation;
old affected dates are removed from active Athena output;
newer import supersedes an old publication retry;
publication retry expiry requires a repairing full import;
cleanup removes only retained batch workspaces.

Live Redshift/S3 Test

With isolated tenant/test prefixes:

upload sample and generated multi-date fixtures to a disposable Fleet folder;
run the real source snapshot service and verify source objects were not downloaded;
execute real Redshift SUPER COPY and transformation;
query both curated targets and compare every mapped column/count;
rerun replacement and verify exact source isolation;
force a pre-commit SQL failure and verify prior rows remain;
run real UNLOAD, staged-object activation, and removed-date cleanup;
always clean test source, snapshot, staging, active Fleet prefixes, and curated rows in teardown.

Do not point destructive tests at a customer folder or a shared source discriminator.

Live Athena Test

Use a configured disposable Athena workgroup/database/table and assert:

both entities are queryable from the stable Fleet path;
tenant_id, tenant_name, _etl_source_type, _etl_batch_id, business_date, and currency_code are present/correct;
Athena counts equal Redshift counts per entity/date/source;
replacement does not double counts;
removed dates disappear;
another source folder remains visible;
the deployed table/partition projection traverses the fixed source-scoped descendant layout.

If the deployed catalog cannot discover the approved key ordering, treat that as a release blocker and update the Glue table location/projection to scan the fixed entity/date root. Do not silently publish unqueryable Parquet and do not change the key contract at this late phase.

Expected-Volume Test

Use fake S3 metadata for at least 36,500 objects across 365 dates and verify:

listing pagination remains bounded;
inventory/manifest generation is deterministic;
one date manifest/COPY is planned per populated date, not one command per object;
details JSON remains bounded and references the external manifest index;
copy concurrency never exceeds configuration;
no full source payload collection/deserialization occurs.

Record representative source snapshot duration, Redshift COPY/validation duration, and publication duration in test/runbook output. Do not create a brittle wall-clock unit-test threshold; define an operational timeout based on measured live behavior.

Regression Verification

Run:

text

dotnet test --project backend/tests/Kds.Ai.Data.Tests/Kds.Ai.Data.Tests.csproj
dotnet test --project backend/tests/Kds.Ai.Integrations.Tests/Kds.Ai.Integrations.Tests.csproj
dotnet test --project backend/tests/Kds.Ai.ApiService.Tests/Kds.Ai.ApiService.Tests/Kds.Ai.ApiService.Tests.csproj
dotnet format Kds.Ai.slnx --include <modified C# files>
dotnet build Kds.Ai.slnx -p:TreatWarningsAsErrors=true

Run commands sequentially to avoid shared build-output locks. Execute live tests at the end with the required environment gate.

Regression assertions must prove:

RedshiftCopyCommandFactory remains Parquet-only;
Brink/KDS target and export catalogs are unchanged;
existing date-scoped ingestion batch lifecycle/current behavior remains green;
no migration appears under tenant curated Redshift bootstrap SQL.

Documentation And Runbook

Update relevant ETL/operator documentation with:

endpoint and publication-retry usage;
server configuration and required IAM permissions;
source upload/snapshot consistency rule;
batch phase meanings and failure categories;
how to identify imported Redshift/Athena rows;
how to inspect manifest index without exposing payloads;
safe retry decision tree;
retained-workspace cleanup behavior;
expected 36,500+ object/year operating envelope;
promotion trigger to standard ETL.

Document removal commands/procedure using exact boundaries only: