# TASK 9: Validate Operational Behavior And Delivery Readiness

## Summary

Validate that the completed KDS CDC pipeline behaves correctly under operational conditions before delivery. This task covers scenario verification for low volume, high volume, redelivery, duplicates, out-of-order input, idempotent retries, checkpoint advancement, quarantine behavior, and delivery-readiness checks across the full stage chain. It also confirms that documented non-goals were not accidentally introduced during implementation. This task does not define new pipeline behavior; it verifies the behavior established by Tasks 2 through 8.

## Current Codebase Findings

- `docs/features/ETL/kds-cdc/implementation-breakdown.md` defines Task 9 as validation/readiness work, not another functional stage.
- `docs/features/ETL/kds-cdc/index.md` already defines the required operational scenarios and expected outcomes:
  - low-volume tenant batch closure,
  - high-volume tenant bounded micro-batching,
  - duplicate CDC message handling,
  - out-of-order or missing sequence failure,
  - raw-write retry behavior,
  - durable-job-creation retry behavior,
  - idempotent `COPY`, curated merge, and curated export retry behavior,
  - record-level quarantine with continued processing for good records,
  - non-goals such as not reusing `SyntheticKds`, not relying on SQS as replay, and not creating one batch per CDC record.
- `backend/Kds.Ai.ApiService/Program.cs` disables background services in integration tests via `options.DisableBackgroundServices()` when `builder.Environment.IsIntegrationTests()` is true. This means worker-driven operational verification cannot rely on automatic hosted-service startup inside current integration-test hosts.
- The same file only auto-registers startup hosted services outside integration tests. This is an important readiness constraint for KDS CDC worker validation: tests must explicitly drive worker behavior or run in a dedicated host shape.
- `backend/tests/Kds.Ai.ApiService.Tests/.../Infrastructure/AppFactory.cs` sets the `IntegrationTests` environment and injects fake AWS/S3 settings. This is the current API-service integration-test seam for deterministic host-based verification.
- Existing live ETL readiness patterns are already present for Brink:
  - `backend/tests/Kds.Ai.Integrations.Tests/BrinkPos/BrinkPosExtractRawLiveTests.cs`
  - `backend/tests/Kds.Ai.Integrations.Tests/BrinkPos/BrinkPosDownstreamLiveTests.cs`

  These tests show the repo preference for:
  - opt-in live verification behind `KDSAI_RUN_LIVE_TESTS=1`,
  - per-test ephemeral infrastructure via Testcontainers/PostgreSQL,
  - explicit host creation with in-memory configuration,
  - stage-by-stage verification of raw, load, staging, and curated effects,
  - durable assertions on `DetailsJson`, S3 paths, batch current-state semantics, and warehouse row outcomes.
- `backend/tests/Kds.Ai.Integrations.Tests/Etl/StageExecutionSupportTests.cs` already verifies checkpoint advancement and failed-package behavior at unit-test level. This is the current shared seam for asserting stage completion/failure behavior.
- `backend/tests/Kds.Ai.Integrations.Tests/Etl/CuratedExportExecutorTests.cs` already verifies curated export success, failure wrapping, and partial-cleanup behavior. This is a direct structural reference for Task 9 export-failure validation.
- `backend/tests/Kds.Ai.Data.Tests/Warehouse/TenantWarehouseBootstrapperTests.cs` currently asserts creation of KDS staging tables and kitchen curated tables, but does not assert a finished KDS CDC operational path by itself.
- Current warehouse bootstrap ties KDS staging resource selection to Brink POS service enablement, so readiness tests must prove KDS CDC tenants get required KDS resources through KDS CDC enablement.
- `backend/Kds.Ai.AppHost/AppHost.cs` exposes `/health` checks for major runtime services. Current readiness shape is infrastructure/service health-oriented, not CDC-flow-specific.
- Across current backend tests, there is no completed KDS CDC operational validation suite yet, and existing live ETL coverage is Brink-oriented and stage-invocation-oriented rather than SQS-worker-driven.

Current repo constraints this task must respect:

1. KDS CDC readiness validation will need a layered test strategy, because current integration tests suppress background workers while current live ETL tests invoke stages/jobs explicitly.
2. Operational validation must prove both stage correctness and worker-entry correctness; existing Brink live tests are useful references but do not cover SQS intake, CDC ordering, or tenant-resolution quarantine.
3. Readiness verification must stay additive. It must not loosen existing Brink, synthetic KDS, or shared ETL behavior in order to make KDS CDC tests easier.

## Dependencies

- `../../docs/features/ETL/kds-cdc/index.md`
- `../../docs/features/ETL/kds-cdc/implementation-breakdown.md`
- `./TASK2.md`
- `./TASK3.md`
- `./TASK4.md`
- `./TASK5.md`
- `./TASK6.md`
- `./TASK7.md`
- `./TASK8.md`

## Source References

- `../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist`
- `../../docs/features/ETL/kds-cdc/index.md#selected-behavior-summary`
- `../../docs/features/ETL/kds-cdc/index.md#checkpointing-and-replay`
- `../../docs/features/ETL/kds-cdc/index.md#quarantine-and-failed-packages`
- `../../docs/features/ETL/kds-cdc/index.md#supported-corner-cases`
- `../../docs/features/ETL/kds-cdc/index.md#non-goals`

## Scope From Breakdown

- verify low-volume, high-volume, redelivery, duplicate, and out-of-order scenarios
- verify idempotent retries across raw, load, curated, and export failure points
- confirm batch boundaries, checkpoint advancement, and quarantine behavior match spec
- confirm excluded behavior is not introduced during implementation

## Files to Create

1. KDS CDC operational validation test suite under `backend/tests/Kds.Ai.Integrations.Tests/KdsCdc/`.
   - Suggested files:
     - `KdsCdcWorkerScenarioTests.cs`
     - `KdsCdcPipelineReadinessTests.cs`
     - `KdsCdcReplayAndRetryTests.cs`
     - `KdsCdcQuarantineBehaviorTests.cs`
     - `KdsCdcOperationalLiveTests.cs`

   Responsibilities:
   - verify end-to-end behavior from worker intake through export where practical,
   - verify stage-to-stage durable retries where full worker flow is unnecessary,
   - keep live tests opt-in and clearly separated from deterministic unit/integration coverage.

2. Optional KDS CDC test infrastructure under `backend/tests/Kds.Ai.Integrations.Tests/KdsCdc/Infrastructure/` or shared backend test infrastructure.
   - Suggested helpers:
     - fake queue client or fake received-message source,
     - deterministic sequence/coordinate fixture builders,
     - test host builder for KDS CDC services,
     - reusable assertions for `IngestionBatch` lineage, checkpoint payloads, quarantine records, failed packages, and S3 path layout.

   Guidance:
   - prefer reuse of existing `AppFactory`, warehouse/data fixtures, and fake storage patterns before introducing new helpers.

3. Optional operational/readiness checklist artifact under `specs/KdsCdcIngestion/` or nearby docs only if a human runbook is needed in addition to automated tests.
   - Examples:
     - delivery-readiness checklist,
     - live-test prerequisites,
     - known limitations or temporary blockers.

   Guidance:
   - only add this if automated tests alone do not communicate launch readiness clearly.

## Files to Modify

- `backend/tests/Kds.Ai.Integrations.Tests/` project files and folders to add KDS CDC scenario coverage.
- `backend/tests/Kds.Ai.ApiService.Tests/.../Infrastructure/AppFactory.cs` or related integration-test infrastructure if KDS CDC host configuration needs additive options, seeded mappings, or fake service registration.
- KDS CDC test seams introduced in Tasks 2 through 8 under `backend/Kds.Ai.ApiService/Jobs/KdsCdc/` and `backend/Kds.Ai.Integrations/KdsCdc/` only where additive hooks are required for deterministic validation.

  Guidance: prefer injectable abstractions and explicit test doubles over test-only production branching.
- Shared test infrastructure only if current fixtures cannot safely express:
  - fake SQS redelivery,
  - duplicate coordinate streams,
  - out-of-order sequence injection,
  - raw/load/curated/export failure injection,
  - replay-entry invocation.
- Potential readiness surfacing points if missing and genuinely needed:
  - `backend/Kds.Ai.AppHost/AppHost.cs`
  - `backend/Kds.Ai.ApiService/Program.cs`

  Guidance: only add health/readiness surfacing if Task 9 discovers an actual observability gap. Do not add decorative health endpoints that do not reflect CDC readiness.

## Required Behavior

### Validation Coverage Matrix

- Task 9 must verify all documented operational scenarios at the right test level.
- At minimum, coverage must include:
  - low-volume tenant batch closes by age limit,
  - high-volume tenant splits into multiple bounded ordered micro-batches,
  - duplicate CDC messages are deduped by source coordinates plus source table and primary key,
  - out-of-order or missing sequence fails safely before Redshift load,
  - malformed or unmapped records are quarantined while good records continue when safe,
  - raw-write failure leaves SQS messages unacknowledged,
  - durable-job-creation failure after raw write leaves SQS messages unacknowledged and preserves duplicate-safe replay semantics,
  - Redshift `COPY` retry reuses same load artifact idempotently,
  - curated merge retry resumes from durable downstream state without re-reading SQS,
  - curated export retry resumes from curated state and cleans incomplete export artifacts when possible,
  - checkpoint advancement occurs only after documented durable success points,
  - replay paths use raw/load/curated durable state rather than SQS retention.
  - mock per-stream ordering is not mistaken for cross-store global ordering.
  - production contract validation requires real source metadata while mock validation accepts the documented mock sequence surrogate.
  - partial SQS acknowledgement deletes only records/slices that reached a durable success or explicit quarantine boundary.
  - oversized or unreadable SQS payloads fail safely.

### Non-Goal Protection

- Task 9 must explicitly verify that implementation did not introduce forbidden behavior:
  - no identification of production CDC as `SyntheticKds`,
  - no use of simulator direct-write path as production CDC ingestion path,
  - no one-ingestion-batch-per-record behavior,
  - no dependency on SQS as long-term replay after acknowledgement,
  - no CDC landing tables added as hidden first-version dependency,
  - no physical deletion of curated rows in normal KDS CDC flow.

### Batch Boundary And Lineage Verification

- Validation must prove actual runtime batch boundaries match the spec:
  - one ETL batch represents a bounded ordered tenant/entity slice,
  - mixed poll windows split correctly by company, tenant, and entity,
  - each framework stage owns its own `IngestionBatch`,
  - `UpstreamBatchId` chain remains intact through all five stages,
  - rerun/current semantics remain correct under retries.

### Checkpoint And Replay Verification

- Validation must prove KDS CDC checkpoints are not wall-clock-only regressions.
- Tests or live verification must confirm persisted checkpoint state includes or resolves back to:
  - source-coordinate facts,
  - SQS ordering facts,
  - last successful stage batch ids,
  - replay pointers to raw/load/curated durable state.
- Replay verification must cover:
  - raw batch replay,
  - load artifact retry,
  - curated retry,
  - coordinate-bounded recovery when implemented.

### Quarantine And Failure Diagnostics Verification

- Validation must confirm record-level quarantine diagnostics and batch-level failed-package diagnostics both contain the fields required by the feature doc.
- Validation must also confirm the boundary stays correct:
  - record-local failures quarantine only the bad records,
  - batch-level failures stop later stages for that tenant/entity slice,
  - unrelated good slices can still succeed.

### Layered Test Strategy

- Task 9 should not force every scenario into one giant end-to-end test.
- Preferred verification layers are:
  - unit tests for ordering, dedupe, checkpoint-payload, and diagnostic-shape logic,
  - integration tests for host wiring, worker orchestration, stage transitions, and persistence effects,
  - live tests for real S3/Redshift and, when available, real queue/infrastructure seams.
- Because current integration hosts disable background services, worker behavior should be validated by either:
  - explicitly invoking worker/message-pump seams in tests, or
  - using a dedicated host/test harness that intentionally enables only the KDS CDC worker under controlled conditions.

### Delivery Readiness Outcome

- Task 9 ends only when the codebase has a credible automated readiness story for KDS CDC.
- That readiness story must make clear:
  - which scenarios are covered by deterministic automated tests,
  - which scenarios are covered by opt-in live tests,
  - which scenarios remain blocked by external infrastructure and must be documented explicitly.
- Temporary blockers are acceptable only if they are called out clearly, mirrored in test skips/messages, and do not hide a missing implementation.

## Proposed Implementation Shape

### Recommended Test Layers

1. Deterministic integration tests under `backend/tests/Kds.Ai.Integrations.Tests/KdsCdc/`:
   - drive fake queue messages into worker or batch-building seams,
   - inject duplicate, redelivery, and out-of-order sequences,
   - assert batch creation, quarantine output, checkpoint advancement, and downstream stage calls.

2. Durable retry tests:
   - start from persisted raw batch state and verify normalize/load replay,
   - start from persisted load state and verify `COPY` retry,
   - start from persisted curated state and verify export retry,
   - assert no replay path depends on original SQS retention.

3. Live readiness tests, gated by `KDSAI_RUN_LIVE_TESTS=1`:
   - mirror existing Brink live-test style,
   - use ephemeral Postgres/Testcontainers plus configured external S3/Redshift,
   - add real KDS CDC readiness coverage only where infrastructure exists,
   - skip clearly when required infrastructure is unavailable.

### Suggested Scenario Matrix

```text
Worker intake and batching:
  - low volume -> age-close batch
  - high volume -> count/byte-bounded split
  - duplicate -> dedupe
  - out-of-order -> fail before load

Resolution and quarantine:
  - missing mapping -> quarantine only bad record
  - ambiguous mapping -> quarantine only bad record
  - bad + good same poll window -> good continues

Durability and retries:
  - raw write fails -> no ack
  - raw succeeds, durable metadata fails -> no ack, duplicate-safe retry
  - load COPY fails -> retry from load artifact
  - curated merge fails -> checkpoint stays behind
  - curated export fails -> retry from curated state, cleanup attempted

Lineage and replay:
  - per-stage batches chained by UpstreamBatchId
  - checkpoint facts source-based, not timestamp-only
  - replay from raw/load/curated works without SQS

Non-goals:
  - no SyntheticKds identity
  - no per-record batches
  - no SQS-as-replay-source
  - no hidden landing tables
```

### Readiness Evidence Direction

- Prefer assertions on durable artifacts and metadata over log-only validation.
- Good readiness evidence includes:
  - `IngestionBatch` rows and `UpstreamBatchId` chain,
  - checkpoint rows and checkpoint metadata payloads,
  - quarantine records and failed-package objects,
  - raw/load/curated S3 path layout,
  - tenant warehouse row outcomes,
  - cleanup of incomplete curated export prefixes,
  - absence of secret leakage in details/diagnostics.

## Testing

- Unit tests for:
  - duplicate detection and redelivery idempotence,
  - out-of-order detection,
  - checkpoint advancement gating,
  - quarantine and failed-package diagnostic shape,
  - non-goal guards such as source identity and batch-count expectations where expressible.
- Integration tests for:
  - worker/message-pump orchestration under deterministic fake queue input,
  - tenant/entity splitting and batch-boundary behavior,
  - good-record continuation when bad records quarantine,
  - replay and retry entry from raw/load/curated durable boundaries,
  - no-ack behavior on raw-write and durable-metadata failures,
  - no downstream stage execution after ordering failure,
  - stage lineage and `IsCurrent` / `ResetOfBatchId` behavior on reruns.
- Data/warehouse tests for:
  - checkpoint persistence shape,
  - failed-package persistence and cleanup behavior,
  - bootstrap/runtime readiness for required KDS staging and curated tables, including any customer-path gaps closed by earlier tasks.
- Live tests, gated by `KDSAI_RUN_LIVE_TESTS=1`, for:
  - real raw/load/curated/export durability where infra exists,
  - end-to-end replay-safe behavior across at least one realistic CDC tenant flow,
  - operational blockers documented as explicit skips when external queue/Debezium wiring is not yet available.
- Regression coverage must confirm Brink POS and synthetic KDS behavior remain unchanged.

## Acceptance Criteria

- Automated coverage exists for all documented KDS CDC operational scenarios at appropriate test layers.
- Low-volume, high-volume, duplicate, redelivery, and out-of-order behaviors are verified against the feature spec.
- Raw, load, curated, and export failure/retry paths are verified as idempotent and replay-safe.
- Batch boundaries, `UpstreamBatchId` lineage, checkpoint advancement, and quarantine behavior are proven to match the spec.
- Tests or readiness checks confirm production CDC is not identified as `SyntheticKds` and other non-goals were not introduced.
- Background-worker validation strategy is explicit and works with current integration-test constraints rather than depending on auto-started hosted services.
- Live readiness coverage is opt-in, clearly gated, and clearly documents any external-infrastructure blockers.
- Required KDS staging/curated schema readiness is verified for the paths KDS CDC uses.
- KDS CDC warehouse bootstrap/readiness is verified independently of Brink POS credential/service enablement.
- Existing Brink POS and synthetic KDS behavior remain unchanged.

## Notes

- Use `index.md#supported-corner-cases` and `index.md#non-goals` as the primary source of truth for Task 9 scenario coverage.
- Existing Brink live ETL tests are the strongest current repository reference for delivery-readiness style, but KDS CDC must add worker-entry, ordering, quarantine, and replay coverage that Brink does not need.
- Keep Task 9 verification additive and explicit. If some readiness scenarios cannot yet run live because external Debezium/SQS infrastructure is unavailable, document that as a real blocker instead of masking it behind broad passing tests.