TASK 9: Validate Operational Behavior And Delivery Readiness

Summary

Validate that the completed KDS CDC pipeline behaves correctly under operational conditions before delivery. This task covers scenario verification for low volume, high volume, redelivery, duplicates, out-of-order input, idempotent retries, checkpoint advancement, quarantine behavior, and delivery-readiness checks across the full stage chain. It also confirms that documented non-goals were not accidentally introduced during implementation. This task does not define new pipeline behavior; it verifies the behavior established by Tasks 2 through 8.

Current Codebase Findings

• docs/features/ETL/kds-cdc/implementation-breakdown.md defines Task 9 as validation/readiness work, not another functional stage.
• docs/features/ETL/kds-cdc/index.md already defines the required operational scenarios and expected outcomes:
• low-volume tenant batch closure,
• high-volume tenant bounded micro-batching,
• duplicate CDC message handling,
• out-of-order or missing sequence failure,
• raw-write retry behavior,
• durable-job-creation retry behavior,
• idempotent COPY, curated merge, and curated export retry behavior,
• record-level quarantine with continued processing for good records,
• non-goals such as not reusing SyntheticKds, not relying on SQS as replay, and not creating one batch per CDC record.
• backend/Kds.Ai.ApiService/Program.cs disables background services in integration tests via options.DisableBackgroundServices() when builder.Environment.IsIntegrationTests() is true. This means worker-driven operational verification cannot rely on automatic hosted-service startup inside current integration-test hosts.
• The same file only auto-registers startup hosted services outside integration tests. This is an important readiness constraint for KDS CDC worker validation: tests must explicitly drive worker behavior or run in a dedicated host shape.
• backend/tests/Kds.Ai.ApiService.Tests/.../Infrastructure/AppFactory.cs sets the IntegrationTests environment and injects fake AWS/S3 settings. This is the current API-service integration-test seam for deterministic host-based verification.
• Existing live ETL readiness patterns are already present for Brink:
• backend/tests/Kds.Ai.Integrations.Tests/BrinkPos/BrinkPosExtractRawLiveTests.cs
• backend/tests/Kds.Ai.Integrations.Tests/BrinkPos/BrinkPosDownstreamLiveTests.cs

These tests show the repo preference for:

• opt-in live verification behind KDSAI_RUN_LIVE_TESTS=1,
• per-test ephemeral infrastructure via Testcontainers/PostgreSQL,
• explicit host creation with in-memory configuration,
• stage-by-stage verification of raw, load, staging, and curated effects,
• durable assertions on DetailsJson, S3 paths, batch current-state semantics, and warehouse row outcomes.
• backend/tests/Kds.Ai.Integrations.Tests/Etl/StageExecutionSupportTests.cs already verifies checkpoint advancement and failed-package behavior at unit-test level. This is the current shared seam for asserting stage completion/failure behavior.
• backend/tests/Kds.Ai.Integrations.Tests/Etl/CuratedExportExecutorTests.cs already verifies curated export success, failure wrapping, and partial-cleanup behavior. This is a direct structural reference for Task 9 export-failure validation.
• backend/tests/Kds.Ai.Data.Tests/Warehouse/TenantWarehouseBootstrapperTests.cs currently asserts creation of KDS staging tables and kitchen curated tables, but does not assert a finished KDS CDC operational path by itself.
• Current warehouse bootstrap ties KDS staging resource selection to Brink POS service enablement, so readiness tests must prove KDS CDC tenants get required KDS resources through KDS CDC enablement.
• backend/Kds.Ai.AppHost/AppHost.cs exposes /health checks for major runtime services. Current readiness shape is infrastructure/service health-oriented, not CDC-flow-specific.
• Across current backend tests, there is no completed KDS CDC operational validation suite yet, and existing live ETL coverage is Brink-oriented and stage-invocation-oriented rather than SQS-worker-driven.

Current repo constraints this task must respect:

1. KDS CDC readiness validation will need a layered test strategy, because current integration tests suppress background workers while current live ETL tests invoke stages/jobs explicitly.
2. Operational validation must prove both stage correctness and worker-entry correctness; existing Brink live tests are useful references but do not cover SQS intake, CDC ordering, or tenant-resolution quarantine.
3. Readiness verification must stay additive. It must not loosen existing Brink, synthetic KDS, or shared ETL behavior in order to make KDS CDC tests easier.

Dependencies

• ../../docs/features/ETL/kds-cdc/index.md
• ../../docs/features/ETL/kds-cdc/implementation-breakdown.md
• ./TASK2.md
• ./TASK3.md
• ./TASK4.md
• ./TASK5.md
• ./TASK6.md
• ./TASK7.md
• ./TASK8.md

Source References

• ../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist
• ../../docs/features/ETL/kds-cdc/index.md#selected-behavior-summary
• ../../docs/features/ETL/kds-cdc/index.md#checkpointing-and-replay
• ../../docs/features/ETL/kds-cdc/index.md#quarantine-and-failed-packages
• ../../docs/features/ETL/kds-cdc/index.md#supported-corner-cases
• ../../docs/features/ETL/kds-cdc/index.md#non-goals

Scope From Breakdown

• verify low-volume, high-volume, redelivery, duplicate, and out-of-order scenarios
• verify idempotent retries across raw, load, curated, and export failure points
• confirm batch boundaries, checkpoint advancement, and quarantine behavior match spec
• confirm excluded behavior is not introduced during implementation

Files to Create

1. KDS CDC operational validation test suite under backend/tests/Kds.Ai.Integrations.Tests/KdsCdc/.
2. Suggested files:
3. KdsCdcWorkerScenarioTests.cs
4. KdsCdcPipelineReadinessTests.cs
5. KdsCdcReplayAndRetryTests.cs
6. KdsCdcQuarantineBehaviorTests.cs
7. KdsCdcOperationalLiveTests.cs

Responsibilities:

• verify end-to-end behavior from worker intake through export where practical,
• verify stage-to-stage durable retries where full worker flow is unnecessary,
• keep live tests opt-in and clearly separated from deterministic unit/integration coverage.

1. Optional KDS CDC test infrastructure under backend/tests/Kds.Ai.Integrations.Tests/KdsCdc/Infrastructure/ or shared backend test infrastructure.
2. Suggested helpers:
3. fake queue client or fake received-message source,
4. deterministic sequence/coordinate fixture builders,
5. test host builder for KDS CDC services,
6. reusable assertions for IngestionBatch lineage, checkpoint payloads, quarantine records, failed packages, and S3 path layout.

Guidance:

• prefer reuse of existing AppFactory, warehouse/data fixtures, and fake storage patterns before introducing new helpers.

1. Optional operational/readiness checklist artifact under specs/KdsCdcIngestion/ or nearby docs only if a human runbook is needed in addition to automated tests.
2. Examples:
3. delivery-readiness checklist,
4. live-test prerequisites,
5. known limitations or temporary blockers.

Guidance:

• only add this if automated tests alone do not communicate launch readiness clearly.

Files to Modify

• backend/tests/Kds.Ai.Integrations.Tests/ project files and folders to add KDS CDC scenario coverage.
• backend/tests/Kds.Ai.ApiService.Tests/.../Infrastructure/AppFactory.cs or related integration-test infrastructure if KDS CDC host configuration needs additive options, seeded mappings, or fake service registration.
• KDS CDC test seams introduced in Tasks 2 through 8 under backend/Kds.Ai.ApiService/Jobs/KdsCdc/ and backend/Kds.Ai.Integrations/KdsCdc/ only where additive hooks are required for deterministic validation.

Guidance: prefer injectable abstractions and explicit test doubles over test-only production branching.

• Shared test infrastructure only if current fixtures cannot safely express:
• fake SQS redelivery,
• duplicate coordinate streams,
• out-of-order sequence injection,
• raw/load/curated/export failure injection,
• replay-entry invocation.
• Potential readiness surfacing points if missing and genuinely needed:
• backend/Kds.Ai.AppHost/AppHost.cs
• backend/Kds.Ai.ApiService/Program.cs

Guidance: only add health/readiness surfacing if Task 9 discovers an actual observability gap. Do not add decorative health endpoints that do not reflect CDC readiness.

Required Behavior

Validation Coverage Matrix

• Task 9 must verify all documented operational scenarios at the right test level.
• At minimum, coverage must include:
• low-volume tenant batch closes by age limit,
• high-volume tenant splits into multiple bounded ordered micro-batches,
• duplicate CDC messages are deduped by source coordinates plus source table and primary key,
• out-of-order or missing sequence fails safely before Redshift load,
• malformed or unmapped records are quarantined while good records continue when safe,
• raw-write failure leaves SQS messages unacknowledged,
• durable-job-creation failure after raw write leaves SQS messages unacknowledged and preserves duplicate-safe replay semantics,
• Redshift COPY retry reuses same load artifact idempotently,
• curated merge retry resumes from durable downstream state without re-reading SQS,
• curated export retry resumes from curated state and cleans incomplete export artifacts when possible,
• checkpoint advancement occurs only after documented durable success points,
• replay paths use raw/load/curated durable state rather than SQS retention.
• mock per-stream ordering is not mistaken for cross-store global ordering.
• production contract validation requires real source metadata while mock validation accepts the documented mock sequence surrogate.
• partial SQS acknowledgement deletes only records/slices that reached a durable success or explicit quarantine boundary.
• oversized or unreadable SQS payloads fail safely.

Non-Goal Protection

• Task 9 must explicitly verify that implementation did not introduce forbidden behavior:
• no identification of production CDC as SyntheticKds,
• no use of simulator direct-write path as production CDC ingestion path,
• no one-ingestion-batch-per-record behavior,
• no dependency on SQS as long-term replay after acknowledgement,
• no CDC landing tables added as hidden first-version dependency,
• no physical deletion of curated rows in normal KDS CDC flow.

Batch Boundary And Lineage Verification

• Validation must prove actual runtime batch boundaries match the spec:
• one ETL batch represents a bounded ordered tenant/entity slice,
• mixed poll windows split correctly by company, tenant, and entity,
• each framework stage owns its own IngestionBatch,
• UpstreamBatchId chain remains intact through all five stages,
• rerun/current semantics remain correct under retries.

Checkpoint And Replay Verification

• Validation must prove KDS CDC checkpoints are not wall-clock-only regressions.
• Tests or live verification must confirm persisted checkpoint state includes or resolves back to:
• source-coordinate facts,
• SQS ordering facts,
• last successful stage batch ids,
• replay pointers to raw/load/curated durable state.
• Replay verification must cover:
• raw batch replay,
• load artifact retry,
• curated retry,
• coordinate-bounded recovery when implemented.

Quarantine And Failure Diagnostics Verification

• Validation must confirm record-level quarantine diagnostics and batch-level failed-package diagnostics both contain the fields required by the feature doc.
• Validation must also confirm the boundary stays correct:
• record-local failures quarantine only the bad records,
• batch-level failures stop later stages for that tenant/entity slice,
• unrelated good slices can still succeed.

Layered Test Strategy

• Task 9 should not force every scenario into one giant end-to-end test.
• Preferred verification layers are:
• unit tests for ordering, dedupe, checkpoint-payload, and diagnostic-shape logic,
• integration tests for host wiring, worker orchestration, stage transitions, and persistence effects,
• live tests for real S3/Redshift and, when available, real queue/infrastructure seams.
• Because current integration hosts disable background services, worker behavior should be validated by either:
• explicitly invoking worker/message-pump seams in tests, or
• using a dedicated host/test harness that intentionally enables only the KDS CDC worker under controlled conditions.

Delivery Readiness Outcome

• Task 9 ends only when the codebase has a credible automated readiness story for KDS CDC.
• That readiness story must make clear:
• which scenarios are covered by deterministic automated tests,
• which scenarios are covered by opt-in live tests,
• which scenarios remain blocked by external infrastructure and must be documented explicitly.
• Temporary blockers are acceptable only if they are called out clearly, mirrored in test skips/messages, and do not hide a missing implementation.

Proposed Implementation Shape

Recommended Test Layers

1. Deterministic integration tests under backend/tests/Kds.Ai.Integrations.Tests/KdsCdc/:
2. drive fake queue messages into worker or batch-building seams,
3. inject duplicate, redelivery, and out-of-order sequences,
4. assert batch creation, quarantine output, checkpoint advancement, and downstream stage calls.

1. Durable retry tests:
2. start from persisted raw batch state and verify normalize/load replay,
3. start from persisted load state and verify COPY retry,
4. start from persisted curated state and verify export retry,
5. assert no replay path depends on original SQS retention.

1. Live readiness tests, gated by KDSAI_RUN_LIVE_TESTS=1:
2. mirror existing Brink live-test style,
3. use ephemeral Postgres/Testcontainers plus configured external S3/Redshift,
4. add real KDS CDC readiness coverage only where infrastructure exists,
5. skip clearly when required infrastructure is unavailable.

Suggested Scenario Matrix

Worker intake and batching:
  - low volume -> age-close batch
  - high volume -> count/byte-bounded split
  - duplicate -> dedupe
  - out-of-order -> fail before load

Resolution and quarantine:
  - missing mapping -> quarantine only bad record
  - ambiguous mapping -> quarantine only bad record
  - bad + good same poll window -> good continues

Durability and retries:
  - raw write fails -> no ack
  - raw succeeds, durable metadata fails -> no ack, duplicate-safe retry
  - load COPY fails -> retry from load artifact
  - curated merge fails -> checkpoint stays behind
  - curated export fails -> retry from curated state, cleanup attempted

Lineage and replay:
  - per-stage batches chained by UpstreamBatchId
  - checkpoint facts source-based, not timestamp-only
  - replay from raw/load/curated works without SQS

Non-goals:
  - no SyntheticKds identity
  - no per-record batches
  - no SQS-as-replay-source
  - no hidden landing tables

Readiness Evidence Direction

• Prefer assertions on durable artifacts and metadata over log-only validation.
• Good readiness evidence includes:
• IngestionBatch rows and UpstreamBatchId chain,
• checkpoint rows and checkpoint metadata payloads,
• quarantine records and failed-package objects,
• raw/load/curated S3 path layout,
• tenant warehouse row outcomes,
• cleanup of incomplete curated export prefixes,
• absence of secret leakage in details/diagnostics.

Testing

• Unit tests for:
• duplicate detection and redelivery idempotence,
• out-of-order detection,
• checkpoint advancement gating,
• quarantine and failed-package diagnostic shape,
• non-goal guards such as source identity and batch-count expectations where expressible.
• Integration tests for:
• worker/message-pump orchestration under deterministic fake queue input,
• tenant/entity splitting and batch-boundary behavior,
• good-record continuation when bad records quarantine,
• replay and retry entry from raw/load/curated durable boundaries,
• no-ack behavior on raw-write and durable-metadata failures,
• no downstream stage execution after ordering failure,
• stage lineage and IsCurrent / ResetOfBatchId behavior on reruns.
• Data/warehouse tests for:
• checkpoint persistence shape,
• failed-package persistence and cleanup behavior,
• bootstrap/runtime readiness for required KDS staging and curated tables, including any customer-path gaps closed by earlier tasks.
• Live tests, gated by KDSAI_RUN_LIVE_TESTS=1, for:
• real raw/load/curated/export durability where infra exists,
• end-to-end replay-safe behavior across at least one realistic CDC tenant flow,
• operational blockers documented as explicit skips when external queue/Debezium wiring is not yet available.
• Regression coverage must confirm Brink POS and synthetic KDS behavior remain unchanged.

Acceptance Criteria

• Automated coverage exists for all documented KDS CDC operational scenarios at appropriate test layers.
• Low-volume, high-volume, duplicate, redelivery, and out-of-order behaviors are verified against the feature spec.
• Raw, load, curated, and export failure/retry paths are verified as idempotent and replay-safe.
• Batch boundaries, UpstreamBatchId lineage, checkpoint advancement, and quarantine behavior are proven to match the spec.
• Tests or readiness checks confirm production CDC is not identified as SyntheticKds and other non-goals were not introduced.
• Background-worker validation strategy is explicit and works with current integration-test constraints rather than depending on auto-started hosted services.
• Live readiness coverage is opt-in, clearly gated, and clearly documents any external-infrastructure blockers.
• Required KDS staging/curated schema readiness is verified for the paths KDS CDC uses.
• KDS CDC warehouse bootstrap/readiness is verified independently of Brink POS credential/service enablement.
• Existing Brink POS and synthetic KDS behavior remain unchanged.

Notes

• Use index.md#supported-corner-cases and index.md#non-goals as the primary source of truth for Task 9 scenario coverage.
• Existing Brink live ETL tests are the strongest current repository reference for delivery-readiness style, but KDS CDC must add worker-entry, ordering, quarantine, and replay coverage that Brink does not need.
• Keep Task 9 verification additive and explicit. If some readiness scenarios cannot yet run live because external Debezium/SQS infrastructure is unavailable, document that as a real blocker instead of masking it behind broad passing tests.