# KDS CDC Debezium Integration - Implementation Tasks ## Overview Implementation of the **KDS CDC Debezium Integration** slice. This task plan is derived from: - `../../docs/features/ETL/kds-cdc/index.md` - `../../docs/features/ETL/kds-cdc/implementation-breakdown.md` - `../../docs/features/ETL/kds-cdc/debezium-payload-compatibility-notes.md` - `../../docs/features/ETL/kds-cdc/performance-and-scaling-notes.md` - sample payloads under `../../docs/features/ETL/kds-cdc/payloads/` Repository analysis for this spec also reviewed the current KDS CDC implementation in: - `backend/Kds.Ai.Integrations/KdsCdc/KdsCdcEnvelopeParser.cs` - `backend/Kds.Ai.Integrations/KdsCdc/Messages/KdsCdcEnvelope.cs` - `backend/Kds.Ai.Integrations/KdsCdc/KdsCdcTenantResolver.cs` - `backend/Kds.Ai.Integrations/KdsCdc/KdsCdcOrderingToken.cs` - `backend/Kds.Ai.Integrations/KdsCdc/KdsCdcMessageOrderingValidator.cs` - `backend/Kds.Ai.Integrations/KdsCdc/KdsCdcDuplicateDetector.cs` - `backend/Kds.Ai.Integrations/KdsCdc/KdsCdcBatchBuilder.cs` - `backend/Kds.Ai.Integrations/KdsCdc/KdsCdcNormalizeLoadJobs.cs` - `backend/Kds.Ai.Integrations/KdsCdc/KdsCdcExtractJobs.cs` - `backend/Kds.Ai.Integrations/KdsCdc/KdsCdcIntakeSupport.cs` - `backend/Kds.Ai.Integrations/KdsCdc/KdsCdcSqsQueueClient.cs` - `backend/Kds.Ai.ApiService/Jobs/KdsCdc/KdsCdcWorkerService.cs` - `backend/Kds.Ai.Data/Entities/KdsCdcSourceTenantMapping.cs` - `backend/Kds.Ai.Data/Repositories/EfKdsCdcSourceTenantMappingRepository.cs` - `backend/tests/Kds.Ai.Integrations.Tests/KdsCdc/` The current branch already has a working KDS CDC ETL spine, but it is still shaped around a simulator/mock contract, transport-group assumptions, and direct SQS-to-tenant batching that do not match the final Debezium-first architecture now documented in `docs/features/ETL/kds-cdc/debezium-payload-compatibility-notes.md`. This slice aligns the integration to current reality: - Debezium payload becomes the only supported intake contract, - SQS processing does not depend on `MessageGroupId`, - `store_guid` is treated as canonical tenant identity, - valid messages are durably captured into landing RAW before tenant promotion, - tenant RAW becomes downstream ETL boundary, - ETL state owns relation indexes, retries, and curated consistency, - simulator-specific code is removed, - downstream normalization and staging are updated to operate on real Debezium row images, - native `item_bumps` are supported directly. ## Source Documents - `../../docs/features/ETL/kds-cdc/index.md` - `../../docs/features/ETL/kds-cdc/implementation-breakdown.md` - `../../docs/features/ETL/kds-cdc/debezium-payload-compatibility-notes.md` - `../../docs/features/ETL/kds-cdc/performance-and-scaling-notes.md` - `../../docs/features/ETL/kds-cdc/payloads/customer_update.json` - `../../docs/features/ETL/kds-cdc/payloads/item_create.json` - `../../docs/features/ETL/kds-cdc/payloads/item_update.json` - `../../docs/features/ETL/kds-cdc/payloads/item_bump_create.json` - `../../docs/features/ETL/kds-cdc/payloads/order_create.json` - `../../docs/features/ETL/kds-cdc/payloads/order_update.json` - `../../docs/features/ETL/kds-cdc/payloads/order_delete.json` ## Spec Name This implementation spec should be referred to as **KDS CDC Debezium Integration**. ## Fixed Decisions - Real Debezium payload is the target contract. Simulator/mock payload is no longer a long-term supported path. - `store_guid == tenant_id` for KDS CDC and is treated as globally unique and stable. - KDS CDC tenant resolution must no longer depend on `source_account_id`. - Queue is FIFO-class, but processing correctness, tenant routing, and ETL batching must not depend on `MessageGroupId`. - `SequenceNumber` may be preserved when present, but correctness must not depend on it. - Debezium `before` / `after` row images are the source of truth for normalization. - Orders and customers can resolve tenant directly from payload `store_guid`. - Items currently do not carry `store_guid` in sample payloads; tenant resolution for items must use relation-based enrichment, most likely through `order_guid -> store_guid` correlation. - Native `item_bumps` payloads exist and must be supported directly. - SQS ack happens after landing RAW durability, not after full downstream ETL completion. - Landing RAW is first durable boundary and replay source for unresolved routing. - Tenant RAW is second durable boundary and input to ETL stages. - Curated correctness comes from rebuilding affected targets from tenant staging state. - Existing simulator/mock module, mock parser branch, mock producer, and mock-only tests can be removed once Debezium coverage replaces them. - Existing downstream ETL stages (`extract_raw`, `normalize_load`, `load_to_staging`, `staging_to_curated`, `curated_to_s3`) remain the target runtime model. - Existing curated export narrowing fix must be preserved. ## Scope This task plan includes: - replacing simulator-shaped intake models with Debezium-native contract models, - removing processing dependence on `MessageGroupId`, - simplifying tenant identity to `store_guid == tenant_id`, - adding landing RAW capture and tenant RAW promotion boundaries, - adding entity-aware tenant resolution for payloads that do not include `store_guid` directly, - redesigning production ordering and duplicate semantics around Debezium metadata and durable ETL state, - updating normalization to read Debezium row images, - supporting native `item_bumps`, - rebuilding curated targets from tenant staging state, - removing simulator code and converting tests/docs to Debezium-first coverage. In short, this slice ends when the KDS CDC integration is intentionally built for real Debezium payloads and no longer depends on simulator-specific contracts. ## Out of Scope The following items are explicitly excluded from this task plan: - redesigning the whole ETL framework, - changing the narrowed curated export behavior already delivered, - large-scale performance redesign beyond changes required for Debezium correctness, - production infrastructure rollout of FIFO queue settings, - creating a brand-new external admin UI, - inventing synthetic source data generators to replace removed simulator code. --- ## Task Checklist ### Phase 1: Debezium Contract And Transport Compatibility - [ ] [TASK 1](./TASK1.md) - Replace simulator contract assumptions with Debezium-native intake models Sources: - `../../docs/features/ETL/kds-cdc/debezium-payload-compatibility-notes.md` - `../../docs/features/ETL/kds-cdc/payloads/*.json` - `../../docs/features/ETL/kds-cdc/index.md#stage-1-debezium-to-sqs-fifo` ### Phase 2: Tenant Identity And Entity Correlation - [ ] [TASK 2](./TASK2.md) - Simplify tenant identity to `store_guid` and define durable routing resolution for child entities Sources: - `../../docs/features/ETL/kds-cdc/debezium-payload-compatibility-notes.md` - `../../docs/features/ETL/kds-cdc/index.md#stage-3-tenant-resolution` - `../../docs/features/ETL/kds-cdc/performance-and-scaling-notes.md` ### Phase 3: Landing RAW, Promotion, Deduplication, And Intake Durability - [ ] [TASK 3](./TASK3.md) - Rework intake durability, landing RAW, promotion flow, ordering metadata, and raw persistence for Debezium semantics Sources: - `../../docs/features/ETL/kds-cdc/index.md#stage-2-worker-pull-and-micro-batch-creation` - `../../docs/features/ETL/kds-cdc/index.md#stage-4-store-to-s3-raw-extract_raw` - `../../docs/features/ETL/kds-cdc/debezium-payload-compatibility-notes.md` ### Phase 4: Debezium Normalization And ETL-State-Based Curated Rebuild - [ ] [TASK 4](./TASK4.md) - Update normalization/load path for Debezium row images, native `item_bumps`, and curated rebuild from staging state Sources: - `../../docs/features/ETL/kds-cdc/payloads/*.json` - `../../docs/features/ETL/kds-cdc/index.md#stage-6-normalize-to-s3-load-normalize_load` - `../../docs/features/ETL/warehouse-data/logic-control-kds-staging-model/kds_item_bumps.md` ### Phase 5: Simulator Removal, Tests, And Documentation Alignment - [ ] [TASK 5](./TASK5.md) - Remove simulator path and replace it with Debezium-first tests and docs Sources: - `../../docs/features/ETL/kds-cdc/debezium-payload-compatibility-notes.md` - `../../docs/features/ETL/kds-cdc/index.md` - `../../docs/features/ETL/kds-cdc/performance-and-scaling-notes.md` --- ## Dependency Graph ```text Phase 1 +-- TASK 1 freezes Debezium models, parser direction, and transport-independent intake shape Phase 2 +-- TASK 2 tenant identity and child-entity routing correlation <-- depends on TASK 1 Phase 3 +-- TASK 3 landing RAW, promotion, ordering metadata, and raw durability <-- depends on TASK 1, TASK 2 Phase 4 +-- TASK 4 normalization, native item_bumps, and curated rebuild <-- depends on TASK 1, TASK 2, TASK 3 Phase 5 +-- TASK 5 simulator removal, coverage, docs <-- depends on TASK 1 through TASK 4 ``` --- ## Recommended Session Order | Session | Tasks | Description | |---|---|---| | 1 | 1 | Freeze Debezium models, parser seam, and transport-indifferent intake behavior | | 2 | 2 | Simplify tenant identity and add durable child-entity routing rules | | 3 | 3 | Introduce landing RAW, promotion flow, and Debezium-native raw durability semantics | | 4 | 4 | Convert normalization/load path, support native `item_bumps`, and rebuild curated from staging state | | 5 | 5 | Remove simulator code, rewrite tests, and finish doc alignment | --- ## Key Design Decisions ### Debezium-Only Contract Current parser and models are shaped around a custom/mock envelope with fields like `source_account_id`, top-level `store_guid`, `primary_key`, and custom sequence metadata. Real payloads do not match that shape. This spec treats that mismatch as a migration, not a coexistence problem. ### Transport Metadata Must Not Be Intake Contract Current parser rejects messages without `MessageGroupId` or `SequenceNumber`. Final design does not allow transport metadata to control tenant routing, correctness, or ETL progression. The code should preserve transport metadata when present, but landing RAW capture and downstream ETL must still work without it. ### `store_guid` Is Canonical Tenant Identity, But Child Entities Still Need Correlation Orders and customers include `store_guid` directly. Items and some item-bump flows may not. So removing `source_account_id` solves one problem, but not all tenant-routing problems. The plan therefore includes explicit durable relation-based resolution for child traffic. ### Landing RAW And Tenant RAW Are Required Boundaries Current worker couples intake too tightly to immediate tenant batching. Final design requires two durable boundaries: landing RAW for fast append-only capture, then tenant RAW for ETL consumption after tenant routing is safe. ### Curated Correctness Comes From Staging-State Rebuild Current curated SQL is sensitive to parent-before-child timing. Final design fixes this by rebuilding affected curated targets from tenant staging state rather than treating one arrival window as final semantic truth. ### Remove Simulator Instead Of Carrying Two Contracts The user explicitly wants simulator-related code removed. This spec therefore favors direct replacement over dual-mode compatibility. ### Preserve Existing ETL Spine The integration should keep current stage structure, pipeline launcher, and downstream batch lineage rather than inventing a new ETL runtime.