KDS CDC Ingestion - Implementation Tasks

Overview

Implementation of the KDS CDC Ingestion delivery slice.

This task plan is derived from:

• ../../docs/features/ETL/kds-cdc/index.md
• ../../docs/features/ETL/kds-cdc/implementation-breakdown.md
• ../../docs/features/ETL/etl-source-adapter-framework.md
• ../../docs/features/ETL/how-to-add-etl-source-adapter.md

Repository analysis for this spec also reviewed the current framework and active reference implementation in:

• backend/Kds.Ai.Integrations/Etl/EtlPipelineRunner.cs
• backend/Kds.Ai.Integrations/Etl/IEtlSourceAdapter.cs
• backend/Kds.Ai.Integrations/Etl/IEtlStage.cs
• backend/Kds.Ai.Integrations/Etl/EtlStageNames.cs
• backend/Kds.Ai.Integrations/Etl/EtlSourceDescriptor.cs
• backend/Kds.Ai.Integrations/DependencyInjectionExtensions.cs
• backend/Kds.Ai.Integrations/BrinkPos/BrinkPosSourceAdapter.cs
• backend/Kds.Ai.Integrations/BrinkPos/BrinkPosExtractStage.cs
• backend/Kds.Ai.Integrations/BrinkPos/BrinkPosNormalizeLoadStage.cs
• backend/Kds.Ai.Integrations/BrinkPos/BrinkPosLoadToStagingStage.cs
• backend/Kds.Ai.Integrations/BrinkPos/BrinkPosStagingToCuratedStage.cs
• backend/Kds.Ai.Integrations/BrinkPos/BrinkPosCuratedToS3Stage.cs
• backend/Kds.Ai.ApiService/Jobs/BrinkPosEtlJobs.cs
• backend/Kds.Ai.Data/Enums/IngestionSourceType.cs
• backend/Kds.Ai.Data/Enums/IngestionMode.cs
• backend/Kds.Ai.Warehouse/Services/Synthetic/SyntheticKdsFeedService.cs

The target slice introduces a production KDS CDC ETL path that uses the existing ETL source-adapter framework instead of the current synthetic KDS bootstrap path.

The slice outcome is:

• a distinct production source identity for KDS CDC,
• framework-native stage execution through extract_raw, normalize_load, load_to_staging, staging_to_curated, and curated_to_s3,
• runtime wiring that keeps production CDC separate from SyntheticKds,
• a migration path where later tasks can implement CDC intake, tenant resolution, raw persistence, normalization, Redshift load, curated merge, export, replay, and operations without revisiting adapter identity.

Source Documents

• ../../docs/features/ETL/kds-cdc/index.md
• ../../docs/features/ETL/kds-cdc/implementation-breakdown.md
• ../../docs/features/ETL/etl-source-adapter-framework.md
• ../../docs/features/ETL/how-to-add-etl-source-adapter.md

Spec Name

This implementation spec should be referred to as KDS CDC Ingestion.

Fixed Decisions

• KDS CDC is a new production ETL source adapter, not an extension of the synthetic KDS feed.
• Production CDC identity must remain separate from IngestionSourceType.SyntheticKds and any SyntheticKds* service path.
• The storage-safe adapter SourceType string is kds_cdc.
• The typed source enum value should be added as a new production entry such as IngestionSourceType.KdsCdc.
• The ingestion mode for this pipeline is IngestionMode.Cdc.
• KDS CDC must use the existing ETL adapter framework and stage runner, not a one-off orchestration path.
• KDS CDC stage names must use the shared framework stage constants from EtlStageNames, not source-local string copies.
• KDS CDC must include curated_to_s3 from the start.
• The five framework stages each own their own IngestionBatch lifecycle and chain via UpstreamBatchId.
• BusinessDate for CDC stage batches is processing date, not a source business date.
• Ordering scope must be explicit. Mock FIFO ordering is per source_account_id + store_guid message group; production ordering must use Debezium/source coordinates to prove whatever global ordering scope is required before load.
• Mock and production CDC contracts are separate validation profiles. kds-cdc-mock/v1 may use mock stream sequence as coordinate surrogate; production Debezium payloads must carry source database/schema/table and real source coordinates.
• KDS CDC bootstrap and tenant warehouse readiness must not depend on Brink POS credential enablement.
• Later tasks may add source-specific catalogs, CDC contracts, and worker entrypoints, but Task 1 only establishes adapter identity and runtime integration seams.
• Existing Brink POS behavior must remain unchanged.
• Existing synthetic KDS bootstrap flow must remain available and unchanged.

Scope

This task plan includes:

• defining KDS CDC adapter identity in framework terms,
• registering KDS CDC in the ETL framework with IngestionMode.Cdc,
• defining the five-stage runtime shape for KDS CDC,
• wiring KDS CDC adapter resolution into the existing DI and runtime selection pattern,
• introducing the first CDC-specific entrypoint seam that later worker or manual dispatch can call,
• preserving separation from synthetic KDS bootstrap code,
• preparing later implementation tasks for SQS intake, micro-batching, mapping, raw persistence, load, curated, export, replay, and operations.

In short, this slice ends when the codebase has a clear implementation plan for a real kds_cdc adapter that fits current ETL framework seams and does not leak into SyntheticKds identity.

Out of Scope

The following items are explicitly excluded from this task plan:

• implementing CDC SQS polling,
• implementing tenant mapping,
• implementing raw S3 writes,
• implementing durable job creation,
• implementing normalization logic,
• implementing Redshift COPY,
• implementing curated SQL or UNLOAD,
• redesigning framework-wide pipeline request shape unless required by a later concrete task,
• removing or migrating synthetic KDS bootstrap code,
• reworking Brink POS runtime patterns beyond what is needed as a reference.

Task Checklist

Phase 1: Adapter Identity And Runtime Integration

• [ ] TASK 1 - Establish KDS CDC adapter identity and pipeline integration

Sources:

• ../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist
• ../../docs/features/ETL/kds-cdc/index.md#selected-behavior-summary
• ../../docs/features/ETL/kds-cdc/index.md#batch-lifecycle-and-lineage
• ../../docs/features/ETL/kds-cdc/index.md#non-goals
• ../../docs/features/ETL/etl-source-adapter-framework.md
• ../../docs/features/ETL/how-to-add-etl-source-adapter.md

Phase 2: CDC Intake And Ordered Batch Formation

• [ ] TASK 2 - Implement SQS FIFO intake, validation, and ordered micro-batching

Sources:

• ../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist
• ../../docs/features/ETL/kds-cdc/index.md#stage-1-debezium-to-sqs-fifo
• ../../docs/features/ETL/kds-cdc/index.md#stage-2-worker-pull-and-micro-batch-creation
• ../../docs/features/ETL/kds-cdc/index.md#supported-corner-cases
• [ ] TASK 3 - Prepare mock CDC contract and producer for SQS-based integration

Sources:

• ../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist
• ../../docs/features/ETL/kds-cdc/index.md#stage-1-debezium-to-sqs-fifo
• ../../docs/features/ETL/kds-cdc/index.md#stage-2-worker-pull-and-micro-batch-creation
• ../../docs/features/ETL/kds-cdc/index.md#selected-behavior-summary

Phase 3: Resolution And Durable Raw Boundary

• [ ] TASK 4 - Implement tenant resolution and quarantine boundaries

Sources:

• ../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist
• ../../docs/features/ETL/kds-cdc/index.md#stage-3-tenant-resolution
• ../../docs/features/ETL/kds-cdc/index.md#quarantine-and-failed-packages
• ../../docs/features/ETL/kds-cdc/index.md#supported-corner-cases
• [ ] TASK 5 - Persist raw CDC batches and create durable ingestion job state

Sources:

• ../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist
• ../../docs/features/ETL/kds-cdc/index.md#stage-4-store-to-s3-raw-extract_raw
• ../../docs/features/ETL/kds-cdc/index.md#stage-5-durable-internal-job-creation
• ../../docs/features/ETL/kds-cdc/index.md#checkpointing-and-replay

Phase 4: Downstream Warehouse Stages

• [ ] TASK 6 - Build normalization and Redshift staging load path

Sources:

• ../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist
• ../../docs/features/ETL/kds-cdc/index.md#stage-6-normalize-to-s3-load-normalize_load
• ../../docs/features/ETL/kds-cdc/index.md#stage-7-load-to-source-shaped-staging-load_to_staging
• ../../docs/features/ETL/kds-cdc/index.md#selected-behavior-summary
• [ ] TASK 7 - Implement curated transformation and curated-to-S3 export

Sources:

• ../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist
• ../../docs/features/ETL/kds-cdc/index.md#stage-8-staging-to-curated-staging_to_curated
• ../../docs/features/ETL/kds-cdc/index.md#stage-9-curated-to-s3-curated_to_s3
• ../../docs/features/ETL/kds-cdc/index.md#selected-behavior-summary

Phase 5: Replay, Lineage, And Readiness

• [ ] TASK 8 - Add checkpointing, replay, failure handling, and lineage support

Sources:

• ../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist
• ../../docs/features/ETL/kds-cdc/index.md#batch-lifecycle-and-lineage
• ../../docs/features/ETL/kds-cdc/index.md#checkpointing-and-replay
• ../../docs/features/ETL/kds-cdc/index.md#quarantine-and-failed-packages
• ../../docs/features/ETL/kds-cdc/index.md#supported-corner-cases
• [ ] TASK 9 - Validate operational behavior and delivery readiness

Sources:

• ../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist
• ../../docs/features/ETL/kds-cdc/index.md#selected-behavior-summary
• ../../docs/features/ETL/kds-cdc/index.md#supported-corner-cases
• ../../docs/features/ETL/kds-cdc/index.md#non-goals

Dependency Graph

Phase 1
    +-- TASK 1 freezes source identity, stage list, and framework registration seam

Phase 2
    |-- TASK 2 ordered CDC intake <-- depends on TASK 1
    +-- TASK 3 mock CDC producer contract <-- depends on TASK 1; contract models can start before Task 2 queue implementation, publish seam aligns with Task 2

Phase 3
    |-- TASK 4 tenant resolution and quarantine <-- depends on TASK 2
    +-- TASK 5 raw durability and durable job state <-- depends on TASK 2, TASK 4

Phase 4
    |-- TASK 6 normalize and load to staging <-- depends on TASK 5
    +-- TASK 7 curated merge and curated export <-- depends on TASK 6

Phase 5
    |-- TASK 8 checkpoints, replay, lineage, failures <-- depends on TASK 5, TASK 6, TASK 7
    +-- TASK 9 operational validation <-- depends on TASK 2 through TASK 8

Recommended Session Order

Key Design Decisions

New Production Identity

The main feature doc is explicit: production CDC must not be identified as synthetic KDS. This means both storage-facing source string and typed source enum must reflect a new production identity.

Reuse Current ETL Framework

The repository already has a working adapter runner, shared stage names, shared stage support, and a reference adapter in Brink. KDS CDC should plug into that seam rather than create a second ETL architecture.

Keep Synthetic Path Separate

SyntheticKdsFeedService is a bootstrap warehouse sync that writes directly through a narrow load_to_staging style flow. It is not a production adapter and must not become the base class or identity for CDC.

Five Stage Model Is Already Supported

The current framework already exposes EtlStageNames.CuratedToS3, and the Brink adapter already includes five stage wrappers. KDS CDC should align to that full list from task 1 rather than add export as an afterthought.

Entry Point Can Differ From Brink

Brink currently enters from BrinkPosEtlJobs, which selects the adapter by source string. KDS CDC will likely enter from a background worker rather than a scheduler/manual API, but it should still resolve and invoke the adapter through IEtlPipelineRunner and IEtlSourceAdapter seams.

Task 1 Freezes Seams, Not Full CDC Behavior

This first task should settle identity, adapter registration, stage wiring, and pipeline invocation boundaries. It should not speculate deeply about payload parsing or SQS details beyond what the main feature doc already fixes.

Contract And Ordering Clarifications

The feature uses "global order" as a safety requirement, but implementation tasks must define the concrete scope. In mock mode, FIFO order is per MessageGroupId = source_account_id:store_guid; that does not prove total cross-store ordering. Production mode must either prove order by Debezium/source coordinates across the intended scope or fail safely before Redshift load.

Mock contracts and production Debezium contracts must coexist as separate validation profiles so temporary mock traffic does not get rejected for intentionally missing real Debezium offsets, while production traffic still requires source database/schema/table and coordinate metadata.