# KDS CDC Ingestion - Implementation Tasks

## Overview

Implementation of the **KDS CDC Ingestion** delivery slice.

This task plan is derived from:

- `../../docs/features/ETL/kds-cdc/index.md`
- `../../docs/features/ETL/kds-cdc/implementation-breakdown.md`
- `../../docs/features/ETL/etl-source-adapter-framework.md`
- `../../docs/features/ETL/how-to-add-etl-source-adapter.md`

Repository analysis for this spec also reviewed the current framework and active reference implementation in:

- `backend/Kds.Ai.Integrations/Etl/EtlPipelineRunner.cs`
- `backend/Kds.Ai.Integrations/Etl/IEtlSourceAdapter.cs`
- `backend/Kds.Ai.Integrations/Etl/IEtlStage.cs`
- `backend/Kds.Ai.Integrations/Etl/EtlStageNames.cs`
- `backend/Kds.Ai.Integrations/Etl/EtlSourceDescriptor.cs`
- `backend/Kds.Ai.Integrations/DependencyInjectionExtensions.cs`
- `backend/Kds.Ai.Integrations/BrinkPos/BrinkPosSourceAdapter.cs`
- `backend/Kds.Ai.Integrations/BrinkPos/BrinkPosExtractStage.cs`
- `backend/Kds.Ai.Integrations/BrinkPos/BrinkPosNormalizeLoadStage.cs`
- `backend/Kds.Ai.Integrations/BrinkPos/BrinkPosLoadToStagingStage.cs`
- `backend/Kds.Ai.Integrations/BrinkPos/BrinkPosStagingToCuratedStage.cs`
- `backend/Kds.Ai.Integrations/BrinkPos/BrinkPosCuratedToS3Stage.cs`
- `backend/Kds.Ai.ApiService/Jobs/BrinkPosEtlJobs.cs`
- `backend/Kds.Ai.Data/Enums/IngestionSourceType.cs`
- `backend/Kds.Ai.Data/Enums/IngestionMode.cs`
- `backend/Kds.Ai.Warehouse/Services/Synthetic/SyntheticKdsFeedService.cs`

The target slice introduces a production KDS CDC ETL path that uses the existing ETL source-adapter framework instead of the current synthetic KDS bootstrap path.

The slice outcome is:

- a distinct production source identity for KDS CDC,
- framework-native stage execution through `extract_raw`, `normalize_load`, `load_to_staging`, `staging_to_curated`, and `curated_to_s3`,
- runtime wiring that keeps production CDC separate from `SyntheticKds`,
- a migration path where later tasks can implement CDC intake, tenant resolution, raw persistence, normalization, Redshift load, curated merge, export, replay, and operations without revisiting adapter identity.

## Source Documents

- `../../docs/features/ETL/kds-cdc/index.md`
- `../../docs/features/ETL/kds-cdc/implementation-breakdown.md`
- `../../docs/features/ETL/etl-source-adapter-framework.md`
- `../../docs/features/ETL/how-to-add-etl-source-adapter.md`

## Spec Name

This implementation spec should be referred to as **KDS CDC Ingestion**.

## Fixed Decisions

- KDS CDC is a new production ETL source adapter, not an extension of the synthetic KDS feed.
- Production CDC identity must remain separate from `IngestionSourceType.SyntheticKds` and any `SyntheticKds*` service path.
- The storage-safe adapter `SourceType` string is `kds_cdc`.
- The typed source enum value should be added as a new production entry such as `IngestionSourceType.KdsCdc`.
- The ingestion mode for this pipeline is `IngestionMode.Cdc`.
- KDS CDC must use the existing ETL adapter framework and stage runner, not a one-off orchestration path.
- KDS CDC stage names must use the shared framework stage constants from `EtlStageNames`, not source-local string copies.
- KDS CDC must include `curated_to_s3` from the start.
- The five framework stages each own their own `IngestionBatch` lifecycle and chain via `UpstreamBatchId`.
- `BusinessDate` for CDC stage batches is processing date, not a source business date.
- Ordering scope must be explicit. Mock FIFO ordering is per `source_account_id + store_guid` message group; production ordering must use Debezium/source coordinates to prove whatever global ordering scope is required before load.
- Mock and production CDC contracts are separate validation profiles. `kds-cdc-mock/v1` may use mock stream sequence as coordinate surrogate; production Debezium payloads must carry source database/schema/table and real source coordinates.
- KDS CDC bootstrap and tenant warehouse readiness must not depend on Brink POS credential enablement.
- Later tasks may add source-specific catalogs, CDC contracts, and worker entrypoints, but Task 1 only establishes adapter identity and runtime integration seams.
- Existing Brink POS behavior must remain unchanged.
- Existing synthetic KDS bootstrap flow must remain available and unchanged.

## Scope

This task plan includes:

- defining KDS CDC adapter identity in framework terms,
- registering KDS CDC in the ETL framework with `IngestionMode.Cdc`,
- defining the five-stage runtime shape for KDS CDC,
- wiring KDS CDC adapter resolution into the existing DI and runtime selection pattern,
- introducing the first CDC-specific entrypoint seam that later worker or manual dispatch can call,
- preserving separation from synthetic KDS bootstrap code,
- preparing later implementation tasks for SQS intake, micro-batching, mapping, raw persistence, load, curated, export, replay, and operations.

In short, this slice ends when the codebase has a clear implementation plan for a real `kds_cdc` adapter that fits current ETL framework seams and does not leak into `SyntheticKds` identity.

## Out of Scope

The following items are explicitly excluded from this task plan:

- implementing CDC SQS polling,
- implementing tenant mapping,
- implementing raw S3 writes,
- implementing durable job creation,
- implementing normalization logic,
- implementing Redshift `COPY`,
- implementing curated SQL or `UNLOAD`,
- redesigning framework-wide pipeline request shape unless required by a later concrete task,
- removing or migrating synthetic KDS bootstrap code,
- reworking Brink POS runtime patterns beyond what is needed as a reference.

---

## Task Checklist

### Phase 1: Adapter Identity And Runtime Integration

- [ ] [TASK 1](./TASK1.md) - Establish KDS CDC adapter identity and pipeline integration
  Sources:
  - `../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist`
  - `../../docs/features/ETL/kds-cdc/index.md#selected-behavior-summary`
  - `../../docs/features/ETL/kds-cdc/index.md#batch-lifecycle-and-lineage`
  - `../../docs/features/ETL/kds-cdc/index.md#non-goals`
  - `../../docs/features/ETL/etl-source-adapter-framework.md`
  - `../../docs/features/ETL/how-to-add-etl-source-adapter.md`

### Phase 2: CDC Intake And Ordered Batch Formation

- [ ] [TASK 2](./TASK2.md) - Implement SQS FIFO intake, validation, and ordered micro-batching
  Sources:
  - `../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist`
  - `../../docs/features/ETL/kds-cdc/index.md#stage-1-debezium-to-sqs-fifo`
  - `../../docs/features/ETL/kds-cdc/index.md#stage-2-worker-pull-and-micro-batch-creation`
  - `../../docs/features/ETL/kds-cdc/index.md#supported-corner-cases`
- [ ] [TASK 3](./TASK3.md) - Prepare mock CDC contract and producer for SQS-based integration
  Sources:
  - `../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist`
  - `../../docs/features/ETL/kds-cdc/index.md#stage-1-debezium-to-sqs-fifo`
  - `../../docs/features/ETL/kds-cdc/index.md#stage-2-worker-pull-and-micro-batch-creation`
  - `../../docs/features/ETL/kds-cdc/index.md#selected-behavior-summary`

### Phase 3: Resolution And Durable Raw Boundary

- [ ] [TASK 4](./TASK4.md) - Implement tenant resolution and quarantine boundaries
  Sources:
  - `../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist`
  - `../../docs/features/ETL/kds-cdc/index.md#stage-3-tenant-resolution`
  - `../../docs/features/ETL/kds-cdc/index.md#quarantine-and-failed-packages`
  - `../../docs/features/ETL/kds-cdc/index.md#supported-corner-cases`
- [ ] [TASK 5](./TASK5.md) - Persist raw CDC batches and create durable ingestion job state
  Sources:
  - `../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist`
  - `../../docs/features/ETL/kds-cdc/index.md#stage-4-store-to-s3-raw-extract_raw`
  - `../../docs/features/ETL/kds-cdc/index.md#stage-5-durable-internal-job-creation`
  - `../../docs/features/ETL/kds-cdc/index.md#checkpointing-and-replay`

### Phase 4: Downstream Warehouse Stages

- [ ] [TASK 6](./TASK6.md) - Build normalization and Redshift staging load path
  Sources:
  - `../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist`
  - `../../docs/features/ETL/kds-cdc/index.md#stage-6-normalize-to-s3-load-normalize_load`
  - `../../docs/features/ETL/kds-cdc/index.md#stage-7-load-to-source-shaped-staging-load_to_staging`
  - `../../docs/features/ETL/kds-cdc/index.md#selected-behavior-summary`
- [ ] [TASK 7](./TASK7.md) - Implement curated transformation and curated-to-S3 export
  Sources:
  - `../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist`
  - `../../docs/features/ETL/kds-cdc/index.md#stage-8-staging-to-curated-staging_to_curated`
  - `../../docs/features/ETL/kds-cdc/index.md#stage-9-curated-to-s3-curated_to_s3`
  - `../../docs/features/ETL/kds-cdc/index.md#selected-behavior-summary`

### Phase 5: Replay, Lineage, And Readiness

- [ ] [TASK 8](./TASK8.md) - Add checkpointing, replay, failure handling, and lineage support
  Sources:
  - `../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist`
  - `../../docs/features/ETL/kds-cdc/index.md#batch-lifecycle-and-lineage`
  - `../../docs/features/ETL/kds-cdc/index.md#checkpointing-and-replay`
  - `../../docs/features/ETL/kds-cdc/index.md#quarantine-and-failed-packages`
  - `../../docs/features/ETL/kds-cdc/index.md#supported-corner-cases`
- [ ] [TASK 9](./TASK9.md) - Validate operational behavior and delivery readiness
  Sources:
  - `../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist`
  - `../../docs/features/ETL/kds-cdc/index.md#selected-behavior-summary`
  - `../../docs/features/ETL/kds-cdc/index.md#supported-corner-cases`
  - `../../docs/features/ETL/kds-cdc/index.md#non-goals`

---

## Dependency Graph

```text
Phase 1
    +-- TASK 1 freezes source identity, stage list, and framework registration seam

Phase 2
    |-- TASK 2 ordered CDC intake <-- depends on TASK 1
    +-- TASK 3 mock CDC producer contract <-- depends on TASK 1; contract models can start before Task 2 queue implementation, publish seam aligns with Task 2

Phase 3
    |-- TASK 4 tenant resolution and quarantine <-- depends on TASK 2
    +-- TASK 5 raw durability and durable job state <-- depends on TASK 2, TASK 4

Phase 4
    |-- TASK 6 normalize and load to staging <-- depends on TASK 5
    +-- TASK 7 curated merge and curated export <-- depends on TASK 6

Phase 5
    |-- TASK 8 checkpoints, replay, lineage, failures <-- depends on TASK 5, TASK 6, TASK 7
    +-- TASK 9 operational validation <-- depends on TASK 2 through TASK 8
```

---

## Recommended Session Order

| Session | Tasks | Description |
|---|---|---|
| 1 | 1 | Freeze adapter identity, stage list, and framework registration shape |
| 2 | 2, 3 | Build SQS worker intake and mock producer contract in parallel-compatible seams |
| 3 | 4, 5 | Add tenant resolution, raw durability, and durable resume state |
| 4 | 6 | Deliver raw-to-load and Redshift staging behavior |
| 5 | 7, 8 | Deliver curated merge/export and full lifecycle metadata |
| 6 | 9 | Validate corner cases, retries, ordering, and non-goals |

---

## Key Design Decisions

### New Production Identity

The main feature doc is explicit: production CDC must not be identified as synthetic KDS. This means both storage-facing source string and typed source enum must reflect a new production identity.

### Reuse Current ETL Framework

The repository already has a working adapter runner, shared stage names, shared stage support, and a reference adapter in Brink. KDS CDC should plug into that seam rather than create a second ETL architecture.

### Keep Synthetic Path Separate

`SyntheticKdsFeedService` is a bootstrap warehouse sync that writes directly through a narrow `load_to_staging` style flow. It is not a production adapter and must not become the base class or identity for CDC.

### Five Stage Model Is Already Supported

The current framework already exposes `EtlStageNames.CuratedToS3`, and the Brink adapter already includes five stage wrappers. KDS CDC should align to that full list from task 1 rather than add export as an afterthought.

### Entry Point Can Differ From Brink

Brink currently enters from `BrinkPosEtlJobs`, which selects the adapter by source string. KDS CDC will likely enter from a background worker rather than a scheduler/manual API, but it should still resolve and invoke the adapter through `IEtlPipelineRunner` and `IEtlSourceAdapter` seams.

### Task 1 Freezes Seams, Not Full CDC Behavior

This first task should settle identity, adapter registration, stage wiring, and pipeline invocation boundaries. It should not speculate deeply about payload parsing or SQS details beyond what the main feature doc already fixes.

### Contract And Ordering Clarifications

The feature uses "global order" as a safety requirement, but implementation tasks must define the concrete scope. In mock mode, FIFO order is per `MessageGroupId = source_account_id:store_guid`; that does not prove total cross-store ordering. Production mode must either prove order by Debezium/source coordinates across the intended scope or fail safely before Redshift load.

Mock contracts and production Debezium contracts must coexist as separate validation profiles so temporary mock traffic does not get rejected for intentionally missing real Debezium offsets, while production traffic still requires source database/schema/table and coordinate metadata.