# KDS CDC Implementation Breakdown

## Purpose

Track main implementation tasks for KDS CDC ingestion. This is a planning checklist, not a detailed execution spec.

## Task Checklist

- [ ] 1. Establish KDS CDC adapter identity and pipeline integration
  Scope:
  - introduce production source identity for KDS CDC and keep it separate from `SyntheticKds`
  - register adapter in existing ETL framework with ingestion mode `Cdc`
  - define stage list and adapter wiring so KDS CDC follows framework-native stage execution
  Sources:
  - [Selected Behavior Summary](./index.md#selected-behavior-summary)
  - [Batch Lifecycle And Lineage](./index.md#batch-lifecycle-and-lineage)
  - [Non-Goals](./index.md#non-goals)

- [ ] 2. Implement SQS FIFO intake, validation, and ordered micro-batching
  Scope:
  - poll Debezium CDC messages from cross-account SQS FIFO
  - validate required message metadata and detect malformed, duplicate, missing, or out-of-order records
  - build bounded micro-batches using count, age, and memory/visibility limits instead of one batch per record
  - preserve global ordering guarantees before downstream load begins
  Sources:
  - [Stage 1: Debezium To SQS FIFO](./index.md#stage-1-debezium-to-sqs-fifo)
  - [Stage 2: Worker Pull And Micro-Batch Creation](./index.md#stage-2-worker-pull-and-micro-batch-creation)
  - [Supported Corner Cases](./index.md#supported-corner-cases)

- [ ] 3. Prepare mock CDC contract and producer for SQS-based integration
  Scope:
  - define mock Debezium-style CDC payload contract used during pre-infrastructure delivery
  - build traffic generation/mock publishing so simulated CDC load enters through SQS rather than bypassing worker entry path
  - keep worker integration pointed at SQS so mock mode exercises same intake, ordering, and acknowledgment behavior as production path
  - keep mocking isolated to producer-side contract generation and do not replace SQS integration with in-memory shortcuts
  Sources:
  - [Stage 1: Debezium To SQS FIFO](./index.md#stage-1-debezium-to-sqs-fifo)
  - [Stage 2: Worker Pull And Micro-Batch Creation](./index.md#stage-2-worker-pull-and-micro-batch-creation)
  - [Selected Behavior Summary](./index.md#selected-behavior-summary)

- [ ] 4. Implement tenant resolution and quarantine boundaries
  Scope:
  - map source customer/account to `company_id` and `store_guid` to `tenant_id`
  - block implicit or ambiguous tenant inference
  - quarantine record-local mapping failures with enough metadata for diagnosis and replay
  - ensure unresolved records never reach tenant-scoped S3 paths or Redshift loads
  Sources:
  - [Stage 3: Tenant Resolution](./index.md#stage-3-tenant-resolution)
  - [Quarantine And Failed Packages](./index.md#quarantine-and-failed-packages)
  - [Supported Corner Cases](./index.md#supported-corner-cases)

- [ ] 5. Persist raw CDC batches and create durable ingestion job state
  Scope:
  - write accepted CDC envelopes to tenant/entity-scoped S3 raw storage
  - preserve replay-safe source metadata in raw artifacts
  - create durable internal batch/job metadata after raw write succeeds
  - acknowledge SQS messages only after raw persistence and durable resume state exist
  Sources:
  - [Stage 4: Store To S3 Raw (`extract_raw`)](./index.md#stage-4-store-to-s3-raw-extract_raw)
  - [Stage 5: Durable Internal Job Creation](./index.md#stage-5-durable-internal-job-creation)
  - [Checkpointing And Replay](./index.md#checkpointing-and-replay)

- [ ] 6. Build normalization and Redshift staging load path
  Scope:
  - transform raw CDC envelopes into load-ready Parquet artifacts split by tenant and entity
  - preserve delete flags, dedupe keys, and ordering metadata during normalization
  - load source-shaped staging tables with idempotent Redshift `COPY`-based behavior
  - fail safely when valid load artifacts or ordered staging application cannot be guaranteed
  Sources:
  - [Stage 6: Normalize To S3 Load (`normalize_load`)](./index.md#stage-6-normalize-to-s3-load-normalize_load)
  - [Stage 7: Load To Source-Shaped Staging (`load_to_staging`)](./index.md#stage-7-load-to-source-shaped-staging-load_to_staging)
  - [Selected Behavior Summary](./index.md#selected-behavior-summary)

- [ ] 7. Implement curated transformation and curated-to-S3 export
  Scope:
  - apply idempotent staging-to-curated merges for KDS warehouse targets
  - preserve production KDS CDC source identity and soft-delete semantics in curated layer
  - export curated data to tenant/entity-partitioned S3 Parquet with Redshift `UNLOAD`
  - treat curated export as part of first delivery, not later follow-up work
  Sources:
  - [Stage 8: Staging To Curated (`staging_to_curated`)](./index.md#stage-8-staging-to-curated-staging_to_curated)
  - [Stage 9: Curated To S3 (`curated_to_s3`)](./index.md#stage-9-curated-to-s3-curated_to_s3)
  - [Selected Behavior Summary](./index.md#selected-behavior-summary)

- [ ] 8. Add checkpointing, replay, failure handling, and lineage support
  Scope:
  - implement per-stage batch lifecycle, upstream chaining, and explicit completion/failure behavior
  - persist checkpoints using Debezium/source coordinates and SQS ordering metadata
  - support replay from raw, load, curated, and coordinate-bounded recovery points
  - standardize quarantine and failed-package diagnostics across record-level and batch-level failures
  Sources:
  - [Batch Lifecycle And Lineage](./index.md#batch-lifecycle-and-lineage)
  - [Checkpointing And Replay](./index.md#checkpointing-and-replay)
  - [Quarantine And Failed Packages](./index.md#quarantine-and-failed-packages)
  - [Supported Corner Cases](./index.md#supported-corner-cases)

- [ ] 9. Validate operational behavior and delivery readiness
  Scope:
  - verify low-volume, high-volume, redelivery, duplicate, and out-of-order scenarios
  - verify idempotent retries across raw, load, curated, and export failure points
  - confirm batch boundaries, checkpoint advancement, and quarantine behavior match spec
  - confirm excluded behavior is not introduced during implementation
  Sources:
  - [Selected Behavior Summary](./index.md#selected-behavior-summary)
  - [Supported Corner Cases](./index.md#supported-corner-cases)
  - [Non-Goals](./index.md#non-goals)