KDS CDC Implementation Breakdown

Purpose

Track main implementation tasks for KDS CDC ingestion. This is a planning checklist, not a detailed execution spec.

Task Checklist

• [ ] 1. Establish KDS CDC adapter identity and pipeline integration

Scope:

• introduce production source identity for KDS CDC and keep it separate from SyntheticKds
• register adapter in existing ETL framework with ingestion mode Cdc
• define stage list and adapter wiring so KDS CDC follows framework-native stage execution

Sources:

• Selected Behavior Summary
• Batch Lifecycle And Lineage
• Non-Goals

• [ ] 2. Implement SQS FIFO intake, validation, and ordered micro-batching

Scope:

• poll Debezium CDC messages from cross-account SQS FIFO
• validate required message metadata and detect malformed, duplicate, missing, or out-of-order records
• build bounded micro-batches using count, age, and memory/visibility limits instead of one batch per record
• preserve global ordering guarantees before downstream load begins

Sources:

• Stage 1: Debezium To SQS FIFO
• Stage 2: Worker Pull And Micro-Batch Creation
• Supported Corner Cases

• [ ] 3. Prepare mock CDC contract and producer for SQS-based integration

Scope:

• define mock Debezium-style CDC payload contract used during pre-infrastructure delivery
• build traffic generation/mock publishing so simulated CDC load enters through SQS rather than bypassing worker entry path
• keep worker integration pointed at SQS so mock mode exercises same intake, ordering, and acknowledgment behavior as production path
• keep mocking isolated to producer-side contract generation and do not replace SQS integration with in-memory shortcuts

Sources:

• Stage 1: Debezium To SQS FIFO
• Stage 2: Worker Pull And Micro-Batch Creation
• Selected Behavior Summary

• [ ] 4. Implement tenant resolution and quarantine boundaries

Scope:

• map source customer/account to company_id and store_guid to tenant_id
• block implicit or ambiguous tenant inference
• quarantine record-local mapping failures with enough metadata for diagnosis and replay
• ensure unresolved records never reach tenant-scoped S3 paths or Redshift loads

Sources:

• Stage 3: Tenant Resolution
• Quarantine And Failed Packages
• Supported Corner Cases

• [ ] 5. Persist raw CDC batches and create durable ingestion job state

Scope:

• write accepted CDC envelopes to tenant/entity-scoped S3 raw storage
• preserve replay-safe source metadata in raw artifacts
• create durable internal batch/job metadata after raw write succeeds
• acknowledge SQS messages only after raw persistence and durable resume state exist

Sources:

• Stage 4: Store To S3 Raw (extract_raw)
• Stage 5: Durable Internal Job Creation
• Checkpointing And Replay

• [ ] 6. Build normalization and Redshift staging load path

Scope:

• transform raw CDC envelopes into load-ready Parquet artifacts split by tenant and entity
• preserve delete flags, dedupe keys, and ordering metadata during normalization
• load source-shaped staging tables with idempotent Redshift COPY-based behavior
• fail safely when valid load artifacts or ordered staging application cannot be guaranteed

Sources:

• Stage 6: Normalize To S3 Load (normalize_load)
• Stage 7: Load To Source-Shaped Staging (load_to_staging)
• Selected Behavior Summary

• [ ] 7. Implement curated transformation and curated-to-S3 export

Scope:

• apply idempotent staging-to-curated merges for KDS warehouse targets
• preserve production KDS CDC source identity and soft-delete semantics in curated layer
• export curated data to tenant/entity-partitioned S3 Parquet with Redshift UNLOAD
• treat curated export as part of first delivery, not later follow-up work

Sources:

• Stage 8: Staging To Curated (staging_to_curated)
• Stage 9: Curated To S3 (curated_to_s3)
• Selected Behavior Summary

• [ ] 8. Add checkpointing, replay, failure handling, and lineage support

Scope:

• implement per-stage batch lifecycle, upstream chaining, and explicit completion/failure behavior
• persist checkpoints using Debezium/source coordinates and SQS ordering metadata
• support replay from raw, load, curated, and coordinate-bounded recovery points
• standardize quarantine and failed-package diagnostics across record-level and batch-level failures

Sources:

• Batch Lifecycle And Lineage
• Checkpointing And Replay
• Quarantine And Failed Packages
• Supported Corner Cases

• [ ] 9. Validate operational behavior and delivery readiness

Scope:

• verify low-volume, high-volume, redelivery, duplicate, and out-of-order scenarios
• verify idempotent retries across raw, load, curated, and export failure points
• confirm batch boundaries, checkpoint advancement, and quarantine behavior match spec
• confirm excluded behavior is not introduced during implementation

Sources:

• Selected Behavior Summary
• Supported Corner Cases
• Non-Goals