TASK 3: Prepare Mock CDC Contract And Producer For SQS-Based Integration

Summary

Define and implement the mock CDC producer seam that emits KDS source-shaped row changes into SQS FIFO using the temporary mock contract document as source of truth.

This task makes mock traffic enter through the same runtime boundary planned for production KDS CDC: ordered row-level SQS messages consumed by the KDS CDC worker. It freezes contract models, producer responsibilities, simulator translation behavior, sequence generation rules, and customer-event gap handling without replacing the worker entry path with in-memory shortcuts.

This task does not implement real Debezium connectivity, real source database CDC offsets, or final production producer infrastructure.

Current Codebase Findings

Repository analysis shows the current simulator path publishes directly into the synthetic warehouse feed rather than through SQS.

• Kds.Ai.Warehouse/Models/Synthetic/SimulatorWarehousePublishRequest.cs is the current simulator publish contract. It carries TenantId, StoreGuid, WarehouseFeedMode, and collections of Order, Item, and ItemBump rows.
• backend/simulator/Kds.Ai.Simulator.Core/OrderSimulator.cs currently creates Order, Item, and ItemBump rows in memory, persists them to the source KDS database, and then calls ISimulatorWarehousePublisher.PublishAsync(...) with WarehouseFeedMode.Incremental.
• Kds.Ai.Warehouse/Services/Synthetic/SimulatorWarehousePublisher.cs immediately converts that batch request into a synthetic staging write, creates an ingestion batch using IngestionMode.SyntheticFeed and IngestionSourceType.SyntheticKds, and writes straight into the current warehouse path.
• Kds.Ai.Warehouse/Services/Synthetic/SyntheticKdsSourceToStagingMapper.cs already contains mapping logic from Order, Item, and ItemBump entities into source-shaped staging rows. This is useful reference material for payload field parity, but it is intentionally the wrong runtime boundary for production CDC.
• Kds.Ai.WarehouseDataModel/Entities/Order.cs, Item.cs, ItemBump.cs, and Customer.cs match the fields described in mock-cdc-sqs-contracts.md, confirming the mock contract is intentionally aligned to current warehouse source row shapes.
• Customer contract is defined in docs, but current OrderSimulator does not generate or publish customer rows.
• Task 2 analysis already established that no SQS runtime or Amazon.SQS dependency exists yet, so mock producer work must stay aligned to the same queue-facing contracts as the worker spec.

These findings mean Task 3 should introduce a translation layer from current simulator publish requests into ordered SQS FIFO CDC envelopes, rather than extending the synthetic warehouse publisher as the long-term runtime path.

Dependencies

• docs/features/ETL/kds-cdc/index.md
• docs/features/ETL/kds-cdc/implementation-breakdown.md
• docs/features/ETL/kds-cdc/mock-cdc-sqs-contracts.md
• specs/KdsCdcIngestion/TASK1.md
• specs/KdsCdcIngestion/TASK2.md for the shared queue publish/consume seam; mock contract models may be implemented before the Task 2 queue implementation is complete.

Source References

• ../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist
• ../../docs/features/ETL/kds-cdc/index.md#stage-1-debezium-to-sqs-fifo
• ../../docs/features/ETL/kds-cdc/index.md#stage-2-worker-pull-and-micro-batch-creation
• ../../docs/features/ETL/kds-cdc/index.md#selected-behavior-summary
• ../../docs/features/ETL/kds-cdc/mock-cdc-sqs-contracts.md
• ../../docs/features/ETL/kds-cdc/mock-cdc-sqs-contracts.md#queue-contract
• ../../docs/features/ETL/kds-cdc/mock-cdc-sqs-contracts.md#cdc-envelope-contract
• ../../docs/features/ETL/kds-cdc/mock-cdc-sqs-contracts.md#simulator-mapping
• ../../docs/features/ETL/kds-cdc/mock-cdc-sqs-contracts.md#mock-producer-rules
• ../../docs/features/ETL/kds-cdc/mock-cdc-sqs-contracts.md#known-gaps-and-decisions

Scope From Breakdown

This task covers only the third implementation-breakdown item:

• define mock Debezium-style CDC payload contract used during pre-infrastructure delivery
• build traffic generation/mock publishing so simulated CDC load enters through SQS rather than bypassing worker entry path
• keep worker integration pointed at SQS so mock mode exercises same intake, ordering, and acknowledgment behavior as production path
• keep mocking isolated to producer-side contract generation and do not replace SQS integration with in-memory shortcuts

Files to Create

1. Mock CDC contract models

Create strongly typed mock CDC contract models that match mock-cdc-sqs-contracts.md.

Recommended logical location:

• backend/Kds.Ai.Integrations/KdsCdc/Mock/Contracts/

Recommended logical types:

• KdsCdcMockEnvelope
• KdsCdcMockPrimaryKey
• KdsCdcMockSequence
• KdsCdcMockFlags
• entity-specific payload models if separate types are preferred for validation clarity:
• KdsCdcMockOrderPayload
• KdsCdcMockItemPayload
• KdsCdcMockItemBumpPayload
• KdsCdcMockCustomerPayload

Important rule:

• contract model names may be C#-friendly,
• serialized JSON field names must match contract document exactly,
• contract version must begin with kds-cdc-mock/v1.

2. Mock CDC producer seam

Create a producer abstraction that emits FIFO-ready messages instead of writing directly to the synthetic warehouse feed.

This seam is the explicit batch-to-per-record translation layer called out as a known gap in mock-cdc-sqs-contracts.md: the current SimulatorWarehousePublishRequest is batch-oriented, while SQS CDC requires one envelope per changed row. KdsCdcMockEnvelopeFactory (or equivalent) is the named deliverable that owns this translation.

Recommended logical locations:

• backend/Kds.Ai.Integrations/KdsCdc/Mock/
• backend/Kds.Ai.Warehouse/Abstractions/ only if existing simulator abstraction is intentionally repurposed

Recommended logical types:

• IKdsCdcMockProducer
• KdsCdcMockProducer
• IKdsCdcMockEnvelopeFactory
• KdsCdcMockEnvelopeFactory
• IKdsCdcSequenceAllocator / KdsCdcSequenceAllocator

Minimum producer responsibilities:

• translate simulator/source rows into one envelope per changed row,
• assign FIFO metadata,
• assign deterministic deduplication id,
• preserve per-stream sequence monotonicity,
• publish to SQS through queue client seam from Task 2 or a compatible producer-side queue abstraction.

3. Producer-specific queue publish seam

If Task 2 queue abstraction is receive-oriented, create complementary publish seam for mock producer.

Recommended logical types:

• IKdsCdcQueuePublisher
• KdsCdcSqsQueuePublisher
• KdsCdcOutboundMessage

Minimum responsibilities:

• publish message body,
• set MessageGroupId,
• set deterministic MessageDeduplicationId,
• preserve ordering across published messages in a stream.

4. Mock customer event fixture path

Because current simulator does not emit customer rows, create an explicit seam for generating customers events outside the normal order loop when needed.

Possible shapes:

• fixture-based producer helper,
• separate Customer mock generator,
• test-only event factory.

Do not block orders/items/item_bumps implementation on customer simulation parity.

Files to Modify

5. Simulator publish abstraction

Review and likely modify:

• backend/Kds.Ai.Warehouse/Abstractions/ISimulatorWarehousePublisher.cs
• backend/Kds.Ai.Warehouse/Models/Synthetic/SimulatorWarehousePublishRequest.cs

Decision point this task must settle:

• either repurpose existing simulator publisher abstraction into queue-oriented mock CDC producer contract,
• or introduce a new dedicated mock CDC publisher abstraction and migrate simulator call sites to it.

Preferred direction:

• add dedicated KDS CDC mock producer abstraction rather than overloading a synthetic warehouse publisher name with new semantics.

6. Simulator traffic source

Modify:

• backend/simulator/Kds.Ai.Simulator.Core/OrderSimulator.cs
• backend/simulator/Kds.Ai.Simulator.Cli/Simulation/SimulateCommand.cs
• backend/simulator/Kds.Ai.Simulator.Web/Services/SimulationService.cs

Current simulator runtime resolves ISimulatorWarehousePublisher and publishes direct warehouse batches. Task 3 should redirect simulator-side mirroring to the mock CDC producer seam so traffic enters through SQS.

Important runtime rule:

• simulator should still create realistic row changes in source DB,
• producer translation happens after row change creation,
• no special in-memory shortcut should bypass the queue.

7. DI registration

Modify relevant DI registration points, likely including:

• backend/Kds.Ai.Warehouse/DependencyInjectionExtensions.cs
• backend/Kds.Ai.Integrations/DependencyInjectionExtensions.cs
• simulator host startup files if producer lives outside existing modules.

Task 3 should ensure simulator/web/CLI paths resolve the mock producer and queue publisher cleanly.

8. AppHost and local configuration seams

Review and likely modify:

• backend/Kds.Ai.AppHost/AppHostExtensions.cs
• backend/Kds.Ai.AppHost/AppHost.cs

Local development must be able to configure:

• queue URL or mock queue target,
• any producer enablement flag,
• source account id defaults for simulator traffic where needed.

9. Tests

Add or modify tests under:

• backend/tests/Kds.Ai.Integrations.Tests/KdsCdc/
• backend/tests/Kds.Ai.Warehouse.Tests/ if simulator publisher abstractions remain there
• simulator test projects if producer translation behavior is verified at simulator boundary

Required Behavior

Contract source of truth

docs/features/ETL/kds-cdc/mock-cdc-sqs-contracts.md is authoritative for mock payload shape.

Task 3 must not invent a second mock envelope format.

One row -> one message

Mock producer must emit one SQS FIFO message per changed source row.

This applies to:

• orders
• items
• item_bumps
• customers when separate generator path exists

Producer must not batch multiple changed rows into one envelope body.

Required queue metadata

Producer must set SQS FIFO metadata exactly per contract intent:

• MessageGroupId = {source_account_id}:{store_guid}
• MessageDeduplicationId = {entity}:{primary_key}:{sequence}
• body = UTF-8 JSON envelope

Task 3 should freeze exact string construction rules in code-facing spec so worker and producer tests agree.

Envelope fields

Serialized envelope must include:

• contract_version
• event_id
• source_type
• source_account_id
• store_guid
• entity
• operation
• occurred_at_utc
• primary_key
• sequence
• flags
• payload

Field values must satisfy documented constraints:

• contract_version = kds-cdc-mock/v1
• source_type = kds_cdc
• flags.is_snapshot = false
• primary_key.guid must equal payload guid
• sequence.stream_key must equal FIFO group key

Entity coverage

Mock producer must support the four documented entities, but implementation may phase them as follows:

1. orders, items, item_bumps from current simulator row flow are required in first working path,
2. customers contract model is required now even if event generation uses separate fixture/helper path initially.

Task 3 should explicitly document that missing first-class simulator customer generation is known and acceptable for initial mock phase.

Source-shaped payload parity

Payloads must remain source-shaped and aligned with current KDS entity fields.

Use current warehouse entity classes and mapper behavior as parity reference, especially:

• Order
• Item
• ItemBump
• Customer
• SyntheticKdsSourceToStagingMapper

Do not reshape payloads into curated or staging-specific destination DTOs at producer boundary.

Sequence allocation

Mock phase ordering rule is explicit body sequence plus FIFO ordering.

Important ordering scope:

• mock sequence is scoped to source_account_id + store_guid, matching FIFO MessageGroupId,
• this proves per-store stream order only,
• producer tests must not claim total cross-store global ordering unless a later contract changes message grouping or adds a true global source coordinate.

Task 3 should define sequence generation per source_account_id + store_guid stream:

• monotonic increasing sequence.stream_position
• stable sequence.stream_key
• sequence.source_timestamp_utc derived from event occurrence time

If one simulator action changes multiple rows, producer must emit several ordered messages using recommended ordering from the contract doc.

Mock source metadata profile

The mock envelope remains intentionally lighter than production Debezium payloads.

• entity identifies the source table class for mock traffic.
• sequence is the mock coordinate surrogate consumed by Task 2.
• Real Debezium database/schema/table/offset metadata is not required in kds-cdc-mock/v1 unless the contract document is updated.
• If adding an optional source object with database/schema/table placeholders is cheap and useful for forward compatibility, it must remain optional for kds-cdc-mock/v1 consumers.

Source account defaults and mapping seed

Current simulator publish requests do not contain source_account_id. Task 3 must provide deterministic local/dev configuration for producer source account identity.

Required behavior:

• source account id is configured explicitly, not inferred from tenant id,
• AppHost/local options expose a default source account id for simulator traffic,
• Task 4 mapping seed/config can map that source_account_id + store_guid to platform company/tenant.

Message size protection

Producer must check serialized SQS body size before publish when feasible.

• Oversize mock messages should fail with clear diagnostics.
• Producer must not silently truncate payloads to fit SQS limits.

Operation semantics

Mock producer may use upsert for create/update in first version.

Delete behavior rules:

• physical deletes not required,
• soft delete may be emitted as operation = delete or operation = upsert plus flags.is_deleted = true,
• consumer must still rely on is_deleted semantics.

Task 3 should choose one default producer behavior and keep it consistent.

Runtime direction

Mock mode must still exercise production-like runtime direction:

• simulator or helper generates traffic,
• traffic is published to SQS FIFO,
• KDS CDC worker consumes from SQS,
• worker owns intake, ordering validation, batching, and later acknowledgement.

Task 3 must not introduce an in-memory consumer shortcut between producer and worker.

Separation from synthetic direct warehouse publish

Production-aligned mock CDC path must not identify emitted events as synthetic feed batches.

This task should stop short of deleting old synthetic publisher code unless user asks, but it must clearly position that path as legacy/bootstrap behavior rather than CDC mock runtime.

Proposed Implementation Shape

Recommended translation flow:

OrderSimulator / simulator helper
  -> create source rows as today
  -> translate each changed row into KdsCdcMockEnvelope
  -> allocate stream sequence per source_account_id + store_guid
  -> publish ordered FIFO messages to SQS
  -> KDS CDC worker from Task 2 consumes same messages

Recommended producer contract shape:

public interface IKdsCdcMockProducer
{
    Task PublishAsync(KdsCdcMockPublishRequest request, CancellationToken ct = default);
}

Recommended request shape:

KdsCdcMockPublishRequest
  SourceAccountId
  StoreGuid
  OccurredAtUtc
  Orders
  Items
  ItemBumps
  Customers

Recommended sequence allocator seam:

public interface IKdsCdcSequenceAllocator
{
    ValueTask<long> GetNextAsync(string sourceAccountId, string storeGuid, CancellationToken ct = default);
}

Recommended outbound message shape:

KdsCdcOutboundMessage
  MessageGroupId
  MessageDeduplicationId
  BodyJson

Important notes:

• prefer new producer abstraction over renaming existing synthetic publisher abstraction,
• keep serialization deterministic for testability,
• keep contract version centralized in one constant,
• preserve room to swap mock sequence with real Debezium coordinates later without changing payload row shapes.

Testing

Unit tests

Add tests for:

• envelope serialization matches documented field names and required top-level structure,
• contract_version serializes as kds-cdc-mock/v1,
• source_type serializes as kds_cdc,
• MessageGroupId equals {source_account_id}:{store_guid},
• MessageDeduplicationId equals {entity}:{primary_key}:{sequence},
• primary_key.guid equals payload guid for each entity,
• sequence allocator increments monotonically per store stream,
• order create flow emits recommended order: orders, then items, then item_bumps,
• completion/update flow emits recommended order when multiple row updates occur,
• soft delete events preserve is_deleted semantics,
• customer contract model can serialize valid example payload even if simulator does not yet generate customers.

Integration/composition tests

Add tests proving:

• simulator-facing code resolves mock CDC producer from DI,
• publishing simulator change set results in expected number of outbound FIFO messages,
• no mock CDC publish path calls synthetic direct warehouse staging writer in the new SQS mode,
• worker-facing consumer tests can deserialize producer-generated envelopes without contract mismatch.

Not required in task 3

This task does not require:

• live AWS SQS publish tests,
• real Debezium offset tests,
• customer simulator generation parity,
• retirement of old synthetic publisher runtime,
• downstream ETL stage tests beyond contract compatibility with Task 2 intake.

Acceptance Criteria

• [ ] Strongly typed mock CDC envelope contract exists and matches mock-cdc-sqs-contracts.md
• [ ] Mock producer emits one SQS FIFO message per changed row
• [ ] Producer publishes through SQS-facing abstraction, not direct in-memory worker shortcut
• [ ] MessageGroupId uses {source_account_id}:{store_guid}
• [ ] MessageDeduplicationId uses deterministic {entity}:{primary_key}:{sequence} form
• [ ] Envelope source_type is kds_cdc
• [ ] Envelope contract_version is kds-cdc-mock/v1
• [ ] Envelope primary_key.guid matches payload guid
• [ ] Producer supports orders, items, and item_bumps from current simulator flow
• [ ] customers contract is defined now, even if produced through separate mock path initially
• [ ] Sequence values increase monotonically per store stream
• [ ] Mock ordering scope is documented as per source_account_id + store_guid stream, not cross-store global order
• [ ] Source account id defaults for simulator traffic are explicit and compatible with Task 4 mapping seed/config
• [ ] Producer fails clearly on oversized SQS message bodies rather than truncating
• [ ] Recommended multi-row ordering rules are documented and test-covered
• [ ] Simulator or helper traffic enters through SQS path rather than direct synthetic warehouse publish path
• [ ] Tests cover contract serialization, sequence generation, FIFO metadata, and simulator-to-envelope translation

Notes

• Follow docs/features/ETL/kds-cdc/mock-cdc-sqs-contracts.md as exact contract source of truth.
• Use current simulator entity generation as payload source, not as runtime transport model.
• Synthetic warehouse publisher and mapper are reference material for field parity only; they are not target runtime path for CDC.
• Keep mock-specific names explicit so future real Debezium contract can coexist during migration.
• Prefer additive producer seam over destructive replacement of current simulator publishing code unless later user request says to remove old path.