# TASK 3: Prepare Mock CDC Contract And Producer For SQS-Based Integration

## Summary

Define and implement the mock CDC producer seam that emits KDS source-shaped row changes into SQS FIFO using the temporary mock contract document as source of truth.

This task makes mock traffic enter through the same runtime boundary planned for production KDS CDC: ordered row-level SQS messages consumed by the KDS CDC worker. It freezes contract models, producer responsibilities, simulator translation behavior, sequence generation rules, and customer-event gap handling without replacing the worker entry path with in-memory shortcuts.

This task does not implement real Debezium connectivity, real source database CDC offsets, or final production producer infrastructure.

## Current Codebase Findings

Repository analysis shows the current simulator path publishes directly into the synthetic warehouse feed rather than through SQS.

- `Kds.Ai.Warehouse/Models/Synthetic/SimulatorWarehousePublishRequest.cs` is the current simulator publish contract. It carries `TenantId`, `StoreGuid`, `WarehouseFeedMode`, and collections of `Order`, `Item`, and `ItemBump` rows.
- `backend/simulator/Kds.Ai.Simulator.Core/OrderSimulator.cs` currently creates `Order`, `Item`, and `ItemBump` rows in memory, persists them to the source KDS database, and then calls `ISimulatorWarehousePublisher.PublishAsync(...)` with `WarehouseFeedMode.Incremental`.
- `Kds.Ai.Warehouse/Services/Synthetic/SimulatorWarehousePublisher.cs` immediately converts that batch request into a synthetic staging write, creates an ingestion batch using `IngestionMode.SyntheticFeed` and `IngestionSourceType.SyntheticKds`, and writes straight into the current warehouse path.
- `Kds.Ai.Warehouse/Services/Synthetic/SyntheticKdsSourceToStagingMapper.cs` already contains mapping logic from `Order`, `Item`, and `ItemBump` entities into source-shaped staging rows. This is useful reference material for payload field parity, but it is intentionally the wrong runtime boundary for production CDC.
- `Kds.Ai.WarehouseDataModel/Entities/Order.cs`, `Item.cs`, `ItemBump.cs`, and `Customer.cs` match the fields described in `mock-cdc-sqs-contracts.md`, confirming the mock contract is intentionally aligned to current warehouse source row shapes.
- `Customer` contract is defined in docs, but current `OrderSimulator` does not generate or publish customer rows.
- Task 2 analysis already established that no SQS runtime or `Amazon.SQS` dependency exists yet, so mock producer work must stay aligned to the same queue-facing contracts as the worker spec.

These findings mean Task 3 should introduce a translation layer from current simulator publish requests into ordered SQS FIFO CDC envelopes, rather than extending the synthetic warehouse publisher as the long-term runtime path.

## Dependencies

- `docs/features/ETL/kds-cdc/index.md`
- `docs/features/ETL/kds-cdc/implementation-breakdown.md`
- `docs/features/ETL/kds-cdc/mock-cdc-sqs-contracts.md`
- `specs/KdsCdcIngestion/TASK1.md`
- `specs/KdsCdcIngestion/TASK2.md` for the shared queue publish/consume seam; mock contract models may be implemented before the Task 2 queue implementation is complete.

## Source References

- `../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist`
- `../../docs/features/ETL/kds-cdc/index.md#stage-1-debezium-to-sqs-fifo`
- `../../docs/features/ETL/kds-cdc/index.md#stage-2-worker-pull-and-micro-batch-creation`
- `../../docs/features/ETL/kds-cdc/index.md#selected-behavior-summary`
- `../../docs/features/ETL/kds-cdc/mock-cdc-sqs-contracts.md`
- `../../docs/features/ETL/kds-cdc/mock-cdc-sqs-contracts.md#queue-contract`
- `../../docs/features/ETL/kds-cdc/mock-cdc-sqs-contracts.md#cdc-envelope-contract`
- `../../docs/features/ETL/kds-cdc/mock-cdc-sqs-contracts.md#simulator-mapping`
- `../../docs/features/ETL/kds-cdc/mock-cdc-sqs-contracts.md#mock-producer-rules`
- `../../docs/features/ETL/kds-cdc/mock-cdc-sqs-contracts.md#known-gaps-and-decisions`

## Scope From Breakdown

This task covers only the third implementation-breakdown item:

- define mock Debezium-style CDC payload contract used during pre-infrastructure delivery
- build traffic generation/mock publishing so simulated CDC load enters through SQS rather than bypassing worker entry path
- keep worker integration pointed at SQS so mock mode exercises same intake, ordering, and acknowledgment behavior as production path
- keep mocking isolated to producer-side contract generation and do not replace SQS integration with in-memory shortcuts

## Files to Create

### 1. Mock CDC contract models

Create strongly typed mock CDC contract models that match `mock-cdc-sqs-contracts.md`.

Recommended logical location:

- `backend/Kds.Ai.Integrations/KdsCdc/Mock/Contracts/`

Recommended logical types:

- `KdsCdcMockEnvelope`
- `KdsCdcMockPrimaryKey`
- `KdsCdcMockSequence`
- `KdsCdcMockFlags`
- entity-specific payload models if separate types are preferred for validation clarity:
  - `KdsCdcMockOrderPayload`
  - `KdsCdcMockItemPayload`
  - `KdsCdcMockItemBumpPayload`
  - `KdsCdcMockCustomerPayload`

Important rule:

- contract model names may be C#-friendly,
- serialized JSON field names must match contract document exactly,
- contract version must begin with `kds-cdc-mock/v1`.

### 2. Mock CDC producer seam

Create a producer abstraction that emits FIFO-ready messages instead of writing directly to the synthetic warehouse feed.

This seam is the explicit batch-to-per-record translation layer called out as a known gap in `mock-cdc-sqs-contracts.md`: the current `SimulatorWarehousePublishRequest` is batch-oriented, while SQS CDC requires one envelope per changed row. `KdsCdcMockEnvelopeFactory` (or equivalent) is the named deliverable that owns this translation.

Recommended logical locations:

- `backend/Kds.Ai.Integrations/KdsCdc/Mock/`
- `backend/Kds.Ai.Warehouse/Abstractions/` only if existing simulator abstraction is intentionally repurposed

Recommended logical types:

- `IKdsCdcMockProducer`
- `KdsCdcMockProducer`
- `IKdsCdcMockEnvelopeFactory`
- `KdsCdcMockEnvelopeFactory`
- `IKdsCdcSequenceAllocator` / `KdsCdcSequenceAllocator`

Minimum producer responsibilities:

- translate simulator/source rows into one envelope per changed row,
- assign FIFO metadata,
- assign deterministic deduplication id,
- preserve per-stream sequence monotonicity,
- publish to SQS through queue client seam from Task 2 or a compatible producer-side queue abstraction.

### 3. Producer-specific queue publish seam

If Task 2 queue abstraction is receive-oriented, create complementary publish seam for mock producer.

Recommended logical types:

- `IKdsCdcQueuePublisher`
- `KdsCdcSqsQueuePublisher`
- `KdsCdcOutboundMessage`

Minimum responsibilities:

- publish message body,
- set `MessageGroupId`,
- set deterministic `MessageDeduplicationId`,
- preserve ordering across published messages in a stream.

### 4. Mock customer event fixture path

Because current simulator does not emit customer rows, create an explicit seam for generating `customers` events outside the normal order loop when needed.

Possible shapes:

- fixture-based producer helper,
- separate `Customer` mock generator,
- test-only event factory.

Do not block orders/items/item_bumps implementation on customer simulation parity.

## Files to Modify

### 5. Simulator publish abstraction

Review and likely modify:

- `backend/Kds.Ai.Warehouse/Abstractions/ISimulatorWarehousePublisher.cs`
- `backend/Kds.Ai.Warehouse/Models/Synthetic/SimulatorWarehousePublishRequest.cs`

Decision point this task must settle:

- either repurpose existing simulator publisher abstraction into queue-oriented mock CDC producer contract,
- or introduce a new dedicated mock CDC publisher abstraction and migrate simulator call sites to it.

Preferred direction:

- add dedicated KDS CDC mock producer abstraction rather than overloading a synthetic warehouse publisher name with new semantics.

### 6. Simulator traffic source

Modify:

- `backend/simulator/Kds.Ai.Simulator.Core/OrderSimulator.cs`
- `backend/simulator/Kds.Ai.Simulator.Cli/Simulation/SimulateCommand.cs`
- `backend/simulator/Kds.Ai.Simulator.Web/Services/SimulationService.cs`

Current simulator runtime resolves `ISimulatorWarehousePublisher` and publishes direct warehouse batches. Task 3 should redirect simulator-side mirroring to the mock CDC producer seam so traffic enters through SQS.

Important runtime rule:

- simulator should still create realistic row changes in source DB,
- producer translation happens after row change creation,
- no special in-memory shortcut should bypass the queue.

### 7. DI registration

Modify relevant DI registration points, likely including:

- `backend/Kds.Ai.Warehouse/DependencyInjectionExtensions.cs`
- `backend/Kds.Ai.Integrations/DependencyInjectionExtensions.cs`
- simulator host startup files if producer lives outside existing modules.

Task 3 should ensure simulator/web/CLI paths resolve the mock producer and queue publisher cleanly.

### 8. AppHost and local configuration seams

Review and likely modify:

- `backend/Kds.Ai.AppHost/AppHostExtensions.cs`
- `backend/Kds.Ai.AppHost/AppHost.cs`

Local development must be able to configure:

- queue URL or mock queue target,
- any producer enablement flag,
- source account id defaults for simulator traffic where needed.

### 9. Tests

Add or modify tests under:

- `backend/tests/Kds.Ai.Integrations.Tests/KdsCdc/`
- `backend/tests/Kds.Ai.Warehouse.Tests/` if simulator publisher abstractions remain there
- simulator test projects if producer translation behavior is verified at simulator boundary

## Required Behavior

### Contract source of truth

`docs/features/ETL/kds-cdc/mock-cdc-sqs-contracts.md` is authoritative for mock payload shape.

Task 3 must not invent a second mock envelope format.

### One row -> one message

Mock producer must emit one SQS FIFO message per changed source row.

This applies to:

- `orders`
- `items`
- `item_bumps`
- `customers` when separate generator path exists

Producer must not batch multiple changed rows into one envelope body.

### Required queue metadata

Producer must set SQS FIFO metadata exactly per contract intent:

- `MessageGroupId = {source_account_id}:{store_guid}`
- `MessageDeduplicationId = {entity}:{primary_key}:{sequence}`
- body = UTF-8 JSON envelope

Task 3 should freeze exact string construction rules in code-facing spec so worker and producer tests agree.

### Envelope fields

Serialized envelope must include:

- `contract_version`
- `event_id`
- `source_type`
- `source_account_id`
- `store_guid`
- `entity`
- `operation`
- `occurred_at_utc`
- `primary_key`
- `sequence`
- `flags`
- `payload`

Field values must satisfy documented constraints:

- `contract_version = kds-cdc-mock/v1`
- `source_type = kds_cdc`
- `flags.is_snapshot = false`
- `primary_key.guid` must equal payload `guid`
- `sequence.stream_key` must equal FIFO group key

### Entity coverage

Mock producer must support the four documented entities, but implementation may phase them as follows:

1. `orders`, `items`, `item_bumps` from current simulator row flow are required in first working path,
2. `customers` contract model is required now even if event generation uses separate fixture/helper path initially.

Task 3 should explicitly document that missing first-class simulator customer generation is known and acceptable for initial mock phase.

### Source-shaped payload parity

Payloads must remain source-shaped and aligned with current KDS entity fields.

Use current warehouse entity classes and mapper behavior as parity reference, especially:

- `Order`
- `Item`
- `ItemBump`
- `Customer`
- `SyntheticKdsSourceToStagingMapper`

Do not reshape payloads into curated or staging-specific destination DTOs at producer boundary.

### Sequence allocation

Mock phase ordering rule is explicit body sequence plus FIFO ordering.

Important ordering scope:

- mock sequence is scoped to `source_account_id + store_guid`, matching FIFO `MessageGroupId`,
- this proves per-store stream order only,
- producer tests must not claim total cross-store global ordering unless a later contract changes message grouping or adds a true global source coordinate.

Task 3 should define sequence generation per `source_account_id + store_guid` stream:

- monotonic increasing `sequence.stream_position`
- stable `sequence.stream_key`
- `sequence.source_timestamp_utc` derived from event occurrence time

If one simulator action changes multiple rows, producer must emit several ordered messages using recommended ordering from the contract doc.

### Mock source metadata profile

The mock envelope remains intentionally lighter than production Debezium payloads.

- `entity` identifies the source table class for mock traffic.
- `sequence` is the mock coordinate surrogate consumed by Task 2.
- Real Debezium database/schema/table/offset metadata is not required in `kds-cdc-mock/v1` unless the contract document is updated.
- If adding an optional `source` object with database/schema/table placeholders is cheap and useful for forward compatibility, it must remain optional for `kds-cdc-mock/v1` consumers.

### Source account defaults and mapping seed

Current simulator publish requests do not contain `source_account_id`. Task 3 must provide deterministic local/dev configuration for producer source account identity.

Required behavior:

- source account id is configured explicitly, not inferred from tenant id,
- AppHost/local options expose a default source account id for simulator traffic,
- Task 4 mapping seed/config can map that `source_account_id + store_guid` to platform company/tenant.

### Message size protection

Producer must check serialized SQS body size before publish when feasible.

- Oversize mock messages should fail with clear diagnostics.
- Producer must not silently truncate payloads to fit SQS limits.

### Operation semantics

Mock producer may use `upsert` for create/update in first version.

Delete behavior rules:

- physical deletes not required,
- soft delete may be emitted as `operation = delete` or `operation = upsert` plus `flags.is_deleted = true`,
- consumer must still rely on `is_deleted` semantics.

Task 3 should choose one default producer behavior and keep it consistent.

### Runtime direction

Mock mode must still exercise production-like runtime direction:

- simulator or helper generates traffic,
- traffic is published to SQS FIFO,
- KDS CDC worker consumes from SQS,
- worker owns intake, ordering validation, batching, and later acknowledgement.

Task 3 must not introduce an in-memory consumer shortcut between producer and worker.

### Separation from synthetic direct warehouse publish

Production-aligned mock CDC path must not identify emitted events as synthetic feed batches.

This task should stop short of deleting old synthetic publisher code unless user asks, but it must clearly position that path as legacy/bootstrap behavior rather than CDC mock runtime.

## Proposed Implementation Shape

Recommended translation flow:

```text
OrderSimulator / simulator helper
  -> create source rows as today
  -> translate each changed row into KdsCdcMockEnvelope
  -> allocate stream sequence per source_account_id + store_guid
  -> publish ordered FIFO messages to SQS
  -> KDS CDC worker from Task 2 consumes same messages
```

Recommended producer contract shape:

```csharp
public interface IKdsCdcMockProducer
{
    Task PublishAsync(KdsCdcMockPublishRequest request, CancellationToken ct = default);
}
```

Recommended request shape:

```text
KdsCdcMockPublishRequest
  SourceAccountId
  StoreGuid
  OccurredAtUtc
  Orders
  Items
  ItemBumps
  Customers
```

Recommended sequence allocator seam:

```csharp
public interface IKdsCdcSequenceAllocator
{
    ValueTask<long> GetNextAsync(string sourceAccountId, string storeGuid, CancellationToken ct = default);
}
```

Recommended outbound message shape:

```text
KdsCdcOutboundMessage
  MessageGroupId
  MessageDeduplicationId
  BodyJson
```

Important notes:

- prefer new producer abstraction over renaming existing synthetic publisher abstraction,
- keep serialization deterministic for testability,
- keep contract version centralized in one constant,
- preserve room to swap mock `sequence` with real Debezium coordinates later without changing payload row shapes.

## Testing

### Unit tests

Add tests for:

- envelope serialization matches documented field names and required top-level structure,
- `contract_version` serializes as `kds-cdc-mock/v1`,
- `source_type` serializes as `kds_cdc`,
- `MessageGroupId` equals `{source_account_id}:{store_guid}`,
- `MessageDeduplicationId` equals `{entity}:{primary_key}:{sequence}`,
- `primary_key.guid` equals payload `guid` for each entity,
- sequence allocator increments monotonically per store stream,
- order create flow emits recommended order: `orders`, then `items`, then `item_bumps`,
- completion/update flow emits recommended order when multiple row updates occur,
- soft delete events preserve `is_deleted` semantics,
- customer contract model can serialize valid example payload even if simulator does not yet generate customers.

### Integration/composition tests

Add tests proving:

- simulator-facing code resolves mock CDC producer from DI,
- publishing simulator change set results in expected number of outbound FIFO messages,
- no mock CDC publish path calls synthetic direct warehouse staging writer in the new SQS mode,
- worker-facing consumer tests can deserialize producer-generated envelopes without contract mismatch.

### Not required in task 3

This task does not require:

- live AWS SQS publish tests,
- real Debezium offset tests,
- customer simulator generation parity,
- retirement of old synthetic publisher runtime,
- downstream ETL stage tests beyond contract compatibility with Task 2 intake.

## Acceptance Criteria

- [ ] Strongly typed mock CDC envelope contract exists and matches `mock-cdc-sqs-contracts.md`
- [ ] Mock producer emits one SQS FIFO message per changed row
- [ ] Producer publishes through SQS-facing abstraction, not direct in-memory worker shortcut
- [ ] `MessageGroupId` uses `{source_account_id}:{store_guid}`
- [ ] `MessageDeduplicationId` uses deterministic `{entity}:{primary_key}:{sequence}` form
- [ ] Envelope `source_type` is `kds_cdc`
- [ ] Envelope `contract_version` is `kds-cdc-mock/v1`
- [ ] Envelope `primary_key.guid` matches payload `guid`
- [ ] Producer supports `orders`, `items`, and `item_bumps` from current simulator flow
- [ ] `customers` contract is defined now, even if produced through separate mock path initially
- [ ] Sequence values increase monotonically per store stream
- [ ] Mock ordering scope is documented as per `source_account_id + store_guid` stream, not cross-store global order
- [ ] Source account id defaults for simulator traffic are explicit and compatible with Task 4 mapping seed/config
- [ ] Producer fails clearly on oversized SQS message bodies rather than truncating
- [ ] Recommended multi-row ordering rules are documented and test-covered
- [ ] Simulator or helper traffic enters through SQS path rather than direct synthetic warehouse publish path
- [ ] Tests cover contract serialization, sequence generation, FIFO metadata, and simulator-to-envelope translation

## Notes

- Follow `docs/features/ETL/kds-cdc/mock-cdc-sqs-contracts.md` as exact contract source of truth.
- Use current simulator entity generation as payload source, not as runtime transport model.
- Synthetic warehouse publisher and mapper are reference material for field parity only; they are not target runtime path for CDC.
- Keep mock-specific names explicit so future real Debezium contract can coexist during migration.
- Prefer additive producer seam over destructive replacement of current simulator publishing code unless later user request says to remove old path.