Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)

Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Implementation Specs (157)

Detailed engineering and feature implementation specifications.

implementation-specs/specs/KdsCdcDurableRelationRouting/TASK3.md 4,389 bytes · Updated Oct 23, 2026 10:44 AM EDT

Raw File

TASK 3: Add Landing RAW And Durable Promotion State

Summary

Move valid CDC durability ahead of tenant resolution and add durable landing-to-tenant promotion state so unresolved children no longer depend on SQS receipt visibility for eventual success.

Current Codebase Findings

Repository analysis shows:

current branch still mostly writes raw data after tenant resolution succeeds,
unresolved child rows remain in SQS retry/visibility loop,
worker acknowledgment is still coupled too tightly to resolved handoff behavior,
branch docs already identify landing RAW plus tenant RAW promotion as target architecture.

Dependencies

TASK 1
TASK 2

Source References

../KdsCdcDebeziumIntegration/TASK3.md
../../docs/features/ETL/kds-cdc/branch-direction-and-flow.md#7-raw-capture-and-handoff
../../docs/features/ETL/kds-cdc/debezium-payload-compatibility-notes.md

Scope From Breakdown

This task covers:

landing RAW write before tenant promotion,
durable promotion-state persistence,
ack behavior tied to landing/quarantine durability,
idempotent promotion from landing RAW to tenant RAW.

Files to Create

landing/promotion contracts under backend/Kds.Ai.Integrations/KdsCdc/ or .../Landing/
promotion-state entities/configuration/repositories under backend/Kds.Ai.Data/
landing RAW metadata contracts if new raw record shape or object family is needed

Files to Modify

backend/Kds.Ai.ApiService/Jobs/KdsCdc/KdsCdcWorkerService.cs
backend/Kds.Ai.Integrations/KdsCdc/KdsCdcExtractJobs.cs
backend/Kds.Ai.Integrations/KdsCdc/Raw/KdsCdcRawArtifactWriter.cs
backend/Kds.Ai.Integrations/KdsCdc/KdsCdcIntakeSupport.cs
any checkpoint/details models affected by landing/promotion state
tests under backend/tests/Kds.Ai.Integrations.Tests/KdsCdc/ and backend/tests/Kds.Ai.Data.Tests/

Required Behavior

Landing RAW first

For valid supported Debezium payloads:

parse and validate message
write landing RAW durably
write promotion-state row durably
acknowledge SQS
resolve tenant and promote from durable state

Promotion-state durability

Promotion state must record enough information to continue without SQS.

Minimum fields:

stable Debezium identity
landing RAW object key
source database/table
operation
primary key
source coordinates
comparable ordering token when available
current promotion status
retry reason
retry count
last attempt timestamp
resolved tenant/store/company when known
tenant RAW object key when promoted

Promotion status model

Need explicit statuses at minimum:

pending
tenant_unresolved
promoted
quarantined
failed

Ack behavior

Required ack behavior:

landing RAW write fails: do not acknowledge SQS
quarantine durability fails: do not acknowledge SQS
landing RAW and promotion-state write succeed: acknowledge SQS even if tenant still unresolved
tenant promotion transient failure after durable landing: do not rely on SQS for retry

Promotion idempotency

If same Debezium message is received again after landing was already persisted:

do not create duplicate logical promotion state,
do not create duplicate tenant RAW logical event,
converge on existing landing/promotion record by stable Debezium identity.

Testing

Add tests for:

landing RAW written before tenant promotion,
acknowledged after landing success for unresolved child,
no acknowledge when landing write fails,
no acknowledge when durable quarantine write fails,
duplicate redelivery converges on same promotion state,
promotion creates tenant RAW only once,
unsupported or malformed payload still uses durable quarantine path.

Acceptance Criteria

[ ] Landing RAW becomes first durable boundary for valid messages
[ ] SQS ack depends on landing/quarantine durability, not immediate tenant resolution success
[ ] Promotion-state persistence exists and is durable
[ ] Unresolved child rows leave SQS hot path after landing success
[ ] Promotion is idempotent across SQS redelivery
[ ] Tenant RAW remains explicit downstream boundary
[ ] Tests cover landing-first, ack behavior, and idempotent promotion semantics

Notes

Do not redesign downstream ETL stages after tenant RAW here.
Keep unsupported/malformed payload quarantine behavior explicit and durable.