Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)

Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Implementation Specs (157)

Detailed engineering and feature implementation specifications.

implementation-specs/specs/KdsCdcDurableRelationRouting/TASK4.md 3,423 bytes · Updated Oct 23, 2026 10:44 AM EDT

Raw File

TASK 4: Add Promotion Retry Worker And Observability

Summary

Add durable retry execution for unresolved landing records and surface the diagnostics needed to operate relation-based tenant routing safely.

Current Codebase Findings

Repository/runtime analysis shows:

unresolved child rows currently cycle through SQS visibility and duplicate detection,
FIFO head-of-line blocking remains visible in logs,
there is no durable retry queue/state independent of SQS,
operational logs can show resolved into 0 tenant slices, but do not yet expose relation miss backlog as first-class durable state.

Dependencies

TASK 1
TASK 2
TASK 3

Source References

../KdsCdcIngestion/TASK8.md
../KdsCdcIngestion/TASK9.md
../../docs/features/ETL/kds-cdc/branch-direction-and-flow.md

Scope From Breakdown

This task covers:

retry worker/promotion coordinator,
retry scheduling fields,
relation-resolution diagnostics,
metrics and logging for unresolved backlog and stale writes.

Files to Create

retry/promotion coordinator under backend/Kds.Ai.Integrations/KdsCdc/
optional hosted service under API service jobs if retry should run continuously
supporting metrics or telemetry helpers if needed

Files to Modify

worker or pipeline launch seams consuming promotion state
promotion-state persistence models if retry metadata fields are missing
tests under backend/tests/Kds.Ai.Integrations.Tests/KdsCdc/

Required Behavior

Durable retry loop

Need background retry flow that:

scans landing/promotion rows in retryable states,
retries tenant resolution using durable relation indexes,
promotes rows when relation becomes available,
records retry count and timestamps,
keeps final quarantine/failure separate from retryable unresolved state.

Retry metadata

Need at minimum:

RetryCount
LastAttemptAtUtc
NextAttemptAtUtc
RetryReason

Diagnostics and telemetry

Need visibility into:

relation lookup hits/misses by entity,
number of tenant_unresolved records,
age of oldest unresolved record,
relation writes,
stale relation writes ignored,
relation store changes on newer writes,
promotion successes/failures.

Logging behavior

Logs should distinguish:

missing direct store_guid,
missing order relation,
missing item relation,
transient promotion failure,
stale relation write ignored,
duplicate landing/promotion convergence.

Testing

Add tests for:

unresolved landing row retries later and promotes when relation appears,
retry metadata updates across attempts,
promoted row is not promoted twice,
stale relation write does not corrupt promotion retry result,
diagnostics surface relation-miss reason by entity,
FIFO/SQS is no longer required for successful delayed promotion.

Acceptance Criteria

[ ] Retryable unresolved rows can be retried from durable state without SQS
[ ] Promotion succeeds automatically after relation appears
[ ] Retry metadata is persisted and updated
[ ] Logs and metrics clearly distinguish relation misses from malformed data
[ ] Stale relation-write and store-change anomalies are visible
[ ] Tests cover delayed resolution and retry promotion behavior

Notes

This task is about operational durability and visibility, not retention policy.
Avoid introducing noisy log spam without structured reason fields.