# TASK 4: Add Promotion Retry Worker And Observability

## Summary

Add durable retry execution for unresolved landing records and surface the diagnostics needed to operate relation-based tenant routing safely.

## Current Codebase Findings

Repository/runtime analysis shows:

- unresolved child rows currently cycle through SQS visibility and duplicate detection,
- FIFO head-of-line blocking remains visible in logs,
- there is no durable retry queue/state independent of SQS,
- operational logs can show `resolved into 0 tenant slices`, but do not yet expose relation miss backlog as first-class durable state.

## Dependencies

- TASK 1
- TASK 2
- TASK 3

## Source References

- `../KdsCdcIngestion/TASK8.md`
- `../KdsCdcIngestion/TASK9.md`
- `../../docs/features/ETL/kds-cdc/branch-direction-and-flow.md`

## Scope From Breakdown

This task covers:

- retry worker/promotion coordinator,
- retry scheduling fields,
- relation-resolution diagnostics,
- metrics and logging for unresolved backlog and stale writes.

## Files to Create

- retry/promotion coordinator under `backend/Kds.Ai.Integrations/KdsCdc/`
- optional hosted service under API service jobs if retry should run continuously
- supporting metrics or telemetry helpers if needed

## Files to Modify

- worker or pipeline launch seams consuming promotion state
- promotion-state persistence models if retry metadata fields are missing
- tests under `backend/tests/Kds.Ai.Integrations.Tests/KdsCdc/`

## Required Behavior

### Durable retry loop

Need background retry flow that:

- scans landing/promotion rows in retryable states,
- retries tenant resolution using durable relation indexes,
- promotes rows when relation becomes available,
- records retry count and timestamps,
- keeps final quarantine/failure separate from retryable unresolved state.

### Retry metadata

Need at minimum:

- `RetryCount`
- `LastAttemptAtUtc`
- `NextAttemptAtUtc`
- `RetryReason`

### Diagnostics and telemetry

Need visibility into:

- relation lookup hits/misses by entity,
- number of `tenant_unresolved` records,
- age of oldest unresolved record,
- relation writes,
- stale relation writes ignored,
- relation store changes on newer writes,
- promotion successes/failures.

### Logging behavior

Logs should distinguish:

- missing direct `store_guid`,
- missing order relation,
- missing item relation,
- transient promotion failure,
- stale relation write ignored,
- duplicate landing/promotion convergence.

## Testing

Add tests for:

- unresolved landing row retries later and promotes when relation appears,
- retry metadata updates across attempts,
- promoted row is not promoted twice,
- stale relation write does not corrupt promotion retry result,
- diagnostics surface relation-miss reason by entity,
- FIFO/SQS is no longer required for successful delayed promotion.

## Acceptance Criteria

- [ ] Retryable unresolved rows can be retried from durable state without SQS
- [ ] Promotion succeeds automatically after relation appears
- [ ] Retry metadata is persisted and updated
- [ ] Logs and metrics clearly distinguish relation misses from malformed data
- [ ] Stale relation-write and store-change anomalies are visible
- [ ] Tests cover delayed resolution and retry promotion behavior

## Notes

- This task is about operational durability and visibility, not retention policy.
- Avoid introducing noisy log spam without structured reason fields.