# TASK 2: Implement Replay Coordinator And Downstream Stage Execution Paths

## Summary

Implement the execution engine that turns a resolved replay plan into real stage runs. This task closes the main functional gap identified by branch audit: replay metadata exists, but replay execution does not.

## Current Codebase Findings

- `KdsCdcPipelineLauncher` currently launches KDS CDC through `EtlPipelineRunner` and always starts from `extract_raw`.
- `EtlPipelineRunner` supports `StopAfterStage`, but not mid-pipeline start.
- KDS CDC stage wrappers depend on prior stage outputs being present in the current pipeline context.
- KDS CDC stage job interfaces already allow direct execution of downstream stages via `KdsCdcStageJobInput(UpstreamBatchId, ...)`.
- This means current framework seams naturally support:
  - replay from raw batch -> call `normalize_load`, then downstream stages
  - replay from load batch -> call `load_to_staging`, then downstream stages
  - replay from curated-pre-export batch -> call `curated_to_s3`

Current repo constraints this task must respect:

1. Replay must create normal rerun batches and preserve lifecycle semantics.
2. Replay should not bypass stage job implementations or directly mutate batch tables.
3. Any generic ETL runner change must be justified by an actual KDS CDC replay need.

## Dependencies

- [TASK 1](./TASK1.md)
- `../KdsCdcIngestion/TASK8.md`

## Source References

- `../../docs/features/ETL/kds-cdc/index.md#checkpointing-and-replay`
- `../../docs/features/ETL/kds-cdc/index.md#supported-corner-cases`
- `../KdsCdcIngestion/TASK8.md`

## Files to Create

1. Replay execution seam under `backend/Kds.Ai.Integrations/KdsCdc/Replay/`:
   - `IKdsCdcReplayCoordinator`
   - `KdsCdcReplayCoordinator`
   - optional `KdsCdcReplayExecutionResult`

   Responsibilities:
   - execute replay plans,
   - call the correct KDS CDC stage jobs in order,
   - preserve bounded execution semantics,
   - report completed rerun batch ids.

2. Optional small internal helpers if needed:
   - `KdsCdcReplayStageExecutor`
   - `KdsCdcReplayPlanExtensions`

   Guidance:
   - prefer keeping logic in one coordinator unless helpers clearly reduce duplication.

## Files to Modify

- `backend/Kds.Ai.Integrations/DependencyInjectionExtensions.cs`
- `backend/Kds.Ai.Integrations/KdsCdc/KdsCdcPipelineLauncher.cs` only if raw replay benefits from a shared launch helper
- KDS CDC stage-job or support files only where additive replay-specific diagnostics are concretely needed

## Required Behavior

### Execution Paths

Replay coordinator must support at least these execution paths:

- raw batch replay:
  - reopen durable raw stage point
  - run `normalize_load -> load_to_staging -> staging_to_curated -> curated_to_s3`
- load artifact retry:
  - start at `load_to_staging`
  - continue `staging_to_curated -> curated_to_s3`
- curated retry:
  - start at `staging_to_curated` or `curated_to_s3` depending on resolved durable state
- tenant/entity replay:
  - execute the bounded set of raw batches in deterministic order
- coordinate replay:
  - execute the bounded set of raw batches covering the requested coordinate range in deterministic order

### Ordering And Boundedness

- Multi-batch replay must run in durable coordinate/order sequence.
- Coordinator must reject overlapping or duplicate candidate execution that would violate deterministic order.
- Replay must not silently widen the operator-requested bound.

### Lifecycle And Checkpoint Semantics

- Replay runs must use ordinary stage jobs so batch completion/failure, checkpoint advancement, and replay-index updates continue through existing code.
- Failed replay must stop later stages for the affected execution path.
- Successful replay reruns must rely on existing `IsCurrent` and `ResetOfBatchId` semantics rather than custom replacement logic.

### No SQS Dependency

- Replay coordinator must not read from `IKdsCdcQueueClient`.
- Replay after acknowledgement must succeed even if original transport messages no longer exist.

## Testing

- Integration tests for:
  - raw replay invokes stages from `normalize_load` onward in order,
  - load retry starts at `load_to_staging`,
  - curated retry starts at correct downstream stage,
  - multi-batch coordinate replay runs candidates in deterministic order,
  - replay failure stops later stages,
  - successful replay updates current batch semantics through existing lifecycle services,
  - replay does not touch queue client.

## Acceptance Criteria

- Replay execution exists as a concrete coordinator, not only metadata lookup.
- Raw, load, and curated replay levels each have a real execution path.
- Replay uses durable artifacts and existing stage jobs rather than SQS or direct table mutation.
- Replay preserves existing batch lifecycle, lineage, and checkpoint behavior.

## Notes

- Prefer KDS CDC-specific orchestration over generic `EtlPipelineRunner` redesign unless a real blocker appears.
- If a generic mid-pipeline runner seam becomes necessary, keep it minimal and prove it with KDS CDC replay usage immediately.