Spensara Spensara Knowledge Base & Documentation Repository
System Overview Brink & Borg Master Index (722 Docs) Spensara Platform Project Tracker

Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)
Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Categories

Implementation Specs (157)

Detailed engineering and feature implementation specifications.
implementation-specs/specs/KdsCdcReplayExecution/TASK2.md 5,098 bytes · Updated Oct 23, 2026 10:44 AM EDT
Raw File

TASK 2: Implement Replay Coordinator And Downstream Stage Execution Paths

Summary

Implement the execution engine that turns a resolved replay plan into real stage runs. This task closes the main functional gap identified by branch audit: replay metadata exists, but replay execution does not.

Current Codebase Findings

  • KdsCdcPipelineLauncher currently launches KDS CDC through EtlPipelineRunner and always starts from extract_raw.
  • EtlPipelineRunner supports StopAfterStage, but not mid-pipeline start.
  • KDS CDC stage wrappers depend on prior stage outputs being present in the current pipeline context.
  • KDS CDC stage job interfaces already allow direct execution of downstream stages via KdsCdcStageJobInput(UpstreamBatchId, ...).
  • This means current framework seams naturally support:
  • replay from raw batch -> call normalize_load, then downstream stages
  • replay from load batch -> call load_to_staging, then downstream stages
  • replay from curated-pre-export batch -> call curated_to_s3

Current repo constraints this task must respect:

  1. Replay must create normal rerun batches and preserve lifecycle semantics.
  2. Replay should not bypass stage job implementations or directly mutate batch tables.
  3. Any generic ETL runner change must be justified by an actual KDS CDC replay need.

Dependencies

  • TASK 1
  • ../KdsCdcIngestion/TASK8.md

Source References

  • ../../docs/features/ETL/kds-cdc/index.md#checkpointing-and-replay
  • ../../docs/features/ETL/kds-cdc/index.md#supported-corner-cases
  • ../KdsCdcIngestion/TASK8.md

Files to Create

  1. Replay execution seam under backend/Kds.Ai.Integrations/KdsCdc/Replay/:
  2. IKdsCdcReplayCoordinator
  3. KdsCdcReplayCoordinator
  4. optional KdsCdcReplayExecutionResult

Responsibilities:

  • execute replay plans,
  • call the correct KDS CDC stage jobs in order,
  • preserve bounded execution semantics,
  • report completed rerun batch ids.
  1. Optional small internal helpers if needed:
  2. KdsCdcReplayStageExecutor
  3. KdsCdcReplayPlanExtensions

Guidance:

  • prefer keeping logic in one coordinator unless helpers clearly reduce duplication.

Files to Modify

  • backend/Kds.Ai.Integrations/DependencyInjectionExtensions.cs
  • backend/Kds.Ai.Integrations/KdsCdc/KdsCdcPipelineLauncher.cs only if raw replay benefits from a shared launch helper
  • KDS CDC stage-job or support files only where additive replay-specific diagnostics are concretely needed

Required Behavior

Execution Paths

Replay coordinator must support at least these execution paths:

  • raw batch replay:
  • reopen durable raw stage point
  • run normalize_load -> load_to_staging -> staging_to_curated -> curated_to_s3
  • load artifact retry:
  • start at load_to_staging
  • continue staging_to_curated -> curated_to_s3
  • curated retry:
  • start at staging_to_curated or curated_to_s3 depending on resolved durable state
  • tenant/entity replay:
  • execute the bounded set of raw batches in deterministic order
  • coordinate replay:
  • execute the bounded set of raw batches covering the requested coordinate range in deterministic order

Ordering And Boundedness

  • Multi-batch replay must run in durable coordinate/order sequence.
  • Coordinator must reject overlapping or duplicate candidate execution that would violate deterministic order.
  • Replay must not silently widen the operator-requested bound.

Lifecycle And Checkpoint Semantics

  • Replay runs must use ordinary stage jobs so batch completion/failure, checkpoint advancement, and replay-index updates continue through existing code.
  • Failed replay must stop later stages for the affected execution path.
  • Successful replay reruns must rely on existing IsCurrent and ResetOfBatchId semantics rather than custom replacement logic.

No SQS Dependency

  • Replay coordinator must not read from IKdsCdcQueueClient.
  • Replay after acknowledgement must succeed even if original transport messages no longer exist.

Testing

  • Integration tests for:
  • raw replay invokes stages from normalize_load onward in order,
  • load retry starts at load_to_staging,
  • curated retry starts at correct downstream stage,
  • multi-batch coordinate replay runs candidates in deterministic order,
  • replay failure stops later stages,
  • successful replay updates current batch semantics through existing lifecycle services,
  • replay does not touch queue client.

Acceptance Criteria

  • Replay execution exists as a concrete coordinator, not only metadata lookup.
  • Raw, load, and curated replay levels each have a real execution path.
  • Replay uses durable artifacts and existing stage jobs rather than SQS or direct table mutation.
  • Replay preserves existing batch lifecycle, lineage, and checkpoint behavior.

Notes

  • Prefer KDS CDC-specific orchestration over generic EtlPipelineRunner redesign unless a real blocker appears.
  • If a generic mid-pipeline runner seam becomes necessary, keep it minimal and prove it with KDS CDC replay usage immediately.