Spensara Spensara Knowledge Base & Documentation Repository
System Overview Brink & Borg Master Index (722 Docs) Spensara Platform Project Tracker

Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)
Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Categories

Implementation Specs (157)

Detailed engineering and feature implementation specifications.
implementation-specs/specs/EtlSourceAdapterFramework/TASK1.md 4,728 bytes · Updated Oct 23, 2026 10:44 AM EDT
Raw File

TASK 1: Extract Generic Stage Failure And Checkpoint Support

Summary

Extract the repeated stage failure and checkpoint logic from the current Brink implementation into a generic shared helper or service.

Implementation Status

Status: implemented.

Verified in code:

  • shared helper exists in backend/Kds.Ai.Integrations/Etl/StageExecutionSupport.cs,
  • shared requests exist in StageCheckpointAdvanceRequest.cs and StageBatchFailureRequest.cs,
  • Brink downstream stages use IStageExecutionSupport in BrinkPosNormalizeLoadJobs.cs, BrinkPosLoadToStagingJobs.cs, and BrinkPosStagingToCuratedJobs.cs,
  • regression tests exist in backend/tests/Kds.Ai.Integrations.Tests/Etl/StageExecutionSupportTests.cs.

The current best candidate is BrinkPosStageExecutionSupport. This task should make it source-agnostic and reusable by later framework stages without changing stage-specific execution logic yet.

Dependencies

Files to Modify

1. Shared stage support

  • backend/Kds.Ai.Integrations/BrinkPos/BrinkPosStageExecutionSupport.cs
  • shared abstractions or new shared helpers under a framework-oriented backend location such as:
  • backend/Kds.Ai.Etl/
  • or backend/Kds.Ai.Integrations.Abstractions/
  • or another minimal shared location chosen in implementation

2. Current Brink stage callers

  • backend/Kds.Ai.Integrations/BrinkPos/BrinkPosNormalizeLoadJobs.cs
  • backend/Kds.Ai.Integrations/BrinkPos/BrinkPosLoadToStagingJobs.cs
  • backend/Kds.Ai.Integrations/BrinkPos/BrinkPosStagingToCuratedJobs.cs

3. Tests

  • add or extend tests under:
  • backend/tests/Kds.Ai.Integrations.Tests/
  • backend/tests/Kds.Ai.Data.Tests/

Required Behavior

Source-agnostic checkpoint support

The extracted helper must no longer hardcode Brink-specific source identity.

It must accept the source descriptor or equivalent source identity as input.

Source-agnostic failure support

The extracted helper must support writing failed packages and failing batches for any source.

It must preserve:

  • stage name,
  • tenant and business date,
  • original artifact reference,
  • exception type and message,
  • trigger and requester details when provided.

Minimal-abstraction rule

Do not create a large framework layer in this task.

This task should extract only the already repeated helper behavior and keep stage-specific orchestration in place.

Proposed Contracts

Suggested minimal shared contracts:

csharp
public interface IStageExecutionSupport
{
    Task AdvanceCheckpointAsync(StageCheckpointAdvanceRequest request, CancellationToken ct = default);
    Task FailStageBatchAsync(StageBatchFailureRequest request, CancellationToken ct = default);
}

public sealed record StageCheckpointAdvanceRequest(
    EtlSourceDescriptor Source,
    string TenantId,
    DateOnly BusinessDate,
    string PipelineStage,
    Guid BatchId,
    DateTimeOffset CompletedAt);

public sealed record StageBatchFailureRequest(
    EtlSourceDescriptor Source,
    Guid BatchId,
    string CompanyId,
    string TenantId,
    DateOnly BusinessDate,
    string PipelineStage,
    string Trigger,
    string? RequestedBy,
    DateTimeOffset StartedAt,
    Exception Exception,
    string OriginalArtifactReferenceUri,
    IReadOnlyDictionary<string, string> Metadata,
    object Diagnostics);

The important point is not exact naming. The important point is that source identity becomes input, not hardcoded implementation detail.

Proposed Execution Shape

Expected flow in a stage after this extraction:

  1. stage-specific code prepares metadata and diagnostics payload,
  2. stage-specific code calls shared AdvanceCheckpointAsync(...) on success,
  3. stage-specific code calls shared FailStageBatchAsync(...) on failure,
  4. the shared helper writes the failed package and fails the batch consistently for any source.

Migration Guidance

Apply the extraction in this order:

  1. introduce shared request records and interface,
  2. move logic from BrinkPosStageExecutionSupport into the shared implementation,
  3. replace the three current Brink downstream stage call sites,
  4. delete or reduce the Brink-specific helper to a compatibility wrapper only if needed temporarily.

Acceptance Criteria

  • repeated failure and checkpoint logic is extracted from the Brink-specific helper,
  • the new helper is source-agnostic,
  • existing Brink stages use the new helper,
  • regression tests cover checkpoint advancement and failure packaging behavior.

Notes

  • This task should not yet change how stages are sequenced.
  • This task should not yet introduce a full generic pipeline runner.
  • The helper should remain thin and orchestration-neutral.