Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)

Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Implementation Specs (157)

Detailed engineering and feature implementation specifications.

implementation-specs/specs/EtlSourceAdapterFramework/TASK3.md 4,899 bytes · Updated Oct 23, 2026 10:44 AM EDT

Raw File

TASK 3: Introduce Pipeline Runner And ETL Source Adapter Contracts

Summary

Introduce the core runtime contracts that separate pipeline mechanics from source-specific stage composition.

Implementation Status

Status: implemented.

Verified in code:

shared runtime contracts exist in backend/Kds.Ai.Integrations/Etl/IEtlPipelineRunner.cs, IEtlSourceAdapter.cs, and IEtlStage.cs,
shared runner exists in backend/Kds.Ai.Integrations/Etl/EtlPipelineRunner.cs,
stage context and result passing exist in EtlStageContext.cs and EtlStageResult.cs,
backend/Kds.Ai.ApiService/Jobs/BrinkPosEtlJobs.cs now resolves the Brink adapter and invokes IEtlPipelineRunner,
stage ordering and upstream-result flow are covered in backend/tests/Kds.Ai.Integrations.Tests/Etl/EtlPipelineRunnerTests.cs.

This task should add the minimal framework-facing contracts for:

pipeline runner,
ETL stage,
ETL source adapter.

The goal is to replace hardcoded source orchestration with a generic pipeline runtime seam without fully rewriting all stage internals in the same step.

Dependencies

Files to Modify

1. Shared runtime contracts

create or modify framework-oriented files under a backend location such as:
backend/Kds.Ai.Etl/
or backend/Kds.Ai.Integrations.Abstractions/

2. Orchestration entrypoints

backend/Kds.Ai.ApiService/Jobs/BrinkPosEtlJobs.cs
optionally related endpoint or dispatch files if orchestration wiring changes

3. Dependency injection registration

DI modules or service registration files touched by ETL runtime composition

4. Tests

add or extend tests under:
backend/tests/Kds.Ai.ApiService.Tests/
backend/tests/Kds.Ai.Integrations.Tests/

Required Behavior

Pipeline runner contract

Introduce a shared runtime contract capable of executing stages in order and passing upstream state between stages.

ETL stage contract

Introduce a stage contract that makes stage identity explicit and allows stages to run under a shared pipeline runtime.

ETL source adapter contract

Introduce a source adapter contract that owns source stage composition.

The adapter should expose the stages that make up its pipeline.

Minimal migration rule

This task should create the framework seam first.

It does not need to fully migrate Brink stage internals yet, but it must make the new orchestration boundary real and usable.

Proposed Contracts

Suggested minimal shared runtime contracts:

csharp

public interface IEtlPipelineRunner
{
    Task<EtlPipelineResult> RunAsync(EtlPipelineRequest request, CancellationToken ct = default);
}

public interface IEtlSourceAdapter
{
    EtlSourceDescriptor Descriptor { get; }
    IReadOnlyList<IEtlStage> GetStages();
}

public interface IEtlStage
{
    string StageName { get; }
    Task<EtlStageResult> ExecuteAsync(EtlStageContext context, CancellationToken ct = default);
}

public sealed record EtlPipelineRequest(
    string CompanyId,
    string TenantId,
    DateOnly? BusinessDate,
    string Trigger,
    bool ForceReset,
    string? RequestedBy,
    IEtlSourceAdapter Adapter);

public sealed record EtlStageContext(
    EtlPipelineRequest Pipeline,
    IReadOnlyDictionary<string, EtlStageResult> CompletedStages);

The exact EtlStageContext shape may differ. What matters is that a stage can read shared pipeline inputs and prior stage results without hardcoded method signatures between every pair of stages.

Proposed Execution Shape

Expected pipeline runner flow:

receive a PipelineRequest with source adapter and trigger context,
iterate stages from adapter.GetStages() in declared order,
build a stage context that contains pipeline input and prior results,
execute stage,
store result by stage name,
return a final pipeline result with per-stage outputs.

Brink Migration Shape For This Task

At the end of this task, BrinkPosEtlJobs should be able to do something conceptually equivalent to: