Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)

Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Implementation Specs (157)

Detailed engineering and feature implementation specifications.

implementation-specs/specs/EtlSourceAdapterFramework/TASK4.md 6,070 bytes · Updated Oct 23, 2026 10:44 AM EDT

Raw File

TASK 4: Standardize Source Descriptor, Diagnostics Envelopes, And Catalog Contracts

Summary

Standardize the metadata contracts that source adapters use to plug into the ETL framework.

Implementation Status

Status: implemented.

Implemented:

shared source descriptor exists in backend/Kds.Ai.Integrations/Etl/EtlSourceDescriptor.cs,
shared diagnostics envelope type exists in backend/Kds.Ai.Integrations/Etl/StageDiagnosticsEnvelope.cs,
shared catalog contracts exist in IExtractOperationCatalog.cs, INormalizationContractCatalog.cs, IStagingTableCatalog.cs, and ICuratedTargetCatalog.cs,
Brink catalogs expose shared contract shapes in BrinkPosEndpointCatalog.cs, BrinkPosNormalizationCatalog.cs, BrinkPosStagingTableCatalog.cs, and BrinkPosCuratedTargetCatalog.cs,
catalog consistency tests exist in backend/tests/Kds.Ai.Integrations.Tests/BrinkPos/BrinkPosEtlCatalogTests.cs.

Completed by:

persisted stage details now use StageDiagnosticsEnvelope<TDetails> in BrinkPosNormalizeLoadJobs.cs, BrinkPosLoadToStagingJobs.cs, and BrinkPosStagingToCuratedJobs.cs,
shared serialization helpers exist in backend/Kds.Ai.Integrations/Etl/StageDiagnosticsJson.cs,
upstream stage-detail readers now support both the new envelope shape and legacy plain detail payloads so existing persisted data is still readable during transition.

This task should define the shared concept of:

source descriptor,
common stage diagnostics envelope,
extract catalog,
normalization catalog,
staging catalog,
curated target catalog.

The framework should own the structure of these contracts, while adapters should continue to own their content.

Dependencies

Files to Modify

1. Shared metadata contracts

framework-oriented files under a backend location such as:
backend/Kds.Ai.Etl/
or backend/Kds.Ai.Integrations.Abstractions/

2. Brink source metadata classes

backend/Kds.Ai.Integrations/BrinkPos/BrinkPosEndpointCatalog.cs
backend/Kds.Ai.Integrations/BrinkPos/BrinkPosNormalizationCatalog.cs
backend/Kds.Ai.Integrations/BrinkPos/BrinkPosStagingTableCatalog.cs
backend/Kds.Ai.Integrations/BrinkPos/BrinkPosCuratedTargetCatalog.cs
backend/Kds.Ai.Integrations/BrinkPos/Models/BrinkPosEtlStageConstants.cs

3. Tests

add or extend tests under:
backend/tests/Kds.Ai.Integrations.Tests/BrinkPos/

Required Behavior

Shared source descriptor

Create a shared source descriptor that represents source identity without hardcoding Brink values into framework helpers.

Shared diagnostics envelope

Create a common stage diagnostics envelope that can carry:

stage identity,
source identity,
tenant/company/business date,
upstream batch id,
artifact references,
status,
timestamps,
stage-specific nested diagnostics when needed.

Shared catalog contracts

Create shared interfaces or contracts for:

extract operation catalogs,
normalization contract catalogs,
staging table catalogs,
curated target catalogs.

Adapter-owned content rule

Brink metadata content must remain in the Brink adapter. This task should only change ownership of the contract shape, not centralize source definitions.

Proposed Contracts

Suggested shared metadata contracts:

csharp

public sealed record EtlSourceDescriptor(
    string SourceType,
    IngestionSourceType IngestionSourceType,
    IngestionMode DefaultIngestionMode);

public sealed record StageDetailsEnvelope<TDetails>(
    EtlSourceDescriptor Source,
    string CompanyId,
    string TenantId,
    DateOnly BusinessDate,
    string PipelineStage,
    Guid BatchId,
    Guid? UpstreamBatchId,
    string Status,
    DateTimeOffset StartedAt,
    DateTimeOffset CompletedAt,
    TDetails Details);

public interface IExtractOperationCatalog
{
    IReadOnlyList<ExtractOperationDefinition> GetOperations();
}

public interface INormalizationContractCatalog
{
    IReadOnlyList<NormalizationContractDefinition> GetContracts();
}

public interface IStagingTableCatalog
{
    IReadOnlyList<StagingTableDefinition> GetTables();
}

public interface ICuratedTargetCatalog
{
    IReadOnlyList<CuratedTargetDefinition> GetTargets();
}

Suggested shared definition records: