# TASK 4: Standardize Source Descriptor, Diagnostics Envelopes, And Catalog Contracts

## Summary

Standardize the metadata contracts that source adapters use to plug into the ETL framework.

## Implementation Status

Status: implemented.

Implemented:

- shared source descriptor exists in `backend/Kds.Ai.Integrations/Etl/EtlSourceDescriptor.cs`,
- shared diagnostics envelope type exists in `backend/Kds.Ai.Integrations/Etl/StageDiagnosticsEnvelope.cs`,
- shared catalog contracts exist in `IExtractOperationCatalog.cs`, `INormalizationContractCatalog.cs`, `IStagingTableCatalog.cs`, and `ICuratedTargetCatalog.cs`,
- Brink catalogs expose shared contract shapes in `BrinkPosEndpointCatalog.cs`, `BrinkPosNormalizationCatalog.cs`, `BrinkPosStagingTableCatalog.cs`, and `BrinkPosCuratedTargetCatalog.cs`,
- catalog consistency tests exist in `backend/tests/Kds.Ai.Integrations.Tests/BrinkPos/BrinkPosEtlCatalogTests.cs`.

Completed by:

- persisted stage details now use `StageDiagnosticsEnvelope<TDetails>` in `BrinkPosNormalizeLoadJobs.cs`, `BrinkPosLoadToStagingJobs.cs`, and `BrinkPosStagingToCuratedJobs.cs`,
- shared serialization helpers exist in `backend/Kds.Ai.Integrations/Etl/StageDiagnosticsJson.cs`,
- upstream stage-detail readers now support both the new envelope shape and legacy plain detail payloads so existing persisted data is still readable during transition.

This task should define the shared concept of:

- source descriptor,
- common stage diagnostics envelope,
- extract catalog,
- normalization catalog,
- staging catalog,
- curated target catalog.

The framework should own the structure of these contracts, while adapters should continue to own their content.

## Dependencies

- [TASK 0](./TASK0.md)
- [TASK 1](./TASK1.md)
- [TASK 2](./TASK2.md)

## Files to Modify

### 1. Shared metadata contracts

- framework-oriented files under a backend location such as:
  - `backend/Kds.Ai.Etl/`
  - or `backend/Kds.Ai.Integrations.Abstractions/`

### 2. Brink source metadata classes

- `backend/Kds.Ai.Integrations/BrinkPos/BrinkPosEndpointCatalog.cs`
- `backend/Kds.Ai.Integrations/BrinkPos/BrinkPosNormalizationCatalog.cs`
- `backend/Kds.Ai.Integrations/BrinkPos/BrinkPosStagingTableCatalog.cs`
- `backend/Kds.Ai.Integrations/BrinkPos/BrinkPosCuratedTargetCatalog.cs`
- `backend/Kds.Ai.Integrations/BrinkPos/Models/BrinkPosEtlStageConstants.cs`

### 3. Tests

- add or extend tests under:
  - `backend/tests/Kds.Ai.Integrations.Tests/BrinkPos/`

## Required Behavior

### Shared source descriptor

Create a shared source descriptor that represents source identity without hardcoding Brink values into framework helpers.

### Shared diagnostics envelope

Create a common stage diagnostics envelope that can carry:

- stage identity,
- source identity,
- tenant/company/business date,
- upstream batch id,
- artifact references,
- status,
- timestamps,
- stage-specific nested diagnostics when needed.

### Shared catalog contracts

Create shared interfaces or contracts for:

- extract operation catalogs,
- normalization contract catalogs,
- staging table catalogs,
- curated target catalogs.

### Adapter-owned content rule

Brink metadata content must remain in the Brink adapter. This task should only change ownership of the contract shape, not centralize source definitions.

## Proposed Contracts

Suggested shared metadata contracts:

```csharp
public sealed record EtlSourceDescriptor(
    string SourceType,
    IngestionSourceType IngestionSourceType,
    IngestionMode DefaultIngestionMode);

public sealed record StageDetailsEnvelope<TDetails>(
    EtlSourceDescriptor Source,
    string CompanyId,
    string TenantId,
    DateOnly BusinessDate,
    string PipelineStage,
    Guid BatchId,
    Guid? UpstreamBatchId,
    string Status,
    DateTimeOffset StartedAt,
    DateTimeOffset CompletedAt,
    TDetails Details);

public interface IExtractOperationCatalog
{
    IReadOnlyList<ExtractOperationDefinition> GetOperations();
}

public interface INormalizationContractCatalog
{
    IReadOnlyList<NormalizationContractDefinition> GetContracts();
}

public interface IStagingTableCatalog
{
    IReadOnlyList<StagingTableDefinition> GetTables();
}

public interface ICuratedTargetCatalog
{
    IReadOnlyList<CuratedTargetDefinition> GetTargets();
}
```

Suggested shared definition records:

```csharp
public sealed record ExtractOperationDefinition(
    string StreamName,
    string OperationName,
    bool IsRequired);

public sealed record NormalizationContractDefinition(
    string ContractName,
    string SchemaVersion);

public sealed record StagingTableDefinition(
    string ContractName,
    string TableName,
    string SchemaVersion,
    string DeleteSql,
    string CopySqlTemplateId);
```

## Proposed Execution Shape

After this task:

1. framework helpers consume `EtlSourceDescriptor` instead of source constants,
2. stage details are serialized through a shared outer envelope,
3. Brink catalogs remain in `Kds.Ai.Integrations.BrinkPos`,
4. Brink catalogs expose their entries through shared catalog contracts,
5. tests verify that Brink catalog content is structurally complete and internally consistent.

## Catalog Consistency Guidance

At minimum, tests in this task should verify:

- every staging table definition references an existing normalization contract,
- every curated target dependency references declared staging tables,
- every active extract operation required by Brink has matching downstream catalog ownership where applicable.

## Acceptance Criteria

- source descriptor is shared and reusable,
- diagnostics envelope is shared and reusable,
- shared catalog contracts exist,
- Brink catalogs implement or align to the shared contracts,
- tests cover catalog completeness or catalog-shape consistency where practical.

## Notes

- Keep the framework contracts minimal.
- Do not use this task to invent speculative metadata required by sources that do not exist yet.
- Prefer shared outer envelopes plus source-specific nested detail records over one giant universal diagnostic schema.