Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)

Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Implementation Specs (157)

Detailed engineering and feature implementation specifications.

implementation-specs/specs/BrinkPosOrdersFirstEtlJob/TASK4.3.md 2,398 bytes · Updated Oct 23, 2026 10:44 AM EDT

Raw File

TASK 4.3: Implement Load Artifact Persistence And Normalize-Load Diagnostics

Summary

Implement the persistence boundary for the first normalize_load stage: write the generated Parquet contracts to S3 load/, record final artifact URIs, and publish safe stage diagnostics that later stages can consume.

Dependencies

TASK 4.2 - GetOrders relational decomposition and normalization catalog

Files to Create

1. Load artifact writer and diagnostics models

Create or finalize the component that persists emitted load artifacts and returns contract-level diagnostics.

Recommended logical service:

BrinkPosLoadArtifactWriter

Required diagnostics details:

contract name
schema version
row count
load/ URI inventory
timing
safe status summary

Files to Modify

2. Shared ETL storage load path

Modify the shared S3 storage path so Parquet artifacts are written into the standard load/ layout and can be referenced by downstream load_to_staging work.

3. Normalize-load batch details

Modify or finalize the normalize_load details_json shape so it preserves upstream raw references and produced load artifact inventory.

Required Behavior

generated Parquet contracts are written to S3 load/
final artifact URIs are recorded in batch details
if any required contract write fails, the endpoint fails and the full normalize_load batch fails
already written load artifacts may remain for diagnostics and replay
diagnostics must not store secrets
batch metadata must preserve enough detail for TASK5.md to resolve contract-to-table inputs

Testing

Unit tests

Add unit tests for:

load/ path generation and recording
contract inventory and row-count diagnostics
failure behavior when one required artifact write fails
diagnostics sanitization
downstream-readable batch details

Acceptance Criteria

[x] Parquet load artifacts are written to S3 load/
[x] Artifact URIs, schema versions, and row counts are recorded in diagnostics
[x] Write failures fail the endpoint and stage correctly
[x] Diagnostics stay secret-free and downstream-readable
[x] Unit tests cover persistence and diagnostics behavior

Notes

This subtask creates the first immutable load/ replay input for Redshift staging.
Keep storage ownership and stage diagnostics outside the Brink adapter.