Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)

Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Implementation Specs (157)

Detailed engineering and feature implementation specifications.

implementation-specs/specs/BrinkPosOrdersFirstEtlJob/TASK4.4.md 2,340 bytes · Updated Oct 23, 2026 10:44 AM EDT

Raw File

TASK 4.4: Add Unit Tests And Live E2E Validation For Normalize-Load

Summary

Add the verification layer for the first raw -> load ETL boundary. This subtask closes TASK 4 by proving both locally and against real integrations that normalize_load works for Brink Sales2.GetOrders.

Dependencies

TASK 4.1 - normalize-load orchestration and upstream raw-batch resolution
TASK 4.2 - GetOrders relational decomposition and normalization catalog
TASK 4.3 - load artifact persistence and normalize-load diagnostics

Files to Create

1. Normalize-load verification suites

Create or finalize the test suites needed for:

unit coverage of orchestration and decomposition rules
unit coverage of load artifact persistence and diagnostics
live E2E coverage of real extract_raw -> normalize_load execution

Required Behavior

unit tests must cover the final normalize_load flow assembled from TASK 4.1-TASK 4.3
live E2E must verify the real integration path whenever Aspire-managed secrets are available
the live test should prove that a real persisted raw artifact can be transformed into Parquet load/ output and recorded in batch metadata

Testing

Unit tests

Ensure the final unit suite covers:

upstream raw-batch selection
normalization-context construction
contract decomposition and ETL metadata envelope
load artifact persistence and diagnostics
failure behavior for required contract writes

Live E2E

Add a live E2E test that verifies at least:

one real extract_raw batch can serve as the upstream source
normalize_load reads the persisted raw artifact instead of re-calling Brink
expected Parquet load/ artifacts are written to S3
batch metadata points to the upstream raw batch and produced load/ artifacts

Acceptance Criteria

[x] Unit-test coverage exists for the assembled normalize_load flow
[x] Live E2E exists for real persisted-raw to Parquet normalization
[x] Live E2E verifies artifact inventory and metadata linkage
[x] TASK 4 is verifiable as a completed ETL stage

Notes

This subtask is mandatory because normalize_load is a finished ETL boundary.
Keep integration parsing/mapping verification separate from Warehouse orchestration and storage verification.