Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)

Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Implementation Specs (157)

Detailed engineering and feature implementation specifications.

implementation-specs/specs/BrinkPosOrdersFirstEtlJob/TASK6.4.md 2,439 bytes · Updated Oct 23, 2026 10:44 AM EDT

Raw File

TASK 6.4: Add Unit Tests And Live E2E Validation For Staging-To-Curated

Summary

Add the verification layer for the first staging -> curated ETL boundary. This subtask closes TASK 6 by proving both locally and against real integrations that staging_to_curated works for curated.pos_orders and curated.pos_order_items.

Dependencies

TASK 6.1 - staging-to-curated orchestration and upstream load-to-staging batch resolution
TASK 6.2 - curated target dependency matrix and v1 field behavior
TASK 6.3 - curated SQL catalog, execution, and diagnostics

Files to Create

1. Staging-to-curated verification suites

Create or finalize the test suites needed for:

unit coverage of orchestration and dependency rules
unit coverage of SQL selection, target execution, and diagnostics
live E2E coverage of real load_to_staging -> staging_to_curated execution

Required Behavior

unit tests must cover the final staging_to_curated flow assembled from TASK 6.1-TASK 6.3
live E2E must verify the real integration path whenever Aspire-managed secrets are available
the live test should prove that a real staged tenant-day dataset can be transformed into curated Redshift rows and recorded in batch metadata

Testing

Unit tests

Ensure the final unit suite covers:

upstream load-to-staging batch selection
curated target catalog and dependency matrix
deferred-enrichment field behavior
curated SQL selection, replacement strategy, and failure behavior

Live E2E

Add a live E2E test that verifies at least:

one real load_to_staging batch can serve as the upstream source
staging_to_curated reads Redshift staging state and persisted upstream metadata instead of S3 artifacts
expected curated.pos_orders and curated.pos_order_items rows are created
batch metadata points to the upstream load-to-staging batch and per-target curated results

Acceptance Criteria

[x] Unit-test coverage exists for the assembled staging_to_curated flow
[x] Live E2E exists for real staging-to-curated Redshift execution
[x] Live E2E verifies target results and metadata linkage
[x] TASK 6 is verifiable as a completed ETL stage

Notes

This subtask is mandatory because staging_to_curated is a finished ETL boundary.
Keep dependency/mapping verification separate from actual warehouse execution verification.