Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)

Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Implementation Specs (157)

Detailed engineering and feature implementation specifications.

implementation-specs/specs/BrinkPosOrdersFirstEtlJob/TASK5.3.md 2,751 bytes · Updated Oct 23, 2026 10:44 AM EDT

Raw File

TASK 5.3: Implement Redshift Staging Load Execution, Diagnostics, And Rerun Semantics

Summary

Implement the execution boundary for the first load_to_staging stage: run Redshift COPY for the resolved table set, apply the chosen rerun-safe staging replacement strategy, and publish safe per-table diagnostics that later stages can consume.

Dependencies

TASK 5.2 - staging table catalog and Redshift COPY command construction

Files to Create

1. Staging-load executor and diagnostics models

Create or finalize the components that execute staging-load SQL and return table-level diagnostics.

Recommended logical service:

RedshiftStagingLoadExecutor

Required diagnostics details:

contract name
target table name
schema version
input artifact URIs or prefixes
timing
safe status summary
copySqlTemplateId or equivalent execution identifier

Files to Modify

2. Warehouse execution path

Modify the tenant warehouse execution path so load_to_staging can run the chosen replacement strategy and then execute COPY statements for every required staging table.

3. Load-to-staging batch details

Modify or finalize the load_to_staging details_json shape so it preserves upstream normalize-load references and per-table staging execution results.

Required Behavior

the chosen v1 staging replacement strategy is applied consistently before or alongside COPY as required
Redshift COPY executes for every required table in the resolved upstream inventory
if any required table load fails, the endpoint fails and the full load_to_staging batch fails
already written upstream load/ artifacts remain for replay and diagnostics
diagnostics must not store secrets
batch metadata must preserve enough detail for TASK6.md to resolve staging_to_curated inputs confidently

Testing

Unit tests

Add unit tests for:

execution ordering of replacement strategy and COPY
per-table success and failure diagnostics recording
failure behavior when one required table load fails
rerun-safe stage behavior and lifecycle integration
diagnostics sanitization

Acceptance Criteria

[x] Redshift staging load executes for every required table in the resolved upstream inventory
[x] The chosen replacement strategy is applied consistently for reruns
[x] Table-load failures fail the endpoint and stage correctly
[x] Diagnostics stay secret-free and downstream-readable
[x] Unit tests cover execution, rerun semantics, and diagnostics behavior

Notes

This subtask creates the first warehouse-loaded staging replay input for curated SQL.
Keep Redshift execution ownership in shared Warehouse code rather than in the Brink adapter.