Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)

Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Implementation Specs (157)

Detailed engineering and feature implementation specifications.

implementation-specs/specs/CuratedToS3Stage/TASK3.md 2,763 bytes · Updated Oct 23, 2026 10:44 AM EDT

Raw File

TASK 3: Add Framework Stage Integration And Brink `curated_to_s3` Pipeline Wiring

Summary

Add executable curated_to_s3 stage support into framework runtime and prove it through Brink pipeline wiring.

Dependencies

TASK 1 - Add shared Redshift UNLOAD executor and curated S3 path usage
TASK 2 - Add Brink curated export catalog for first entities

Files to Create

1. Stage and jobs classes

Create Brink stage runtime types equivalent to:

BrinkPosCuratedToS3Stage
BrinkPosCuratedToS3Jobs
small details/result model if stage diagnostics need it

Files to Modify

2. Brink source adapter

Update adapter stage list so curated_to_s3 runs after staging_to_curated.

3. Dependency injection

4. Framework runtime model if needed

Update shared ETL runtime shape only as much as needed so curated_to_s3 is real framework stage, not Brink-only special case.

Progress Checklist

[ ] BrinkPosCuratedToS3Stage created
[ ] BrinkPosCuratedToS3Jobs created
[ ] Stage result/details model created if needed
[ ] Brink adapter stage order updated
[ ] DI wiring added for new stage/jobs
[ ] Shared runtime updated if needed for real curated_to_s3 stage support
[ ] Stage uses shared export executor
[ ] Stage uses Brink export catalog
[ ] Stage success advances checkpoint
[ ] Stage failure writes failed package and fails batch
[ ] Parent pipeline success boundary updated to include export stage
[ ] Unit tests added for stage order, success, and failure paths

Required Behavior

stage consumes upstream successful staging_to_curated output,
stage resolves export targets from Brink export catalog,
stage invokes shared export executor for each target,
stage advances checkpoint only on full success,
stage writes failed package and fails batch on export exception,
framework runtime can host same stage for future adapters without redesign,
parent pipeline result is not successful if export stage fails.

Testing

Unit tests

Add unit tests for:

stage ordering inside Brink adapter,
stage consuming upstream result,
success path calling checkpoint support,
failure path calling failed package writer and batch fail lifecycle.

Acceptance Criteria

Brink adapter includes curated_to_s3 after staging_to_curated,
framework runtime treats curated_to_s3 as real stage boundary,
stage uses shared export executor and Brink catalog,
success advances checkpoint,
failure marks batch failed,
pipeline no longer ends successfully before export.

Notes

Reuse existing stage support patterns.
Do not duplicate failure/checkpoint plumbing.