Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)

Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Implementation Specs (157)

Detailed engineering and feature implementation specifications.

implementation-specs/specs/CuratedToS3Stage/TASK.md 6,085 bytes · Updated Oct 23, 2026 10:44 AM EDT

Raw File

Curated To S3 Stage - Implementation Tasks

Overview

Implementation of curated_to_s3 ETL stage framework plus first Brink adoption.

This slice extends current ETL framework with real export stage after staging_to_curated and proves it through current Brink pipeline.

Current repo state:

ETL framework already reserves EtlStageNames.CuratedToS3.
Active Brink pipeline stops at staging_to_curated.
S3 storage already supports curated-object keys and writes.
docs/ETL.md expects Redshift UNLOAD from tenant curated tables to S3 Parquet.

This slice outcome is:

framework-owned export contracts and execution seam exist,
export list stays adapter-owned,
new entities can be added for Brink through catalog extension rather than stage rewrites,
other source systems can adopt same stage model by providing export catalog content,
Brink pipeline exports curated.pos_orders and curated.pos_order_items as first adopter,
Redshift UNLOAD writes Parquet directly to standard S3 curated/ layout,
parent ETL batch succeeds only after export succeeds.

Source Documents

../../docs/ETL.md
../../docs/features/ETL/etl-source-adapter-framework.md
../../docs/features/ETL/progress-gap-vs-etl-architecture.md

Spec Name

This implementation spec should be referred to as Curated To S3 Stage.

Fixed Decisions

This slice delivers framework support plus Brink first adoption only.
Framework must support future non-Brink adapters without stage redesign.
Brink is first adopter, not one-off implementation target.
First exported entities are curated.pos_orders and curated.pos_order_items only.
Export list stays adapter-owned. Execution mechanics and catalog contracts stay shared.
Export mechanism is Redshift UNLOAD to Parquet in S3.
Application code must not read curated rows and write Parquet itself.
Parent ETL batch succeeds only after curated_to_s3 succeeds.
Export metadata should reuse source adapter catalog pattern with framework-owned contract.
S3 layout should reuse existing project curated path standard.
Athena, Glue, partition projection, and cross-tenant query routing stay out of scope.

Scope

This task includes:

freezing minimal delivery contract for curated_to_s3,
adding framework-owned export contracts for adapter catalogs,
adding shared executor for Redshift UNLOAD,
adding adapter-owned export catalog for Brink as first adopter,
adding new curated_to_s3 stage support into source-adapter runtime model,
adding new Brink stage and jobs wiring,
adding regression coverage for export success, failure, and stage ordering,
validating S3 key layout against current curated-path standard.

In short, this slice ends when ETL framework has reusable curated_to_s3 stage model and current Brink pipeline can adopt it for first curated exports without starting Athena/Glue work.

Out of Scope

The following items are explicitly excluded from this task:

Athena query execution,
Glue Data Catalog tables,
partition projection setup,
access-control tables,
multi-tenant query routing,
adding more curated entities beyond pos_orders and pos_order_items in first adopter implementation,
redesigning whole ETL framework,
app-side Parquet export path.

Task Checklist

Phase 0: Contract Freeze

[ ] TASK 0 - Freeze minimal curated_to_s3 delivery contract

Phase 1: Shared Export Mechanics

[ ] TASK 1 - Add framework-owned export contracts, shared Redshift UNLOAD executor, and S3 path builder usage

Phase 2: Adapter Export Metadata

[ ] TASK 2 - Add adapter export catalog model and Brink first-adopter catalog

Phase 3: Stage Runtime Integration

[ ] TASK 3 - Add framework stage integration and Brink curated_to_s3 pipeline wiring

Phase 4: Validation

[ ] TASK 4 - Add regression coverage for export success, failure, and contract shape

Dependency Graph

text

Phase 0 (Contract Freeze)
    +-- freezes scope and success boundary before runtime work starts

Phase 1 (Shared Export Mechanics) <-- depends on Phase 0
    +-- adds reusable export contracts and UNLOAD execution seam

Phase 2 (Adapter Export Metadata) <-- depends on Phase 0
    +-- adds adapter-owned export target definitions under shared contract

Phase 3 (Stage Runtime Integration) <-- depends on Phases 1, 2
    +-- introduces executable curated_to_s3 stage in framework runtime and Brink pipeline

Phase 4 (Validation) <-- depends on Phase 3
    +-- locks behavior with unit and integration-style tests

Recommended Session Order

Session	Tasks	Description
1	0	Freeze minimal contract and non-goals
2	1	Add shared export contracts and `UNLOAD` execution seam
3	2	Add adapter export catalog model through Brink first adopter
4	3	Add stage runtime support, then Brink stage and adapter wiring
5	4	Add regression coverage and verify S3 contract

Key Design Decisions

Export Is Real Warehouse Behavior

This slice should use Redshift UNLOAD, not app-side row export. Exported Parquet must come directly from tenant curated tables.

Shared Mechanics, Adapter-Owned Targets

Execution, connection handling, common command shaping, and export catalog contracts are shared concerns. Export target list and source-specific SELECT semantics stay in adapter catalog.

Success Boundary Includes Export

staging_to_curated is not final success anymore. Green ETL run means curated export also completed.

Minimal First Scope

Brink is first adopter. First exported entities are only curated.pos_orders and curated.pos_order_items. Breadth comes later through catalog extension after framework seam proves itself.

Reuse Existing S3 Standard

This slice should not invent another curated S3 layout. It should plug into existing curated object path standard already used in project docs and storage helpers.