# Curated To S3 Stage - Implementation Tasks

## Overview

Implementation of `curated_to_s3` ETL stage framework plus first Brink adoption.

This slice extends current ETL framework with real export stage after `staging_to_curated` and proves it through current Brink pipeline.

Current repo state:

- ETL framework already reserves `EtlStageNames.CuratedToS3`.
- Active Brink pipeline stops at `staging_to_curated`.
- S3 storage already supports curated-object keys and writes.
- `docs/ETL.md` expects Redshift `UNLOAD` from tenant `curated` tables to S3 Parquet.

This slice outcome is:

- framework-owned export contracts and execution seam exist,
- export list stays adapter-owned,
- new entities can be added for Brink through catalog extension rather than stage rewrites,
- other source systems can adopt same stage model by providing export catalog content,
- Brink pipeline exports `curated.pos_orders` and `curated.pos_order_items` as first adopter,
- Redshift `UNLOAD` writes Parquet directly to standard S3 `curated/` layout,
- parent ETL batch succeeds only after export succeeds.

## Source Documents

- `../../docs/ETL.md`
- `../../docs/features/ETL/etl-source-adapter-framework.md`
- `../../docs/features/ETL/progress-gap-vs-etl-architecture.md`

## Spec Name

This implementation spec should be referred to as **Curated To S3 Stage**.

## Fixed Decisions

- This slice delivers framework support plus Brink first adoption only.
- Framework must support future non-Brink adapters without stage redesign.
- Brink is first adopter, not one-off implementation target.
- First exported entities are `curated.pos_orders` and `curated.pos_order_items` only.
- Export list stays adapter-owned. Execution mechanics and catalog contracts stay shared.
- Export mechanism is Redshift `UNLOAD` to Parquet in S3.
- Application code must not read curated rows and write Parquet itself.
- Parent ETL batch succeeds only after `curated_to_s3` succeeds.
- Export metadata should reuse source adapter catalog pattern with framework-owned contract.
- S3 layout should reuse existing project curated path standard.
- Athena, Glue, partition projection, and cross-tenant query routing stay out of scope.

## Scope

This task includes:

- freezing minimal delivery contract for `curated_to_s3`,
- adding framework-owned export contracts for adapter catalogs,
- adding shared executor for Redshift `UNLOAD`,
- adding adapter-owned export catalog for Brink as first adopter,
- adding new `curated_to_s3` stage support into source-adapter runtime model,
- adding new Brink stage and jobs wiring,
- adding regression coverage for export success, failure, and stage ordering,
- validating S3 key layout against current curated-path standard.

In short, this slice ends when ETL framework has reusable `curated_to_s3` stage model and current Brink pipeline can adopt it for first curated exports without starting Athena/Glue work.

## Out of Scope

The following items are explicitly excluded from this task:

- Athena query execution,
- Glue Data Catalog tables,
- partition projection setup,
- access-control tables,
- multi-tenant query routing,
- adding more curated entities beyond `pos_orders` and `pos_order_items` in first adopter implementation,
- redesigning whole ETL framework,
- app-side Parquet export path.

---

## Task Checklist

### Phase 0: Contract Freeze

- [ ] [TASK 0](./TASK0.md) - Freeze minimal `curated_to_s3` delivery contract

### Phase 1: Shared Export Mechanics

- [ ] [TASK 1](./TASK1.md) - Add framework-owned export contracts, shared Redshift `UNLOAD` executor, and S3 path builder usage

### Phase 2: Adapter Export Metadata

- [ ] [TASK 2](./TASK2.md) - Add adapter export catalog model and Brink first-adopter catalog

### Phase 3: Stage Runtime Integration

- [ ] [TASK 3](./TASK3.md) - Add framework stage integration and Brink `curated_to_s3` pipeline wiring

### Phase 4: Validation

- [ ] [TASK 4](./TASK4.md) - Add regression coverage for export success, failure, and contract shape

---

## Dependency Graph

```text
Phase 0 (Contract Freeze)
    +-- freezes scope and success boundary before runtime work starts

Phase 1 (Shared Export Mechanics) <-- depends on Phase 0
    +-- adds reusable export contracts and UNLOAD execution seam

Phase 2 (Adapter Export Metadata) <-- depends on Phase 0
    +-- adds adapter-owned export target definitions under shared contract

Phase 3 (Stage Runtime Integration) <-- depends on Phases 1, 2
    +-- introduces executable curated_to_s3 stage in framework runtime and Brink pipeline

Phase 4 (Validation) <-- depends on Phase 3
    +-- locks behavior with unit and integration-style tests
```

---

## Recommended Session Order

| Session | Tasks | Description |
|---------|-------|-------------|
| 1 | 0 | Freeze minimal contract and non-goals |
| 2 | 1 | Add shared export contracts and `UNLOAD` execution seam |
| 3 | 2 | Add adapter export catalog model through Brink first adopter |
| 4 | 3 | Add stage runtime support, then Brink stage and adapter wiring |
| 5 | 4 | Add regression coverage and verify S3 contract |

---

## Key Design Decisions

### Export Is Real Warehouse Behavior

This slice should use Redshift `UNLOAD`, not app-side row export. Exported Parquet must come directly from tenant `curated` tables.

### Shared Mechanics, Adapter-Owned Targets

Execution, connection handling, common command shaping, and export catalog contracts are shared concerns. Export target list and source-specific `SELECT` semantics stay in adapter catalog.

### Success Boundary Includes Export

`staging_to_curated` is not final success anymore. Green ETL run means curated export also completed.

### Minimal First Scope

Brink is first adopter. First exported entities are only `curated.pos_orders` and `curated.pos_order_items`. Breadth comes later through catalog extension after framework seam proves itself.

### Reuse Existing S3 Standard

This slice should not invent another curated S3 layout. It should plug into existing curated object path standard already used in project docs and storage helpers.