# ETL Source Adapter Framework - Implementation Tasks ## Overview Implementation of the **ETL Source Adapter Framework** slice. This task plan is derived from analysis of the current ETL implementation in: - `backend/Kds.Ai.ApiService/Jobs/BrinkPosEtlJobs.cs` - `backend/Kds.Ai.Integrations/BrinkPos/BrinkPosExtractJobs.cs` - `backend/Kds.Ai.Integrations/BrinkPos/BrinkPosNormalizeLoadJobs.cs` - `backend/Kds.Ai.Integrations/BrinkPos/BrinkPosLoadToStagingJobs.cs` - `backend/Kds.Ai.Integrations/BrinkPos/BrinkPosStagingToCuratedJobs.cs` - `backend/Kds.Ai.Integrations/BrinkPos/BrinkPosStageExecutionSupport.cs` - `backend/Kds.Ai.Integrations/BrinkPos/BrinkPosEndpointCatalog.cs` - `backend/Kds.Ai.Integrations/BrinkPos/BrinkPosNormalizationCatalog.cs` - `backend/Kds.Ai.Integrations/BrinkPos/BrinkPosStagingTableCatalog.cs` - `backend/Kds.Ai.Integrations/BrinkPos/BrinkPosCuratedTargetCatalog.cs` - `backend/Kds.Ai.Integrations/BrinkPos/Models/BrinkPosEtlStageConstants.cs` - `backend/Kds.Ai.Warehouse/Services/Normalization/RawToLoadNormalizationStep.cs` - `backend/Kds.Ai.Warehouse/Services/RedshiftStagingLoadExecutor.cs` and from the ETL documentation set in: - `docs/ETL.md` - `docs/features/ETL/index.md` - `docs/features/ETL/etl-source-adapter-framework.md` - `docs/features/ETL/warehouse-data/index.md` - `docs/features/ETL/warehouse-data/curated-model/index.md` The target slice extracts a small internal ETL framework that keeps common pipeline mechanics in one place and moves source-specific behavior into adapter packages. ## Current Status Status after code review on 2026-06-01: - [x] TASK 0 is implemented - [x] TASK 1 is implemented - [x] TASK 2 is implemented - [x] TASK 3 is implemented - [x] TASK 4 is implemented - [x] TASK 5 is implemented - [x] TASK 6 is implemented Overall assessment: - the ETL framework seam is real and in active use by the Brink POS path, - the shared runner, shared stage support, shared curated executor, shared diagnostics envelope, Brink adapter, DI wiring, and core regression tests are present, - persisted Brink stage details now use `StageDiagnosticsEnvelope` while keeping backward-compatible reads for earlier plain detail payloads. The target outcome is: - adding a new data source should mostly mean registering an adapter, - ETL stages should reuse the same batch lifecycle, checkpoint, failure, and warehouse execution model, - source packages should own only extraction, source contracts, and source-to-curated mapping rules. ## Source Documents - `./TASK0.md` - `../../docs/ETL.md` - `../../docs/features/ETL/index.md` - `../../docs/features/ETL/etl-source-adapter-framework.md` - repository review of current Brink POS ETL and warehouse code ## Spec Name This implementation spec should be referred to as **ETL Source Adapter Framework**. ## Fixed Decisions - This slice extracts reusable ETL framework mechanics from the current Brink POS implementation. - The first source that must fit the framework is the existing Brink POS pipeline. - This slice should keep the framework intentionally small and pragmatic. - Shared abstractions should be introduced only where current code already shows repeatable patterns. - The framework must separate pipeline mechanics from source-specific behavior. - Batch lifecycle, checkpoint advancement, failure packaging, and tenant warehouse routing are framework concerns. - Source credential models, transport clients, business-date semantics, parser internals, and curated SQL content are adapter concerns. - Existing shared services in `Kds.Ai.Warehouse` and `Kds.Ai.Data` should be reused instead of replaced where they already fit the target model. - The current stage model remains the baseline: `extract_raw`, `normalize_load`, `load_to_staging`, `staging_to_curated`. - The framework should reserve a first-class place for a future curated export stage required by `docs/ETL.md`. - The current Brink pipeline should be refactored toward adapter registration rather than kept as a permanently hardcoded orchestration chain. - Catalog content remains source-owned, but catalog concepts and contracts should become framework-owned. - The framework should not force premature unification of source-specific semantics. - The framework should improve maintainability and source onboarding speed without redesigning the entire ETL architecture. - This slice should end with implementation-ready framework seams, not only a documentation boundary. - All framework extractions must include regression coverage. ## Scope This task includes: - defining the split between ETL framework core and source adapter responsibilities, - introducing a shared pipeline runner concept, - introducing a shared stage execution template, - standardizing source descriptor and source metadata contracts, - extracting a generic checkpoint and failure helper from the current Brink stage support, - extracting a generic executor for transactional `staging -> curated` target execution, - standardizing the concept of extract, normalization, staging, and curated catalogs, - defining the minimal adapter contract required to onboard a new data source, - refactoring the current Brink pipeline toward the adapter model, - keeping room for a future `curated_to_s3` or `publish_curated` stage aligned with `docs/ETL.md`, - adding regression coverage for the extracted framework seams and Brink migration path. In short, this slice ends when the codebase has a clear framework boundary for ETL pipeline mechanics, a clear adapter boundary for source-specific behavior, and a realistic migration path from the current Brink-first implementation to a multi-source-ready model. ## Out of Scope The following items are explicitly excluded from this task: - adding new non-Brink source integrations, - broadening Brink runtime scope beyond the current active path just to validate the framework, - redesigning source credential storage, - redesigning warehouse foundation or Redshift tenant routing, - changing the semantic contract of curated business tables unless needed by framework boundaries, - implementing Athena, Glue, or cross-tenant query behavior, - redesigning replay control-plane behavior, - replacing current low-level warehouse executors that are already reusable, - forcing one parser or extraction model across all source types. --- ## Task Checklist ### Phase 0: Framework Boundary And Migration Guardrails - [x] [TASK 0](./TASK0.md) - Freeze framework boundary, source adapter ownership, and Brink migration target ### Phase 1: Shared Stage Mechanics - [x] [TASK 1](./TASK1.md) - Extract generic stage failure and checkpoint support - [x] [TASK 2](./TASK2.md) - Extract generic transactional curated target executor ### Phase 2: Pipeline Runtime And Adapter Contracts - [x] [TASK 3](./TASK3.md) - Introduce pipeline runner and ETL source adapter contracts - [x] [TASK 4](./TASK4.md) - Standardize source descriptor, diagnostics envelopes, and catalog contracts ### Phase 3: Brink Migration To The Framework - [x] [TASK 5](./TASK5.md) - Migrate the current Brink POS pipeline to the framework contracts ### Phase 4: Validation And Forward Compatibility - [x] [TASK 6](./TASK6.md) - Add regression coverage and reserve the future publish stage boundary --- ## Dependency Graph ```text Phase 0 (Framework Boundary And Migration Guardrails) +-- freezes what becomes framework-owned vs adapter-owned Phase 1 (Shared Stage Mechanics) |-- TASK 1 generic stage failure/checkpoint support <-- depends on TASK 0 +-- TASK 2 generic curated target executor <-- depends on TASK 0 Phase 2 (Pipeline Runtime And Adapter Contracts) |-- TASK 3 pipeline runner and source adapter contracts <-- depends on TASK 0, TASK 1 +-- TASK 4 source descriptor, diagnostics, and catalog contracts <-- depends on TASK 0, TASK 1, TASK 2 Phase 3 (Brink Migration To The Framework) +-- TASK 5 migrate Brink to framework contracts <-- depends on TASK 1, TASK 2, TASK 3, TASK 4 Phase 4 (Validation And Forward Compatibility) +-- TASK 6 regression coverage and future publish-stage boundary <-- depends on TASK 5 ``` --- ## Recommended Session Order | Session | Tasks | Description | |---------|-------|-------------| | 1 | 0, 1 | Freeze framework boundary and extract the smallest shared stage helper first | | 2 | 2 | Extract the generic transactional curated executor | | 3 | 3, 4 | Introduce the pipeline runtime and adapter contracts | | 4 | 5 | Migrate the current Brink pipeline to the new framework seams | | 5 | 6 | Add regression coverage and lock the future publish-stage boundary | --- ## Key Design Decisions ### Small Framework Over Big Rewrite This slice should extract only the abstractions already justified by the current Brink implementation. It should not attempt to redesign the ETL platform from scratch. ### Pipeline Mechanics Are Shared Batch lifecycle, checkpointing, failure handling, tenant warehouse resolution, and transactional execution are repeatable system behavior and should be owned by the framework. ### Source Knowledge Stays In Adapters Credentials, transport, endpoint semantics, parser logic, source contracts, and source-specific mappings should stay with the adapter. ### Catalog Structure Should Be Shared, Not Catalog Content The system should standardize what an extract catalog, normalization catalog, staging catalog, and curated target catalog look like, but should not centralize source-specific entries. ### Brink Is The Migration Baseline The current Brink POS implementation is the first concrete migration target and the main proof that the framework boundary is practical. ### Keep Room For Publish Or Export Stage The architecture in `docs/ETL.md` already expects curated export for Athena and cross-tenant use. The framework should reserve that stage in its design even if current runtime code does not implement it yet.