# TASK 2: Extract Raw XML ## Summary Implement the first real ETL data boundary for the Brink POS orders flow: live extraction of raw SOAP XML and immutable persistence to S3 `raw/`. This task starts from the tenant-day child job contract defined in `TASK1.md` and turns it into a real `extract_raw` execution step for: - `Sales2.GetCurrentBusinessDate` - `Sales2.GetOrders` This task owns: - child extract orchestration for one tenant-day run - business-date resolution - Warehouse-side binding resolution and Brink extract boundary usage - streaming raw capture to S3 - persisted-raw validation - extract-stage diagnostics This task does not implement `raw -> load`, Redshift `COPY`, curated SQL, or curated export yet. This task must preserve the architectural boundary introduced by the source-binding refactor: - Warehouse/app-layer resolves tenant context, loads the active source binding, validates it, and builds `BrinkConnectionDefinition` - `Kds.Ai.Integrations.BrinkPos` receives that connection context from the outside and only performs Brink protocol work - ETL batch semantics, S3 ownership, and replay-boundary semantics stay outside the Brink integration module ## Dependencies - [TASK 0](./TASK0.md) - Module 0 decision record - [TASK 1](./TASK1.md) - source binding and dispatch ## Subtasks - [x] [TASK 2.1](./TASK2.1.md) - Implement child extract orchestration and business-date resolution - [x] [TASK 2.2](./TASK2.2.md) - Implement Brink raw capture boundary for `GetOrders` - [x] [TASK 2.3](./TASK2.3.md) - Implement S3 raw persistence and persisted-raw validation - [x] [TASK 2.4](./TASK2.4.md) - Add unit tests and live E2E validation for `extract_raw` ## Files to Create ### 1. Extract job orchestration contracts and services Create or finalize the Warehouse/app-layer services that execute one tenant-day `extract_raw` run. Required logical responsibilities: - child extract job entry - resolve active Brink binding for the tenant-day run through Warehouse/app-layer source-binding services - business-date resolver - extract orchestrator - endpoint catalog for the narrowed v1 scope - endpoint executor Recommended logical services: - `BrinkPosExtractJobs` - `BrinkPosBusinessDateResolver` - `BrinkPosExtractOrchestrator` - `BrinkPosEndpointCatalog` - `BrinkPosEndpointExecutor` ### 2. Raw capture and validation seam Create or finalize the seam between Brink raw extraction and generic ETL storage. Required logical responsibilities: - capture response stream through `IBrinkRawClient` using externally provided `BrinkConnectionDefinition` - persist original XML through shared ETL S3 storage - reopen persisted object from storage - validate XML, SOAP fault state, and Brink business success using Brink-specific validation components without moving S3 or batch semantics into `Kds.Ai.Integrations.BrinkPos` ### 3. Extract-stage result and diagnostics models Create or finalize the minimum result/diagnostics shape needed for one `extract_raw` endpoint attempt and one full tenant-day extract batch. These models belong to ETL/Warehouse-facing code, not to `Kds.Ai.Integrations.BrinkPos`. Required minimum details: - `company_id` - `tenant_id` - resolved `business_date` - `pipeline_stage = extract_raw` - `stream_name` - `service_name` - `operation_name` - `raw_s3_path` - `started_at` - `completed_at` - endpoint status and validation outcome ## Files to Modify ### 4. Brink integration boundary Modify the Brink integration so the ETL path uses: - `IBrinkPosFacade` only for `GetCurrentBusinessDate` - `IBrinkRawClient` for `GetOrders` raw capture - externally provided `BrinkConnectionDefinition` or a factory-created bound Brink client for every ETL call path This task should not keep a parallel typed extract path for `GetOrders` in the ETL flow. This task must not add or keep the following inside `Kds.Ai.Integrations.BrinkPos` for the ETL path: - tenant resolution - source-binding lookup from operational DB - `ISourceConnectionRepository` - `IRuntimeTenantResolver` - S3 raw persistence ownership - ETL batch diagnostics models ### 5. S3 raw storage integration Modify the shared ETL storage path so raw capture writes immutable XML into the standard S3 `raw/` layout from `docs/ETL.md`. ### 6. Batch metadata touch points Use the existing ingestion-batch foundation where possible, but this task may need to extend or adapt the extract-stage metadata written during execution. Any remaining gaps in batch/checkpoint richness should be carried forward into `TASK3.md` rather than blocking raw extraction design here. ## Required Behavior ### Scope rule This task is strict v1 `extract_raw` only for: - `GetCurrentBusinessDate` as required helper - `GetOrders` as the only replayable extract endpoint No other Brink endpoint belongs to this task. ### Connection ownership rule - Warehouse/app-layer resolves the active binding and builds `BrinkConnectionDefinition` - ETL orchestration passes that connection context into the Brink integration boundary - `Kds.Ai.Integrations.BrinkPos` must not resolve tenant context or load source bindings from the operational database for this flow - the exact API shape may be either explicit `BrinkConnectionDefinition` parameters or a factory-created bound Brink client, but the connection context must come from outside the integration module ### Business-date rule For one child extract request: 1. if `BusinessDate` is provided, use it exactly 2. otherwise call `GetCurrentBusinessDate` 3. use the resolved value for the full tenant-day extract run ### Extract boundary rule - `IBrinkPosFacade` is control-only in this task - `IBrinkRawClient` is the only ETL extraction API for `GetOrders` - both consume externally provided Brink connection context - job-layer code must not build SOAP envelopes, routes, headers, or auth rules - integration-layer code must not own ETL batch state, S3 persistence, or tenant-binding lookup ### Raw capture rule - `GetOrders` must be captured as streaming raw SOAP XML - original XML must be persisted to S3 `raw/` without mutation - the persisted raw artifact is the replay boundary - transport success alone is not enough for endpoint success ### Validation rule Endpoint success requires persisted-raw validation after upload. Validation must confirm at least: - XML is well formed - no SOAP fault is present - Brink business success semantics are satisfied Responsibility split: - Brink-specific payload inspection and validation logic may live in `Kds.Ai.Integrations.BrinkPos` - reopening the persisted S3 object, recording `raw_s3_path`, and deciding extract-stage success or failure belong to ETL/Warehouse orchestration ### Orchestration rule - one child extract job executes one tenant-day extract boundary - fail-fast behavior is acceptable because v1 includes one replayable endpoint only - endpoint diagnostics must be written in a form that later stages can reference ## Testing ### Unit tests Unit tests are mandatory for this task. Add unit tests for: - Warehouse-side connection resolution handoff into the Brink integration boundary - business-date resolution with explicit date override - business-date resolution through `GetCurrentBusinessDate` - endpoint catalog for the narrowed v1 scope - raw capture orchestration using `IBrinkRawClient` - persisted-raw validation success and failure - SOAP fault failure handling - Brink business-failure handling - raw capture diagnostics and S3 path recording ### Live E2E tests This task ends at a real ETL boundary, so live E2E is required whenever Aspire-managed secrets and integrations are available. Add live E2E coverage that verifies at least: - the child extract flow can resolve real Brink connection data - `GetCurrentBusinessDate` works against real Brink when `BusinessDate` is omitted - `GetOrders` calls the real Brink API - the returned XML is written to S3 `raw/` - the persisted raw object can be reopened and validated successfully Recommended live verification shape: - run one tenant-day extract against real Brink - assert that one expected raw artifact exists in S3 - assert that metadata and diagnostics point to the persisted object - assert that no secret values are written into S3 metadata or diagnostics - assert that the ETL path does not depend on DB-coupled connection resolution inside `Kds.Ai.Integrations.BrinkPos` ## Acceptance Criteria - [x] Child extract job entry exists for the tenant-day `extract_raw` stage - [x] Warehouse/app-layer resolves the active Brink binding and supplies connection context to the integration boundary - [x] `GetCurrentBusinessDate` is used when `BusinessDate` is omitted - [x] `GetOrders` is captured through `IBrinkRawClient` - [x] `Kds.Ai.Integrations.BrinkPos` no longer owns tenant resolution or source-binding lookup for this flow - [x] ETL orchestration does not use typed `GetOrders` DTO extraction - [x] Original raw SOAP XML is written to S3 `raw/` - [x] Persisted raw validation is required before endpoint success - [x] Extract diagnostics capture stream, service, operation, timing, and raw path - [x] Unit tests cover business-date resolution, raw capture, and validation behavior - [x] Live E2E verifies real Brink extraction and raw S3 persistence - [x] Solution builds without errors ## Notes - Keep this task strict: `extract_raw` only. - Do not widen scope to normalization or batch-lifecycle redesign inside this task. - This is the first task where live E2E against real services is mandatory unless a concrete blocker is documented. - Prefer the ongoing source-binding refactor direction: `Kds.Ai.Integrations.BrinkPos` should consume externally provided connection context rather than resolve DB-backed bindings internally.