Spensara Spensara Knowledge Base & Documentation Repository
System Overview Brink & Borg Master Index (722 Docs) Spensara Platform Project Tracker

Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)
Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Categories

Branches (328)

Branch-specific documentation across extract-raw-xml, feature-fleet-import, kds_cdc_improvments, life-cycle, and tenat-admin.
branches/extract-raw-xml/implementation-specs/specs/BrinkPosOrdersFirstEtlJob/TASK2.md 9,773 bytes · Updated Oct 23, 2026 10:44 AM EDT
Raw File

TASK 2: Extract Raw XML

Summary

Implement the first real ETL data boundary for the Brink POS orders flow: live extraction of raw SOAP XML and immutable persistence to S3 raw/.

This task starts from the tenant-day child job contract defined in TASK1.md and turns it into a real extract_raw execution step for:

  • Sales2.GetCurrentBusinessDate
  • Sales2.GetOrders

This task owns:

  • child extract orchestration for one tenant-day run
  • business-date resolution
  • Warehouse-side binding resolution and Brink extract boundary usage
  • streaming raw capture to S3
  • persisted-raw validation
  • extract-stage diagnostics

This task does not implement raw -> load, Redshift COPY, curated SQL, or curated export yet.

This task must preserve the architectural boundary introduced by the source-binding refactor:

  • Warehouse/app-layer resolves tenant context, loads the active source binding, validates it, and builds BrinkConnectionDefinition
  • Kds.Ai.Integrations.BrinkPos receives that connection context from the outside and only performs Brink protocol work
  • ETL batch semantics, S3 ownership, and replay-boundary semantics stay outside the Brink integration module

Dependencies

  • TASK 0 - Module 0 decision record
  • TASK 1 - source binding and dispatch

Subtasks

  • [ ] TASK 2.1 - Implement child extract orchestration and business-date resolution
  • [ ] TASK 2.2 - Implement Brink raw capture boundary for GetOrders
  • [ ] TASK 2.3 - Implement S3 raw persistence and persisted-raw validation
  • [ ] TASK 2.4 - Add unit tests and live E2E validation for extract_raw

Files to Create

1. Extract job orchestration contracts and services

Create or finalize the Warehouse/app-layer services that execute one tenant-day extract_raw run.

Required logical responsibilities:

  • child extract job entry
  • resolve active Brink binding for the tenant-day run through Warehouse/app-layer source-binding services
  • business-date resolver
  • extract orchestrator
  • endpoint catalog for the narrowed v1 scope
  • endpoint executor

Recommended logical services:

  • BrinkPosExtractJobs
  • BrinkPosBusinessDateResolver
  • BrinkPosExtractOrchestrator
  • BrinkPosEndpointCatalog
  • BrinkPosEndpointExecutor

2. Raw capture and validation seam

Create or finalize the seam between Brink raw extraction and generic ETL storage.

Required logical responsibilities:

  • capture response stream through IBrinkRawClient using externally provided BrinkConnectionDefinition
  • persist original XML through shared ETL S3 storage
  • reopen persisted object from storage
  • validate XML, SOAP fault state, and Brink business success using Brink-specific validation components without moving S3 or batch semantics into Kds.Ai.Integrations.BrinkPos

3. Extract-stage result and diagnostics models

Create or finalize the minimum result/diagnostics shape needed for one extract_raw endpoint attempt and one full tenant-day extract batch.

These models belong to ETL/Warehouse-facing code, not to Kds.Ai.Integrations.BrinkPos.

Required minimum details:

  • company_id
  • tenant_id
  • resolved business_date
  • pipeline_stage = extract_raw
  • stream_name
  • service_name
  • operation_name
  • raw_s3_path
  • started_at
  • completed_at
  • endpoint status and validation outcome

Files to Modify

4. Brink integration boundary

Modify the Brink integration so the ETL path uses:

  • IBrinkPosFacade only for GetCurrentBusinessDate
  • IBrinkRawClient for GetOrders raw capture
  • externally provided BrinkConnectionDefinition or a factory-created bound Brink client for every ETL call path

This task should not keep a parallel typed extract path for GetOrders in the ETL flow.

This task must not add or keep the following inside Kds.Ai.Integrations.BrinkPos for the ETL path:

  • tenant resolution
  • source-binding lookup from operational DB
  • ISourceConnectionRepository
  • IRuntimeTenantResolver
  • S3 raw persistence ownership
  • ETL batch diagnostics models

5. S3 raw storage integration

Modify the shared ETL storage path so raw capture writes immutable XML into the standard S3 raw/ layout from docs/ETL.md.

6. Batch metadata touch points

Use the existing ingestion-batch foundation where possible, but this task may need to extend or adapt the extract-stage metadata written during execution.

Any remaining gaps in batch/checkpoint richness should be carried forward into TASK3.md rather than blocking raw extraction design here.

Required Behavior

Scope rule

This task is strict v1 extract_raw only for:

  • GetCurrentBusinessDate as required helper
  • GetOrders as the only replayable extract endpoint

No other Brink endpoint belongs to this task.

Connection ownership rule

  • Warehouse/app-layer resolves the active binding and builds BrinkConnectionDefinition
  • ETL orchestration passes that connection context into the Brink integration boundary
  • Kds.Ai.Integrations.BrinkPos must not resolve tenant context or load source bindings from the operational database for this flow
  • the exact API shape may be either explicit BrinkConnectionDefinition parameters or a factory-created bound Brink client, but the connection context must come from outside the integration module

Business-date rule

For one child extract request:

  1. if BusinessDate is provided, use it exactly
  2. otherwise call GetCurrentBusinessDate
  3. use the resolved value for the full tenant-day extract run

Extract boundary rule

  • IBrinkPosFacade is control-only in this task
  • IBrinkRawClient is the only ETL extraction API for GetOrders
  • both consume externally provided Brink connection context
  • job-layer code must not build SOAP envelopes, routes, headers, or auth rules
  • integration-layer code must not own ETL batch state, S3 persistence, or tenant-binding lookup

Raw capture rule

  • GetOrders must be captured as streaming raw SOAP XML
  • original XML must be persisted to S3 raw/ without mutation
  • the persisted raw artifact is the replay boundary
  • transport success alone is not enough for endpoint success

Validation rule

Endpoint success requires persisted-raw validation after upload.

Validation must confirm at least:

  • XML is well formed
  • no SOAP fault is present
  • Brink business success semantics are satisfied

Responsibility split:

  • Brink-specific payload inspection and validation logic may live in Kds.Ai.Integrations.BrinkPos
  • reopening the persisted S3 object, recording raw_s3_path, and deciding extract-stage success or failure belong to ETL/Warehouse orchestration

Orchestration rule

  • one child extract job executes one tenant-day extract boundary
  • fail-fast behavior is acceptable because v1 includes one replayable endpoint only
  • endpoint diagnostics must be written in a form that later stages can reference

Testing

Unit tests

Unit tests are mandatory for this task.

Add unit tests for:

  • Warehouse-side connection resolution handoff into the Brink integration boundary
  • business-date resolution with explicit date override
  • business-date resolution through GetCurrentBusinessDate
  • endpoint catalog for the narrowed v1 scope
  • raw capture orchestration using IBrinkRawClient
  • persisted-raw validation success and failure
  • SOAP fault failure handling
  • Brink business-failure handling
  • raw capture diagnostics and S3 path recording

Live E2E tests

This task ends at a real ETL boundary, so live E2E is required whenever Aspire-managed secrets and integrations are available.

Add live E2E coverage that verifies at least:

  • the child extract flow can resolve real Brink connection data
  • GetCurrentBusinessDate works against real Brink when BusinessDate is omitted
  • GetOrders calls the real Brink API
  • the returned XML is written to S3 raw/
  • the persisted raw object can be reopened and validated successfully

Recommended live verification shape:

  • run one tenant-day extract against real Brink
  • assert that one expected raw artifact exists in S3
  • assert that metadata and diagnostics point to the persisted object
  • assert that no secret values are written into S3 metadata or diagnostics
  • assert that the ETL path does not depend on DB-coupled connection resolution inside Kds.Ai.Integrations.BrinkPos

Acceptance Criteria

  • [ ] Child extract job entry exists for the tenant-day extract_raw stage
  • [ ] Warehouse/app-layer resolves the active Brink binding and supplies connection context to the integration boundary
  • [ ] GetCurrentBusinessDate is used when BusinessDate is omitted
  • [ ] GetOrders is captured through IBrinkRawClient
  • [ ] Kds.Ai.Integrations.BrinkPos no longer owns tenant resolution or source-binding lookup for this flow
  • [ ] ETL orchestration does not use typed GetOrders DTO extraction
  • [ ] Original raw SOAP XML is written to S3 raw/
  • [ ] Persisted raw validation is required before endpoint success
  • [ ] Extract diagnostics capture stream, service, operation, timing, and raw path
  • [ ] Unit tests cover business-date resolution, raw capture, and validation behavior
  • [ ] Live E2E verifies real Brink extraction and raw S3 persistence
  • [ ] Solution builds without errors

Notes

  • Keep this task strict: extract_raw only.
  • Do not widen scope to normalization or batch-lifecycle redesign inside this task.
  • This is the first task where live E2E against real services is mandatory unless a concrete blocker is documented.
  • Prefer the ongoing source-binding refactor direction: Kds.Ai.Integrations.BrinkPos should consume externally provided connection context rather than resolve DB-backed bindings internally.