Spensara Spensara Knowledge Base & Documentation Repository
System Overview Brink & Borg Master Index (722 Docs) Spensara Platform Project Tracker

Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)
Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Categories

Implementation Specs (157)

Detailed engineering and feature implementation specifications.
implementation-specs/specs/BrinkPosOrdersFirstEtlJob/TASK0.md 7,128 bytes · Updated Oct 23, 2026 10:44 AM EDT
Raw File

TASK 0 - Module 0 Decision Record

Purpose

Freeze the true v1 scope for the first Brink POS ETL job before implementation work starts in dispatch, extraction, normalization, Redshift load, and curated mapping.

This record overrides the broader and partly outdated Brink POS groundwork documents for this specific implementation slice.

Decision Summary

The first Brink POS ETL job is strictly limited to:

  • orders
  • order_items

The job must run daily and therefore always supports business-date resolution through:

  • Sales2.GetCurrentBusinessDate
  • Sales2.GetOrders

The first implemented end-to-end path is:

  1. resolve business date
  2. extract raw XML
  3. persist raw XML to S3 raw/
  4. normalize to Parquet in S3 load/
  5. load Parquet to Redshift staging
  6. map staging -> curated
  7. expose correct data in:
  8. curated.pos_orders
  9. curated.pos_order_items

Testing is a mandatory part of every module:

  • every module must include unit tests
  • every completed ETL stage should include live E2E validation whenever the stage can be verified against real integrations
  • Aspire secrets are the standard way to access AWS services and Brink POS API during live verification

In Scope

  • one daily tenant-day ETL job for Brink POS
  • Sales2.GetCurrentBusinessDate as required business-date helper
  • Sales2.GetOrders as the main transactional source
  • full raw XML preservation in S3
  • Parquet load/ artifacts derived from GetOrders
  • Redshift staging load from Parquet
  • Redshift curated mapping for:
  • curated.pos_orders
  • curated.pos_order_items
  • stage-level metadata, checkpoints, rerun semantics, and traceability
  • unit tests for every module
  • live E2E verification for each completed ETL stage whenever practical through Aspire-managed secrets and integrations

Out Of Scope

  • every Brink endpoint other than GetCurrentBusinessDate and GetOrders
  • kitchen contracts and kitchen-oriented curated outputs
  • item, destination, tender, tax, employee, modifier, or table enrichment endpoints
  • widening the first slice to any additional business contract beyond orders and order_items
  • treating old Brink groundwork docs as binding scope for this implementation

Final Curated Targets

The curated target names for v1 are already fixed and must not be renamed in this slice:

  • curated.pos_orders
  • curated.pos_order_items

Source of truth:

  • backend/Kds.Ai.Warehouse/Sql/Bootstrap/Tenant/Curated/0612_create_curated_pos_orders.sql
  • backend/Kds.Ai.Warehouse/Sql/Bootstrap/Tenant/Curated/0611_create_curated_pos_order_items.sql

Source Of Truth For V1 Contracts

For this slice, the source of truth is:

  • the existing curated bootstrap SQL for pos_orders and pos_order_items
  • the existing staging models under backend/integrations/Kds.Ai.Integrations.BrinkPos/Models/Staging/Sales2/GetOrders
  • the curated model docs and Brink mapping docs used as semantic reference

The broader documents under:

  • docs/features/ETL/epics/06-source-adapter-brink-pos-api/01-first-job-foundation/

are treated only as supporting material. They may remain broader than this actual implementation scope.

Staging And Load Interpretation

For this slice, the pipeline layers are interpreted as follows:

  • S3 raw/ stores the original XML payload from Brink
  • S3 load/ stores processed Parquet contracts
  • Redshift staging stores data loaded from those Parquet contracts
  • Redshift curated stores business-facing mappings built from Redshift staging

Important rule:

  • the existing GetOrders staging model is reused
  • Module 0 does not redesign staging
  • later implementation work must only verify completeness and correctness of the existing staging model against:
  • Brink portal documentation
  • temp/GetOrders_WithKnownBusinessDate_CallsRealBrinkEndpoint_2024-10-06.xml

Field Coverage Decision

For Module 0, field-scope decisions are frozen as follows:

  • the job targets the full existing curated contracts for curated.pos_orders and curated.pos_order_items
  • the job uses the full existing GetOrders staging contract already modeled in code
  • fields that cannot be populated correctly from GetOrders plus GetCurrentBusinessDate alone are allowed to be NULL in v1
  • missing enrichment from other Brink endpoints is not a blocker for the first slice

This means:

  • no extra endpoint is added just to backfill one field
  • nullable output is preferred over hidden scope expansion
  • any later enrichment must be added as a separate, atomic scope extension

Extensibility Rule

The first slice must be easy to extend.

Future addition of a new Brink contract should be:

  • fast
  • isolated
  • atomic
  • non-breaking for the existing orders and order_items flow

This implies:

  • orchestration is stage-based and contract-based
  • adding a new endpoint later must not require redesign of the current GetOrders path
  • adding enrichment later must be possible without invalidating the current v1 delivery

Testing Rule

Testing is part of the delivery contract, not a later hardening task.

Unit tests

  • every module must include unit tests
  • unit tests should verify module-local rules, mappings, validation, orchestration branching, and failure semantics

Live E2E tests

  • every module should include live E2E coverage when the module represents a completed ETL step that can be validated against real services
  • live E2E tests should use Aspire-managed secrets and environment wiring to reach:
  • Brink POS API
  • AWS services such as S3
  • later Redshift when the stage requires it

Examples

  • extract_raw should have a live E2E test that calls real Brink POS endpoints and verifies that raw XML is persisted to S3
  • normalize_load should have a live E2E test that starts from a real raw artifact and verifies written Parquet output
  • load_to_staging should have a live E2E test that verifies Redshift staging load from produced Parquet
  • staging_to_curated should have a live E2E test that verifies curated rows after SQL mapping

Practical rule

  • if a module ends with a clearly verifiable ETL boundary, that module should not be treated as complete without E2E validation unless there is a concrete technical blocker
  • if a live E2E test is temporarily not possible, the blocker must be documented explicitly in the module task

Success Criteria For Module 0

Module 0 is complete when the team treats the following as fixed:

  • the first Brink ETL slice is only orders and order_items
  • GetCurrentBusinessDate is required for daily execution support
  • GetOrders is the main transactional source
  • curated targets are fixed as:
  • curated.pos_orders
  • curated.pos_order_items
  • the existing GetOrders staging model is reused and later only verified, not redesigned
  • fields missing non-GetOrders enrichment are allowed to remain NULL in v1
  • older groundwork docs are non-authoritative when they widen the scope beyond this decision record
  • every module requires unit tests, and completed ETL stages require live E2E verification whenever feasible