Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)

Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Branches (328)

Branch-specific documentation across extract-raw-xml, feature-fleet-import, kds_cdc_improvments, life-cycle, and tenat-admin.

branches/life-cycle/implementation-specs/specs/BrinkPosOrdersFirstEtlJob/TASK3.md 11,656 bytes · Updated Oct 23, 2026 10:44 AM EDT

Raw File

TASK 3: Batch Lifecycle, Checkpoints, And Failure Packaging

Summary

Finalize the reusable ETL runtime contract for stage-level tenant-day batches across the Brink orders flow.

This task turns the metadata decisions from TASK1.md and the first real replay boundary from TASK2.md into a concrete stage lifecycle model for:

extract_raw
normalize_load
load_to_staging
staging_to_curated
publish_curated

This task owns:

per-stage tenant-day batch identity
upstream-batch references between stages
current-batch and supersede semantics for reruns
checkpoint advancement rules
failed-package creation and safe metadata packaging under failed/
minimum traceability requirements across raw, load, staging, curated, and publish outputs

This task does not implement normalization mapping, Redshift SQL, or curated export logic itself. It defines the lifecycle contract those later tasks must use.

Dependencies

TASK 1 - source binding and dispatch
TASK 2 - extract raw XML

Subtasks

[ ] TASK 3.1 - Extend stage batch and checkpoint contracts
[ ] TASK 3.2 - Implement current-state, supersede, and upstream-batch semantics
[ ] TASK 3.3 - Implement failed-package handling and stage traceability
[ ] TASK 3.4 - Add unit tests and live verification for lifecycle semantics

Files to Create

1. Stage lifecycle services and contracts

Create or finalize the ETL/Warehouse-facing services that own stage batch creation, completion, rerun linkage, checkpoint updates, and failed-package publication.

Required logical responsibilities:

stage batch factory or lifecycle coordinator
current successful batch resolver
upstream batch resolver for downstream stages
checkpoint writer/reader
failed-package writer

Recommended logical services:

IngestionBatchLifecycleService
CurrentIngestionBatchResolver
StageUpstreamBatchResolver
IngestionCheckpointService
FailedPackageWriter

2. Stage batch metadata models

Create or finalize the minimum models needed so every stage batch can be queried and replayed without inferring state from ad-hoc JSON.

Required minimum details:

company_id
tenant_id
business_date
pipeline_stage
lifecycle status and timestamps
details_json
error_details
reset_of_batch_id
is_current
explicit upstream batch reference for downstream stages

3. Failed-package payload models

Create or finalize the minimum payloads written to failed/ for a failed batch attempt.

Required minimum artifacts:

error.json
optional diagnostics.json
references to already-written upstream artifacts when available

Files to Modify

4. Ingestion batch persistence and repository surface

Modify the ingestion-batch model, mapping, and query surface so stage-level current-state resolution, rerun lineage, and upstream-batch selection are first-class operations.

Required repository/query capabilities:

create a stage batch for one tenant-day logical key
resolve the current successful batch for one stage and tenant-day key
resolve historical superseded and failed batches
link a successful rerun to the prior current successful batch
read or write the explicit upstream batch reference

5. Checkpoint persistence and update path

Modify the checkpoint path so checkpoints remain tenant-scoped and stage-specific, and advance only after full stage success.

This task should keep the logical checkpoint contract generic and should not introduce Brink-specific watermark names into the shared checkpoint model.

6. Shared ETL storage failed-package path

Modify the shared storage path so failed batch attempts can publish the minimum safe package under failed/ without copying raw payloads when an existing artifact reference is already available.

7. Stage orchestration touch points

Modify the stage orchestration from TASK2.md and prepare later stages so they call the shared lifecycle contract instead of implementing ad-hoc is_current, checkpoint, or failed-package rules in each module.

Required Behavior

Scope rule

This task defines one lifecycle contract reused by every stage in this slice:

extract_raw
normalize_load
load_to_staging
staging_to_curated
publish_curated

The task may be implemented incrementally starting from extract_raw, but the contract must already fit all listed stages.

Stage identity rule

every stage owns its own tenant-day batch identity
the logical batch key is tenant-scoped and stage-scoped
company_id is stored for traceability, but current-state and checkpoint resolution stay tenant-scoped
the logical key components are tenant_id + business_date + pipeline_stage
the exact serialized checkpoint key format may vary, but it must be deterministic and derived from those components

Upstream reference rule

every stage after extract_raw must store an explicit reference to the successful upstream batch it consumed
downstream stages must resolve upstream input from either:
an explicitly requested upstream batch id
the current successful upstream batch for the same tenant-day key
downstream stages must not infer upstream input from newest-created rows only

Current-state rule

is_current = true marks the current successful batch for one tenant-day stage key
only successful batches may become current
failed batches may never become current
partial work may persist artifacts for diagnostics, but partial completion is never current or publishable
downstream readers must resolve current state explicitly rather than using the newest batch record

Supersede rule

rerun creates a new batch; it does not overwrite the previous batch record
a successful rerun must link to the previous current successful batch through reset_of_batch_id or an equivalent explicit lineage field
when the rerun succeeds, the previous current successful batch becomes is_current = false
supersede semantics are stage-local; rerunning one stage does not automatically mutate current-state flags for other stages
prior successful and failed batches remain queryable for audit and debugging

Checkpoint rule

checkpoints are tenant-scoped and stage-specific
each stage advances its own checkpoint only after the full tenant-day stage succeeds
failed or partial stage attempts must not advance checkpoint state
stream_name and endpoint detail remain diagnostics only and must not become part of the primary stage checkpoint key
checkpoint advancement is the single unambiguous success point for stage freshness

Failure packaging rule

failed-package creation is mandatory at the failed batch-attempt level
batch-level failed packaging may aggregate one or many endpoint or target failures
the failed package must include at least error.json
diagnostics.json should be written when safe endpoint or target detail exists
when an upstream raw, load, or curated artifact already exists, the failed package should reference it instead of duplicating it
failed-package payloads and object metadata must not contain secrets, tokens, auth headers, or raw request headers

Traceability rule

The lifecycle contract must preserve enough metadata to trace one tenant-day run across stages.

Minimum traceability expectations:

request/trigger context
batch id per stage
upstream batch id per downstream stage
raw artifact URI references
load artifact URI references
staging target or SQL execution identifiers
curated target identifiers
publish/export artifact URI references

Replay rule

replay starts from persisted upstream artifacts, not from in-memory DTOs
stage rerun may start from an explicitly selected successful upstream batch or from the current successful upstream batch
failed packages support investigation and replay selection, but they do not replace the upstream persisted artifact as the replay boundary

Testing

Unit tests

Unit tests are mandatory for this task.

Add unit tests for:

stage batch creation for each pipeline stage key
current-batch resolution for one tenant-day stage
successful rerun superseding the previous current batch
failed rerun leaving the previous current batch unchanged
upstream batch selection by explicit id and by current-state lookup
checkpoint advancement only after full success
checkpoint non-advancement on failure or partial completion
failed-package creation with artifact references instead of payload copying
diagnostics and failed-package sanitization
repository/query behavior for historical superseded and failed batches

Live verification

This task defines operational behavior across real ETL boundaries, so live verification is required whenever Aspire-managed integrations from TASK2.md are available.

Add live verification that proves at least:

a successful rerun of the same tenant-day extract_raw stage supersedes the previous successful batch
current-batch resolution returns the latest successful non-superseded batch rather than the newest row blindly
checkpoint state advances only after a successful stage completion
a failed tenant-day attempt produces a failed/ package with safe metadata and artifact references
failure packaging can be correlated back to the batch and any persisted raw artifact without leaking secrets

Recommended live verification shape:

run one successful tenant-day extract_raw batch
rerun the same tenant-day with ForceReset and assert current/superseded transitions
trigger or simulate one controlled failed attempt and assert failed/ packaging plus unchanged checkpoint state
assert the resulting metadata is enough to identify batch lineage and replay inputs

If a controlled live failure cannot be induced safely in one environment, document the blocker explicitly and keep the unit-test coverage mandatory.

Acceptance Criteria

[ ] Stage batch contract exists for extract_raw, normalize_load, load_to_staging, staging_to_curated, and publish_curated
[ ] Each stage uses a tenant-day logical batch key based on tenant_id, business_date, and pipeline_stage
[ ] Downstream stages store an explicit upstream batch reference
[ ] Current successful batch resolution exists and does not depend on newest-row inference
[ ] Successful reruns supersede only prior successful batches of the same stage and tenant-day key
[ ] Failed or partial batches never become current
[ ] Checkpoints advance only after full stage success
[ ] Failed stage attempts do not advance checkpoints
[ ] Failed batch attempts create a minimal safe package under failed/
[ ] Failed packages reference existing artifacts when available instead of duplicating them
[ ] Lifecycle metadata supports traceability across batch, artifact, and upstream lineage boundaries
[ ] Unit tests cover supersede semantics, checkpoint rules, upstream selection, and failure packaging
[ ] Live verification covers rerun/current-state and failed-package behavior, or a concrete blocker is documented
[ ] Solution builds without errors

Notes

Keep the checkpoint model generic; do not encode Brink-only concepts into shared ETL metadata.
Keep stage current-state semantics explicit and queryable; downstream code must not rely on creation order alone.
Preserve immutable upstream artifacts on rerun and failure.
Keep failed-package storage minimal and safe; richer operational tooling belongs to later tasks.