Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)

Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Implementation Specs (157)

Detailed engineering and feature implementation specifications.

implementation-specs/specs/BrinkPosOrdersFirstEtlJob/TASK3.3.md 2,843 bytes · Updated Oct 23, 2026 10:44 AM EDT

Raw File

TASK 3.3: Implement Failed-Package Handling And Stage Traceability

Summary

Implement the minimum failed-package and cross-stage traceability model so failed tenant-day stage attempts can be investigated and correlated back to their persisted upstream artifacts.

This subtask closes the operational gap left after raw capture by making failure output safe, structured, and stage-aware.

Dependencies

TASK 2 - extract raw XML
TASK 3.1 - stage batch and checkpoint contracts

Files to Create

1. Failed-package payload writers

Create or finalize the components that write the minimum batch-level failed package under failed/.

Required artifacts:

error.json
optional diagnostics.json
references to existing raw, load, or curated artifacts when available

Recommended logical service:

FailedPackageWriter

Files to Modify

2. Shared ETL storage failed path

Modify the shared storage path so failed packages are created by batch attempt and can reference existing artifacts rather than duplicating payload content.

3. Stage diagnostics payload shape

Modify or finalize details_json conventions so traceability exists across:

trigger and requester context
stage batch ids
upstream batch ids
persisted artifact URIs
stage target or endpoint execution summaries

Required Behavior

failed-package creation happens at the failed batch-attempt level
error.json includes batch scope, stage, business date, failure kind, and message
diagnostics.json carries safe stage or endpoint context when available
failed packages must reference existing persisted artifacts when they already exist
failed packages must not copy raw payloads into a parallel location when a stable upstream reference already exists
diagnostics and failed artifacts must not include secrets or auth material
batch metadata must preserve enough traceability to follow one tenant-day run across stage and artifact boundaries

Testing

Unit tests

Add unit tests for:

error.json minimum shape
diagnostics.json minimum shape
original artifact reference behavior
no secret leakage in failed payloads or metadata
traceability links from batch to upstream batch and persisted artifact URIs

Acceptance Criteria

[x] Failed batch attempts create error.json
[x] Safe diagnostics payload can be created when contextual details exist
[x] Existing upstream artifacts are referenced rather than copied
[x] Failed-package and diagnostics payloads are sanitized
[x] Batch metadata is traceable across stage and artifact boundaries
[x] Unit tests cover failed-package and traceability behavior

Notes

Keep failure packaging minimal in v1.
Richer operator workflows or endpoint-level standalone failed packages belong to later tasks.