# TASK 3.3: Implement Failed-Package Handling And Stage Traceability

## Summary

Implement the minimum failed-package and cross-stage traceability model so failed tenant-day stage attempts can be investigated and correlated back to their persisted upstream artifacts.

This subtask closes the operational gap left after raw capture by making failure output safe, structured, and stage-aware.

## Dependencies

- [TASK 2](./TASK2.md) - extract raw XML
- [TASK 3.1](./TASK3.1.md) - stage batch and checkpoint contracts

## Files to Create

### 1. Failed-package payload writers

Create or finalize the components that write the minimum batch-level failed package under `failed/`.

Required artifacts:

- `error.json`
- optional `diagnostics.json`
- references to existing raw, load, or curated artifacts when available

Recommended logical service:

- `FailedPackageWriter`

## Files to Modify

### 2. Shared ETL storage failed path

Modify the shared storage path so failed packages are created by batch attempt and can reference existing artifacts rather than duplicating payload content.

### 3. Stage diagnostics payload shape

Modify or finalize `details_json` conventions so traceability exists across:

- trigger and requester context
- stage batch ids
- upstream batch ids
- persisted artifact URIs
- stage target or endpoint execution summaries

## Required Behavior

- failed-package creation happens at the failed batch-attempt level
- `error.json` includes batch scope, stage, business date, failure kind, and message
- `diagnostics.json` carries safe stage or endpoint context when available
- failed packages must reference existing persisted artifacts when they already exist
- failed packages must not copy raw payloads into a parallel location when a stable upstream reference already exists
- diagnostics and failed artifacts must not include secrets or auth material
- batch metadata must preserve enough traceability to follow one tenant-day run across stage and artifact boundaries

## Testing

### Unit tests

Add unit tests for:

- `error.json` minimum shape
- `diagnostics.json` minimum shape
- original artifact reference behavior
- no secret leakage in failed payloads or metadata
- traceability links from batch to upstream batch and persisted artifact URIs

## Acceptance Criteria

- [ ] Failed batch attempts create `error.json`
- [ ] Safe diagnostics payload can be created when contextual details exist
- [ ] Existing upstream artifacts are referenced rather than copied
- [ ] Failed-package and diagnostics payloads are sanitized
- [ ] Batch metadata is traceable across stage and artifact boundaries
- [ ] Unit tests cover failed-package and traceability behavior

## Notes

- Keep failure packaging minimal in v1.
- Richer operator workflows or endpoint-level standalone failed packages belong to later tasks.