Spensara Spensara Knowledge Base & Documentation Repository
System Overview Brink & Borg Master Index (722 Docs) Spensara Platform Project Tracker

Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)
Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Categories

Implementation Specs (157)

Detailed engineering and feature implementation specifications.
implementation-specs/specs/RedshiftSyntheticDataFeed/TASK2.2.md 2,697 bytes · Updated Oct 23, 2026 10:44 AM EDT
Raw File

TASK 2.2: Add Idempotency, Checkpoint, and Full-Refresh Behavior

Summary

Add the operational behavior that makes the synthetic feed safe to rerun.

This task adds minimal checkpointing and/or deterministic full-refresh support. It does not implement production replay, DLQ, or backfill tooling.

Dependencies

  • TASK 2.1 (tenant-resolved Redshift curated writer)

Files to Create

1. Checkpoint/full-refresh support

Create services or methods needed to:

  • read synthetic feed checkpoints,
  • update checkpoints only after successful writes,
  • run a deterministic full refresh for the configured tenant/store.

Exact placement depends on the metadata abstractions created in Redshift Foundation.

Files to Modify

2. Feed service orchestration

Extend the synthetic feed orchestration to:

  • create/update ingestion batch skeleton records where available,
  • use checkpoint values for incremental sync,
  • support full-refresh mode,
  • update checkpoints only after success,
  • include row counts in the feed result.

Required Behavior

Minimum acceptable behavior:

  • deterministic keys:
  • order_id for orders,
  • order_item_id for order items,
  • rerunning the same source window does not duplicate rows,
  • updated source rows are reflected in Redshift,
  • failed feed runs do not advance checkpoints,
  • full-refresh mode is tenant-scoped and source-store-scoped,
  • full-refresh mode does not affect unrelated tenants or stores.

Recommended checkpoint identity:

  • source_type = synthetic-kds
  • checkpoint_key = store:{store_guid}:orders
  • checkpoint_key = store:{store_guid}:order_items

Recommended batch metadata:

  • ingestion_mode = synthetic-feed
  • source_type = synthetic-kds
  • tenant_id = configured target tenant
  • tenant_database = resolved target database
  • raw_s3_path = null
  • load_s3_path = null

Acceptance Criteria

  • [x] Feed can run safely more than once for the same source rows
  • [x] Feed can update Redshift rows when source rows change
  • [x] Feed supports deterministic full-refresh mode
  • [x] Full refresh is scoped to the selected tenant/store
  • [x] Incremental checkpoint behavior exists or a clear full-refresh-only mode is documented
  • [x] Checkpoints advance only after successful Redshift writes
  • [x] Failed runs do not advance checkpoints
  • [x] Batch metadata is recorded where the slice-01 skeleton is available
  • [x] Batch metadata does not require S3 paths
  • [x] Solution builds without errors

Notes

  • This task is intentionally minimal operational hardening.
  • Do not build production replay, backfill, DLQ, or S3 raw retention here.
  • Idempotency matters more than sophisticated scheduling in this slice.