Spensara Spensara Knowledge Base & Documentation Repository
System Overview Brink & Borg Master Index (722 Docs) Spensara Platform Project Tracker

Repository Docs & Knowledge Base

Consolidated Spensara documentation repository (722 Markdown & architecture documents in ../repository)
Built Word Documents (.docx): Spensara Overview (.docx) Capability Assessment (.docx) Brink POS Ingestion Analysis (.docx) The Borg Ingestion Architecture (.docx)

Categories

Branches (328)

Branch-specific documentation across extract-raw-xml, feature-fleet-import, kds_cdc_improvments, life-cycle, and tenat-admin.
branches/feature-fleet-import/implementation-specs/specs/ImportFromS3ToRedshift/TASK0.md 7,717 bytes · Updated Oct 23, 2026 10:44 AM EDT
Raw File

TASK 0: Freeze Fleet Import Contracts, Constants, Options, And Phase Details

Implementation Status

Completed: 2026-07-22

  • Added frozen Fleet constants, canonical source identity normalization, server-bound options, and startup validation under backend/Kds.Ai.Integrations/FleetS3/.
  • Added immutable snapshot, Redshift replacement, Athena publication, orchestration, cleanup, durable details, and bounded diagnostic contracts without AWS SDK or Npgsql types.
  • Added web-JSON details serialization with exact frozen snake_case phase values.
  • Added focused TUnit coverage for constants, identity, option safety, all phase round-trips, bounded diagnostics, and forbidden sensitive/payload detail fields.
  • Verification: Kds.Ai.Integrations.Tests passed 159 runnable tests; 11 opt-in live tests were skipped. The targeted warnings-as-errors build passed with zero warnings.
  • Repository-wide warnings-as-errors build is currently blocked outside TASK 0 by existing NU1903 findings for Microsoft.OpenApi 2.0.0 in Kds.Ai.ApiService and its test project.

Summary

Create the small, Fleet-specific contract layer used by storage, Redshift, publication, orchestration, and API work. Freeze names and invariants before parallel implementation begins.

Dependencies

Files To Create

Recommended logical locations:

  • backend/Kds.Ai.Integrations/FleetS3/FleetS3ImportConstants.cs
  • backend/Kds.Ai.Integrations/FleetS3/Configuration/FleetS3ImportOptions.cs
  • backend/Kds.Ai.Integrations/FleetS3/Configuration/FleetS3ImportOptionsValidator.cs
  • backend/Kds.Ai.Integrations/FleetS3/Models/FleetS3ImportInput.cs
  • backend/Kds.Ai.Integrations/FleetS3/Models/FleetS3ImportPhase.cs
  • backend/Kds.Ai.Integrations/FleetS3/Models/FleetS3ImportDetails.cs
  • source snapshot request/result and manifest-index summary records used by Task 2;
  • Redshift request, durable replacement-intent, result, and reconciliation records used by Task 3;
  • publication request/result/journal records used by Task 4;
  • focused bounded diagnostic records used by Task 5.

If S3 implementation options must live with the Warehouse S3 implementation, place the storage-only options under Kds.Ai.Warehouse/Configuration and expose them through a Fleet storage registration extension. Do not introduce circular project references.

Fixed Constants

NameValue
source systemfleet_s3
source discriminator prefixfleet_s3:
pipeline stagefleet_s3_direct_to_curated
schema version1.0
currencyUSD
order targetcurated.pos_orders
item targetcurated.pos_order_items
default source bucketkds-warehouse-ai-test
default source rootfleet/
ingestion modeFileUpload

Validate sourceFolder with ^[A-Za-z0-9][A-Za-z0-9_-]{0,63}$ and preserve its exact case. The canonical value is the validated submitted value; do not lowercase it because S3 keys are case-sensitive. Use it consistently in S3 boundaries, store_id, terminal validation, operational source scope, and _etl_source_type.

Required Options

Server-controlled options must cover:

  • source bucket and fixed fleet/ root;
  • batch workspace root outside query-visible curated/ data;
  • bounded conditional-copy concurrency;
  • maximum source object size, strictly below Redshift's 4 MiB JSON object limit;
  • maximum validation sample count and maximum persisted error length;
  • Redshift/import command timeout when the shared timeout is not sufficient;
  • retained workspace duration after success/failure;
  • publication retry lifetime, after which retained artifacts may be removed and a new full import is required;
  • staged Athena export root outside query-visible curated/ data.

Validation must reject:

  • blank bucket/root/workspace values;
  • a workspace under the customer source root;
  • a staging workspace under the stable query-visible Fleet prefix;
  • non-positive concurrency, timeout, size, sample, or retention values;
  • source object limit greater than or equal to 4,194,304 bytes.

The caller must never override these options.

Durable Details Contract

FleetS3ImportDetails is a versioned JSON document stored in IngestionBatch.DetailsJson. Use web JSON conventions and include a details version. It must remain small and contain summaries/URIs rather than one record per source object.

Minimum fields:

  • details version and importer version;
  • batch, customer, tenant, tenant database, requested-by identity;
  • canonical source folder, source scope, override flag;
  • current phase and phase timestamps;
  • source prefix, snapshot/index URI, source object count/bytes;
  • min/max business date and populated date count;
  • candidate order/item counts and bounded validation summary;
  • old/new affected date sets or an S3 URI when the set is externalized;
  • durable replacement-intent ID/status, recorded before permanent Redshift mutation;
  • deleted/inserted Redshift counts;
  • redshiftCommitted and commit timestamp;
  • per-date/entity publication journal or journal URI;
  • publication completed flag;
  • cleanup eligibility timestamp;
  • bounded failure category/message and retryability.

Freeze these phases:

text
reserved
snapshotting
snapshot_ready
loading_redshift
validating
replacing_redshift
redshift_committed
publishing_athena
completed
failed_pre_commit
failed_publication
publication_expired
superseded

Fixed Athena Publication Contract

Task 0 must verify the deployed Glue/Athena table location for each entity before Tasks 2 through 4 begin. The fixed active object key is:

text
curated/company_id={customerId}/tenant_id={tenantId}/entity={entity}/
dt={yyyy-MM-dd}/source_type=fleet_s3/source_id={canonicalSourceFolder}/data.parquet

The table's partition location must resolve to or above the corresponding dt={yyyy-MM-dd}/ directory so Athena recursively scans the source descendants. If the deployed table does not do this, update the environment's Glue table location/projection as a release prerequisite. Do not make runtime key ordering configurable and do not defer the decision to Task 7.

Public Service Contracts

Freeze narrowly scoped interfaces before implementations:

  • source snapshot service: request -> immutable snapshot/index result;
  • Redshift import executor: snapshot + warehouse + lineage -> replacement result;
  • replacement-intent checkpoint: persist old/new/affected dates and expected counts before Redshift deletes or commit;
  • Athena publisher: replacement result + lineage -> publication result;
  • import orchestrator: execute/resume one reserved batch;
  • workspace cleanup service.

Do not expose AWS SDK or Npgsql types through these contracts.

Testing

Add unit tests for:

  • source folder validation and normalization;
  • exact-case folder behavior, including distinct Front-Counter and front-counter prefixes;
  • exact source scope/discriminator generation and 100-character limit;
  • options validation;
  • details JSON round-trip for every phase;
  • fixed Athena key and deployed table-location compatibility;
  • bounded diagnostics behavior;
  • no credentials, SQL, or full payload fields in details models.

Acceptance Criteria

  • all later tasks share one constants/options/details contract;
  • source identity cannot drift between Redshift, S3, and operational metadata;
  • options are server-controlled and fail at startup when unsafe;
  • details are versioned and resumable without containing the complete source inventory;
  • no generic ETL contract is changed only to accommodate Fleet.