TASK 4: Implement Tenant Resolution And Quarantine Boundaries

Summary

Implement the KDS CDC tenant-resolution seam that maps source customer/account and store_guid into platform company_id, tenant_id, and tenant warehouse identity before any tenant-scoped raw S3 path or Redshift load is created.

This task freezes the mapping contract, explicit resolution rules, ambiguous/missing mapping handling, and the quarantine boundary for record-local mapping failures. It also defines how unresolved records are diagnosed and replayed without leaking into tenant-scoped raw/load/curated paths.

This task does not yet implement raw S3 persistence, durable job creation, normalization, Redshift COPY, curated merges, or checkpoint replay mechanics.

Current Codebase Findings

Repository analysis shows the codebase already has tenant warehouse resolution, but it does not yet have the source-account plus store_guid mapping seam required by the KDS CDC feature.

• backend/Kds.Ai.Warehouse/Abstractions/ITenantWarehouseResolver.cs and backend/Kds.Ai.Warehouse/Services/MetadataTenantWarehouseResolver.cs only resolve tenant_id -> tenant database using ITenantWarehouseMappingRepository.
• backend/Kds.Ai.Data/Repositories/ITenantWarehouseMappingRepository.cs and EfTenantWarehouseMappingRepository.cs store only tenant-to-warehouse metadata: TenantId, TenantDatabase, activation state, and provisioning state.
• backend/Kds.Ai.Data/Entities/Customer.cs and ICustomerRepository.cs expose tenant-facing customer data such as TenantId and Company, but there is no current repository that maps external KDS source account/store identity into platform company/tenant identity.
• backend/Kds.Ai.Warehouse/Services/Synthetic/SyntheticKdsFeedService.cs and SimulatorWarehousePublisher.cs require a caller-supplied tenant id up front and then resolve warehouse from that tenant. This is opposite of KDS CDC, where source account/store must be resolved into tenant identity first.
• backend/Kds.Ai.Integrations/Etl/StageExecutionSupport.cs, backend/Kds.Ai.Warehouse/Services/Storage/FailedPackageWriter.cs, backend/Kds.Ai.Warehouse/Models/Storage/FailedPackageWriteRequest.cs, and backend/Kds.Ai.Integrations/Etl/StageBatchFailureRequest.cs all assume failure packaging is stage-scoped and already has CompanyId, TenantId, BatchId, and PipelineStage.
• backend/Kds.Ai.Warehouse/Services/Storage/S3EtlStorage.cs builds raw, load, curated, and failed paths with mandatory company_id and tenant_id key segments.
• backend/Kds.Ai.Data/Models/Ingestion/CreateIngestionBatchRequest.cs requires TenantId and TenantDatabase when creating stage batches, with optional CompanyId.

These findings matter for Task 4:

1. current warehouse mapping is downstream of tenant identity, not a replacement for source-account/store resolution,
2. unresolved or ambiguous CDC records cannot reuse current stage-level failed-package writer directly because that writer requires tenant-scoped identity,
3. KDS CDC needs a new explicit mapping seam plus a pre-tenant quarantine seam before Task 5 raw persistence begins.

Dependencies

• docs/features/ETL/kds-cdc/index.md
• docs/features/ETL/kds-cdc/implementation-breakdown.md
• specs/KdsCdcIngestion/TASK2.md
• specs/KdsCdcIngestion/TASK3.md

Source References

• ../../docs/features/ETL/kds-cdc/implementation-breakdown.md#task-checklist
• ../../docs/features/ETL/kds-cdc/index.md#stage-3-tenant-resolution
• ../../docs/features/ETL/kds-cdc/index.md#quarantine-and-failed-packages
• ../../docs/features/ETL/kds-cdc/index.md#supported-corner-cases
• ../../docs/features/ETL/kds-cdc/index.md#selected-behavior-summary

Scope From Breakdown

This task covers only the fourth implementation-breakdown item:

• map source customer/account to company_id and store_guid to tenant_id
• block implicit or ambiguous tenant inference
• quarantine record-local mapping failures with enough metadata for diagnosis and replay
• ensure unresolved records never reach tenant-scoped S3 paths or Redshift loads

Files to Create

1. KDS CDC source-to-tenant resolution seam

Create a dedicated resolution seam for KDS CDC source identity.

Recommended logical location:

• backend/Kds.Ai.Integrations/KdsCdc/Resolution/

Recommended logical types:

• IKdsCdcTenantResolver
• KdsCdcTenantResolver
• KdsCdcTenantResolutionRequest
• KdsCdcTenantResolutionResult
• KdsCdcResolvedTenantContext

Minimum responsibilities:

• accept source-facing identity from CDC message metadata,
• resolve source_account_id -> company_id,
• resolve source_account_id + store_guid -> tenant_id,
• resolve tenant_id -> tenant database through existing warehouse resolver or repository seam,
• return explicit success, missing-mapping, or ambiguous-mapping outcomes,
• keep mapping logic separate from worker polling and later raw persistence logic.

2. Source mapping repository or service contract

Create a persistence-facing seam for the mapping table or mapping service described by the feature doc.

Recommended logical locations:

• backend/Kds.Ai.Data/Repositories/
• backend/Kds.Ai.Data/Entities/
• backend/Kds.Ai.Data/Configurations/

Recommended logical types:

• IKdsCdcSourceTenantMappingRepository
• EfKdsCdcSourceTenantMappingRepository
• KdsCdcSourceTenantMapping

Recommended fields for the mapping record:

• source account/customer id
• source store_guid
• resolved company_id
• resolved tenant_id
• optional source system discriminator if future KDS variants exist
• activation flag or effective state if mappings can be disabled
• audit timestamps

Important rule:

• mapping key must be explicit and deterministic,
• Task 4 must not depend on fuzzy matching against customer names, company names, or inferred store ownership.

3. Pre-tenant quarantine seam

Create a quarantine seam for mapping failures that happen before tenant identity is known.

Recommended logical locations:

• backend/Kds.Ai.Integrations/KdsCdc/Quarantine/
• or backend/Kds.Ai.Warehouse/Services/Storage/ only if storage behavior is intentionally shared

Recommended logical types:

• IKdsCdcQuarantineWriter
• KdsCdcQuarantineWriter
• KdsCdcQuarantineRecord
• KdsCdcQuarantineReason

Why new seam is needed:

• current IFailedPackageWriter assumes CompanyId and TenantId already exist,
• Task 4 failures occur before tenant-safe S3 path creation,
• unresolved records still need durable diagnostics and replay reference.

Task 4 should explicitly choose one of these directions:

1. dedicated unresolved quarantine storage path not requiring tenant id,
2. generalized failed-package writer expanded to support pre-tenant failures.

Preferred direction:

• use dedicated KDS CDC quarantine seam for unresolved records instead of widening stage-scoped failed-package behavior too early.

Files to Modify

4. Integration wiring from worker intake

Modify likely KDS CDC intake types from Task 2 so accepted messages flow through tenant resolution before any downstream raw handoff.

Expected areas:

• backend/Kds.Ai.ApiService/Jobs/KdsCdc/ worker or pump types introduced by Task 2
• backend/Kds.Ai.Integrations/KdsCdc/ intake models and orchestration seams

Required flow change:

• parse and validate message,
• resolve company and tenant,
• quarantine mapping failures,
• only resolved records proceed into tenant-scoped batch shaping for Task 5.

5. Data model and repository registration

Modify:

• backend/Kds.Ai.Data/KdsAiDbContext.cs
• backend/Kds.Ai.Data/DependencyInjectionExtensions.cs
• any new EF configuration and migration files needed for source-tenant mapping persistence

Task 4 should define whether mapping storage lives in kdsaidb as first-party configuration metadata. Current repo patterns suggest yes.

6. Warehouse resolution composition

Review and likely modify:

• backend/Kds.Ai.Warehouse/Services/MetadataTenantWarehouseResolver.cs
• backend/Kds.Ai.Warehouse/Abstractions/ITenantWarehouseResolver.cs
• backend/Kds.Ai.Data/Repositories/ITenantWarehouseMappingRepository.cs

Important boundary:

• do not overload ITenantWarehouseResolver with source-account lookup semantics,
• keep source-account/store resolution as upstream seam,
• reuse existing tenant-to-database resolution only after tenant_id is already resolved.

7. Dependency injection and configuration

Modify likely:

• backend/Kds.Ai.Integrations/DependencyInjectionExtensions.cs
• backend/Kds.Ai.ApiService/Program.cs only if resolution/quarantine services are host-registered there
• backend/Kds.Ai.AppHost/AppHostExtensions.cs
• backend/Kds.Ai.AppHost/AppHost.cs

Configuration may need to cover:

• enablement of KDS CDC mapping repository/service,
• local seed or default mapping data for mock CDC flows,
• unresolved-record quarantine bucket/prefix if separate from stage failed packages.

Task 4 must include a dev/local seed strategy for the mock contract path:

• seed or configure source_account_id + store_guid -> company_id + tenant_id,
• align seeded source_account_id with Task 3 simulator producer defaults,
• keep seed data explicit so tests do not infer tenant from ambient simulator request tenancy.

8. Tests

Add or modify tests under:

• backend/tests/Kds.Ai.Integrations.Tests/KdsCdc/
• backend/tests/Kds.Ai.Data.Tests/
• backend/tests/Kds.Ai.ApiService.Tests/ if host/composition behavior is verified there

Required Behavior

Resolution must happen before tenant-scoped artifacts

Task 4 must enforce feature-doc rule that tenant resolution happens before:

• tenant-scoped raw S3 path creation,
• tenant Redshift database resolution for load,
• ingestion batch creation that requires tenant identity,
• failed-package writes that assume tenant-scoped storage.

Unresolved records must stop before Task 5 raw persistence path.

Explicit mapping rules

Feature doc mapping rule is fixed:

source customer/account -> company_id
source store_guid -> tenant_id within company_id
company_id + tenant_id -> tenant Redshift database

Task 4 must keep these as explicit mappings, not inferred equivalence rules.

Important consequences:

• source field names do not need to literally be company_id or tenant_id,
• mapping can come from table or service,
• mapping result must produce platform company_id and tenant_id values used by later S3 and Redshift paths.

No implicit or ambiguous inference

Task 4 must explicitly reject:

• missing source account mapping,
• missing store_guid mapping within known company,
• multiple tenants matching same source identity,
• best-effort fallback to default tenant,
• deriving tenant from simulator defaults or ambient request tenancy,
• deriving company from customer display name or other fuzzy field.

If mapping is missing or ambiguous, record is quarantined and not loaded.

Resolved tenant context shape

A successful resolution result must carry enough identity for later tasks to proceed without another ambiguous lookup round.

Minimum resolved context:

• CompanyId
• TenantId
• TenantDatabase
• source account/customer id
• source StoreGuid
• mapping evidence identifier or record details useful for diagnostics

Task 4 should freeze whether inactive tenant warehouse mapping is treated as:

1. mapping failure quarantine, or
2. downstream batch failure.

Recommended direction:

• treat missing/ambiguous source mapping as record-local quarantine,
• treat resolved tenant with missing/inactive warehouse mapping as blocking failure for that resolved tenant slice because tenant identity is known but runtime destination is unusable.

Quarantine metadata requirements

Mapping-failure quarantine must include enough data for diagnosis and replay, matching feature doc intent.

Required metadata:

• source customer/account value
• source store_guid
• source table/entity
• source primary key
• Debezium source coordinates
• SQS message id
• SQS FIFO group id and ordering value
• raw payload or raw payload pointer when available
• reason mapping failed
• whether failure was missing mapping or ambiguous mapping

If Task 4 occurs before raw S3 write, quarantine record should preserve original message body or safe pointer so mapping can be repaired and replayed later.

Record-local vs batch-level boundary

Task 4 must preserve main feature distinction:

• missing or ambiguous source mapping is record-local quarantine,
• good records in same poll window continue,
• unresolved records never force all good records to fail,
• later batch-level failures still use stage lifecycle plus failed-package flow once tenant identity exists.

Task 4 should clearly separate:

1. pre-tenant quarantine for bad records,
2. tenant-scoped stage failures after resolution succeeds.

No unresolved writes to tenant paths

Task 4 must guarantee unresolved records never reach:

• raw/company_id=.../tenant_id=...
• load/company_id=.../tenant_id=...
• curated/company_id=.../tenant_id=...
• tenant warehouse databases in Redshift
• stage-batch creation calls requiring TenantId and TenantDatabase

This is core safety boundary for multi-tenant correctness.

Fit with Task 2 and Task 5

Task 4 should preserve this handoff model:

1. Task 2 intake validates message transport/body/order shape,
2. Task 4 resolves tenant identity and quarantines mapping failures,
3. Task 5 persists only resolved records into raw S3 and durable batch state.

Task 4 must not collapse these responsibilities into one opaque worker step.

Proposed Implementation Shape

Recommended high-level flow:

KdsCdc worker intake
  -> parsed/validated CDC message
  -> IKdsCdcTenantResolver.ResolveAsync(...)
      -> success: resolved company_id + tenant_id + tenant database
      -> missing/ambiguous mapping: quarantine record with full source diagnostics
  -> only resolved records continue to tenant/entity micro-batch handoff for Task 5

Recommended resolver contract shape:

public interface IKdsCdcTenantResolver
{
    Task<KdsCdcTenantResolutionResult> ResolveAsync(
        KdsCdcTenantResolutionRequest request,
        CancellationToken ct = default);
}

Recommended resolution request shape:

KdsCdcTenantResolutionRequest
  SourceAccountId
  StoreGuid
  Entity
  PrimaryKey
  MessageId
  MessageGroupId
  SequenceValue
  SourceCoordinates

Recommended resolution result shape:

KdsCdcTenantResolutionResult
  Outcome: Resolved | MissingMapping | AmbiguousMapping | InactiveTenantWarehouse
  CompanyId
  TenantId
  TenantDatabase
  Diagnostics

Recommended quarantine writer shape:

public interface IKdsCdcQuarantineWriter
{
    Task WriteAsync(KdsCdcQuarantineRecord record, CancellationToken ct = default);
}

Important design notes:

• keep source mapping lookup separate from warehouse lookup,
• keep resolver pure enough for unit testing,
• do not require CreateIngestionBatchRequest before resolution succeeds,
• do not force unresolved records through StageExecutionSupport.FailStageBatchAsync, because no tenant-scoped stage batch should exist yet.

Testing

Unit tests

Add tests for:

• resolver maps valid source_account_id + store_guid to expected company_id, tenant_id, and tenant database,
• missing source account mapping returns quarantine outcome,
• missing store mapping within known company returns quarantine outcome,
• ambiguous mapping returns quarantine outcome,
• resolver does not fall back to default tenant or ambient tenant context,
• inactive or missing tenant warehouse mapping produces documented outcome,
• quarantine record includes required source diagnostics and reason,
• unresolved record cannot produce tenant-scoped raw/load key inputs,
• resolved record carries enough identity for later batch creation.

Repository/data tests

Add tests for:

• mapping repository stores and retrieves source account plus store_guid mappings deterministically,
• uniqueness rules prevent ambiguous active mappings for same source identity,
• disabled/inactive mappings are handled per chosen resolution policy.

Integration/composition tests

Add tests proving:

• Task 2 worker-facing composition routes unresolved records to quarantine writer,
• good and bad records in same intake window split correctly,
• unresolved records do not call tenant-scoped stage-batch creation or tenant-scoped S3 writers,
• resolved records continue toward downstream handoff,
• current stage-scoped failed-package path remains reserved for post-resolution stage failures.

Not required in task 4

This task does not require:

• raw S3 persistence tests,
• durable job creation tests,
• Redshift load tests,
• curated merge/export tests,
• replay orchestration implementation,
• UI/admin workflows for maintaining mappings unless later user request adds them.

Acceptance Criteria

• [ ] KDS CDC has explicit source-account/store to company/tenant resolution seam
• [ ] Mapping rule follows source customer/account -> company_id and store_guid -> tenant_id within company_id
• [ ] No implicit tenant inference or best-effort fallback is allowed
• [ ] Missing mappings are quarantined with full source diagnostics
• [ ] Ambiguous mappings are quarantined with full source diagnostics
• [ ] Unresolved records never create tenant-scoped S3 paths or Redshift loads
• [ ] Resolver produces tenant database identity for resolved records, directly or via existing warehouse resolver seam
• [ ] Pre-tenant quarantine seam exists and is not incorrectly forced through current tenant-scoped failed-package writer
• [ ] Good records can continue even when bad records in same intake window are quarantined
• [ ] Local/dev mapping seed or configuration supports mock producer source_account_id + store_guid values explicitly
• [ ] Tests cover valid resolution, missing mapping, ambiguous mapping, and unresolved-write blocking behavior

Notes

• Follow docs/features/ETL/kds-cdc/index.md#stage-3-tenant-resolution as source of truth.
• Current ITenantWarehouseResolver is downstream resolver for known tenant id; it is not sufficient by itself for CDC source-account/store mapping.
• Current IFailedPackageWriter is tenant-scoped and batch-scoped; unresolved CDC mapping failures need a separate pre-tenant quarantine path unless later work intentionally generalizes storage contracts.
• Keep Task 4 focused on identity resolution and quarantine boundary. Raw persistence starts in Task 5.