# Fleet Pre-Ingestion Processing Architecture

## 1. Purpose

This document describes the proposed **Fleet pre-ingestion process** for customer-provided data files.

Fleet is a customer-facing pre-ingestion and schema-onboarding zone. It is used when a customer-side system uploads files whose structure must be identified, validated, versioned, and approved before the data can enter the standard ETL pipeline.

The key architectural decision is:

```text
Fleet is a pre-ingestion and schema contract onboarding zone.
The standard ETL starts only after a Fleet upload is validated and published to raw/source_type=fleet.
```

This means the platform keeps two raw representations:

1. **Fleet landing copy** — the original customer-provided upload exactly as received.
2. **Standard ETL raw copy** — the accepted and validated package published by KDSAI into the internal ETL structure.

The Fleet customer contract uses an atomic upload model:

```text
one upload package = one source data file = one manifest = one processing batch
```

Multiple source data files in one customer manifest are not supported in the initial architecture. If the customer needs to send several datasets, entities, or Excel worksheets, each one must be sent as a separate upload with a separate `upload_id`.

---

## 2. Key Architecture Rules

1. Customers upload only to Fleet landing.
2. Customers upload exactly one source data file per upload package.
3. Customers include exactly one `metadata/upload-manifest.json` per upload package.
4. Customers upload the manifest last; the manifest is the completion signal.
5. Customers do not send `contract_id` or `contract_version`.
6. Customers declare the source file structure using `schema.version`.
7. KDSAI owns schema approval, validation, mapping, and the internal approved contract registry.
8. KDSAI checks whether `company_id + tenant_id + dataset + schema.version` is already approved.
9. KDSAI publishes to standard ETL raw only after the schema version is approved and data validation passes.
10. The standard ETL worker does not negotiate schemas. It processes only Fleet packages that were already approved and published to internal raw.

---

## 3. High-Level Flow

```mermaid
flowchart TD
    A["Customer Fleet System"] --> B["Fleet Landing Zone<br/>one data file + manifest"]
    B --> C["Manifest Validation"]
    C --> D["Discovery, Profiling<br/>and Schema Fingerprint"]
    D --> E{"schema.version<br/>approved?"}
    E -- "No" --> F["Proposed Internal Schema<br/>and Mapping"]
    F --> G["Schema Review / Approval"]
    G --> H["Internal Approved Contract Registry"]
    E -- "Yes" --> I["Structure Check<br/>Against Approved Version"]
    H --> I
    I --> J{"Structure matches?"}
    J -- "No" --> K["Schema Drift<br/>Quarantine / Rejection"]
    J -- "Yes" --> L["Data Validation"]
    L --> M{"Validation passed?"}
    M -- "No" --> N["Validation Errors<br/>Quarantine / Rejection"]
    M -- "Yes" --> O["Fleet Ready Package"]
    O --> P["Publish to Standard ETL Raw"]
    P --> Q["Standard ETL Worker"]
    Q --> R["S3 Load-Ready Files"]
    R --> S["Tenant Redshift staging"]
    S --> T["Tenant Redshift curated"]
    T --> U["S3 curated Parquet"]
```

---

## 4. Bucket Strategy

### 4.1 Customer-visible bucket model

The architecture should use the same bucket and prefix model as the customer upload contract. The customer-facing upload location is a Fleet prefix inside the environment warehouse bucket.

Current customer-facing bucket:

```text
s3://kds-warehouse-ai-test/
```

Customer uploads go only to:

```text
s3://kds-warehouse-ai-test/fleet/landing/company_id={company_id}/tenant_id={tenant_id}/dataset={dataset}/upload_id={upload_id}/received_at={yyyy-MM-ddTHH-mm-ssZ}/
```

Example:

```text
s3://kds-warehouse-ai-test/fleet/landing/company_id=company-123/tenant_id=tenant-456/dataset=employees/upload_id=upl_20260602_001/received_at=2026-06-02T10-00-00Z/
```

### 4.2 Prefix responsibilities

| Prefix | Responsibility | Writer | Reader | Customer access |
|---|---|---|---|---|
| `fleet/landing/` | Customer-facing Fleet upload area | Customer system / Fleet upload process | Fleet onboarding service | Write only to assigned tenant/dataset prefix |
| `fleet/profiling/` | Profiling and schema discovery artifacts | Fleet onboarding service | Fleet onboarding service / operators | No customer write access |
| `fleet/schemas/` | KDSAI-owned proposed and approved schema artifacts | Fleet onboarding service / operators | Fleet onboarding service / ETL worker | No customer write access |
| `fleet/validation/` | Validation reports and rejected rows | Fleet onboarding service | Fleet onboarding service / support | No customer write access |
| `fleet/ready/` | Validated package ready for ETL publication | Fleet onboarding service | Fleet onboarding service | No customer write access |
| `fleet/quarantine/` | Rejected uploads and investigation artifacts | Fleet onboarding service | Fleet onboarding service / support | No customer write access |
| `raw/` | Internal standard ETL raw packages | Fleet onboarding service / ingestion adapters | ETL worker | No customer access |
| `load/` | Load-ready files for Redshift | ETL worker | ETL worker / Redshift | No customer access |
| `curated/` | Curated Parquet exported from Redshift | ETL worker / Redshift unload | Athena / analytics services | No customer access |
| `failed/` | Standard ETL failure artifacts | ETL worker | ETL worker / support | No customer access |

This keeps the customer contract simple: customers know only their assigned `fleet/landing/...` prefix. All other prefixes are internal KDSAI implementation details.

### 4.3 Security boundary

The initial architecture uses the `s3://kds-warehouse-ai-test/` bucket with strict prefix-level permissions. Separate physical buckets can still be introduced later if isolation, lifecycle policies, or account boundaries require it, but the customer-facing contract should remain stable and continue to point to the assigned `fleet/landing/...` prefix.

---

## 5. Fleet Prefix Structure

```text
s3://kds-warehouse-ai-test/fleet/
  landing/
    company_id={company_id}/
      tenant_id={tenant_id}/
        dataset={dataset}/
          upload_id={upload_id}/
            received_at={yyyy-MM-ddTHH-mm-ssZ}/
              source/
                original.xlsx
                original.csv
                original.json
                original.jsonl
                original.xml
                optional_schema.json
              metadata/
                upload-manifest.json

  profiling/
    company_id={company_id}/
      tenant_id={tenant_id}/
        dataset={dataset}/
          upload_id={upload_id}/
            inferred-schema.json
            schema-fingerprint.json
            data-profile.json
            sample-preview.json
            issues.json

  schemas/
    company_id={company_id}/
      tenant_id={tenant_id}/
        dataset={dataset}/
          schema_version={schema_version}/
            schema.json
            mapping.json
            status.json

  validation/
    company_id={company_id}/
      tenant_id={tenant_id}/
        dataset={dataset}/
          schema_version={schema_version}/
            batch_id={batch_id}/
              validation-report.json
              rejected-rows.jsonl
              accepted-preview.jsonl

  ready/
    company_id={company_id}/
      tenant_id={tenant_id}/
        dataset={dataset}/
          schema_version={schema_version}/
            batch_id={batch_id}/
              publish-manifest.json
              normalized/
                records.jsonl
              original/
                original.{extension}
              validation/
                validation-report.json

  quarantine/
    company_id={company_id}/
      tenant_id={tenant_id}/
        dataset={dataset}/
          upload_id={upload_id}/
            reason.json
            original/
```

Notes:

- `source-metadata.json` is not part of the initial customer contract.
- ZIP packages are not supported in the initial customer upload contract.
- `schemas/` contains KDSAI-owned approved or proposed schema artifacts. Customers do not write to this prefix.
- `schema_version` in S3 paths is the customer-declared `schema.version`, not a customer-facing KDSAI contract identifier.

---

## 6. Standard ETL Prefix Structure After Fleet Publication

After a Fleet upload is approved and validated, the Fleet onboarding service publishes a standard raw package into the internal standard ETL prefixes in the same `s3://kds-warehouse-ai-test/` bucket.

```text
s3://kds-warehouse-ai-test/
  raw/
    company_id={company_id}/
      tenant_id={tenant_id}/
        source_type=fleet/
          dataset={dataset}/
            schema_version={schema_version}/
              dt={yyyy-MM-dd}/
                batch_id={batch_id}/
                  manifest.json
                  schema-ref.json
                  data/
                    records.jsonl
                  original/
                    original.{extension}
                  validation/
                    validation-report.json

  load/
    company_id={company_id}/
      tenant_id={tenant_id}/
        source_type=fleet/
          dataset={dataset}/
            schema_version={schema_version}/
              dt={yyyy-MM-dd}/
                batch_id={batch_id}/
                  part-000.parquet

  curated/
    company_id={company_id}/
      tenant_id={tenant_id}/
        entity={entity}/
          dt={yyyy-MM-dd}/
            batch_id={batch_id}/
              part-000.parquet

  failed/
    company_id={company_id}/
      tenant_id={tenant_id}/
        source_type=fleet/
          dataset={dataset}/
            batch_id={batch_id}/
              error.json
              original-payload.json
```

The internal raw path uses `schema_version` because this is the stable customer-declared structure identifier. KDSAI may additionally store an internal contract ID in operational metadata, but this is not required in the customer upload contract and should not be required from the customer.

---

## 7. Processing Stages

### Stage 1 — File Arrival in Fleet Landing

The customer uploads files into the assigned Fleet landing prefix:

```text
s3://kds-warehouse-ai-test/fleet/landing/company_id={company_id}/tenant_id={tenant_id}/dataset={dataset}/upload_id={upload_id}/received_at={timestamp}/
```

The required customer package structure is:

```text
source/
  <one source data file>
  <optional schema file>

metadata/
  upload-manifest.json
```

Supported source file types in the initial contract:

```text
.xlsx
.csv
.json
.jsonl
.xml
```

At this point, the file is not ready for the standard ETL. It is only a customer-provided input that must be validated and, when necessary, onboarded as a schema version.

---

### Stage 2 — Upload Manifest Validation

The Fleet onboarding service reads `metadata/upload-manifest.json`.

The upload manifest is responsible for telling the platform:

- who sent the data,
- which company and tenant the data belongs to,
- which dataset the file represents,
- which customer-declared `schema.version` the file follows,
- which single source file should be processed,
- how the source file should be parsed,
- whether an optional schema file was provided,
- whether the upload corrects a previous upload.

Manifest validation must check at least:

- `manifest_version` exists,
- `upload_id` exists and was not reused with different files,
- `company_id`, `tenant_id`, and `dataset` exist,
- manifest values match the assigned S3 prefix,
- `schema.version` exists,
- exactly one `data_file` is declared,
- the declared source file exists,
- the declared file type is supported,
- required parsing metadata exists for the file type,
- optional `schema_file` exists when declared.

If required metadata is missing or inconsistent, the upload is moved to Fleet quarantine and is not published to standard ETL raw.

---

### Stage 3 — Discovery, Profiling, and Schema Fingerprint

The platform analyzes the uploaded data file and creates profiling artifacts:

```text
profiling/.../inferred-schema.json
profiling/.../schema-fingerprint.json
profiling/.../data-profile.json
profiling/.../sample-preview.json
profiling/.../issues.json
```

This stage may detect:

- file format,
- worksheet name for Excel files,
- delimiter and encoding for CSV files,
- column names,
- column order,
- likely data types,
- null counts,
- sample values,
- duplicate columns,
- invalid rows,
- potential primary keys,
- date/time format candidates,
- decimal separator issues,
- structural drift from an approved schema version.

KDSAI should compute a normalized schema fingerprint/hash from structural properties such as:

```text
file_type + sheet_name + delimiter + encoding + column_names + column_order + normalized_types + primary_key rules
```

The fingerprint is used to detect whether a file that declares an existing `schema.version` still matches the approved structure.

---

### Stage 4 — Schema Version Resolution

KDSAI checks whether the combination below is already approved in the internal registry:

```text
company_id + tenant_id + dataset + schema.version
```

There are three main outcomes.

| Outcome | Meaning | Action |
|---|---|---|
| Approved schema version exists and structure matches | The customer sent a known file structure | Continue to data validation |
| Schema version is new | The customer intentionally introduced a new structure | Create proposed schema and mapping, then wait for approval |
| Schema version is known but structure does not match | The customer changed the file structure without increasing `schema.version` | Stop as schema drift and reject or quarantine |

The customer owns the declared `schema.version`. KDSAI owns approval and validation of that version.

---

### Stage 5 — Proposed Schema and Mapping

If the incoming `schema.version` is not approved yet, the platform creates a proposed internal schema and mapping.

The proposal may use:

- inferred schema from profiling,
- optional customer-provided `schema_file`,
- existing mappings from previous schema versions,
- onboarding operator decisions,
- business rules for target curated entities.

The output of this stage is a candidate internal contract:

```text
schemas/.../schema.json
schemas/.../mapping.json
schemas/.../status.json
```

The contract remains inactive until approved.

---

### Stage 6 — Schema Approval

A schema version must be approved before data can be published to standard ETL raw.

The approved internal contract defines:

- company ID,
- tenant ID,
- dataset name,
- customer-declared `schema.version`,
- schema fingerprint/hash,
- source file format and parsing rules,
- source columns,
- normalized target columns,
- data types,
- required fields,
- primary key or business key rules,
- load strategy,
- validation rules,
- mapping to Redshift staging and curated entities,
- optional transformation hints.

Approval should be stored both:

1. as S3 artifacts under `fleet/schemas`, and
2. in operational metadata tables.

KDSAI may assign an internal contract ID for operational tracking, but this ID is not required from the customer and should not be part of the customer-facing upload contract.

---

### Stage 7 — Data Validation Against Approved Schema Version

Once the schema version is approved, the platform validates uploaded data against the approved internal contract.

Validation should check at least:

- required columns exist,
- no unexpected structural changes exist for the declared schema version,
- required values are not null,
- data types can be converted,
- primary key or business key columns are present,
- duplicate records are handled according to the contract,
- date/time formats are parseable,
- numeric fields are parseable,
- file row count matches the manifest when provided,
- file hash matches the manifest when provided,
- tenant and company metadata are consistent.

Validation outputs:

```text
validation-report.json
rejected-rows.jsonl
accepted-preview.jsonl
```

If validation fails, the batch is not published to standard ETL raw.

---

### Stage 8 — Fleet Ready Package

If validation passes, the platform creates a ready package:

```text
ready/.../publish-manifest.json
ready/.../normalized/records.jsonl
ready/.../original/original.{extension}
ready/.../validation/validation-report.json
```

The ready package is the boundary between Fleet onboarding and standard ETL.

It contains:

- normalized accepted data,
- reference to the approved schema version,
- reference to the original uploaded source file,
- validation report,
- publishing instructions.

---

### Stage 9 — Publish to Standard ETL Raw

The Fleet onboarding service publishes the ready package into:

```text
s3://kds-warehouse-ai-test/raw/company_id={company_id}/tenant_id={tenant_id}/source_type=fleet/dataset={dataset}/schema_version={schema_version}/...
```

This creates the official raw package for the standard ETL process.

The published raw package should contain:

```text
manifest.json
schema-ref.json
data/records.jsonl
original/original.{extension}
validation/validation-report.json
```

This is the point where standard ETL starts.

---

### Stage 10 — Standard ETL Processing

The standard ETL worker processes the `raw/source_type=fleet` package like any other supported source:

1. Reads raw `manifest.json`.
2. Reads `schema-ref.json`.
3. Converts normalized raw data to load-ready Parquet.
4. Writes files to `load/`.
5. Loads tenant Redshift `staging`.
6. Applies the approved mapping and load strategy.
7. Transforms data into tenant Redshift `curated`.
8. Exports curated Parquet to S3 `curated/` if required for cross-tenant Athena queries.
9. Marks the batch as successful only after the complete ETL path succeeds.

---

## 8. File Contracts and Internal Artifacts

### 8.1 Customer `upload-manifest.json`

Used in the Fleet landing zone.

Purpose:

- declares company and tenant context,
- declares dataset,
- declares customer-owned `schema.version`,
- describes exactly one source data file,
- provides parsing metadata,
- optionally references an upload correction,
- optionally declares an uploaded schema file.

Example:

```json
{
  "manifest_version": "1.0",
  "upload_id": "upl_20260602_001",
  "company_id": "company-123",
  "tenant_id": "tenant-456",
  "dataset": "employees",
  "source_system": "customer-fleet-agent",
  "created_at_utc": "2026-06-02T10:00:00Z",
  "schema": {
    "version": "1.0.0",
    "previous_version": null,
    "change_description": "Initial schema for the employees dataset."
  },
  "data_file": {
    "file_name": "employees.xlsx",
    "relative_path": "source/employees.xlsx",
    "file_type": "excel",
    "sheet_name": "Employees",
    "has_header": true,
    "row_count": 1520,
    "sha256": "REPLACE_WITH_FILE_HASH"
  },
  "schema_file": {
    "file_name": "employees_schema.json",
    "relative_path": "source/employees_schema.json",
    "file_type": "json",
    "sha256": "REPLACE_WITH_FILE_HASH"
  }
}
```

`manifest_type` is not required in the initial customer upload contract because there is only one customer-facing Fleet landing manifest type.

---

### 8.2 Optional customer `schema_file`

The optional schema file is input for KDSAI schema proposal and approval.

It is recommended when:

- this is the first upload of a dataset,
- this is a new `schema.version`,
- the file structure changed,
- a column was added, removed, renamed, or changed,
- the customer wants to help KDSAI approve the schema faster.

The schema file does not need to match the final KDSAI internal contract exactly.

---

### 8.3 Internal `schema.json`

Used for KDSAI-owned approved schema versions.

Example:

```json
{
  "internal_contract_id": "fleet_employees_1_0_0",
  "company_id": "company-123",
  "tenant_id": "tenant-456",
  "dataset": "employees",
  "schema_version": "1.0.0",
  "schema_hash": "REPLACE_WITH_SCHEMA_HASH",
  "entity": "employees",
  "format": "jsonl",
  "load_strategy": "merge",
  "primary_key": ["employee_number"],
  "event_time_column": "updated_at",
  "columns": [
    {
      "source_name": "Employee no.",
      "target_name": "employee_number",
      "type": "string",
      "required": true
    },
    {
      "source_name": "First name",
      "target_name": "first_name",
      "type": "string",
      "required": true
    },
    {
      "source_name": "Employment date",
      "target_name": "employment_date",
      "type": "date",
      "required": false,
      "formats": ["yyyy-MM-dd", "dd.MM.yyyy"]
    }
  ]
}
```

---

### 8.4 Internal `mapping.json`

Maps source file columns to normalized target columns and curated entities.

Example:

```json
{
  "internal_contract_id": "fleet_employees_1_0_0",
  "dataset": "employees",
  "schema_version": "1.0.0",
  "source_format": "excel",
  "default_sheet_name": "Employees",
  "target_entity": "employees",
  "columns": [
    {
      "source_name": "Employee no.",
      "target_name": "employee_number"
    },
    {
      "source_name": "First name",
      "target_name": "first_name"
    },
    {
      "source_name": "Employment date",
      "target_name": "employment_date"
    }
  ]
}
```

---

### 8.5 `validation-report.json`

Created by the platform after validation.

Example:

```json
{
  "batch_id": "batch_20260602_001",
  "upload_id": "upl_20260602_001",
  "company_id": "company-123",
  "tenant_id": "tenant-456",
  "dataset": "employees",
  "schema_version": "1.0.0",
  "internal_contract_id": "fleet_employees_1_0_0",
  "status": "valid",
  "validated_at_utc": "2026-06-02T10:35:00Z",
  "total_rows": 1520,
  "accepted_rows": 1520,
  "rejected_rows": 0,
  "warnings": [],
  "errors": []
}
```

---

### 8.6 `publish-manifest.json`

Created by the platform when a Fleet package is ready to be published to standard ETL raw.

Example:

```json
{
  "manifest_version": "1.0",
  "batch_id": "batch_20260602_001",
  "upload_id": "upl_20260602_001",
  "company_id": "company-123",
  "tenant_id": "tenant-456",
  "source_type": "fleet",
  "dataset": "employees",
  "schema_version": "1.0.0",
  "internal_contract_id": "fleet_employees_1_0_0",
  "created_at_utc": "2026-06-02T10:40:00Z",
  "validation_status": "valid",
  "normalized_file": {
    "relative_path": "normalized/records.jsonl",
    "format": "jsonl",
    "row_count": 1520,
    "sha256": "REPLACE_WITH_FILE_HASH"
  },
  "original_file": {
    "source_path": "s3://kds-warehouse-ai-test/fleet/landing/company_id=company-123/tenant_id=tenant-456/dataset=employees/upload_id=upl_20260602_001/received_at=2026-06-02T10-00-00Z/source/employees.xlsx",
    "file_name": "employees.xlsx",
    "file_type": "excel",
    "sha256": "REPLACE_WITH_FILE_HASH"
  },
  "target_raw_path": "s3://kds-warehouse-ai-test/raw/company_id=company-123/tenant_id=tenant-456/source_type=fleet/dataset=employees/schema_version=1.0.0/dt=2026-06-02/batch_id=batch_20260602_001/"
}
```

---

### 8.7 Standard ETL `manifest.json`

Created under the standard ETL raw path.

Example:

```json
{
  "manifest_version": "1.0",
  "batch_id": "batch_20260602_001",
  "upload_id": "upl_20260602_001",
  "company_id": "company-123",
  "tenant_id": "tenant-456",
  "source_type": "fleet",
  "dataset": "employees",
  "schema_version": "1.0.0",
  "created_at_utc": "2026-06-02T10:42:00Z",
  "data_file": {
    "relative_path": "data/records.jsonl",
    "format": "jsonl",
    "row_count": 1520,
    "sha256": "REPLACE_WITH_FILE_HASH"
  },
  "original_file": {
    "relative_path": "original/employees.xlsx",
    "file_type": "excel",
    "sha256": "REPLACE_WITH_FILE_HASH"
  },
  "validation_report": {
    "relative_path": "validation/validation-report.json",
    "status": "valid"
  },
  "processing": {
    "expected_next_step": "convert_to_load_ready",
    "target_load_format": "parquet",
    "target_redshift_schema": "staging"
  }
}
```

---

### 8.8 `schema-ref.json`

Created under the standard ETL raw path.

Example:

```json
{
  "dataset": "employees",
  "schema_version": "1.0.0",
  "internal_contract_id": "fleet_employees_1_0_0",
  "schema_path": "s3://kds-warehouse-ai-test/fleet/schemas/company_id=company-123/tenant_id=tenant-456/dataset=employees/schema_version=1.0.0/schema.json",
  "mapping_path": "s3://kds-warehouse-ai-test/fleet/schemas/company_id=company-123/tenant_id=tenant-456/dataset=employees/schema_version=1.0.0/mapping.json",
  "status": "approved"
}
```

---

## 9. File Format Handling

### 9.1 Source formats accepted in Fleet

Fleet landing may accept:

| Format | Extension | Notes |
|---|---|---|
| Excel | `.xlsx` | One worksheet per upload; `sheet_name` and `has_header` are required in the manifest |
| CSV | `.csv` | `delimiter`, `encoding`, and `has_header` are required in the manifest |
| JSON | `.json` | Must contain structured data records |
| JSON Lines | `.jsonl` | Preferred for large semi-structured data |
| XML | `.xml` | Treated as source data; mapping rules are handled by KDSAI |

ZIP packages are not supported in the initial Fleet customer upload contract.

### 9.2 Internal normalized format

Fleet ready packages should use:

```text
normalized/records.jsonl
```

Standard ETL should then convert it into:

```text
load/.../part-000.parquet
```

The platform should not require customers to send Parquet in the initial Fleet phase, but Parquet can be supported later if the customer system can generate it reliably and the contract is extended.

---

## 10. Schema Versioning and Warehouse Impact

`schema.version` identifies the structure of the customer source file. It does not directly define the Redshift curated table version.

KDSAI may map multiple approved schema versions of the same dataset into one stable curated entity.

Example:

```text
employees schema.version=1.0.0 -> curated.employees
employees schema.version=1.1.0 -> curated.employees
employees schema.version=2.0.0 -> curated.employees, only if a new mapping and load strategy are approved
```

Recommended handling by change type:

| Source schema change | Example | Architecture handling |
|---|---|---|
| New dataset | First `employees` file with `schema.version=1.0.0` | Profile, propose schema, approve, create mapping, then load |
| Optional column added | `middle_name` added in `1.1.0` | Add mapping; optionally add nullable curated column if useful |
| Required column added | `employee_status` added in `1.2.0` | Approve new validation rules; curated column should usually be nullable for historical data |
| Column renamed | `employee_number` renamed to `employee_id` | Keep curated column stable if business meaning is the same; update source-to-target mapping |
| Type or format changed | Date format changed from `yyyy-MM-dd` to `dd.MM.yyyy` | Update parser/validation for new schema version; curated type can remain stable |
| Primary key changed | Key changed from `employee_number` to composite key | Treat as breaking change; require explicit approval and new load strategy |
| Same version but different structure | Customer sends `1.0.0` but file contains renamed column | Treat as schema drift; stop before ETL publication |
| Same structure but corrected values | Data correction with same columns | Keep same `schema.version`; use correction metadata and a new `upload_id` |

Default warehouse strategy:

```text
staging = may be schema-version-aware and close to source shape
curated = stable business model queried by agents and monitors
serving = optional stable published views over curated data
```

Do not create a new curated table for every `schema.version` by default. New curated entities should be created only when the business meaning changes enough that the existing curated model cannot represent the data safely.

---

## 11. Batch Status Model

Recommended Fleet status model:

```text
Uploaded
ManifestValidated
Profiled
WaitingForSchemaApproval
SchemaApproved
SchemaDriftDetected
ValidationRunning
ValidationFailed
Validated
ReadyToPublish
PublishedToRaw
StandardEtlQueued
StandardEtlSucceeded
StandardEtlFailed
Quarantined
Rejected
```

The standard ETL should keep its own batch lifecycle after `PublishedToRaw`.

Customer-visible states can be simplified to:

| State | Meaning |
|---|---|
| `Uploaded` | Files and manifest were received in Fleet landing |
| `Rejected` | Manifest, file structure, schema version, or data validation failed |
| `WaitingForSchemaApproval` | A new schema version requires approval before processing |
| `AcceptedForEtl` | Data passed Fleet validation and was published to standard ETL raw |
| `Loaded` | Standard ETL loaded the data into the warehouse |

---

## 12. Error Handling

### 12.1 Quarantine and rejection reasons

Examples:

```text
MissingUploadManifest
InvalidUploadManifest
UnknownCompanyId
UnknownTenantId
CompanyTenantMismatch
UnsupportedFileType
UnreadableFile
MissingDataFile
MultipleDataFilesNotSupported
MissingSchemaVersion
SchemaVersionNotApproved
SchemaDriftDetected
SchemaValidationFailed
RowValidationFailed
FileHashMismatch
DuplicateUploadId
DuplicateBatchId
```

### 12.2 Quarantine output

```text
quarantine/.../reason.json
quarantine/.../original/
```

Example `reason.json`:

```json
{
  "upload_id": "upl_20260602_001",
  "status": "quarantined",
  "reason_code": "SchemaDriftDetected",
  "message": "The upload declares employees schema.version 1.0.0, but the detected file structure does not match the approved schema version.",
  "detected_at_utc": "2026-06-02T10:05:00Z"
}
```

---

## 13. Operational Metadata

S3 is not enough to manage the process. The platform should also store operational metadata in a database.

Recommended tables:

```text
fleet_uploads
fleet_upload_files
fleet_schema_versions
fleet_schema_columns
fleet_schema_mappings
fleet_validation_runs
fleet_batches
fleet_publications
etl_batches
```

Minimum metadata for each Fleet upload:

| Field | Description |
|---|---|
| `upload_id` | Customer-generated idempotency key |
| `batch_id` | Internal processing batch ID |
| `company_id` | Company context |
| `tenant_id` | Tenant context |
| `dataset` | Business dataset name |
| `schema_version` | Customer-declared source file structure version |
| `schema_hash` | KDSAI-computed fingerprint of the detected source structure |
| `internal_contract_id` | Optional KDSAI-owned internal contract identifier |
| `status` | Current processing status |
| `landing_path` | Original Fleet landing path |
| `ready_path` | Fleet ready package path |
| `raw_path` | Published standard ETL raw path |
| `created_at_utc` | Upload creation time |
| `published_at_utc` | Raw publication time |
| `row_count` | Total accepted rows |
| `error_code` | Error code if failed |

Minimum metadata for each approved schema version:

| Field | Description |
|---|---|
| `company_id` | Company context |
| `tenant_id` | Tenant context |
| `dataset` | Business dataset name |
| `schema_version` | Customer-declared structure version |
| `previous_schema_version` | Previous version if provided or inferred |
| `schema_hash` | Approved source structure fingerprint |
| `status` | Pending / Approved / Rejected / Deprecated |
| `target_entity` | Target curated entity |
| `load_strategy` | Append, merge, replace, or another approved load strategy |
| `primary_key` | Key rules used by validation and merge logic |
| `schema_path` | S3 path to `schema.json` |
| `mapping_path` | S3 path to `mapping.json` |
| `created_at_utc` | Creation timestamp |
| `approved_at_utc` | Approval timestamp |

---

## 14. Idempotency Rules

The platform should enforce the following:

1. `upload_id` must be unique per tenant.
2. `batch_id` must be unique globally or per tenant.
3. The same `upload_id` cannot be reused with different files.
4. Files inside an existing `upload_id` path must not be replaced.
5. If the customer needs to correct data, they must send a new upload.
6. Corrections should reference the previous upload in the root-level `correction` object in `upload-manifest.json`.
7. `schema.version` must not be used as an upload sequence, export date, or business date.
8. `schema.version` changes only when the source file structure changes.

Example correction metadata inside `upload-manifest.json`:

```json
{
  "upload_id": "upl_20260602_002",
  "schema": {
    "version": "1.0.0"
  },
  "correction": {
    "is_correction": true,
    "correction_of_upload_id": "upl_20260602_001",
    "correction_reason": "Corrected employment date values."
  }
}
```

---

## 15. Security Requirements

Recommended security rules:

1. Customers should write only to their assigned Fleet landing prefix.
2. Customers should not have write access to `schemas`, `profiling`, `validation`, `ready`, `quarantine`, or internal ETL prefixes.
3. Customers should not write directly to standard ETL `raw`.
4. The Fleet onboarding service should be the only writer to `raw/source_type=fleet`.
5. All objects should be encrypted at rest.
6. S3 object versioning should be enabled for Fleet landing and standard raw.
7. Uploads should preferably use presigned URLs or tightly scoped IAM roles.
8. The platform must validate that `company_id`, `tenant_id`, and `dataset` in the manifest match the authorized upload prefix.

---

## 16. Design Principles

1. **Fleet accepts unknowns; standard raw accepts only validated packages.**
2. **One customer upload produces one processing batch.**
3. **The original customer file is preserved.**
4. **The standard ETL raw package is also preserved.**
5. **Customer-owned `schema.version` represents source file structure, not upload sequence.**
6. **KDSAI owns schema approval, mapping, validation, and internal contract registry.**
7. **The standard ETL does not negotiate schemas.**
8. **Schema approval happens before publishing to raw.**
9. **The ETL worker should not guess customer intent.**
10. **The platform owns tenant routing and validation.**
11. **Batches are immutable.**
12. **Replay must be possible from standard raw, and investigation must be possible from Fleet landing.**
13. **Staging may be schema-version-aware; curated should remain as stable as possible.**