Repository Docs & Knowledge Base

# Fleet Pre-Ingestion Processing Architecture

1. Purpose

This document describes the proposed Fleet pre-ingestion process for customer-provided data files.

Fleet is a customer-facing pre-ingestion and schema-onboarding zone. It is used when a customer-side system uploads files whose structure must be identified, validated, versioned, and approved before the data can enter the standard ETL pipeline.

The key architectural decision is:

Fleet is a pre-ingestion and schema contract onboarding zone.
The standard ETL starts only after a Fleet upload is validated and published to raw/source_type=fleet.

This means the platform keeps two raw representations:

1. Fleet landing copy — the original customer-provided upload exactly as received.
2. Standard ETL raw copy — the accepted and validated package published by KDSAI into the internal ETL structure.

The Fleet customer contract uses an atomic upload model:

one upload package = one source data file = one manifest = one processing batch

Multiple source data files in one customer manifest are not supported in the initial architecture. If the customer needs to send several datasets, entities, or Excel worksheets, each one must be sent as a separate upload with a separate upload_id.

2. Key Architecture Rules

1. Customers upload only to Fleet landing.
2. Customers upload exactly one source data file per upload package.
3. Customers include exactly one metadata/upload-manifest.json per upload package.
4. Customers upload the manifest last; the manifest is the completion signal.
5. Customers do not send contract_id or contract_version.
6. Customers declare the source file structure using schema.version.
7. KDSAI owns schema approval, validation, mapping, and the internal approved contract registry.
8. KDSAI checks whether company_id + tenant_id + dataset + schema.version is already approved.
9. KDSAI publishes to standard ETL raw only after the schema version is approved and data validation passes.
10. The standard ETL worker does not negotiate schemas. It processes only Fleet packages that were already approved and published to internal raw.

3. High-Level Flow

flowchart TD
    A["Customer Fleet System"] --> B["Fleet Landing Zone<br/>one data file + manifest"]
    B --> C["Manifest Validation"]
    C --> D["Discovery, Profiling<br/>and Schema Fingerprint"]
    D --> E{"schema.version<br/>approved?"}
    E -- "No" --> F["Proposed Internal Schema<br/>and Mapping"]
    F --> G["Schema Review / Approval"]
    G --> H["Internal Approved Contract Registry"]
    E -- "Yes" --> I["Structure Check<br/>Against Approved Version"]
    H --> I
    I --> J{"Structure matches?"}
    J -- "No" --> K["Schema Drift<br/>Quarantine / Rejection"]
    J -- "Yes" --> L["Data Validation"]
    L --> M{"Validation passed?"}
    M -- "No" --> N["Validation Errors<br/>Quarantine / Rejection"]
    M -- "Yes" --> O["Fleet Ready Package"]
    O --> P["Publish to Standard ETL Raw"]
    P --> Q["Standard ETL Worker"]
    Q --> R["S3 Load-Ready Files"]
    R --> S["Tenant Redshift staging"]
    S --> T["Tenant Redshift curated"]
    T --> U["S3 curated Parquet"]

4. Bucket Strategy

4.1 Customer-visible bucket model

The architecture should use the same bucket and prefix model as the customer upload contract. The customer-facing upload location is a Fleet prefix inside the environment warehouse bucket.

Current customer-facing bucket:

s3://kds-warehouse-ai-test/

Customer uploads go only to:

s3://kds-warehouse-ai-test/fleet/landing/company_id={company_id}/tenant_id={tenant_id}/dataset={dataset}/upload_id={upload_id}/received_at={yyyy-MM-ddTHH-mm-ssZ}/

Example:

s3://kds-warehouse-ai-test/fleet/landing/company_id=company-123/tenant_id=tenant-456/dataset=employees/upload_id=upl_20260602_001/received_at=2026-06-02T10-00-00Z/

4.2 Prefix responsibilities

This keeps the customer contract simple: customers know only their assigned fleet/landing/... prefix. All other prefixes are internal KDSAI implementation details.

4.3 Security boundary

The initial architecture uses the s3://kds-warehouse-ai-test/ bucket with strict prefix-level permissions. Separate physical buckets can still be introduced later if isolation, lifecycle policies, or account boundaries require it, but the customer-facing contract should remain stable and continue to point to the assigned fleet/landing/... prefix.

5. Fleet Prefix Structure

s3://kds-warehouse-ai-test/fleet/
  landing/
    company_id={company_id}/
      tenant_id={tenant_id}/
        dataset={dataset}/
          upload_id={upload_id}/
            received_at={yyyy-MM-ddTHH-mm-ssZ}/
              source/
                original.xlsx
                original.csv
                original.json
                original.jsonl
                original.xml
                optional_schema.json
              metadata/
                upload-manifest.json

  profiling/
    company_id={company_id}/
      tenant_id={tenant_id}/
        dataset={dataset}/
          upload_id={upload_id}/
            inferred-schema.json
            schema-fingerprint.json
            data-profile.json
            sample-preview.json
            issues.json

  schemas/
    company_id={company_id}/
      tenant_id={tenant_id}/
        dataset={dataset}/
          schema_version={schema_version}/
            schema.json
            mapping.json
            status.json

  validation/
    company_id={company_id}/
      tenant_id={tenant_id}/
        dataset={dataset}/
          schema_version={schema_version}/
            batch_id={batch_id}/
              validation-report.json
              rejected-rows.jsonl
              accepted-preview.jsonl

  ready/
    company_id={company_id}/
      tenant_id={tenant_id}/
        dataset={dataset}/
          schema_version={schema_version}/
            batch_id={batch_id}/
              publish-manifest.json
              normalized/
                records.jsonl
              original/
                original.{extension}
              validation/
                validation-report.json

  quarantine/
    company_id={company_id}/
      tenant_id={tenant_id}/
        dataset={dataset}/
          upload_id={upload_id}/
            reason.json
            original/

Notes:

• source-metadata.json is not part of the initial customer contract.
• ZIP packages are not supported in the initial customer upload contract.
• schemas/ contains KDSAI-owned approved or proposed schema artifacts. Customers do not write to this prefix.
• schema_version in S3 paths is the customer-declared schema.version, not a customer-facing KDSAI contract identifier.

6. Standard ETL Prefix Structure After Fleet Publication

After a Fleet upload is approved and validated, the Fleet onboarding service publishes a standard raw package into the internal standard ETL prefixes in the same s3://kds-warehouse-ai-test/ bucket.

s3://kds-warehouse-ai-test/
  raw/
    company_id={company_id}/
      tenant_id={tenant_id}/
        source_type=fleet/
          dataset={dataset}/
            schema_version={schema_version}/
              dt={yyyy-MM-dd}/
                batch_id={batch_id}/
                  manifest.json
                  schema-ref.json
                  data/
                    records.jsonl
                  original/
                    original.{extension}
                  validation/
                    validation-report.json

  load/
    company_id={company_id}/
      tenant_id={tenant_id}/
        source_type=fleet/
          dataset={dataset}/
            schema_version={schema_version}/
              dt={yyyy-MM-dd}/
                batch_id={batch_id}/
                  part-000.parquet

  curated/
    company_id={company_id}/
      tenant_id={tenant_id}/
        entity={entity}/
          dt={yyyy-MM-dd}/
            batch_id={batch_id}/
              part-000.parquet

  failed/
    company_id={company_id}/
      tenant_id={tenant_id}/
        source_type=fleet/
          dataset={dataset}/
            batch_id={batch_id}/
              error.json
              original-payload.json

The internal raw path uses schema_version because this is the stable customer-declared structure identifier. KDSAI may additionally store an internal contract ID in operational metadata, but this is not required in the customer upload contract and should not be required from the customer.

7. Processing Stages

Stage 1 — File Arrival in Fleet Landing

The customer uploads files into the assigned Fleet landing prefix:

s3://kds-warehouse-ai-test/fleet/landing/company_id={company_id}/tenant_id={tenant_id}/dataset={dataset}/upload_id={upload_id}/received_at={timestamp}/

The required customer package structure is:

source/
  <one source data file>
  <optional schema file>

metadata/
  upload-manifest.json

Supported source file types in the initial contract:

.xlsx
.csv
.json
.jsonl
.xml

At this point, the file is not ready for the standard ETL. It is only a customer-provided input that must be validated and, when necessary, onboarded as a schema version.

Stage 2 — Upload Manifest Validation

The Fleet onboarding service reads metadata/upload-manifest.json.

The upload manifest is responsible for telling the platform:

• who sent the data,
• which company and tenant the data belongs to,
• which dataset the file represents,
• which customer-declared schema.version the file follows,
• which single source file should be processed,
• how the source file should be parsed,
• whether an optional schema file was provided,
• whether the upload corrects a previous upload.

Manifest validation must check at least:

• manifest_version exists,
• upload_id exists and was not reused with different files,
• company_id, tenant_id, and dataset exist,
• manifest values match the assigned S3 prefix,
• schema.version exists,
• exactly one data_file is declared,
• the declared source file exists,
• the declared file type is supported,
• required parsing metadata exists for the file type,
• optional schema_file exists when declared.

If required metadata is missing or inconsistent, the upload is moved to Fleet quarantine and is not published to standard ETL raw.

Stage 3 — Discovery, Profiling, and Schema Fingerprint

The platform analyzes the uploaded data file and creates profiling artifacts:

profiling/.../inferred-schema.json
profiling/.../schema-fingerprint.json
profiling/.../data-profile.json
profiling/.../sample-preview.json
profiling/.../issues.json

This stage may detect:

• file format,
• worksheet name for Excel files,
• delimiter and encoding for CSV files,
• column names,
• column order,
• likely data types,
• null counts,
• sample values,
• duplicate columns,
• invalid rows,
• potential primary keys,
• date/time format candidates,
• decimal separator issues,
• structural drift from an approved schema version.

KDSAI should compute a normalized schema fingerprint/hash from structural properties such as:

file_type + sheet_name + delimiter + encoding + column_names + column_order + normalized_types + primary_key rules

The fingerprint is used to detect whether a file that declares an existing schema.version still matches the approved structure.

Stage 4 — Schema Version Resolution

KDSAI checks whether the combination below is already approved in the internal registry:

company_id + tenant_id + dataset + schema.version

There are three main outcomes.

The customer owns the declared schema.version. KDSAI owns approval and validation of that version.

Stage 5 — Proposed Schema and Mapping

If the incoming schema.version is not approved yet, the platform creates a proposed internal schema and mapping.

The proposal may use:

• inferred schema from profiling,
• optional customer-provided schema_file,
• existing mappings from previous schema versions,
• onboarding operator decisions,
• business rules for target curated entities.

The output of this stage is a candidate internal contract:

schemas/.../schema.json
schemas/.../mapping.json
schemas/.../status.json

The contract remains inactive until approved.

Stage 6 — Schema Approval

A schema version must be approved before data can be published to standard ETL raw.

The approved internal contract defines:

• company ID,
• tenant ID,
• dataset name,
• customer-declared schema.version,
• schema fingerprint/hash,
• source file format and parsing rules,
• source columns,
• normalized target columns,
• data types,
• required fields,
• primary key or business key rules,
• load strategy,
• validation rules,
• mapping to Redshift staging and curated entities,
• optional transformation hints.

Approval should be stored both:

1. as S3 artifacts under fleet/schemas, and
2. in operational metadata tables.

KDSAI may assign an internal contract ID for operational tracking, but this ID is not required from the customer and should not be part of the customer-facing upload contract.

Stage 7 — Data Validation Against Approved Schema Version

Once the schema version is approved, the platform validates uploaded data against the approved internal contract.

Validation should check at least:

• required columns exist,
• no unexpected structural changes exist for the declared schema version,
• required values are not null,
• data types can be converted,
• primary key or business key columns are present,
• duplicate records are handled according to the contract,
• date/time formats are parseable,
• numeric fields are parseable,
• file row count matches the manifest when provided,
• file hash matches the manifest when provided,
• tenant and company metadata are consistent.

Validation outputs:

validation-report.json
rejected-rows.jsonl
accepted-preview.jsonl

If validation fails, the batch is not published to standard ETL raw.

Stage 8 — Fleet Ready Package

If validation passes, the platform creates a ready package:

ready/.../publish-manifest.json
ready/.../normalized/records.jsonl
ready/.../original/original.{extension}
ready/.../validation/validation-report.json

The ready package is the boundary between Fleet onboarding and standard ETL.

It contains:

• normalized accepted data,
• reference to the approved schema version,
• reference to the original uploaded source file,
• validation report,
• publishing instructions.

Stage 9 — Publish to Standard ETL Raw

The Fleet onboarding service publishes the ready package into:

s3://kds-warehouse-ai-test/raw/company_id={company_id}/tenant_id={tenant_id}/source_type=fleet/dataset={dataset}/schema_version={schema_version}/...

This creates the official raw package for the standard ETL process.

The published raw package should contain:

manifest.json
schema-ref.json
data/records.jsonl
original/original.{extension}
validation/validation-report.json

This is the point where standard ETL starts.

Stage 10 — Standard ETL Processing

The standard ETL worker processes the raw/source_type=fleet package like any other supported source:

1. Reads raw manifest.json.
2. Reads schema-ref.json.
3. Converts normalized raw data to load-ready Parquet.
4. Writes files to load/.
5. Loads tenant Redshift staging.
6. Applies the approved mapping and load strategy.
7. Transforms data into tenant Redshift curated.
8. Exports curated Parquet to S3 curated/ if required for cross-tenant Athena queries.
9. Marks the batch as successful only after the complete ETL path succeeds.

8. File Contracts and Internal Artifacts

8.1 Customer upload-manifest.json

Used in the Fleet landing zone.

Purpose:

• declares company and tenant context,
• declares dataset,
• declares customer-owned schema.version,
• describes exactly one source data file,
• provides parsing metadata,
• optionally references an upload correction,
• optionally declares an uploaded schema file.

Example:

{
  "manifest_version": "1.0",
  "upload_id": "upl_20260602_001",
  "company_id": "company-123",
  "tenant_id": "tenant-456",
  "dataset": "employees",
  "source_system": "customer-fleet-agent",
  "created_at_utc": "2026-06-02T10:00:00Z",
  "schema": {
    "version": "1.0.0",
    "previous_version": null,
    "change_description": "Initial schema for the employees dataset."
  },
  "data_file": {
    "file_name": "employees.xlsx",
    "relative_path": "source/employees.xlsx",
    "file_type": "excel",
    "sheet_name": "Employees",
    "has_header": true,
    "row_count": 1520,
    "sha256": "REPLACE_WITH_FILE_HASH"
  },
  "schema_file": {
    "file_name": "employees_schema.json",
    "relative_path": "source/employees_schema.json",
    "file_type": "json",
    "sha256": "REPLACE_WITH_FILE_HASH"
  }
}

manifest_type is not required in the initial customer upload contract because there is only one customer-facing Fleet landing manifest type.

8.2 Optional customer schema_file

The optional schema file is input for KDSAI schema proposal and approval.

It is recommended when:

• this is the first upload of a dataset,
• this is a new schema.version,
• the file structure changed,
• a column was added, removed, renamed, or changed,
• the customer wants to help KDSAI approve the schema faster.

The schema file does not need to match the final KDSAI internal contract exactly.

8.3 Internal schema.json

Used for KDSAI-owned approved schema versions.

Example:

{
  "internal_contract_id": "fleet_employees_1_0_0",
  "company_id": "company-123",
  "tenant_id": "tenant-456",
  "dataset": "employees",
  "schema_version": "1.0.0",
  "schema_hash": "REPLACE_WITH_SCHEMA_HASH",
  "entity": "employees",
  "format": "jsonl",
  "load_strategy": "merge",
  "primary_key": ["employee_number"],
  "event_time_column": "updated_at",
  "columns": [
    {
      "source_name": "Employee no.",
      "target_name": "employee_number",
      "type": "string",
      "required": true
    },
    {
      "source_name": "First name",
      "target_name": "first_name",
      "type": "string",
      "required": true
    },
    {
      "source_name": "Employment date",
      "target_name": "employment_date",
      "type": "date",
      "required": false,
      "formats": ["yyyy-MM-dd", "dd.MM.yyyy"]
    }
  ]
}

8.4 Internal mapping.json

Maps source file columns to normalized target columns and curated entities.

Example:

{
  "internal_contract_id": "fleet_employees_1_0_0",
  "dataset": "employees",
  "schema_version": "1.0.0",
  "source_format": "excel",
  "default_sheet_name": "Employees",
  "target_entity": "employees",
  "columns": [
    {
      "source_name": "Employee no.",
      "target_name": "employee_number"
    },
    {
      "source_name": "First name",
      "target_name": "first_name"
    },
    {
      "source_name": "Employment date",
      "target_name": "employment_date"
    }
  ]
}

8.5 validation-report.json

Created by the platform after validation.

Example:

{
  "batch_id": "batch_20260602_001",
  "upload_id": "upl_20260602_001",
  "company_id": "company-123",
  "tenant_id": "tenant-456",
  "dataset": "employees",
  "schema_version": "1.0.0",
  "internal_contract_id": "fleet_employees_1_0_0",
  "status": "valid",
  "validated_at_utc": "2026-06-02T10:35:00Z",
  "total_rows": 1520,
  "accepted_rows": 1520,
  "rejected_rows": 0,
  "warnings": [],
  "errors": []
}

8.6 publish-manifest.json

Created by the platform when a Fleet package is ready to be published to standard ETL raw.

Example:

{
  "manifest_version": "1.0",
  "batch_id": "batch_20260602_001",
  "upload_id": "upl_20260602_001",
  "company_id": "company-123",
  "tenant_id": "tenant-456",
  "source_type": "fleet",
  "dataset": "employees",
  "schema_version": "1.0.0",
  "internal_contract_id": "fleet_employees_1_0_0",
  "created_at_utc": "2026-06-02T10:40:00Z",
  "validation_status": "valid",
  "normalized_file": {
    "relative_path": "normalized/records.jsonl",
    "format": "jsonl",
    "row_count": 1520,
    "sha256": "REPLACE_WITH_FILE_HASH"
  },
  "original_file": {
    "source_path": "s3://kds-warehouse-ai-test/fleet/landing/company_id=company-123/tenant_id=tenant-456/dataset=employees/upload_id=upl_20260602_001/received_at=2026-06-02T10-00-00Z/source/employees.xlsx",
    "file_name": "employees.xlsx",
    "file_type": "excel",
    "sha256": "REPLACE_WITH_FILE_HASH"
  },
  "target_raw_path": "s3://kds-warehouse-ai-test/raw/company_id=company-123/tenant_id=tenant-456/source_type=fleet/dataset=employees/schema_version=1.0.0/dt=2026-06-02/batch_id=batch_20260602_001/"
}

8.7 Standard ETL manifest.json

Created under the standard ETL raw path.

Example:

{
  "manifest_version": "1.0",
  "batch_id": "batch_20260602_001",
  "upload_id": "upl_20260602_001",
  "company_id": "company-123",
  "tenant_id": "tenant-456",
  "source_type": "fleet",
  "dataset": "employees",
  "schema_version": "1.0.0",
  "created_at_utc": "2026-06-02T10:42:00Z",
  "data_file": {
    "relative_path": "data/records.jsonl",
    "format": "jsonl",
    "row_count": 1520,
    "sha256": "REPLACE_WITH_FILE_HASH"
  },
  "original_file": {
    "relative_path": "original/employees.xlsx",
    "file_type": "excel",
    "sha256": "REPLACE_WITH_FILE_HASH"
  },
  "validation_report": {
    "relative_path": "validation/validation-report.json",
    "status": "valid"
  },
  "processing": {
    "expected_next_step": "convert_to_load_ready",
    "target_load_format": "parquet",
    "target_redshift_schema": "staging"
  }
}

8.8 schema-ref.json

Created under the standard ETL raw path.

Example:

{
  "dataset": "employees",
  "schema_version": "1.0.0",
  "internal_contract_id": "fleet_employees_1_0_0",
  "schema_path": "s3://kds-warehouse-ai-test/fleet/schemas/company_id=company-123/tenant_id=tenant-456/dataset=employees/schema_version=1.0.0/schema.json",
  "mapping_path": "s3://kds-warehouse-ai-test/fleet/schemas/company_id=company-123/tenant_id=tenant-456/dataset=employees/schema_version=1.0.0/mapping.json",
  "status": "approved"
}

9. File Format Handling

9.1 Source formats accepted in Fleet

Fleet landing may accept:

ZIP packages are not supported in the initial Fleet customer upload contract.

9.2 Internal normalized format

Fleet ready packages should use:

normalized/records.jsonl

Standard ETL should then convert it into:

load/.../part-000.parquet

The platform should not require customers to send Parquet in the initial Fleet phase, but Parquet can be supported later if the customer system can generate it reliably and the contract is extended.

10. Schema Versioning and Warehouse Impact

schema.version identifies the structure of the customer source file. It does not directly define the Redshift curated table version.

KDSAI may map multiple approved schema versions of the same dataset into one stable curated entity.

Example:

employees schema.version=1.0.0 -> curated.employees
employees schema.version=1.1.0 -> curated.employees
employees schema.version=2.0.0 -> curated.employees, only if a new mapping and load strategy are approved

Recommended handling by change type:

Default warehouse strategy:

staging = may be schema-version-aware and close to source shape
curated = stable business model queried by agents and monitors
serving = optional stable published views over curated data

Do not create a new curated table for every schema.version by default. New curated entities should be created only when the business meaning changes enough that the existing curated model cannot represent the data safely.

11. Batch Status Model

Recommended Fleet status model:

Uploaded
ManifestValidated
Profiled
WaitingForSchemaApproval
SchemaApproved
SchemaDriftDetected
ValidationRunning
ValidationFailed
Validated
ReadyToPublish
PublishedToRaw
StandardEtlQueued
StandardEtlSucceeded
StandardEtlFailed
Quarantined
Rejected

The standard ETL should keep its own batch lifecycle after PublishedToRaw.

Customer-visible states can be simplified to:

12. Error Handling

12.1 Quarantine and rejection reasons

Examples:

MissingUploadManifest
InvalidUploadManifest
UnknownCompanyId
UnknownTenantId
CompanyTenantMismatch
UnsupportedFileType
UnreadableFile
MissingDataFile
MultipleDataFilesNotSupported
MissingSchemaVersion
SchemaVersionNotApproved
SchemaDriftDetected
SchemaValidationFailed
RowValidationFailed
FileHashMismatch
DuplicateUploadId
DuplicateBatchId

12.2 Quarantine output

quarantine/.../reason.json
quarantine/.../original/

Example reason.json:

{
  "upload_id": "upl_20260602_001",
  "status": "quarantined",
  "reason_code": "SchemaDriftDetected",
  "message": "The upload declares employees schema.version 1.0.0, but the detected file structure does not match the approved schema version.",
  "detected_at_utc": "2026-06-02T10:05:00Z"
}

13. Operational Metadata

S3 is not enough to manage the process. The platform should also store operational metadata in a database.

Recommended tables:

fleet_uploads
fleet_upload_files
fleet_schema_versions
fleet_schema_columns
fleet_schema_mappings
fleet_validation_runs
fleet_batches
fleet_publications
etl_batches

Minimum metadata for each Fleet upload:

Minimum metadata for each approved schema version:

14. Idempotency Rules

The platform should enforce the following:

1. upload_id must be unique per tenant.
2. batch_id must be unique globally or per tenant.
3. The same upload_id cannot be reused with different files.
4. Files inside an existing upload_id path must not be replaced.
5. If the customer needs to correct data, they must send a new upload.
6. Corrections should reference the previous upload in the root-level correction object in upload-manifest.json.
7. schema.version must not be used as an upload sequence, export date, or business date.
8. schema.version changes only when the source file structure changes.

Example correction metadata inside upload-manifest.json:

{
  "upload_id": "upl_20260602_002",
  "schema": {
    "version": "1.0.0"
  },
  "correction": {
    "is_correction": true,
    "correction_of_upload_id": "upl_20260602_001",
    "correction_reason": "Corrected employment date values."
  }
}

15. Security Requirements

Recommended security rules:

1. Customers should write only to their assigned Fleet landing prefix.
2. Customers should not have write access to schemas, profiling, validation, ready, quarantine, or internal ETL prefixes.
3. Customers should not write directly to standard ETL raw.
4. The Fleet onboarding service should be the only writer to raw/source_type=fleet.
5. All objects should be encrypted at rest.
6. S3 object versioning should be enabled for Fleet landing and standard raw.
7. Uploads should preferably use presigned URLs or tightly scoped IAM roles.
8. The platform must validate that company_id, tenant_id, and dataset in the manifest match the authorized upload prefix.

16. Design Principles

1. Fleet accepts unknowns; standard raw accepts only validated packages.
2. One customer upload produces one processing batch.
3. The original customer file is preserved.
4. The standard ETL raw package is also preserved.
5. Customer-owned schema.version represents source file structure, not upload sequence.
6. KDSAI owns schema approval, mapping, validation, and internal contract registry.
7. The standard ETL does not negotiate schemas.
8. Schema approval happens before publishing to raw.
9. The ETL worker should not guess customer intent.
10. The platform owns tenant routing and validation.
11. Batches are immutable.
12. Replay must be possible from standard raw, and investigation must be possible from Fleet landing.
13. Staging may be schema-version-aware; curated should remain as stable as possible.