Repository Docs & Knowledge Base

# Fleet Customer Upload Contract

1. Purpose

This document defines what the customer-side Fleet system must upload to KDSAI so that one source data file can be accepted, validated, and prepared for the standard ETL process.

Fleet is a customer-facing pre-ingestion area. The customer uploads files to Fleet landing only. KDSAI publishes accepted data to the internal standard ETL raw area only after schema approval and data validation pass.

Supported source data formats for the initial contract are:

Excel
CSV
JSON
JSON Lines
XML

2. Core Rules

1. One upload package represents one logical dataset upload.
2. One upload-manifest.json describes exactly one source data file.
3. One upload creates one processing batch in KDSAI.
4. Multiple source data files in one manifest are not supported in the initial contract.
5. If the customer needs to send several datasets, entities, or worksheets, they must send separate uploads with separate upload_id values.
6. The customer declares the dataset schema version in the manifest using schema.version.
7. The customer must increase schema.version whenever the file structure changes.
8. KDSAI owns schema approval, validation, and the internal approved contract registry.
9. The customer uploads all files first and uploads metadata/upload-manifest.json last.
10. KDSAI treats metadata/upload-manifest.json as the signal that the upload is complete.
11. The customer must not overwrite or replace files in an existing upload path.
12. Corrections are sent as new uploads with a new upload_id.

13. The customer must not write directly to standard ETL raw, load, curated, or failed prefixes.

3. Target Upload Location

The customer uploads files only to the assigned Fleet landing prefix.

Recommended structure:

s3://kds-warehouse-ai-test/fleet/landing/company_id={company_id}/tenant_id={tenant_id}/dataset={dataset}/upload_id={upload_id}/received_at={yyyy-MM-ddTHH-mm-ssZ}/

Example:

s3://kds-warehouse-ai-test/fleet/landing/company_id=company-123/tenant_id=tenant-456/dataset=employees/upload_id=upl_20260602_001/received_at=2026-06-02T10-00-00Z/

Required package structure:

source/
  employees.xlsx
  employees_schema.json        # optional

metadata/
  upload-manifest.json         # required; uploaded last

4. Upload Manifest

Every upload must include:

metadata/upload-manifest.json

The manifest tells KDSAI:

• who sent the data,
• which company and tenant the data belongs to,
• which dataset the file represents,
• which customer-declared schema version the file follows,
• which single source file should be processed,
• how the file should be parsed,
• whether the upload corrects a previous upload.

The customer does not send contract_id or contract_version. KDSAI creates and manages the internal approved contract for each accepted company_id + tenant_id + dataset + schema.version combination.

4.1 Example manifest

{
  "manifest_version": "1.0",
  "upload_id": "upl_20260602_001",
  "company_id": "company-123",
  "tenant_id": "tenant-456",
  "dataset": "employees",
  "source_system": "customer-fleet-agent",
  "created_at_utc": "2026-06-02T10:00:00Z",
  "schema": {
    "version": "1.0.0",
    "previous_version": null,
    "change_description": null
  },
  "data_file": {
    "file_name": "employees.xlsx",
    "relative_path": "source/employees.xlsx",
    "file_type": "excel",
    "sheet_name": "Employees",
    "has_header": true,
    "row_count": 1520,
    "sha256": "REPLACE_WITH_FILE_HASH"
  }
}

4.2 Manifest field requirements

5. data_file Requirements

data_file is required and must describe exactly one source data file.

Rules:

1. One manifest cannot contain more than one data_file.
2. One Excel upload should point to one worksheet only.
3. If multiple worksheets must be processed, each worksheet should be sent as a separate upload.
4. Excel files should not contain merged cells, totals, comments, or notes inside the data range.
5. CSV files should use consistent delimiter, encoding, date format, and decimal format.
6. JSON, JSON Lines, and XML files must contain data records, not only schema or metadata.

6. Supported Source File Types

7. Optional Schema File

A schema_file can be provided when the customer knows the expected structure of the source data.

It is recommended when:

• this is the first upload of a dataset,
• this is a new schema.version,
• the file structure has intentionally changed,
• a column was added, removed, renamed, or changed,
• the customer wants to help KDSAI approve the schema faster.

Example:

{
  "schema_file": {
    "file_name": "employees_schema.json",
    "relative_path": "source/employees_schema.json",
    "file_type": "json",
    "sha256": "REPLACE_WITH_FILE_HASH"
  }
}

The schema file does not need to match the final KDSAI internal contract exactly. It is input for schema proposal, review, and approval.

Example schema file content:

{
  "dataset": "employees",
  "columns": [
    {
      "name": "Employee no.",
      "type": "string",
      "required": true
    },
    {
      "name": "First name",
      "type": "string",
      "required": true
    },
    {
      "name": "Employment date",
      "type": "date",
      "required": false,
      "formats": ["yyyy-MM-dd", "dd.MM.yyyy"]
    }
  ]
}

8. Schema Version and Approval Handling

8.1 Ownership model

The customer owns the declared schema version.

KDSAI owns schema approval and validation.

This means:

1. The customer sends schema.version in every upload.
2. The customer increases schema.version whenever the file structure changes.
3. KDSAI checks whether this company_id + tenant_id + dataset + schema.version is already approved internally.
4. If it is approved, KDSAI validates the file against the approved internal contract.
5. If it is not approved yet, KDSAI treats it as a new schema version and routes it to schema profiling, review, and approval.
6. KDSAI publishes the upload to standard ETL raw only after the schema version is approved and the data passes validation.

8.2 Decision table

8.3 Versioning rule

schema.version must represent the structure of the file, not the business date, export date, or upload sequence.

Good examples:

1.0.0  initial employees file structure
1.1.0  added optional middle_name column
2.0.0  renamed employee_number and changed primary key rules

Bad examples:

2026-06-02  daily upload date used as schema version
001         upload sequence used as schema version
latest      non-versioned alias

9. Upload Scenarios

9.1 First upload of a new dataset

Use when KDSAI has not approved any schema version for this dataset yet.

Customer action:

1. Use a new upload_id.
2. Set schema.version, usually to 1.0.0.
3. Provide schema.change_description.
4. Include schema_file when available.

Manifest fragment:

{
  "dataset": "employees",
  "schema": {
    "version": "1.0.0",
    "previous_version": null,
    "change_description": "Initial schema for the employees dataset."
  },
  "schema_file": {
    "file_name": "employees_schema.json",
    "relative_path": "source/employees_schema.json",
    "file_type": "json"
  }
}

KDSAI profiles the file, proposes an internal contract, waits for approval if required, validates the data, and only then publishes accepted data to standard ETL raw.

9.2 Repeated upload with the same structure

Use when the dataset is already onboarded and the file structure has not changed.

Customer action:

1. Use a new upload_id.
2. Send the same schema.version as the approved structure.
3. Do not include schema_file unless requested by KDSAI.
4. Do not change schema.version for ordinary daily, hourly, or repeated data uploads.

Manifest fragment:

{
  "dataset": "employees",
  "schema": {
    "version": "1.0.0"
  }
}

KDSAI validates the file against the approved internal contract for this dataset and schema version. If validation passes, KDSAI publishes the accepted package to standard ETL raw.

9.3 Upload with a changed structure

Use when the dataset is the same business dataset, but the file structure changed.

Examples:

• a column was added,
• a column was removed,
• a column was renamed,
• a column type changed,
• date or decimal format changed,
• Excel sheet name changed,
• CSV delimiter or encoding changed.

Customer action:

1. Upload to the same Fleet landing pattern for the dataset.
2. Use a new upload_id.
3. Increase schema.version.
4. Include schema.previous_version if known.
5. Include schema.change_description.
6. Include schema_file when available.

Manifest fragment:

{
  "dataset": "employees",
  "schema": {
    "version": "1.1.0",
    "previous_version": "1.0.0",
    "change_description": "Added Middle name column and changed Employment date format."
  },
  "schema_file": {
    "file_name": "employees_schema_v1_1_0.json",
    "relative_path": "source/employees_schema_v1_1_0.json",
    "file_type": "json"
  }
}

KDSAI does not publish this upload to standard ETL raw until the new schema version is reviewed, approved, and validated.

9.4 Data correction without schema change

Use when the file structure is unchanged, but previous data values were wrong.

Examples:

• invalid dates were corrected,
• missing values were filled,
• wrong amounts were corrected,
• duplicate records were removed.

Customer action:

1. Use a new upload_id.
2. Use the same schema.version as the corrected upload.
3. Include correction.is_correction=true.
4. Include correction.correction_of_upload_id.
5. Do not increase schema.version unless the file structure also changed.

Manifest fragment:

{
  "upload_id": "upl_20260611_001",
  "dataset": "employees",
  "schema": {
    "version": "1.0.0"
  },
  "correction": {
    "is_correction": true,
    "correction_of_upload_id": "upl_20260603_001",
    "correction_reason": "Corrected invalid employment date values."
  }
}

The correction metadata belongs in metadata/upload-manifest.json.

9.5 Accidental schema drift

Use when the customer sends the same schema.version, but the actual file structure is different from the approved structure.

Example:

Previous approved structure: employees schema.version=1.0.0
Incoming manifest: schema.version=1.0.0
Incoming file: contains an extra required column or renamed column

KDSAI treats this as schema drift. The upload is not processed as a normal data upload.

The customer should either:

• send a corrected file that matches the declared schema.version, or
• send a new upload with a new schema.version and a change description.

10. Upload Completion and Idempotency

The upload is complete only when metadata/upload-manifest.json exists.

Required upload order:

1. Upload the source data file under source/.
2. Upload the optional schema file under source/ if needed.
3. Upload metadata/upload-manifest.json last.

Idempotency rules:

1. upload_id must be unique for the tenant.
2. The same upload_id cannot be reused with different files.
3. Files inside an existing upload_id path must not be replaced.
4. A corrected file must be sent as a new upload.
5. KDSAI may use upload_id, dataset, received_at, schema.version, and file hash to detect duplicate or inconsistent uploads.

11. KDSAI Processing Behavior

After the manifest is uploaded, KDSAI performs the following checks:

1. The manifest exists and is valid JSON.
2. company_id, tenant_id, and dataset are present.
3. The manifest values match the assigned S3 prefix.
4. upload_id has not been reused with different files.
5. Exactly one data_file is declared.
6. The declared source file exists.
7. The file type is supported.
8. Required parsing metadata is present for the file type.
9. schema.version is present.
10. KDSAI checks whether the declared schema version is already approved internally.
11. If the schema version is not approved yet, the upload is routed to schema profiling, review, and approval.
12. If the schema version is approved, the file structure is validated against the approved internal contract.
13. Data validation passes before publication.

KDSAI publishes data to standard ETL raw only after:

the schema version is approved
the source file structure matches the approved schema version
the source file data passes validation
the accepted package is ready for ETL processing

12. Customer-Facing Status Expectation

The customer should consider the upload accepted only after KDSAI reports that the upload was validated and published to the standard ETL raw area.

Uploading files to Fleet landing does not mean that the data has been accepted into the warehouse.

Recommended customer-visible states are:

The exact way of exposing these statuses can be implemented through portal, API, report file, or operational handover. It is not required for the customer-side upload process to query this status before every upload.

13. Key Rule

The customer uploads one source data file plus one manifest to Fleet landing.

The customer owns:

• upload_id,
• dataset,
• declared schema.version,
• source file content,
• correction reference when needed.

KDSAI owns:

• schema approval,
• internal approved contract registry,
• schema version validation,
• data validation,
• publication to standard ETL raw,
• loading into Redshift.