# Fleet Customer Upload Contract

## 1. Purpose

This document defines what the customer-side Fleet system must upload to KDSAI so that one source data file can be accepted, validated, and prepared for the standard ETL process.

Fleet is a customer-facing pre-ingestion area. The customer uploads files to Fleet landing only. KDSAI publishes accepted data to the internal standard ETL raw area only after schema approval and data validation pass.

Supported source data formats for the initial contract are:

```text
Excel
CSV
JSON
JSON Lines
XML
```

---

## 2. Core Rules

1. One upload package represents one logical dataset upload.
2. One `upload-manifest.json` describes exactly one source data file.
3. One upload creates one processing batch in KDSAI.
4. Multiple source data files in one manifest are not supported in the initial contract.
5. If the customer needs to send several datasets, entities, or worksheets, they must send separate uploads with separate `upload_id` values.
6. The customer declares the dataset schema version in the manifest using `schema.version`.
7. The customer must increase `schema.version` whenever the file structure changes.
8. KDSAI owns schema approval, validation, and the internal approved contract registry.
9. The customer uploads all files first and uploads `metadata/upload-manifest.json` last.
10. KDSAI treats `metadata/upload-manifest.json` as the signal that the upload is complete.
11. The customer must not overwrite or replace files in an existing upload path.
12. Corrections are sent as new uploads with a new `upload_id`.
    **13. The customer must not write directly to standard ETL `raw`, `load`, `curated`, or `failed` prefixes.**

---

## 3. Target Upload Location

The customer uploads files only to the assigned Fleet landing prefix.

Recommended structure:

```text
s3://kds-warehouse-ai-test/fleet/landing/company_id={company_id}/tenant_id={tenant_id}/dataset={dataset}/upload_id={upload_id}/received_at={yyyy-MM-ddTHH-mm-ssZ}/
```

Example:

```text
s3://kds-warehouse-ai-test/fleet/landing/company_id=company-123/tenant_id=tenant-456/dataset=employees/upload_id=upl_20260602_001/received_at=2026-06-02T10-00-00Z/
```

Required package structure:

```text
source/
  employees.xlsx
  employees_schema.json        # optional

metadata/
  upload-manifest.json         # required; uploaded last
```

---

## 4. Upload Manifest

Every upload must include:

```text
metadata/upload-manifest.json
```

The manifest tells KDSAI:

- who sent the data,
- which company and tenant the data belongs to,
- which dataset the file represents,
- which customer-declared schema version the file follows,
- which single source file should be processed,
- how the file should be parsed,
- whether the upload corrects a previous upload.

The customer does not send `contract_id` or `contract_version`. KDSAI creates and manages the internal approved contract for each accepted `company_id + tenant_id + dataset + schema.version` combination.

### 4.1 Example manifest

```json
{
  "manifest_version": "1.0",
  "upload_id": "upl_20260602_001",
  "company_id": "company-123",
  "tenant_id": "tenant-456",
  "dataset": "employees",
  "source_system": "customer-fleet-agent",
  "created_at_utc": "2026-06-02T10:00:00Z",
  "schema": {
    "version": "1.0.0",
    "previous_version": null,
    "change_description": null
  },
  "data_file": {
    "file_name": "employees.xlsx",
    "relative_path": "source/employees.xlsx",
    "file_type": "excel",
    "sheet_name": "Employees",
    "has_header": true,
    "row_count": 1520,
    "sha256": "REPLACE_WITH_FILE_HASH"
  }
}
```

### 4.2 Manifest field requirements

| Field | Required | When to provide | Description |
|---|---:|---|---|
| `manifest_version` | Yes | Always | Version of this manifest structure. Current value: `1.0`. |
| `upload_id` | Yes | Always | Customer-generated idempotency key. Must be unique for the tenant. |
| `company_id` | Yes | Always | Company identifier assigned by KDSAI. Must match the assigned S3 prefix. |
| `tenant_id` | Yes | Always | Tenant identifier assigned by KDSAI. Must match the assigned S3 prefix. |
| `dataset` | Yes | Always | Business dataset name, for example `employees`, `orders`, or `inventory`. |
| `source_system` | Recommended | When known | Name of the customer-side system or agent that produced the export. |
| `created_at_utc` | Yes | Always | UTC timestamp when the export or upload package was created. |
| `schema` | Yes | Always | Customer-declared schema information for this dataset upload. |
| `schema.version` | Yes | Always | Customer-owned schema version. Must change whenever the file structure changes. This is not an approval flag. |
| `schema.previous_version` | Conditional | When this upload introduces a new structure based on a previous version | Previous customer-declared schema version, if known. |
| `schema.change_description` | Recommended | For first uploads and changed structures | Human-readable description of the initial schema or structural change. |
| `correction` | Conditional | When this upload corrects a previous upload | Links the corrected upload to the previous upload. |
| `data_file` | Yes | Always | The single source data file to process. |
| `schema_file` | Optional | Recommended for a new dataset or a new schema version | Optional schema description used by KDSAI during schema proposal and approval. |

---

## 5. `data_file` Requirements

`data_file` is required and must describe exactly one source data file.

| Field | Required | Applies to | Description |
|---|---:|---|---|
| `file_name` | Yes | All formats | Name of the uploaded file. |
| `relative_path` | Yes | All formats | Path relative to the upload root, for example `source/employees.xlsx`. |
| `file_type` | Yes | All formats | One of: `excel`, `csv`, `json`, `jsonl`, `xml`. |
| `sheet_name` | Conditional | Excel | Required for Excel files. Identifies the worksheet to process. |
| `has_header` | Conditional | Excel, CSV | Required for Excel and CSV files. Indicates whether the first row contains column names. |
| `encoding` | Conditional | CSV, text-like files | Required for CSV. `UTF-8` is preferred. |
| `delimiter` | Conditional | CSV | Required for CSV. Example: `,`, `;`, or `\t`. |
| `row_count` | Optional | All formats | Expected number of data rows. Used for consistency checks when available. |
| `sha256` | Optional | All formats | File hash. Used to detect incomplete uploads or accidental changes. |

Rules:

1. One manifest cannot contain more than one `data_file`.
2. One Excel upload should point to one worksheet only.
3. If multiple worksheets must be processed, each worksheet should be sent as a separate upload.
4. Excel files should not contain merged cells, totals, comments, or notes inside the data range.
5. CSV files should use consistent delimiter, encoding, date format, and decimal format.
6. JSON, JSON Lines, and XML files must contain data records, not only schema or metadata.

---

## 6. Supported Source File Types

| Type | Extension | Required manifest details | Notes |
|---|---|---|---|
| Excel | `.xlsx` | `sheet_name`, `has_header` | One worksheet per upload. |
| CSV | `.csv` | `delimiter`, `encoding`, `has_header` | UTF-8 is preferred. |
| JSON | `.json` | `file_type=json` | Must contain structured data records. |
| JSON Lines | `.jsonl` | `file_type=jsonl` | Preferred for large semi-structured exports. |
| XML | `.xml` | `file_type=xml` | XML is treated as a source data format. Mapping rules are handled by KDSAI. |

---

## 7. Optional Schema File

A `schema_file` can be provided when the customer knows the expected structure of the source data.

It is recommended when:

- this is the first upload of a dataset,
- this is a new `schema.version`,
- the file structure has intentionally changed,
- a column was added, removed, renamed, or changed,
- the customer wants to help KDSAI approve the schema faster.

Example:

```json
{
  "schema_file": {
    "file_name": "employees_schema.json",
    "relative_path": "source/employees_schema.json",
    "file_type": "json",
    "sha256": "REPLACE_WITH_FILE_HASH"
  }
}
```

The schema file does not need to match the final KDSAI internal contract exactly. It is input for schema proposal, review, and approval.

Example schema file content:

```json
{
  "dataset": "employees",
  "columns": [
    {
      "name": "Employee no.",
      "type": "string",
      "required": true
    },
    {
      "name": "First name",
      "type": "string",
      "required": true
    },
    {
      "name": "Employment date",
      "type": "date",
      "required": false,
      "formats": ["yyyy-MM-dd", "dd.MM.yyyy"]
    }
  ]
}
```

---

## 8. Schema Version and Approval Handling

### 8.1 Ownership model

The customer owns the declared schema version.

KDSAI owns schema approval and validation.

This means:

1. The customer sends `schema.version` in every upload.
2. The customer increases `schema.version` whenever the file structure changes.
3. KDSAI checks whether this `company_id + tenant_id + dataset + schema.version` is already approved internally.
4. If it is approved, KDSAI validates the file against the approved internal contract.
5. If it is not approved yet, KDSAI treats it as a new schema version and routes it to schema profiling, review, and approval.
6. KDSAI publishes the upload to standard ETL raw only after the schema version is approved and the data passes validation.

### 8.2 Decision table

| Customer upload | KDSAI behavior |
|---|---|
| `dataset=employees`, `schema.version=1.0.0`, version already approved | Validate data against the approved internal contract and process if valid. |
| `dataset=employees`, `schema.version=1.1.0`, version not known | Treat as a new schema version, profile/review/approve before ETL publication. |
| Same `schema.version` as before, but file structure changed | Treat as schema drift and reject or quarantine. The customer should send a new upload with a new `schema.version`. |
| Same `schema.version` and same structure, but invalid values | Treat as data validation error, not schema change. |
| Same `schema.version` and same structure, corrected values | Treat as correction if the `correction` object is provided. |

### 8.3 Versioning rule

`schema.version` must represent the structure of the file, not the business date, export date, or upload sequence.

Good examples:

```text
1.0.0  initial employees file structure
1.1.0  added optional middle_name column
2.0.0  renamed employee_number and changed primary key rules
```

Bad examples:

```text
2026-06-02  daily upload date used as schema version
001         upload sequence used as schema version
latest      non-versioned alias
```

---

## 9. Upload Scenarios

### 9.1 First upload of a new dataset

Use when KDSAI has not approved any schema version for this dataset yet.

Customer action:

1. Use a new `upload_id`.
2. Set `schema.version`, usually to `1.0.0`.
3. Provide `schema.change_description`.
4. Include `schema_file` when available.

Manifest fragment:

```json
{
  "dataset": "employees",
  "schema": {
    "version": "1.0.0",
    "previous_version": null,
    "change_description": "Initial schema for the employees dataset."
  },
  "schema_file": {
    "file_name": "employees_schema.json",
    "relative_path": "source/employees_schema.json",
    "file_type": "json"
  }
}
```

KDSAI profiles the file, proposes an internal contract, waits for approval if required, validates the data, and only then publishes accepted data to standard ETL raw.

### 9.2 Repeated upload with the same structure

Use when the dataset is already onboarded and the file structure has not changed.

Customer action:

1. Use a new `upload_id`.
2. Send the same `schema.version` as the approved structure.
3. Do not include `schema_file` unless requested by KDSAI.
4. Do not change `schema.version` for ordinary daily, hourly, or repeated data uploads.

Manifest fragment:

```json
{
  "dataset": "employees",
  "schema": {
    "version": "1.0.0"
  }
}
```

KDSAI validates the file against the approved internal contract for this dataset and schema version. If validation passes, KDSAI publishes the accepted package to standard ETL raw.

### 9.3 Upload with a changed structure

Use when the dataset is the same business dataset, but the file structure changed.

Examples:

- a column was added,
- a column was removed,
- a column was renamed,
- a column type changed,
- date or decimal format changed,
- Excel sheet name changed,
- CSV delimiter or encoding changed.

Customer action:

1. Upload to the same Fleet landing pattern for the dataset.
2. Use a new `upload_id`.
3. Increase `schema.version`.
4. Include `schema.previous_version` if known.
5. Include `schema.change_description`.
6. Include `schema_file` when available.

Manifest fragment:

```json
{
  "dataset": "employees",
  "schema": {
    "version": "1.1.0",
    "previous_version": "1.0.0",
    "change_description": "Added Middle name column and changed Employment date format."
  },
  "schema_file": {
    "file_name": "employees_schema_v1_1_0.json",
    "relative_path": "source/employees_schema_v1_1_0.json",
    "file_type": "json"
  }
}
```

KDSAI does not publish this upload to standard ETL raw until the new schema version is reviewed, approved, and validated.

### 9.4 Data correction without schema change

Use when the file structure is unchanged, but previous data values were wrong.

Examples:

- invalid dates were corrected,
- missing values were filled,
- wrong amounts were corrected,
- duplicate records were removed.

Customer action:

1. Use a new `upload_id`.
2. Use the same `schema.version` as the corrected upload.
3. Include `correction.is_correction=true`.
4. Include `correction.correction_of_upload_id`.
5. Do not increase `schema.version` unless the file structure also changed.

Manifest fragment:

```json
{
  "upload_id": "upl_20260611_001",
  "dataset": "employees",
  "schema": {
    "version": "1.0.0"
  },
  "correction": {
    "is_correction": true,
    "correction_of_upload_id": "upl_20260603_001",
    "correction_reason": "Corrected invalid employment date values."
  }
}
```

The correction metadata belongs in `metadata/upload-manifest.json`.

### 9.5 Accidental schema drift

Use when the customer sends the same `schema.version`, but the actual file structure is different from the approved structure.

Example:

```text
Previous approved structure: employees schema.version=1.0.0
Incoming manifest: schema.version=1.0.0
Incoming file: contains an extra required column or renamed column
```

KDSAI treats this as schema drift. The upload is not processed as a normal data upload.

The customer should either:

- send a corrected file that matches the declared `schema.version`, or
- send a new upload with a new `schema.version` and a change description.

---

## 10. Upload Completion and Idempotency

The upload is complete only when `metadata/upload-manifest.json` exists.

Required upload order:

1. Upload the source data file under `source/`.
2. Upload the optional schema file under `source/` if needed.
3. Upload `metadata/upload-manifest.json` last.

Idempotency rules:

1. `upload_id` must be unique for the tenant.
2. The same `upload_id` cannot be reused with different files.
3. Files inside an existing `upload_id` path must not be replaced.
4. A corrected file must be sent as a new upload.
5. KDSAI may use `upload_id`, `dataset`, `received_at`, `schema.version`, and file hash to detect duplicate or inconsistent uploads.

---

## 11. KDSAI Processing Behavior

After the manifest is uploaded, KDSAI performs the following checks:

1. The manifest exists and is valid JSON.
2. `company_id`, `tenant_id`, and `dataset` are present.
3. The manifest values match the assigned S3 prefix.
4. `upload_id` has not been reused with different files.
5. Exactly one `data_file` is declared.
6. The declared source file exists.
7. The file type is supported.
8. Required parsing metadata is present for the file type.
9. `schema.version` is present.
10. KDSAI checks whether the declared schema version is already approved internally.
11. If the schema version is not approved yet, the upload is routed to schema profiling, review, and approval.
12. If the schema version is approved, the file structure is validated against the approved internal contract.
13. Data validation passes before publication.

KDSAI publishes data to standard ETL raw only after:

```text
the schema version is approved
the source file structure matches the approved schema version
the source file data passes validation
the accepted package is ready for ETL processing
```

---

## 12. Customer-Facing Status Expectation

The customer should consider the upload accepted only after KDSAI reports that the upload was validated and published to the standard ETL raw area.

Uploading files to Fleet landing does not mean that the data has been accepted into the warehouse.

Recommended customer-visible states are:

| State | Meaning |
|---|---|
| `Uploaded` | Files and manifest were received in Fleet landing. |
| `Rejected` | Manifest, file structure, schema version, or data validation failed. |
| `WaitingForSchemaApproval` | A new schema version requires approval before processing. |
| `AcceptedForEtl` | Data passed Fleet validation and was published to standard ETL raw. |
| `Loaded` | Standard ETL loaded the data into the warehouse. |

The exact way of exposing these statuses can be implemented through portal, API, report file, or operational handover. It is not required for the customer-side upload process to query this status before every upload.

---

## 13. Key Rule

The customer uploads one source data file plus one manifest to Fleet landing.

The customer owns:

- `upload_id`,
- `dataset`,
- declared `schema.version`,
- source file content,
- correction reference when needed.

KDSAI owns:

- schema approval,
- internal approved contract registry,
- schema version validation,
- data validation,
- publication to standard ETL raw,
- loading into Redshift.