# Technologies

This document captures the **technology choices** for the agent-based monitoring system and recommends options for **knowledgebase storage + retrieval**.

## Model hosting

**Primary choice: Amazon Bedrock**

- Use Bedrock for model inference (and optionally embeddings, if you want a managed embedding model).
- Agent orchestration remains in the application layer (Microsoft Agent Framework), so the model host is a replaceable backend.

## Agent runtime / orchestration

**Primary choice: Microsoft Agent Framework**

- GitHub: [microsoft/agent-framework](https://github.com/microsoft/agent-framework)
- Docs: [Microsoft Learn: Agent Framework overview](https://learn.microsoft.com/en-us/agent-framework/overview/agent-framework-overview)

Rationale:

- Multi-agent orchestration and workflow primitives with **.NET** support.
- Pluggable **context providers** and **tool integration** patterns.
- Built-in support for **Model Context Protocol (MCP)** clients (for tool connectivity).

Hosting (vendor-neutral):

- Run agents as containerized services (Docker) on ECS/EKS/Kubernetes (or any VM/container platform).
- Keep agent state/specs in your KB stores (below) so compute is stateless and portable.

## Database connectivity

**Primary choice: Model Context Protocol (MCP) servers**

Pattern:

- Each source system (POS, KDS, inventory, etc.) is exposed as an **MCP server** (or a small set of servers).
- The **Data Access Agent** is an **MCP client** and is the only component allowed to touch data sources directly.

Why:

- Keeps schema/tool noise out of other agents’ contexts.
- Allows consistent, permissioned access across heterogeneous sources.

Minimum guardrails to bake into MCP tools:

- Restaurant/tenant scoping (every query requires `restaurant_id` and time window).
- Read-only enforcement and query auditing.
- Prefer returning **feature panels** (aggregates/cohorts/distributions) over raw rows.

## Warehouse & ETL (future)

Current (MVP):

- One **MySQL** instance with multiple databases; a single service account can query all tenants.

Recommended direction for a managed warehouse you operate:

- **Raw zone**: S3 (or S3-compatible) with immutable extracts in open formats (prefer Parquet), partitioned by `restaurant_id`, `source_system`, and date.
- **Serving warehouse (OLAP)**: **ClickHouse** for fast analytical SQL over large, mostly-immutable event data (orders, line items, transactions, etc.).
- **Operational metadata DB**: keep **MySQL** for tenant config, KB structured facts, `MonitorSpec`, run history, and alert dedupe.

Why ClickHouse fits this system:

- Optimized for append-heavy, analytical workloads (lots of `GROUP BY`, time-window scans, cohorts).
- Strong compression and performance for time-series/event-style tables.

Tenant isolation options (in increasing “strictness”):

- **App-layer isolation only**: simplest, but weakest; relies on the Data Access Agent to always enforce `restaurant_id` scoping.
- **Shared ClickHouse cluster, separate DB + user per tenant**: good defense-in-depth while still allowing a unified platform.
- **Deployment per restaurant** (separate warehouse stack per tenant): strongest isolation, but more ops cost; also makes future cross-restaurant comparisons harder unless you add an aggregation layer.

Portability / escape hatch:

- Keeping the raw zone in **Parquet** on S3 lets you move to a lakehouse query engine later (e.g., Iceberg + Trino/Spark) without re-extracting from restaurants.

## Knowledgebase storage + retrieval (recommended)

The system needs *two* kinds of knowledge:

1) **Structured “facts”** (authoritative): schema catalog, join graph, entity mappings, KPI definitions, versions.
2) **Unstructured text** (supporting): manuals/SOPs, vendor docs, “what does this mean?” passages.

### Recommended architecture (AWS-friendly, still portable)

- **Structured KB store**: keep your current **MySQL** (or use PostgreSQL) for:
  - Capabilities Manifest (tables/columns, units, freshness, join edges)
  - KPI Catalog (definitions/targets/exceptions)
  - `MonitorSpec` versions and approvals
  - Run history + alert dedupe memory
- **Document store**: S3 (or any S3-compatible object store like MinIO) for raw manuals and vendor docs (plus extracted text artifacts).
- **Retrieval / vector search** (choose one):
  - **OpenSearch** (AWS OpenSearch Service or self-managed): hybrid retrieval (BM25 + vector) with metadata filters.
  - **ChromaDB** (self-hosted): simple vector DB for chunks + embeddings + metadata (keep MySQL as the system-of-record for structured facts).
  - **Redis Stack (RediSearch vector search)**: fast “serving index” for vectors + metadata filters; keep MySQL/S3 as the source of truth and rebuild/sync the Redis index as needed.
  - **PostgreSQL + pgvector**: simplest “single DB” option if you’re willing to use Postgres for both structured KB + vectors.

Why this works well for your system:

- Structured KB stays queryable and versioned (so specs don’t rot when schemas change).
- A dedicated search index (OpenSearch) or pgvector provides filterable retrieval for manuals/docs without overloading the LLM context.
- The KB/Semantic Agent can return a compact “KB packet” that merges structured facts with a few relevant doc passages.

### Alternative: single-store (simpler ops, fewer moving parts)

If you strongly prefer one primary store:

- **PostgreSQL + pgvector**: store structured KB + chunks + embeddings in one DB.
  - Pros: simplest deployment model, strong transactional semantics, easy versioning.
  - Cons: retrieval quality/ranking/filters are good but typically less specialized than a dedicated search service.
- **OpenSearch-only**: store doc chunks + embeddings + metadata in OpenSearch; keep structured facts in a relational DB anyway.
  - Pros: strong retrieval UX; flexible filtering.
  - Cons: not a great system-of-record for versioned structured facts.

### MVP option: vectors in-memory (fastest to ship)

If the KB is small (manuals + a bit of schema docs), you can start with an **in-memory vector index** inside the KB/Semantic Agent:

- Store doc chunks + metadata in your relational DB (MySQL).
- Store embeddings either:
  - in MySQL as a BLOB (so you can reload quickly on startup), or
  - regenerated on startup if the embedding volume is tiny and cold-start time is acceptable.
- Build an in-memory ANN index (or brute-force similarity for small N) per restaurant/collection at runtime.

This is fine for early prototyping, but you’ll likely want a persistent vector store (Chroma/OpenSearch/Postgres) once you have multiple instances, frequent KB updates, or you need consistent results across restarts.

### Notes on Redis as a “middle option”

Redis can be a good fit when you want **very low-latency retrieval** and you’re comfortable treating the vector index as *derived state*:

- Use Redis primarily as a **serving index / cache** for document chunk vectors and metadata filters (`restaurant_id`, `doc_type`, etc.).
- Keep the source of truth in MySQL/S3; make the Redis index rebuildable from those stores.
- Beware memory sizing (vector indexes can be large) and confirm your chosen Redis deployment supports the required vector search features (e.g., RediSearch).

### Optional (AWS-managed RAG for documents)

If you want a managed “docs RAG” layer that integrates directly with Bedrock:

- **Knowledge Bases for Amazon Bedrock** can handle ingestion + chunking + embeddings + retrieval for unstructured documents.
- You will still want the **structured KB store** (MySQL/PostgreSQL) for schema facts, join graph, KPI catalog, specs, and run history.

### Retrieval strategy (how agents should use the KB)

To avoid hallucinating schema/joins/KPIs:

- **Prefer structured lookups first** (manifest + KPI catalog).
- Use doc retrieval only to:
  - confirm definitions (“prep time” meaning, units, exceptions)
  - reference the manual in explanations
  - discover additional candidate KPIs the structured extractor missed

Practical “KB packet” returned by the KB/Semantic Agent should include:

- Canonical entity mapping (e.g., `Order`, `LineItem`, `Station`) → source tables/columns
- Verified join path(s) with confidence
- KPI definitions relevant to the monitor prompt (targets + exceptions)
- 3–8 short doc excerpts (manual/vendor) with doc IDs + section headings

## Observability (recommended)

- **OpenTelemetry** tracing across orchestrator, MCP calls, and retrieval.
- Persist per-run “evidence panels” (or references) so alerts can be audited later.