Categories
Implementation Specs (157)
- specs/AnomalyDetection/TASK1.1.md
- specs/AnomalyDetection/TASK1.2.md
- specs/AnomalyDetection/TASK1.3.md
- specs/AnomalyDetection/TASK1.4.md
- specs/AnomalyDetection/TASK2.1.md
- specs/AnomalyDetection/TASK2.2.md
- specs/AnomalyDetection/TASK3.1.md
- specs/AnomalyDetection/TASK3.2.md
- specs/AnomalyDetection/TASK3.3.md
- specs/AnomalyDetection/TASK4.1.md
- specs/AnomalyDetection/TASK4.2.md
- specs/AnomalyDetection/TASK4.3.md
- specs/AnomalyDetection/TASK4.4.md
- specs/AnomalyDetection/TASK5.1.md
- specs/AnomalyDetection/TASK5.2.md
- specs/AnomalyDetection/TASK5.3.md
- specs/AnomalyDetection/TASK6.1.md
- specs/AnomalyDetection/TASK6.2.md
- specs/AnomalyDetection/TASK6.3.md
- specs/AnomalyDetection/TASKS.md
- specs/ApiServicePostgresTestStability/TASK.md
- specs/BackendDatabaseConsolidation/TASK.md
- specs/BackendDatabaseConsolidation/TASK1.1.md
- specs/BackendDatabaseConsolidation/TASK1.2.md
- specs/BackendDatabaseConsolidation/TASK1.3.md
- specs/BackendDatabaseConsolidation/TASK2.1.md
- specs/BackendDatabaseConsolidation/TASK2.2.md
- specs/BackendDatabaseConsolidation/TASK2.3.md
- specs/BackendDatabaseConsolidation/TASK3.1.md
- specs/BackendDatabaseConsolidation/TASK3.2.md
- specs/BackendDatabaseConsolidation/TASK4.1.md
- specs/BackendDatabaseConsolidation/TASK4.2.md
- specs/BackendDatabaseConsolidation/TASK4.3.md
- specs/BackendDatabaseConsolidation/TASK5.1.md
- specs/BackendDatabaseConsolidation/TASK5.2.md
- specs/BrinkLaborGetShiftsStaging/TASK.md
- specs/BrinkPosOrdersFirstEtlJob/TASK.md
- specs/BrinkPosOrdersFirstEtlJob/TASK0.md
- specs/BrinkPosOrdersFirstEtlJob/TASK1.md
- specs/BrinkPosOrdersFirstEtlJob/TASK2.1.md
- specs/BrinkPosOrdersFirstEtlJob/TASK2.2.md
- specs/BrinkPosOrdersFirstEtlJob/TASK2.3.md
- specs/BrinkPosOrdersFirstEtlJob/TASK2.4.md
- specs/BrinkPosOrdersFirstEtlJob/TASK2.md
- specs/BrinkPosOrdersFirstEtlJob/TASK3.1.md
- specs/BrinkPosOrdersFirstEtlJob/TASK3.2.md
- specs/BrinkPosOrdersFirstEtlJob/TASK3.3.md
- specs/BrinkPosOrdersFirstEtlJob/TASK3.4.md
- specs/BrinkPosOrdersFirstEtlJob/TASK3.md
- specs/BrinkPosOrdersFirstEtlJob/TASK4.1.md
- specs/BrinkPosOrdersFirstEtlJob/TASK4.2.md
- specs/BrinkPosOrdersFirstEtlJob/TASK4.3.md
- specs/BrinkPosOrdersFirstEtlJob/TASK4.4.md
- specs/BrinkPosOrdersFirstEtlJob/TASK4.md
- specs/BrinkPosOrdersFirstEtlJob/TASK5.1.md
- specs/BrinkPosOrdersFirstEtlJob/TASK5.2.md
- specs/BrinkPosOrdersFirstEtlJob/TASK5.3.md
- specs/BrinkPosOrdersFirstEtlJob/TASK5.4.md
- specs/BrinkPosOrdersFirstEtlJob/TASK5.md
- specs/BrinkPosOrdersFirstEtlJob/TASK6.1.md
- specs/BrinkPosOrdersFirstEtlJob/TASK6.2.md
- specs/BrinkPosOrdersFirstEtlJob/TASK6.3.md
- specs/BrinkPosOrdersFirstEtlJob/TASK6.4.md
- specs/BrinkPosOrdersFirstEtlJob/TASK6.md
- specs/BrinkPosOrdersFirstEtlJob/TASK7.md
- specs/BrinkPosOrdersFirstEtlJob/TASK8.md
- specs/BrinkPosOrdersFirstEtlJob/TASK9.md
- specs/BrinkSettings2ToStaging/TASK.MD
- specs/CuratedToS3Stage/TASK.md
- specs/CuratedToS3Stage/TASK0.md
- specs/CuratedToS3Stage/TASK1.md
- specs/CuratedToS3Stage/TASK2.md
- specs/CuratedToS3Stage/TASK3.md
- specs/CuratedToS3Stage/TASK4.md
- specs/CustomerCredentialCustomHeaders/TASK.md
- specs/CustomerCredentialCustomHeaders/TASK1.1.md
- specs/CustomerCredentialCustomHeaders/TASK1.2.md
- specs/CustomerCredentialCustomHeaders/TASK2.1.md
- specs/CustomerCredentialCustomHeaders/TASK2.2.md
- specs/CustomerCredentialCustomHeaders/TASK3.1.md
- specs/CustomerCredentialTriggeredProvisioning/TASK.md
- specs/CustomerCredentialTriggeredProvisioning/TASK1.1.md
- specs/CustomerCredentialTriggeredProvisioning/TASK1.2.md
- specs/CustomerCredentialTriggeredProvisioning/TASK2.1.md
- specs/CustomerCredentialTriggeredProvisioning/TASK2.2.md
- specs/CustomerCredentialTriggeredProvisioning/TASK3.1.md
- specs/CustomerCredentialTriggeredProvisioning/TASK3.2.md
- specs/CustomerCredentialTriggeredProvisioning/TASK4.1.md
- specs/EtlSourceAdapterFramework/TASK.md
- specs/EtlSourceAdapterFramework/TASK0.md
- specs/EtlSourceAdapterFramework/TASK1.md
- specs/EtlSourceAdapterFramework/TASK2.md
- specs/EtlSourceAdapterFramework/TASK3.md
- specs/EtlSourceAdapterFramework/TASK4.md
- specs/EtlSourceAdapterFramework/TASK5.md
- specs/EtlSourceAdapterFramework/TASK6.md
- specs/KdsCdcDebeziumIntegration/TASK.md
- specs/KdsCdcDebeziumIntegration/TASK1.md
- specs/KdsCdcDebeziumIntegration/TASK2.md
- specs/KdsCdcDebeziumIntegration/TASK3.md
- specs/KdsCdcDebeziumIntegration/TASK4.md
- specs/KdsCdcDebeziumIntegration/TASK5.md
- specs/KdsCdcDurableRelationRouting/TASK.md
- specs/KdsCdcDurableRelationRouting/TASK1.md
- specs/KdsCdcDurableRelationRouting/TASK2.md
- specs/KdsCdcDurableRelationRouting/TASK3.md
- specs/KdsCdcDurableRelationRouting/TASK4.md
- specs/KdsCdcDurableRelationRouting/TASK5.md
- specs/KdsCdcIngestion/TASK.md
- specs/KdsCdcIngestion/TASK1.md
- specs/KdsCdcIngestion/TASK2.md
- specs/KdsCdcIngestion/TASK3.md
- specs/KdsCdcIngestion/TASK4.md
- specs/KdsCdcIngestion/TASK5.md
- specs/KdsCdcIngestion/TASK6.md
- specs/KdsCdcIngestion/TASK7.md
- specs/KdsCdcIngestion/TASK8.md
- specs/KdsCdcIngestion/TASK9.md
- specs/KdsCdcReplayExecution/TASK.md
- specs/KdsCdcReplayExecution/TASK1.md
- specs/KdsCdcReplayExecution/TASK2.md
- specs/KdsCdcReplayExecution/TASK3.md
- specs/KdsCdcReplayExecution/TASK4.md
- specs/KdsMonitorServingContractMigration/TASK.md
- specs/KdsMonitorServingContractMigration/TASK1.1.md
- specs/KdsMonitorServingContractMigration/TASK1.2.md
- specs/KdsMonitorServingContractMigration/TASK2.1.md
- specs/KdsMonitorServingContractMigration/TASK2.2.md
- specs/KdsMonitorServingContractMigration/TASK3.1.md
- specs/KdsMonitorServingContractMigration/TASK3.2.md
- specs/RedshiftFoundation/TASK.md
- specs/RedshiftFoundation/TASK1.1.md
- specs/RedshiftFoundation/TASK1.2.md
- specs/RedshiftFoundation/TASK1.3.md
- specs/RedshiftFoundation/TASK1.4.md
- specs/RedshiftFoundation/TASK2.1.md
- specs/RedshiftFoundation/TASK2.2.md
- specs/RedshiftFoundation/TASK3.1.md
- specs/RedshiftFoundation/TASK3.2.md
- specs/RedshiftFoundation/TASK3.3.md
- specs/RedshiftSyntheticDataFeed/TASK.md
- specs/RedshiftSyntheticDataFeed/TASK1.1.md
- specs/RedshiftSyntheticDataFeed/TASK1.2.md
- specs/RedshiftSyntheticDataFeed/TASK1.3.md
- specs/RedshiftSyntheticDataFeed/TASK2.1.md
- specs/RedshiftSyntheticDataFeed/TASK2.2.md
- specs/RedshiftSyntheticDataFeed/TASK3.1.md
- specs/RedshiftSyntheticDataFeed/TASK3.2.md
- specs/SingleStoreRedshiftRuntimeReadPath/TASK.md
- specs/SingleStoreRedshiftRuntimeReadPath/TASK1.1.md
- specs/SingleStoreRedshiftRuntimeReadPath/TASK1.2.md
- specs/SingleStoreRedshiftRuntimeReadPath/TASK2.1.md
- specs/SingleStoreRedshiftRuntimeReadPath/TASK2.2.md
- specs/SingleStoreRedshiftRuntimeReadPath/TASK2.3.md
- specs/SingleStoreRedshiftRuntimeReadPath/TASK3.1.md
- specs/SingleStoreRedshiftRuntimeReadPath/TASK3.2.md
- specs/SingleStoreRedshiftRuntimeReadPath/TASK4.1.md
Brink Settings2 To Staging
Goal
Add BrinkPOS Settings2 reference data into the Brink ETL flow as raw staging data for Redshift.
This task covers three Settings2 endpoints:
GetEmployeesGetJobsGetItems
The target implementation must follow the newer Brink ETL contract-onboarding pattern already established by BrinkLaborGetShiftsStaging, not the older mapper-only Settings2 path.
Source
- Endpoint:
https://api2.brinkpos.net/Settings2.svc - SOAP Actions:
http://www.brinksoftware.com/webservices/settings/v2/ISettingsWebService2/GetEmployeeshttp://www.brinksoftware.com/webservices/settings/v2/ISettingsWebService2/GetJobshttp://www.brinksoftware.com/webservices/settings/v2/ISettingsWebService2/GetItems
Scope
Onboard Settings2 into the standard Brink ETL pipeline:
extract_rawnormalize_loadload_to_staging
This is not a one-off import path.
Current Repo Findings
Already implemented
GetEmployeestyped facade support exists.GetItemstyped facade support exists.Settings2Clientalready supportsGetEmployeesAsync(...)andGetItemsAsync(...).BrinkSoapEnvelopeBuilderalready supportsSettings2SOAP envelopes.- Legacy staging SQL already exists for some
Settings2entities: staging.brink_settings2_get_employeesstaging.brink_settings2_itemsstaging.brink_settings2_item_groupsstaging.brink_settings2_item_modifier_groupsstaging.brink_settings2_item_taxes- Legacy row models / mappers already exist for some
Settings2endpoints.
Missing or incomplete
GetJobstransport / facade support is not onboarded.IBrinkRawClient/BrinkRawClientdo not supportSettings2raw capture.BrinkPosEndpointCatalogonly includes:Sales2.GetOrdersLabor.GetShiftsBrinkPosEndpointExecutoronly dispatches supported extract endpoints for orders and labor shifts.BrinkPosOperationCataloghas noSettings2ETL operation entries.BrinkPosNormalizationCataloghas noSettings2contract group for this task.BrinkPosStagingTableCataloghas noSettings2table group for this task.- DI currently registers only the new-style staging profiles:
Sales2GetOrdersStagingProfileLaborGetShiftsStagingProfile
Important legacy mismatch
Existing Settings2 employees / items artifacts do not consistently follow the newer Brink staging standard:
- some rows inherit
NormalizedRowBaseinstead ofBrinkStagingRowBase - some tables are missing
_etl_contract_name,_etl_contract_row_type,_etl_row_ordinal - some nested collections are stored as
SUPERJSON blobs instead of separate child artifacts - naming is inconsistent between
get_*and entity-based table names
This task should align the new implementation with the modern Brink staging pattern and only reuse legacy tables where their shape is already compatible or can be upgraded safely.
Required Rules
- Staging is raw, not curated.
- Business columns use Brink field names flattened to
snake_case. - Shared staging/parquet row models should be used; do not introduce separate parquet-only DTOs.
- All new staging rows inherit from
BrinkStagingRowBase. - Include standard
_etl_*metadata fromNormalizedRowBase+BrinkStagingRowBase. - Every new staging table, including every child table, must contain the full modern Brink ETL metadata set:
_etl_batch_id_etl_company_id_etl_tenant_id_etl_source_type_etl_stream_name_etl_operation_name_etl_ingested_at_utc_etl_raw_s3_path_etl_raw_file_name_etl_parser_version_etl_schema_version_etl_contract_name_etl_contract_row_type_etl_row_ordinal- Use
StageDiagnosticsEnvelope<T>details payloads for extract / normalize / load diagnostics. - Plug into the existing Brink ETL job chain, not a special settings-specific pipeline.
- Do not introduce legacy generic metadata columns like
payload_json,record_hash,source_endpoint, orloaded_at_utcunless the framework requires them. - Nested collections should be split into separate child artifacts when the structure is simple and relationally useful.
- For very deep
Itemsubobjects, it is acceptable to keep selected substructures as JSON/SUPER if flattening them all would create excessive scope with little staging value. - If
Settings2is extracted as part of a date-scoped integration run, staging should preserve the observed snapshot for that run rather than only the latest tenant-wide state.
Flow
- Extract from Brink POS
BrinkPosExtractOrchestratorstarts the job.BrinkPosEndpointCatalogprovides the list ofSettings2endpoints to capture.BrinkPosEndpointExecutorcalls Brink SOAP endpoints throughIBrinkRawClient.- Store raw response
- Raw Brink XML is saved to S3 via
IEtlS3Storage.WriteRawAsync(...). - Extract details are persisted as
StageDiagnosticsEnvelope<BrinkPosExtractBatchDetails>. - Normalize / map data
BrinkPosNormalizeLoadJobsreads endpoint-level extract details.- Dedicated staging profiles parse the SOAP payload through
IBrinkXmlEnvelopeParser. - Profiles flatten parent rows and selected child collections into normalized staging row models.
- Write Parquet
ParquetBatchWriter<T>writes staging row models directly into parquet.- Parent and child collections are emitted as separate artifacts.
- Load to Redshift staging
BrinkPosLoadToStagingJobsresolves tables from the normalized contracts.RedshiftStagingLoadExecutorloads parquet files into Redshift staging.- Existing rows are deleted first using
Settings2-appropriate idempotency predicates.
Source Contract
Employee Type
| Data Name | Data Type | Data Description |
|---|---|---|
| Address1 | string | Employee's address |
| Address2 | string | Employee's address |
| AlternateId | string | Alternate external Id for this Employee |
| BirthDate | DateTime? | Employee's birth date |
| CanLoginWithCard | bool | Whether Employee can login with card |
| CanLoginWithFinger | bool | Whether Employee can login with fingerprint |
| CanLoginWithPin | bool | Whether Employee can login with PIN |
| CardNumber | string | Employee card number |
| CellPhone | string | Employee cell phone |
| City | string | Employee city |
| ClockedInDiscountId | int? | Clocked-in discount Id |
| ClockedOutDiscountId | int? | Clocked-out discount Id |
| DisplayName | string | Display name |
| EmailAddress | string | |
| EmployeeUniqueId | Guid | Employee unique identifier |
| ExportToPayroll | bool | Export to payroll flag |
| FirstName | string | First name |
| HealthCardExpirationDate | DateTime? | Health card expiration |
| HireDate | DateTime? | Hire date |
| HomeLocationId | Guid? | Home location |
| HomePhone | string | Home phone |
| Id | int | Employee Id |
| IdentificationVerified | bool | Identity verified flag |
| IsExempt | bool | Tax exempt flag |
| IsSalaried | bool | Salaried flag |
| Jobs | Array of EmployeeJob | Assigned jobs |
| LastName | string | Last name |
| LimitLocations | bool | Whether locations are limited |
| MaritalStatus | byte | Marital status |
| MaximumDailyDiscountAmount | decimal | Maximum daily discount amount |
| MaximumDailyDiscountCount | int | Maximum daily discount count |
| Notes | string | Notes |
| PayrollId | string | Payroll Id |
| Permissions | Array of EmployeePermission | Permissions |
| Pin | string | Always null when returned |
| SecondaryAuthPin | string | Secondary auth PIN |
| State | string | State |
| TaxWitholdingAllowance | int | Withholding allowance |
| Terminated | bool | Terminated flag |
| TerminationDate | DateTime? | Termination date |
| ValidLocations | Array of EmployeeLocation | Valid locations |
| Zip | string | Zip |
EmployeeJob Type
| Data Name | Data Type | Data Description |
|---|---|---|
| Id | int | Employee-job relationship Id |
| JobId | int | Job Id |
| JobUniqueId | Guid | Job unique Id |
| PayRate | decimal | Pay rate |
| SecurityLevelId | int | Security level Id |
| SecurityLevelUniqueId | Guid | Security level unique Id |
EmployeeLocation Type
| Data Name | Data Type | Data Description |
|---|---|---|
| Id | int | Employee-location relationship Id |
| LocationId | Guid | Location Id |
EmployeePermission Type
| Data Name | Data Type | Data Description |
|---|---|---|
| Id | int | Employee-permission relationship Id |
| PermissionId | int | Permission Id |
| PermissionUniqueId | Guid | Permission unique Id |
Job Type
| Data Name | Data Type | Data Description |
|---|---|---|
| AllowAddItem | bool | Allow add item |
| AlternateId | string | Alternate external Id |
| CannotClose | bool | Cannot close flag |
| CanOpenAnyOrder | bool | Can open any order flag |
| CashDrawer | bool | Cash drawer flag |
| CheckoutRequiresApproval | bool | Checkout approval required |
| ClockInRequiresApproval | bool | Clock-in approval required |
| ClockOutRequiresApproval | bool | Clock-out approval required |
| DeclareTips | bool | Can declare tips |
| DefaultScreenId | int? | Default screen Id |
| DefaultSecurityLevelId | int? | Default security level Id |
| DefaultSecurityLevelUniqueId | Guid? | Default security level unique Id |
| DisplayColor | int | ARGB display color |
| ExcludeOnSalesAndLaborReports | bool | Exclude from reports |
| ExemptFromLaborSchedule | bool | Exempt from labor schedule |
| ExportCode | string | Export code |
| GroupItemsBySeat | bool | Group items by seat |
| Id | int | Job Id |
| IndicateTrainingModeOnKitchenChits | bool | Indicate training mode |
| IsBartender | bool | Bartender job |
| IsDeliveryDispatcher | bool | Delivery dispatcher job |
| IsDeliveryDriver | bool | Delivery driver job |
| IsHostess | bool | Hostess job |
| ItemLookup | bool | Item lookup allowed |
| JobUniqueId | Guid | Job unique Id |
| LaneId | int? | Lane Id |
| LimitedDestinationId | int? | Limited destination Id |
| LimitShiftBreakTypes | bool | Limit shift break types |
| LocationType | byte | Location type |
| Menus | Array of JobMenu | Assigned menus |
| Name | string | Job name |
| NoCashTransactions | bool | No cash transactions |
| OrderEntry | bool | Order entry allowed |
| OrderScreenId | int? | Order screen Id |
| RequireTipReporting | bool | Require tip reporting |
| SectionId | int? | Section Id |
| SelfBanking | bool | Self-banking flag |
| SelfVoid | bool | Self-void flag |
| ShiftBreakTypes | Array of JobShiftBreakType | Allowed break types |
| Tabs | bool | Tabs allowed |
| Training | bool | Training job |
JobMenu Type
| Data Name | Data Type | Data Description |
|---|---|---|
| Days | byte? | Days bitmask |
| MenuId | int | Menu Id |
| StartTime | TimeSpan? | Start time |
JobShiftBreakType Type
| Data Name | Data Type | Data Description |
|---|---|---|
| Id | int | Job-break type relationship Id |
| ShiftBreakTypeId | int | Shift break type Id |
Item Type
Use the full Brink Item contract from the source system. The main parent fields and nested child collections listed in the original task remain in scope. Implementation should flatten the parent row and selected child collections as described below.
Proposed Staging Shape
This schema should be treated as a warehouse foundation layer, similar in spirit to how curated.pos_orders and curated.pos_order_items are modeled as separate but joinable tables.
The preferred pattern is:
- keep core entities in separate tables
- bind them through explicit source keys
- avoid embedding reusable child entities into parent JSON unless the child data is too deep, too volatile, or too low-value to justify relational staging
For Settings2, that means employees, jobs, and the simple child collections should stay relational.
Employees contract group
Parent:
staging.brink_settings2_employees
Child tables:
staging.brink_settings2_employee_jobsstaging.brink_settings2_employee_permissionsstaging.brink_settings2_employee_valid_locations
Binding strategy:
employee_jobs.employee_id -> employees.employee_idemployee_permissions.employee_id -> employees.employee_idemployee_valid_locations.employee_id -> employees.employee_idemployee_jobs.job_id -> jobs.job_id
Reasoning:
Jobs,Permissions, andValidLocationsare not just display blobs inside an employee payload.- They are reusable relationship surfaces that future curated models will likely join directly.
- Keeping them relational makes downstream dimensional modeling much easier.
employee_idis not part of the nativeEmployeeJobchild contract itself; it should be copied from the containingEmployeenode during flattening, just like other Brink staging child tables in this repo carry parent context keys such asorder_idorshift_id.
Jobs contract group
Parent:
staging.brink_settings2_jobs
Child tables:
staging.brink_settings2_job_menusstaging.brink_settings2_job_shift_break_types
Binding strategy:
job_menus.job_id -> jobs.job_idjob_shift_break_types.job_id -> jobs.job_id
Reasoning:
- Jobs are a reference/master-data entity.
- Menus and allowed shift break types are repeating child relationships, not attributes that should be hidden inside one serialized column.
Items contract group
Parent:
staging.brink_settings2_items
Child tables to keep relational:
staging.brink_settings2_item_groupsstaging.brink_settings2_item_modifier_groupsstaging.brink_settings2_item_taxes
Binding strategy:
item_groups.item_id -> items.item_iditem_modifier_groups.item_id -> items.item_iditem_taxes.item_id -> items.item_id
Reasoning:
- These are simple repeatable relationship sets and are likely to be reused by downstream menu/catalog and transaction enrichment models.
- This is directly analogous to the warehouse pattern where
pos_ordersandpos_order_itemsstay separate and joinable rather than storing order items inside the order row.
Complex subobjects that may stay as JSON/SUPER on the parent row unless a later task requires deeper flattening:
brand_allocationscomponentscustom_fieldsdestination_availabilitiesincluded_modifiersingredientsmodifier_codes_pricingmodify_panelsselection_panelsskus
Reasoning for keeping these as JSON/SUPER for now:
- they are deeper structures with more nesting and lower immediate reuse value
- flattening all of them now would create many low-signal child tables before we know the curated access patterns
- they can still be parsed later if a concrete curated model needs them
Binding And Warehouse Modeling Strategy
Why not embed subtypes inside employee/job/item rows?
For a warehouse foundation schema, embedding reusable children into parent JSON makes downstream modeling harder:
- joins become harder and slower
- relationship semantics are hidden inside blobs
- curated tables cannot easily reuse child entities independently
- incremental validation and row-count reconciliation become weaker
- temporal comparison between snapshots becomes harder because changes are hidden inside serialized blobs
That is why the recommended pattern is the same one already visible in the warehouse for transactional data:
- parent surface in one table
- repeated child surface in another table
- explicit binding key between them
curated.pos_orders and curated.pos_order_items are a good precedent:
- they are separated because orders and order items serve different analytical purposes
- they are bound through
order_id - downstream models can join them when needed, but they are not forced into one physical shape
For time-varying reference data, the same principle still applies:
- keep the entity relational
- keep the child relationships relational
- carry snapshot context on every row
- let downstream models reconstruct the state "as observed" for a given integration run or snapshot date
The same logic applies here:
- employees are a master entity
- employee-jobs and employee-permissions are relationship tables
- jobs are a master entity
- items are a master entity
- simple repeating collections remain separate child tables
Recommended keys for binding
Employees:
- primary source key:
employee_id - stable alternate source key:
employee_unique_id
Jobs:
- primary source key:
job_id - stable alternate source key:
job_unique_id
Items:
- primary source key:
item_id
Child relations:
- employee child tables bind through
employee_id, populated from the containing employee record during flattening rather than from the nested child payload itself - employee-job also binds to jobs through
job_id - job child tables bind through
job_id - item child tables bind through
item_id
Raw contract fidelity rule
- Keep every native field that exists on the Brink child contract.
- It is acceptable to add parent binding columns during flattening when the API nests a child collection under a parent but does not repeat the parent key inside each child object.
- These added parent binding columns are part of the staging shape, not a claim that the child contract originally exposed them.
- For
EmployeeJob, preserve native fields such asid,job_id,job_unique_id,pay_rate,security_level_id, andsecurity_level_unique_id, and addemployee_idfrom the containing employee record. - The same rule applies to
EmployeePermissionandEmployeeLocationif their nested payloads do not repeat the employee key.
Recommended rule
- Keep relational child tables for simple arrays of identifiers or small structured rows.
- Use JSON/SUPER only for deep nested structures that are not yet known warehouse join surfaces.
- Do not serialize
employee_jobs,employee_permissions,employee_valid_locations,job_menus,job_shift_break_types,item_groups,item_modifier_groups, oritem_taxesinto parent rows. - Nested JSON is acceptable as a convenience copy only for deep payload preservation, not as the primary analytical or historical model for reusable child relationships.
Temporal Snapshot Strategy
Settings2 should be treated as time-varying reference data captured by observation time, not as a timeless current-state lookup.
Best-practice recommendation
- Keep the staging model relational.
- Preserve one logical snapshot per tenant per integration date/run.
- Put the snapshot context on every parent and child row.
- Build any "latest state" convenience surface later from the historical staging data.
Why this is better than embedding more nested objects
- historical diffs are easier to compute row-by-row
- downstream curated models can reproduce the configuration state that existed when a dated integration ran
- joins stay explicit and stable
- shared reference entities like jobs remain reusable across employees and across snapshots
- child rows can be independently validated and reconciled across snapshots
Practical guidance for this task
- Do not switch simple child collections to embedded JSON just because the settings can change over time.
- Instead, version the relational rows by snapshot context.
- If a later consumer wants one self-contained document per employee or item, that can be produced as a derived curated/export shape rather than being the foundation staging model.
Snapshot columns to add
Because the current Settings2 calls in this repo are not business-date parameterized, the historical meaning is "observed during this integration run".
Add snapshot business columns such as:
snapshot_date: the date the integration is representing- optionally
snapshot_captured_at_utc: when the API response was actually observed, if separate from_etl_ingested_at_utc
Rules:
- the same snapshot column(s) must exist on every parent and child table for this task
- child rows bind to parent rows by both entity key and shared snapshot context
- snapshot context is part of the staging grain for
Settings2
Binding with temporal context
Recommended join shape for downstream use:
employee_jobs.employee_id + snapshot_date -> employees.employee_id + snapshot_dateemployee_permissions.employee_id + snapshot_date -> employees.employee_id + snapshot_dateemployee_valid_locations.employee_id + snapshot_date -> employees.employee_id + snapshot_dateemployee_jobs.job_id + snapshot_date -> jobs.job_id + snapshot_datejob_menus.job_id + snapshot_date -> jobs.job_id + snapshot_datejob_shift_break_types.job_id + snapshot_date -> jobs.job_id + snapshot_dateitem_groups.item_id + snapshot_date -> items.item_id + snapshot_dateitem_modifier_groups.item_id + snapshot_date -> items.item_id + snapshot_dateitem_taxes.item_id + snapshot_date -> items.item_id + snapshot_date
Preferred operational bridge:
- use
staging.brink_labor_shiftsto connect employees to jobs for actual worked activity - treat
employee_jobsas assignment/eligibility history, not as the default actual-work attribution bridge - for labor-based attribution, join:
labor_shifts.employee_id + requested_business_date -> employees.employee_id + snapshot_datelabor_shifts.job_id + requested_business_date -> jobs.job_id + snapshot_date- if downstream curated logic uses
business_dateinstead ofrequested_business_date, document that convention explicitly and keep it consistent across joins
Important constraint
- If Brink does not expose a true "effective as-of date" for these endpoints, then
snapshot_daterepresents warehouse observation/integration context, not a guaranteed source-system effective date. - The spec and code should name/document that clearly to avoid overstating temporal precision.
snapshot_dateis the canonical business snapshot column name for this task.
Table Grain
The implementation should treat each staging table as having an explicit row grain.
staging.brink_settings2_employees
- Grain: one row per employee returned by
Settings2.GetEmployees - Native business key:
employee_id - Alternate source key to retain:
employee_unique_id - No duplicated
Jobs,Permissions, orValidLocationsarrays stored as relational business columns on this row
staging.brink_settings2_employee_jobs
- Grain: one row per
EmployeeJobelement nested under one employee - Native child key from API:
id - Parent-derived binding key:
employee_id - Shared reference key:
job_id - Alternate shared reference key to retain:
job_unique_id - Important:
job_idis not unique at the table level because many employees can reference the same job - Practical row identity is the nested relationship row under a specific employee, not
job_idalone
staging.brink_settings2_employee_permissions
- Grain: one row per
EmployeePermissionelement nested under one employee - Native child key from API:
id - Parent-derived binding key:
employee_id - Shared reference key:
permission_id - Alternate shared reference key to retain:
permission_unique_id
staging.brink_settings2_employee_valid_locations
- Grain: one row per
EmployeeLocationelement nested under one employee - Native child key from API:
id - Parent-derived binding key:
employee_id - Shared reference key:
location_id
staging.brink_settings2_jobs
- Grain: one row per job returned by
Settings2.GetJobs - Native business key:
job_id - Alternate source key to retain:
job_unique_id
staging.brink_settings2_job_menus
- Grain: one row per
JobMenuelement nested under one job - Parent-derived binding key:
job_id - Shared reference key:
menu_id - Include
daysandstart_timebecause they are part of the child row grain rather than attributes of the parent job
staging.brink_settings2_job_shift_break_types
- Grain: one row per
JobShiftBreakTypeelement nested under one job - Native child key from API:
id - Parent-derived binding key:
job_id - Shared reference key:
shift_break_type_id
staging.brink_settings2_items
- Grain: one row per item returned by
Settings2.GetItems - Native business key:
item_id - Keep deep nested structures only where they remain intentionally JSON/SUPER for now
staging.brink_settings2_item_groups
- Grain: one row per item-group relation under one item
- Parent-derived binding key:
item_id - Shared reference key:
item_group_id
staging.brink_settings2_item_modifier_groups
- Grain: one row per modifier-group relation under one item
- Parent-derived binding key:
item_id - Native child key from API:
item_modifier_group_id - Shared reference key:
modifier_group_id
staging.brink_settings2_item_taxes
- Grain: one row per tax relation under one item
- Parent-derived binding key:
item_id - Shared reference key:
tax_id
Grain rule for nested collections
- If Brink returns a nested collection under a parent node, the child staging row grain is "one row per nested element under one parent".
- When the child payload does not repeat the parent key, copy the parent key into the child row during flattening.
- Do not treat shared reference keys like
job_id,permission_id,location_id,menu_id, ortax_idas if they made the child row globally unique. - Preserve the child's native
idwhen Brink provides one, because it is part of the raw relationship payload.
Downstream Join Guidance
Use the following downstream rules when building curated models from these staging tables:
orders -> employeesis a safe direct enrichment path whenemployee_idis present on the order andsnapshot_date = business_dateis the agreed warehouse convention.order_items -> itemsis a safe direct enrichment path whenitem_idis present on the transactional row andsnapshot_date = business_dateis the agreed warehouse convention.labor_shifts -> employeesandlabor_shifts -> jobsare the preferred paths for actual worked employee/job attribution.employees -> employee_jobsis a safe path for assignment history.orders -> jobs through employee_jobsis not a safe default join because an employee can have multiple assigned jobs on the same snapshot date.- if order-to-job attribution is required, prefer a labor-based bridge: order -> employee -> labor shift by tenant + employee + date context + time overlap -> job.
Reuse / Migration Guidance
Existing tables that may be reused after review
staging.brink_settings2_itemsstaging.brink_settings2_item_groupsstaging.brink_settings2_item_modifier_groupsstaging.brink_settings2_item_taxes
These should only be reused if the final row model and metadata columns match the new contract-based implementation.
Existing tables likely requiring upgrade or replacement
staging.brink_settings2_get_employees
Reasons:
- table naming does not match the modern contract naming pattern
- metadata columns are incomplete compared with the current Brink staging standard
- legacy implementation stores child collections differently than the proposed contract group
New tables required
staging.brink_settings2_jobsstaging.brink_settings2_job_menusstaging.brink_settings2_job_shift_break_typesstaging.brink_settings2_employee_jobsstaging.brink_settings2_employee_permissionsstaging.brink_settings2_employee_valid_locations
Idempotency Strategy
Unlike orders and labor shifts, Settings2 data is reference / snapshot data.
Recommended delete strategy:
- delete by
_etl_tenant_idplus snapshot context for each target staging table before load - do not wipe all historical snapshots for the tenant when loading one dated integration run
Parent-table example:
DELETE FROM staging.brink_settings2_employees
WHERE _etl_tenant_id = @tenant_id
AND snapshot_date = CAST(@snapshot_date AS DATE);Child-table example:
DELETE FROM staging.brink_settings2_employee_jobs
WHERE _etl_tenant_id = @tenant_id
AND snapshot_date = CAST(@snapshot_date AS DATE);This should be confirmed during implementation, but snapshot-scoped replacement is the expected model for Settings2 when the integration is run for specific dates.
Detailed Implementation Plan
Module 1: Transport and facade support
- Verify existing
GetEmployeesandGetItemstyped transport remains correct. - Add missing
GetJobstyped support: Settings2Client.GetJobsAsync(...)GetJobsResponseIBrinkPosFacade.GetJobsAsync(...)BrinkPosFacade.GetJobsAsync(...)- Add raw capture support to:
IBrinkRawClientBrinkRawClientBrinkOperationRequestBuilder- Add request builders for:
CaptureEmployeesCaptureJobsCaptureItems- Reuse
BrinkSoapEnvelopeBuilder.BuildSimpleSettings2Call(...)where possible.
Module 2: Extract onboarding
- Extend
BrinkPosEndpointCatalogwith three newSettings2endpoints. - Extend
BrinkPosEndpointExecutorto dispatch: Settings2.GetEmployeesSettings2.GetJobsSettings2.GetItems- Keep the existing raw artifact persistence pattern through
IEtlS3Storage.WriteRawAsync(...). - Ensure extract details capture endpoint-level metadata for each
Settings2endpoint usingBrinkPosOperationCatalog. - Ensure extract details carry the snapshot/integration context that the normalize and load stages will use.
Module 3: Operation metadata
- Add operation metadata entries in
BrinkPosOperationCatalogfor the threeSettings2endpoints. - For each entry define:
stream_name- full operation name
target_contract_group- parser version
- request metadata, including snapshot/integration context
- required contracts
- required staging tables
Recommended stream names:
settings2_employeessettings2_jobssettings2_items
Recommended target contract groups:
brink_settings2_get_employeesbrink_settings2_get_jobsbrink_settings2_get_items
Snapshot metadata rule:
- standardize on
snapshot_dateas the canonical business snapshot column and request metadata key for this task
Module 4: Row models
- Add new shared staging/parquet row models inheriting from
BrinkStagingRowBase. - Include the agreed snapshot business column(s) on every parent and child row model for this task.
Employees:
BrinkSettings2EmployeesRowBrinkSettings2EmployeeJobsRowBrinkSettings2EmployeePermissionsRowBrinkSettings2EmployeeValidLocationsRow
Jobs:
BrinkSettings2JobsRowBrinkSettings2JobMenusRowBrinkSettings2JobShiftBreakTypesRow
Items:
- keep or replace
BrinkSettings2GetItemsRowbased on final naming decision - ensure child row models exist for:
- item groups
- item modifier groups
- item taxes
Important:
- If an existing row model is kept, upgrade it to the current Brink staging conventions rather than layering compatibility code.
- Decimal fields should use the same parquet decimal annotations as the modern Brink staging rows.
- Snapshot context must be part of the row grain, not inferred only from batch-level metadata.
Module 5: Staging profiles
- Add dedicated profile executors using
BrinkMultiArtifactProfileBaseandIBrinkXmlEnvelopeParser.
Recommended profiles:
Settings2GetEmployeesStagingProfileSettings2GetJobsStagingProfileSettings2GetItemsStagingProfile
- Each profile should:
- parse the SOAP payload root
- read response
ResultCodeandMessage - emit one parent artifact
- emit child artifacts for selected collections
- stamp every emitted parent and child row with the agreed snapshot/integration context
- apply metadata using
BrinkStagingMetadataApplicator
Module 6: Normalization catalog
- Extend
BrinkPosNormalizationCatalogwith newSettings2contract groups. - Register every emitted artifact contract explicitly.
- Ensure the contract list matches the staging profile output exactly.
Module 7: DI registration
- Register the new
Settings2staging profiles inAddBrinkPosEtlIntegration(). - Keep the
GetShiftspattern as the registration model.
Module 8: Staging table catalog
- Extend
BrinkPosStagingTableCatalogwith table metadata for each newSettings2contract. - Add idempotent delete SQL for parent and child tables.
- Use snapshot-scoped delete predicates (
_etl_tenant_id+ snapshot context), not tenant-wide delete predicates.
Module 9: SQL bootstrap
- Review existing
Settings2SQL scripts for compatibility. - Add new SQL scripts for new tables.
- Alter or replace existing scripts where the schema does not match the modern row model.
Required SQL outcomes:
- all business columns first
- include the agreed snapshot business column(s) such as
snapshot_date - then the full 14-column
_etl_*metadata set listed inRequired Rules _etl_ingested_at_utc TIMESTAMP NOT NULL_etl_row_ordinal BIGINT NOT NULLfor modern contract-based staging artifacts- do not create any new table with the older reduced metadata shape used by some legacy
Settings2scripts
Module 10: Testing
- Update
BrinkPosEndpointCatalogTeststo assertSettings2endpoints are present. - Update
BrinkPosEtlCatalogTeststo cover the new contract groups and staging tables. - Add transport/request builder tests for:
Settings2.GetEmployeesSettings2.GetJobsSettings2.GetItems- Add staging profile tests for parent and child artifact emission.
- Add staging profile tests to verify snapshot context is populated on every emitted parent and child row.
- Add live facade tests for
GetJobsif the facade surface is added. - Add raw capture live tests if XML artifacts in
temp/are part of the expected verification workflow. - Add normalize/load tests for the new
Settings2contract groups and snapshot-scoped delete predicates. - Add downstream-spec tests or validation notes that mark
orders -> jobs through employee_jobsas unsafe andlabor_shifts -> jobsas the preferred actual-work bridge.
Recommended Delivery Order
- Add
GetJobstransport + facade support. - Add raw capture support for
Settings2endpoints. - Add endpoint catalog + endpoint executor support.
- Add operation metadata + normalization contracts + staging table catalog entries.
- Implement
Employeesstaging profile and tables. - Implement
Jobsstaging profile and tables. - Review and modernize
Itemsstaging profile / tables, reusing existing SQL only where compatible. - Add tests.
- Run full backend build and relevant test suites.
Open Decisions To Confirm During Implementation
- Whether to keep legacy table names like
brink_settings2_get_employeesor replace them with entity-based names likebrink_settings2_employees. - Whether
Itemsshould keep selected deep subobjects as JSON/SUPER in this task or flatten more child collections now. - Whether any downstream curated flow already depends on the old
Settings2table names.
Default recommendation:
- use modern entity-based contract/table names for newly onboarded artifacts
- keep
Itemspartially relational and partially JSON/SUPER for deep subobjects - only preserve old names if a concrete downstream dependency is discovered
- standardize on
snapshot_dateacross all parent and child tables before implementation starts
Code References
backend/Kds.Ai.Integrations/BrinkPos/BrinkPosEndpointCatalog.csbackend/Kds.Ai.Integrations/BrinkPos/BrinkPosEndpointExecutor.csbackend/Kds.Ai.Integrations/BrinkPos/BrinkPosOperationCatalog.csbackend/Kds.Ai.Integrations/BrinkPos/BrinkPosNormalizationCatalog.csbackend/Kds.Ai.Integrations/BrinkPos/BrinkPosNormalizeLoadJobs.csbackend/Kds.Ai.Integrations/BrinkPos/BrinkPosLoadToStagingJobs.csbackend/Kds.Ai.Integrations/BrinkPos/BrinkPosStagingTableCatalog.csbackend/Kds.Ai.Integrations/BrinkPos/BrinkPosNormalizationContextFactory.csbackend/integrations/Kds.Ai.Integrations.BrinkPos/Services/Normalization/Staging/Profiles/LaborGetShiftsStagingProfile.csbackend/integrations/Kds.Ai.Integrations.BrinkPos/Services/Facade/Clients/Settings2Client.csbackend/integrations/Kds.Ai.Integrations.BrinkPos/Services/Facade/BrinkRawClient.csbackend/integrations/Kds.Ai.Integrations.BrinkPos/Abstractions/IBrinkRawClient.csbackend/Kds.Ai.Warehouse/Sql/Bootstrap/Tenant/Staging/BrinkPos/0611_create_staging_brink_settings2_get_employees.sqlbackend/Kds.Ai.Warehouse/Sql/Bootstrap/Tenant/Staging/BrinkPos/0612_create_staging_brink_settings2_get_items.sqlspecs/BrinkLaborGetShiftsStaging/TASK.md