Skip to main content

Architecture

MDMongoose is a set of Databricks resources around one Delta table. There is no extra database, cache or server: the UI runs as a Databricks App, the work runs on serverless compute, and Unity Catalog governs every table and grant.

In your lakehouse​

MDMongoose sits between the silver and gold layers of a medallion architecture, as a separate component next to your data pipelines:

MDMongoose between the silver and gold layers

FlowFrom → toHow
ImportSilver → MDMongooseCleaned source tables with change data feed (mdm_staging.src_*) are loaded incrementally by the mdm_load pipeline and job. → Ingestion
Reference dataMDMongoose → SilverReference data maintained in MDMongoose (countries, product groups, mappings) is published as Export nodes, so silver pipelines can join and standardise with it. → Export
Master dataMDMongoose → GoldGolden records and hierarchies are published as materialized views or views in mdm_export, ready for BI, AI and apps. → Export
Manual entryBusiness users & stewards → MDMongooseUsers edit nodes, hierarchies and golden records in the app, one node at a time or in the Excel-like Edit Children grid. → User guide
Excel / CSVMDMongoose → Business users & stewardsAny branch can be downloaded from the app as CSV or Excel; Export nodes also write files to a volume. → Exporting to CSV and Excel

Bronze stays untouched: MDMongoose only reads data that your pipelines have already cleaned into silver.

Inside MDMongoose​

The Databricks resources that make up MDMongoose and how they work together:

MDMongoose architecture

Components​

ComponentDatabricks resourceRole
UI + APIDatabricks App mdm (React + FastAPI)Hierarchy, properties, dialogs; enforces types, structure rules and permissions. Every save is one MERGE on the SQL warehouse.
SQL warehouseServerless (the starter warehouse on Free Edition)Serves every read and save of the app.
MDMNodesDelta table <catalog>.mdm_core.mdmnodesThe whole hierarchy: nodes, definitions, records, golden records. Change data feed and row tracking on.
StagingSchema <catalog>.mdm_stagingYour source tables (src_*, change data feed on) and the pipeline's rec_* tables.
mdm_load pipelineSpark Declarative Pipeline, serverlessOne dataset per Table node: reads the staging change feed, applies expectations, AUTO CDC into rec_<table>.
mdm_load jobServerless jobRefreshes the selected datasets, then merges the new changes into MDMNodes; optionally refreshes golden records.
mdm_action jobServerless jobGolden-record refreshes and custom background actions; then runs mdm_export.
mdm_export jobServerless jobWrites the CSV / Excel files of the Export nodes, then updates the export pipeline.
mdm_refresh jobServerless job, hourly, paused by defaultLoads every table whose Schedule is due.
mdm_init jobServerless jobCreates or upgrades the table, seeds definitions and demo data, grants the app access.
mdm_export pipelineSpark Declarative Pipeline, serverlessSystem views, history and one materialized view per Export node in <catalog>.mdm_export.
exports volumeUC volume in mdm_coreCSV / Excel files of the Export nodes.

The data model​

Every node is one row of MDMNodes:

  • UID — MDM- plus a base-36 number. Numbers up to 100 000 are reserved for system nodes; new UIDs are allocated in blocks, so many writers can work at once.
  • Structure — parent_uid, path (the ancestor UIDs) and path_key (the path as text, e.g. /1/12W/1KE/). The table is clustered by path_key, so a branch is a contiguous range of files: children, subtrees, moves and deletes read only their branch.
  • Node fields as columns — name, types, origin, icon, max child depth, allowed child types, owner, steward, locked, is_deleted, modified_at / modified_by.
  • All other properties in one properties VARIANT column, and a user_defined map that marks values people set by hand.

Definitions are nodes too: a NodeType node holds its property definitions and actions; an Action node holds its Python handler and its Required Level; an Enum node's children are the values of the enum. The app reads them once and merges the definitions of a node's types.

How a save works​

  1. The UI sends the change to the API.
  2. The API checks the rules: types, Allowed Child Type, Max Child Depth, the user's permission level, locks.
  3. One MERGE writes all changed rows — a whole Edit Children batch is one transaction.
  4. If someone else changed the node since you opened it, the save is refused (HTTP 409) and you reload.

How a load works​

Incremental load flow

  1. Your ingestion writes to mdm_staging.src_<name> (change data feed on).
  2. Load Childs (or the schedule) starts the mdm_load job, which refreshes only that table's dataset in the pipeline: the change feed is cleaned, checked against the table's expectations and applied to rec_<name> with AUTO CDC.
  3. The job reads rec_<name>'s own change feed from the version it merged last and writes only those records with one MERGE into the Table node's branch — keeping every UserDefined value.
  4. If the table has Run Golden After Load, the golden records that use it are refreshed; then the mdm_export job refreshes the Export nodes.

More on each stage: Ingestion, Data quality, Export.

Performance​

  • Paging: the UI reads 40 rows at a time; sort and filters run on the server.
  • Caching: the browser keeps visited levels and preloads the properties of the rows on screen; the app caches reads and notices other writers through the table's Delta version.
  • Set-based operations: deleting or moving a subtree is one statement, not one per node.
  • Lookup index: mdm_export.node_index (clustered by UID) finds any node by UID or name quickly.