Ingestion
How source data gets from your systems into the hierarchy. The short version: you land the data in a Delta
table with change data feed; MDMongoose takes it from there, and only changed records are ever written to
MDMNodes.
The four stages
| Stage | Where | What happens |
|---|---|---|
| 1. Landing | mdm_staging.src_<name> — your table | Your ingestion (Lakeflow Connect, a job, a notebook, a streaming table) writes the source data. Change data feed must be on. Inserts, updates, deletes and MERGEs are all fine. |
| 2. Cleaning | mdm_load pipeline → mdm_staging.rec_<table> | Per Table node, a streaming view reads the staging table's change feed, trims values, turns empty strings into NULL and applies the table's expectations. AUTO CDC (SCD 1) applies the surviving changes to rec_<table>, the cleaned current state of the source. |
| 3. Merging | mdm_load job → MDMNodes | The job reads the change feed of rec_<table> since the version it merged last, maps columns to properties, and writes one MERGE into the Table node's branch. |
| 4. Follow-up | mdm_load job | Golden records that use the table are refreshed (Run Golden After Load), then the mdm_export job runs. |
The pipeline is metadata-driven: at every update it reads the Table nodes from MDMNodes and declares one
changes_<table> view and one rec_<table> streaming table for each. A new Table node in the app becomes new
datasets at the next update; nothing is deployed.
Load Childs refreshes only that table's dataset (refresh_selection), so one load never reprocesses the
other sources.
Incremental and full reload
Every Table node has a Load mode (Configure Load). Both modes end in the same set-based MERGE, and in
both only records that really changed are written.
Incremental (change data feed)
- The pipeline update for
rec_<table>is incremental: the streaming read continues the staging table's change feed from where its checkpoint stopped. - The merge job reads
rec_<table>withreadChangeFeedfrom the version after the one it merged last (kept on the Table node as_cdf_version) up to the current version — pinned, so the plan is repeatable on serverless. - Per source key only the latest change counts: an insert or update becomes an upsert, a delete a delete.
- The upserts are joined to the Table's existing records by source key; a record whose
record_hash(a hash of its mapped values) is unchanged is skipped.
Cost: proportional to the number of changes, not to the size of the source.
Full reload
A full reload does not trust the change history. It rebuilds and compares:
- The pipeline runs a full refresh of
rec_<table>(full_refresh_selection): its checkpoint is reset and the staging table is read again from the start, sorec_<table>becomes an exact copy of the current, cleaned source. - The merge job reads all of
rec_<table>(at a pinned version) and all existing records of the Table node, and joins them with a full outer join on the source key:
In rec_<table> | In MDMNodes | Result |
|---|---|---|
| yes | no | insert — a new record with a new UID |
| yes | yes, same record_hash | nothing — unchanged |
| yes | yes, different hash (or deleted before) | update — UserDefined values are kept |
| no | yes | delete — soft delete, hard delete or keep, by the table's When a source row is deleted |
So yes — a full reload computes the delta itself: it compares the complete source with what is in the
hierarchy and writes only the differences. The cost is a full read of the source and of the Table's branch
(which, thanks to clustering by path_key, reads only that branch's files).
The first load of a table is always full, whatever the mode; after that the chosen mode applies.
Which one to choose
| Situation | Mode |
|---|---|
| The staging table is updated with INSERT / UPDATE / MERGE / DELETE and keeps its change feed | Incremental |
The staging table is replaced each time (CREATE OR REPLACE, overwrite, truncate + insert) | Full reload — a replace breaks the change history |
| The change feed was off for a while, or old versions were vacuumed away | Full reload once, then incremental |
| You changed the column mapping, keys or name expression | Full reload once — every record is re-mapped |
| A small reference table (countries, currencies) | Either — full is simple and cheap |
What the merge keeps
- UIDs — a source key always maps to the same record and UID.
- UserDefined values — a value someone edited in the app is never overwritten; the merge keeps the old value for every UserDefined key.
- Locks — locked records are not deleted.
- History — every change is a new version of
MDMNodes(Delta history) and a row inmdm_export.node_history.
Triggers
| Trigger | Runs |
|---|---|
| Load Childs in the app | mdm_load job for that table |
| mdm_refresh job (hourly, paused by default) | every table whose Schedule (cron, UTC) is due |
| Your own job | mdm_load with table_uid = a Table node's UID, or all |
| mdm_init | every table, once, at deployment |
See Importing data for the step-by-step setup.