Skip to main content

Ingestion

How source data gets from your systems into the hierarchy. The short version: you land the data in a Delta table with change data feed; MDMongoose takes it from there, and only changed records are ever written to MDMNodes.

Incremental load flow

The four stages​

StageWhereWhat happens
1. Landingmdm_staging.src_<name> — your tableYour ingestion (Lakeflow Connect, a job, a notebook, a streaming table) writes the source data. Change data feed must be on. Inserts, updates, deletes and MERGEs are all fine.
2. Cleaningmdm_load pipeline → mdm_staging.rec_<table>Per Table node, a streaming view reads the staging table's change feed, trims values, turns empty strings into NULL and applies the table's expectations. AUTO CDC (SCD 1) applies the surviving changes to rec_<table>, the cleaned current state of the source.
3. Mergingmdm_load job → MDMNodesThe job reads the change feed of rec_<table> since the version it merged last, maps columns to properties, and writes one MERGE into the Table node's branch.
4. Follow-upmdm_load jobGolden records that use the table are refreshed (Run Golden After Load), then the mdm_export job runs.

The pipeline is metadata-driven: at every update it reads the Table nodes from MDMNodes and declares one changes_<table> view and one rec_<table> streaming table for each. A new Table node in the app becomes new datasets at the next update; nothing is deployed.

Load Childs refreshes only that table's dataset (refresh_selection), so one load never reprocesses the other sources.

Incremental and full reload​

Every Table node has a Load mode (Configure Load). Both modes end in the same set-based MERGE, and in both only records that really changed are written.

Incremental (change data feed)​

  1. The pipeline update for rec_<table> is incremental: the streaming read continues the staging table's change feed from where its checkpoint stopped.
  2. The merge job reads rec_<table> with readChangeFeed from the version after the one it merged last (kept on the Table node as _cdf_version) up to the current version — pinned, so the plan is repeatable on serverless.
  3. Per source key only the latest change counts: an insert or update becomes an upsert, a delete a delete.
  4. The upserts are joined to the Table's existing records by source key; a record whose record_hash (a hash of its mapped values) is unchanged is skipped.

Cost: proportional to the number of changes, not to the size of the source.

Full reload​

A full reload does not trust the change history. It rebuilds and compares:

  1. The pipeline runs a full refresh of rec_<table> (full_refresh_selection): its checkpoint is reset and the staging table is read again from the start, so rec_<table> becomes an exact copy of the current, cleaned source.
  2. The merge job reads all of rec_<table> (at a pinned version) and all existing records of the Table node, and joins them with a full outer join on the source key:
In rec_<table>In MDMNodesResult
yesnoinsert — a new record with a new UID
yesyes, same record_hashnothing — unchanged
yesyes, different hash (or deleted before)update — UserDefined values are kept
noyesdelete — soft delete, hard delete or keep, by the table's When a source row is deleted

So yes — a full reload computes the delta itself: it compares the complete source with what is in the hierarchy and writes only the differences. The cost is a full read of the source and of the Table's branch (which, thanks to clustering by path_key, reads only that branch's files).

The first load of a table is always full, whatever the mode; after that the chosen mode applies.

Which one to choose​

SituationMode
The staging table is updated with INSERT / UPDATE / MERGE / DELETE and keeps its change feedIncremental
The staging table is replaced each time (CREATE OR REPLACE, overwrite, truncate + insert)Full reload — a replace breaks the change history
The change feed was off for a while, or old versions were vacuumed awayFull reload once, then incremental
You changed the column mapping, keys or name expressionFull reload once — every record is re-mapped
A small reference table (countries, currencies)Either — full is simple and cheap

What the merge keeps​

  • UIDs — a source key always maps to the same record and UID.
  • UserDefined values — a value someone edited in the app is never overwritten; the merge keeps the old value for every UserDefined key.
  • Locks — locked records are not deleted.
  • History — every change is a new version of MDMNodes (Delta history) and a row in mdm_export.node_history.

Triggers​

TriggerRuns
Load Childs in the appmdm_load job for that table
mdm_refresh job (hourly, paused by default)every table whose Schedule (cron, UTC) is due
Your own jobmdm_load with table_uid = a Table node's UID, or all
mdm_initevery table, once, at deployment

See Importing data for the step-by-step setup.