Skip to main content

Monitoring

Everything MDMongoose runs is a standard Databricks job or pipeline, so you monitor it with the tools you already have: Jobs & Pipelines, the pipeline event log, system tables and SQL on the data.

From the app​

  • Background actions (Load Childs, Refresh Golden Records) open a run dialog that follows the job until it finishes and links to the run in Databricks.
  • The Refresh Golden Records dry run shows what a refresh would change, including conflicts and near misses, before you run it.
  • Configure Load shows whether the source still has its change data feed on.

Jobs​

Open Jobs & Pipelines and check the runs of:

JobWatch for
mdm_loadOne run per load. Its output lists, per table, the mode (incremental / full), the records inserted, updated and deleted and the source version merged.
mdm_actionGolden-record refreshes and custom actions. The output summarises records, golden records, created, changed, retired and conflicts.
mdm_exportExport files and the export pipeline (materialized views, history). The first task lists every Export node with its result; it fails if one export failed.
mdm_refreshThe scheduled loads (paused by default).
mdm_initDeployments and upgrades.

Set job notifications (on failure, on duration over a threshold) in each job's settings to get e-mails or webhook alerts.

Data quality​

Expectations (drop row)​

Every Table node can carry Expectations — SQL conditions on the source rows (Configure Load). The mdm_load pipeline applies them as expectations with the DROP action (expect_all_or_drop): a row that fails any rule is dropped before it reaches rec_<table>, so it never creates or changes a record; the rest of the load continues. Every update records, per rule, how many rows passed and failed. How the rules work and how to write them: Architecture › Data quality.

In the UI: Jobs & Pipelines → mdm_load, pick an update, click the table's dataset → Data quality tab (passed and failed rows per rule).

In SQL — the pipeline event log (the pipeline ID is on the pipeline's page):

-- dropped rows per update
SELECT timestamp,
details:flow_progress.metrics.num_output_rows AS rows_written,
details:flow_progress.data_quality.dropped_records AS rows_dropped,
details:flow_progress.data_quality.expectations AS expectations
FROM event_log('<mdm_load pipeline id>')
WHERE event_type = 'flow_progress' AND details:flow_progress.status = 'COMPLETED'
AND details:flow_progress.data_quality IS NOT NULL
ORDER BY timestamp DESC;

-- failures per rule and day
SELECT date_trunc('day', timestamp) AS day, e.dataset, e.name AS rule,
sum(e.passed_records) AS passed, sum(e.failed_records) AS failed
FROM event_log('<mdm_load pipeline id>'),
LATERAL explode(from_json(details:flow_progress.data_quality.expectations::string,
'array<struct<name string, dataset string, passed_records bigint, failed_records bigint>>')) AS t(e)
WHERE event_type = 'flow_progress'
GROUP BY ALL ORDER BY day DESC, failed DESC;

Which rows were dropped: they stay in the staging table — compare it with the cleaned table:

SELECT s.* FROM workspace.mdm_staging.src_sap_kna1 s
LEFT ANTI JOIN workspace.mdm_staging.rec_kna1_2021 r ON r.KUNNR = s.KUNNR;

Alerts: save the failures per rule query in the SQL editor and create a SQL alert on it (e.g. failed > 0 for today), or add a notification to the mdm_load job.

After the load​

  • Golden records — the Refresh Golden Records dry run lists conflicts (values that do not fit the golden property's PropertyType) and near misses; low Match Score values point at records to review.
  • Edits — the app validates every value by its PropertyType; invalid cells block saving.

Exports​

mdm_export runs after loads and golden refreshes. Each Export node shows its Last Run (time, rows or columns written, or the error) and its Location. For the materialized views, check whether a refresh was incremental:

SELECT timestamp, message
FROM event_log(TABLE(workspace.mdm_export.customer_golden_records))
WHERE event_type = 'planning_information'
ORDER BY timestamp DESC;

Checking the data​

A few useful queries on the hierarchy (replace the catalog if yours is different):

-- size of each top-level branch and how much of it is loaded
SELECT path[1] AS branch, origin, count(*) AS nodes
FROM workspace.mdm_core.mdmnodes
WHERE NOT is_deleted
GROUP BY ALL ORDER BY ALL;

-- values people overrode (UserDefined) per node type
SELECT t AS type, count(*) AS nodes_with_overrides
FROM workspace.mdm_core.mdmnodes LATERAL VIEW explode(types) AS t
WHERE NOT is_deleted AND size(map_filter(user_defined, (k, v) -> v)) > 0
GROUP BY t ORDER BY nodes_with_overrides DESC;

-- who changed what today
SELECT modified_by, count(*) FROM workspace.mdm_core.mdmnodes
WHERE modified_at >= current_date() GROUP BY modified_by;

-- source records not matched to any golden record
SELECT r.uid, r.name
FROM workspace.mdm_core.mdmnodes r
WHERE NOT r.is_deleted AND r.origin = 'loaded'
AND NOT EXISTS (SELECT 1 FROM workspace.mdm_core.mdmnodes g
WHERE NOT g.is_deleted AND exists(g.child_links, l -> l.uid = r.uid));

For the change history of single nodes use mdm_export.node_history; for the table as a whole, DESCRIBE HISTORY workspace.mdm_core.mdmnodes.

The app​

  • Compute → Apps → mdm shows the app's state, its deployments and its logs (requests and errors).
  • GET /api/health returns {"ok": true} when the app is up — use it for an uptime check.
  • Slow screens usually mean a cold SQL warehouse: check the warehouse's state and query history (SQL Warehouses → Query history).

Costs​

Every component is serverless. Use the billing system tables (system.billing.usage) filtered by the job and pipeline names mdm_* and the app name to see what MDMongoose costs per day.