Metadata framework guide

The metadata framework builds and runs your medallion layers (Bronze, Silver, Gold, or any 2 to 10 layers you choose) from configuration instead of one-off notebooks. You describe each table once in a small file: where its data comes from, its columns, how to clean and check it, and how to load it. Fabric Desktop by KratosBI runs every table the same way, checks it, writes it to its layer's lakehouse, and logs what happened.

This guide goes from an empty folder to Prod. The reference at the end lists every setting.

Tested and not yet tested. Everything that runs on your PC, including Dev on the PC, the four gates and every load mode, is covered by automated tests on every build. Features that talk to Fabric (promote to Test and Prod, runs in Fabric, Optimize, the semantic model, history and restore, the variable library) are built to Fabric's documented APIs but haven't yet been tried on a live tenant. Several are tagged Preview in the app.

Ideas in one page#

1. Set up a project#

1. Open Data mesh in the left rail. With no project yet, it shows Set up a metadata project.
2. Folder: where the project goes. The default is Documents\FabricDesktop\metadata. Choose a folder inside a Git repository if you have one.
3. Layers: pick 2 (raw, curated), 3 (bronze, silver, gold) or 5 (landing, bronze, silver, gold, semantic). You can add, remove, rename and reorder later.
4. Add the sample tables: with 3 layers, tick this to get customers and orders from Bronze to Gold with sample CSV files, so you can try everything straight away.
5. Create project.

To open a project someone else made (for example, after git clone), use Or open an existing project and give the folder that has layers.toml.

A new project looks like this:

layers.toml                     your layers, in order, and each one's rules
environments.toml               Dev, Test and Prod: workspace, where it runs
exceptions.toml                 approved exceptions to a rule
metadata/entities/*.toml        one file per table
metadata/keys/*.toml            one file per business key, and source_systems.toml
transforms/                     your Python: hooks, custom checks, custom steps
tests/                          tests for that Python
.local/                         Dev's tables on this PC (in .gitignore)

2. Run Dev on your PC#

A new project's Dev keeps its tables in the .local folder, so you can build and run everything with no Fabric workspace at all.

1. Pick Dev in the top bar's Environment picker.
2. Run all (top right) runs every entity in layer order. A dialog shows each table's result: rows written, rows quarantined, or why it failed.
3. Details on a result, or any row on the Runs tab, shows each step with rows and time, and each check with how many rows failed.

With the sample project, the first Run all loads customers and orders through Silver into a Type 2 customer dimension and a sales fact, quarantines a returned order whose amount isn't negative, and warns about an order whose customer doesn't exist yet.

When you're ready to use Fabric for Dev, assign it a workspace (step 7) and switch Tables in to Workspace.

3. Design your layers#

Data mesh > Design > Layers lists your layers in order, with each one's rules. Change them, then Save layers.

Add layer adds one at the end (up to 10). Reordering is refused if a table would end up reading a later layer.

Rules that always hold: a table writes only to its own layer's lakehouse; it reads from earlier layers, or from its own layer when that makes no loop (a fact reading its dimensions); reading a later layer is always rejected.

4. Add entities#

New entity (on any Data mesh page, or Ctrl+K) asks for:

The entity page opens. From there:

• What happens on every run shows the steps for this table. Grey steps are the framework; purple is your code, at a fixed point.
• Config is the TOML file. Edit and Save (or Ctrl+S). Saving runs the save gate: problems show above the editor and the table can't run or be promoted until they're fixed.
• Contract lists the output columns and marks keys, surrogate keys, derived columns and columns your hooks add.
• Lineage shows what it Reads, the table itself, and what it Feeds. Click a table to open it.
• Recent runs shows its last runs in every environment.
• Data in shows a sample of the table as written.
• Run in Dev (or the picked environment) saves and runs just this table. Rebuild with downstream reprocesses it in full and then every table that reads it, in order.
• Delete removes the file. The Delta table stays in the lakehouse until you delete it there.

Where custom logic goes#

Simplest first. The framework still runs every other step, checks the result and writes it.

1. An expression: [[derive]] adds a column from a SQL expression; [[check]] with expr is a row-level rule.
2. A hook: [hooks] after_clean or before_write names a function (df, ctx) that returns rows. Declare the tables it reads and the columns it adds.
3. A custom check: [[check]] python = "transforms.checks:valid_email" returns true for each good row.
4. A custom step: [step] custom = "transforms.sales:build" replaces this table's transform.

Every hook, custom check and custom step needs an owner and a test, or the save gate stops it. Shared helpers go in transforms/common.py.

5. Load modes and decisions#

Facts use [[lookup]] to take a dimension's surrogate key as it was on the fact's date. A fact row whose dimension member doesn't exist gets -1.

Large tables#

An incremental run is built to stay small when the table is big:

• It reads only what it needs. The watermark filter goes into the read itself, so files whose rows are all older are skipped without being opened. A table that reads another entity reads only the columns it uses (plus any new ones, so drift is still noticed). The run's read step says so, for example incremental, modified after 2026-10-01; read 3 columns of 4.
• It rewrites only what changed. New and changed rows go into the table with a Delta MERGE on the key: rows with a key already in the table are replaced, the rest are added, and Parquet files without any of those keys stay as they are. A table without keys gets a plain append. The write step says what happened, for example merged 2000 rows on id: 1000 updated, 1000 new (Delta MERGE rewrote 1 file, added 1).
• It falls back to a rewrite when a MERGE would change the result. If the new rows repeat a key (there's no unique check), or a column's type changed, the run reads the whole table and replaces it, as every run did before 0.20, and the write step says why.

Replayed quarantine rows go in the same way.

Dimensions (scd1, scd2) read only the rows of the business keys that arrived, plus the Unknown member, merge them exactly as before (same keys, versions and Type 2 checks) and write them back with one MERGE on the surrogate key. A Type 2 dimension fed a full snapshot of its source first drops the rows that would change nothing (same tracked values, no new Type 1 value), so only changed and new keys are touched; when none changed, nothing is written at all. A Type 1 dimension takes every row that arrives, so a full snapshot rewrites it, and the write step says most of it changed. The write step says what it read, for example 1000 new, 1000 changed (new versions), 0 Type 1 updates, 4999000 rows unchanged; read 1001 rows of the table for 2000 business keys.

Full loads and rebuilds replace the whole table. A source too big to read at once can be read in batches (Preview): set batch_by under [load] to a column the source has, usually a date such as order_date. A full load, an incremental entity's first load, or a full rebuild then reads the source one batch of that column's values at a time, sorted, with as many rows in a batch as fit in free memory with the copies the later steps make. Each batch goes through every step (contract, keys, cleaning, derived columns, hooks, lookups, checks) on its own, and the batches are streamed into the table, which is replaced in one Delta commit at the end: readers see the old table until the whole load has worked, and a load that stops (a check, an error, a closed app) leaves the table as it was. The run shows a plan batches step, for example 8 batches by order_date, the largest 1,900,000 rows, and one step per step name with the batches added up.

• Batching reads the source column once to count its values, then each batch's rows. When everything fits in one batch, the load is one ordinary read. Measured on a 15 GB PC with a 60 million row table (16 columns, about 13 GB in memory read at once, too big for one read): 6 batches, 52 seconds, peaking at 7.2 GB. The script is examples/benchmark/batched_loads.py.
• A batch never splits one value. When a single value holds more rows than fit, the memory guard stops the run and names that value: batch by a column with more values.
• unique checks also compare each batch with the values earlier batches wrote. A value repeated across batches fails the check in the later batch; in one big read every copy fails. With on_fail = "stop" both stop.
• Hooks see one batch at a time. A hook that needs the whole table (a ranking, a total) gives different results in batches: leave batch_by off for that entity.
• Only Delta sources (a table, a link or another entity) are batched; files are read as before. Dimensions (scd1, scd2) are merged by key and don't use it. Changing batch_by isn't a logic change.
• The table isn't partitioned by the column: batching is how the run reads, not how the table is stored. Parquet files a stopped load had already written stay in the table's folder: they aren't part of any table version, so nothing reads them, but they take space, and the app's Vacuum doesn't remove them yet.

Before it reads a Delta source (a table, a link or another entity), a run estimates the read from the table's Delta log alone: the files left after the watermark, their rows, their size on disk, and the memory the columns it reads take (8 bytes a number or timestamp, 4 a date, about 32 a text value). The read step shows it, for example about 2,000 rows in 1 of 20 files (0.4 MB on disk, 0.1 MB in memory). If three times that estimate (the read plus the copies the later steps make; a 5-million-row first load peaked at 2.5 times) is more than the machine's free memory, the run stops before reading anything and says what to do instead: load it incrementally, read fewer columns, or run the environment in Fabric. To try anyway, set memory_guard = false under [load]; that isn't a logic change, so it doesn't ask whether to rebuild. Free memory is read on Windows and Linux; elsewhere there's no guard. File sources (files, landing) aren't estimated yet.

Measured on one Linux machine (4 cores, 16 GB), local Delta tables, one incremental run with 1,000 changed and 1,000 new rows, using examples/benchmark/large_tables.py:

Both ways end with the same rows. When the changed rows are spread over every file, a MERGE has to rewrite every file too, and the gain is mostly memory. Most of a dimension's remaining time and memory is reading its 5-million-row source snapshot, which both ways do. These are local Delta tables; OneLake adds network time to both.

Loads from files read only new files#

An incremental or dimension load from files (files or landing) remembers each file it loaded: its path, size and modified time, in LH_Meta's meta.loaded_files. Each run then reads only files that are new or have changed since, instead of every file in the folder. The read step says so, for example 1 new file of 30.

• When every file is loaded already, the run succeeds without writing anything: no new files.
• A full backfill reads every file again and records them all.
• Full loads always read every file.
• To read every file on each run anyway, set new_files_only = false under [source]. This isn't a logic change, so it doesn't ask whether to rebuild.

The list is written after the table. A run that fails before its write reads the same files next time. If the table is written but the list can't be, the run says so and the next run reads those files again, which a load with keys absorbs.

When a table's logic changes, an incremental run stops and asks you to choose:

• Apply to new rows only: rows already loaded keep the old logic. _run_id shows which rows used which version.
• Rebuild it and everything downstream (recommended): reprocesses the full source, then every table that reads it, in order.
• Migrate to Type 2 (when a dimension changes from Type 1 to Type 2): keeps every surrogate key and starts each row's history at 1900-01-01.

The table shows Needs a decision on Home and in Runs until you choose.

6. Quality: checks, quarantine and the four gates#

Checks run before every write. Each has an on_fail:

The four gates:

1. Save: every entity is checked against its layer's standards. Columns must exist upstream, so renames and deletes show what breaks. Custom code needs an owner and a test.
2. Write: checks run before every commit. Only the framework writes a managed table; notebooks get an error and can use scratch_ tables.
3. Promote: Dev to Test and Test to Prod need a clean save gate, passing tests, and a green last run of every table's current logic.
4. After: meta.run_log, meta.dq_results, meta.watermarks, meta.schema_history and meta.deployments in LH_Meta, plus _run_id and _loaded_at on every row and the run id and metadata version on every Delta commit.

The table commit comes first and everything in step 4 follows it, together with the quarantined rows. Only the table commit is ever retried, so a failure after it can't put the rows in twice: the run reports the rows as written and says what it couldn't record. Each commit also carries its watermark, so if saving meta.watermarks failed, the next run goes by the commit and doesn't load the same rows again. A run whose table write fails quarantines nothing.

Exceptions. When a rule genuinely doesn't fit, add an approved exception to exceptions.toml with the entity, the rule, a reason, an approver and an expiry date (or permanent). An expired exception fails the save gate again.

[[exception]]
entity = "gold.fact_sales"
rule = "custom_code_test"
reason = "Signed off by finance; test due next sprint"
approver = "Chris"
expires = "2026-11-01"

Tools on the entity page#

Freshness promises#

Freshness says when a table should have had a successful run, in the environments you pick (Test and Prod when left out):

[freshness]
within_hours = 26     # a successful run in the last 26 hours
by = "07:00"          # and one by 7:00 each day, this PC's clock
envs = ["prod"]

Late tables show on Home and in the bell. To get them in Teams, Settings > Notifications > Teams alerts (or Set up Teams alerts on the card) takes the address of a Teams Workflows webhook ("Post to a channel when a webhook request is received"). The app checks at startup and every 10 minutes while it's open. Late and failed tables become incidents (Preview), shared with everyone who can read the environment, and each new incident is posted once, and again when it recovers. An environment that a running agent already checks is left to the agent, so Teams hears about a problem once. With preview features hidden, each late or failed table is posted once, as before.

Reliability objectives (Preview)#

An objective says how often a table's runs must succeed:

[objectives]
criticality = "high"   # critical, high, standard or low
success_rate = 95      # percent of the last `runs` runs; from criticality when left out
runs = 20              # how many recent runs count (at least 5)
envs = ["prod"]        # Test and Prod when left out

Criticality sets the rate when success_rate is left out: critical 99 %, high 95 %, standard 90 %, low none. Skipped runs and runs still going don't count, and fewer than five finished runs say nothing. A table under its objective opens an "Objective missed" incident wherever incidents are evaluated (the agent, for now). A completion time is the freshness promise's by.

Tests (Preview)#

Data mesh > Quality > Tests runs the test_*.py files in the project's tests folder. A test is a function named test_something that uses assert, as in pytest; it can take tmp_path for a scratch folder. Each run imports transforms/ and tests/ fresh, so it tests what you just saved. Promote runs them first and won't promote while one fails.

Table health (Preview)#

Data mesh > Quality > Table health reads every managed table's Delta log in the picked environment: files, size, average file size, V-Order, versions since the last VACUUM and the last OPTIMIZE, with advice.

• In Test and Prod, Optimize runs Fabric's table maintenance job on the workspace's capacity: OPTIMIZE with V-Order (what Direct Lake and the SQL endpoint read fastest), then VACUUM keeping the layer's history days. Tables written from a PC have no V-Order until it runs, so by default every table a Test or Prod run writes is optimized right after the run. Untick Optimize every table a run writes to stop that.
• In Dev on this PC, Compact does the same with delta-rs, without V-Order.

7. Environments and promote#

Data mesh > Deploy > Environments and promote has a card for Dev, Test and Prod.

Promote to a Fabric workspace is in Preview: it is tested against local stores and has not yet been run on a tenant.

Below the cards are Promote Dev to Test and Promote Test to Prod. Each shows whether the promote gate passes and, if not, what to do first. It also lists the tables whose logic differs from what last ran in the target, each with Compare, so you see what a promote does to the numbers before you press it. Under each table it lists what changed in its settings since the version promoted there, by meaning rather than by line: column channel added (string), check returns_are_negative changed on_fail from quarantine to stop. Comments, spacing and the order of settings don't count.

Promote to Test (or Prod):

1. Checks the gate: the project passes the save gate, the tests pass, and every table's last run in the source environment is green on this exact metadata version.
2. Creates any missing lakehouses (LH_<Layer> for each layer, and LH_Meta) in the target workspace, and LH_Keys with any missing key tables in the Keys workspace.
3. Syncs source systems to LH_Keys and publishes the VL_Sources variable library.
4. Pins the target to this metadata version, recorded in meta.deployments with a release record.
5. If the target runs in Fabric, uploads the new version for its runner.

Promote copies configuration, never data and never keys. Each environment issues its own surrogate keys from its own LH_Keys, and Test and Prod load from their own sources. After a promote, run the target to load it.

Release records#

Preview. Every promote records which Git commit it came from, so you can tell later exactly which files are running in Test or Prod. Fabric Desktop by KratosBI reads Git with the git installed on your PC: it never signs in, never fetches and never stores a remote's address. "Not pushed" and "behind" are counted against your upstream branch as of your last fetch.

The record goes into the target's meta.deployments row:

Rows written before this existed have these columns empty.

Released files and rollback#

Preview. A promote also keeps every file of the metadata version (configs, transforms, tests: the files the version is a hash of, without environments.toml and .local) in the target's meta.release_files. Test and Prod then run those files, wherever the run happens: on this PC, on an agent, or in Fabric. Editing your project folder changes what Dev runs, never what Test or Prod run, until you promote again. The first run after a release writes its files once to .local/releases/<env>/<version>/ in the project folder and reuses them. An environment last promoted before releases kept their files runs your folder as before, and only while the folder is on that release.

Roll back... on a Test or Prod Last release card puts that environment back on an earlier release it kept. It lists every release to the environment, shows what goes back entity by entity, and then:

• puts the environment on the earlier release's files, exactly as that environment kept them. Nothing is read from Git, Dev or your folder, and files that don't match their recorded hashes are refused
• records the rollback as a new release in meta.deployments, with a release record of its own whose rollback section names the release it replaced and the one it went back to
• uploads the earlier files for Fabric runs when the environment runs in Fabric

Tables keep their rows. A table whose logic changes asks whether to rebuild on its next run; to put its rows back now, restore it from History. Rolling back checks your workspace roles there, as a promote does. It doesn't apply separate_promoter, because it only goes back to a release that was already promoted. Releases promoted before this version can't be rolled back to: promote them again instead.

Each promote card shows the Last release to its target: the metadata version, the commit and whether it was pushed, the engine, who promoted it and when, and whether the stored record still matches its id (checks out). Copy record copies the whole record.

If the files can't be named by a pushed commit (uncommitted changes, unpushed commits, no upstream branch, or no Git), the promote still goes ahead and the promote card says so. To require a pushed commit instead, set require_commit = true for that environment in environments.toml:

[prod]
require_commit = true

Promote to that environment is then blocked until the files are committed and pushed.

Who can promote#

Preview. Before a promote to Test or Prod writes anything, the promote card checks your workspace role in each workspace the target uses. If Fabric lists you as a Viewer there, it says so and names the workspace's Admins to ask, and the promote is blocked. Fabric would refuse those writes anyway, partway through. If your role may come through a group, or Fabric won't show the workspace's access list, the card shows a note instead and Fabric decides when the promote writes.

To have a second person release to Prod, set separate_promoter = true for Prod in environments.toml:

[prod]
separate_promoter = true

Whoever promoted a metadata version to Test then can't also promote it to Prod. The check compares PC login names in the release record's by column. That makes it a team process check, not a security boundary: to make sure only release managers can change Prod, give only them Contributor or higher on the Prod workspaces.

This PC or Fabric runs each environment#

Each environment card has Runs on: This PC | Fabric, saved as compute in environments.toml. Fabric needs the tables in OneLake, so an environment with This PC only tables runs on the PC until you point it at a workspace.

With Fabric (Preview), Run all (and running one entity, backfills and rebuilds) uploads the metadata project and the app's engine to LH_Meta/Files/_fabric_desktop/current, and starts one notebook in the workspace, nb_fabric_desktop_runner. It's a Fabric Python notebook that runs the same engine against the same lakehouses. The upload is skipped when nothing changed. The app follows the job and shows the results when it finishes; the run log, checks and quarantine land in the same LH_Meta tables, so it doesn't matter where a run happened. A Fabric schedule runs every entity even when your PC is off, from the project as it was last uploaded.

8. Surrogate keys#

Surrogate keys are issued in one place per environment: LH_Keys, in a dedicated Keys workspace (Dev keeps it on the PC). Each business key has its own key table, sk.<name>, with three columns: <name>_sk (INT), <name>_bk (the business key, unique) and create_date. Customer ABC gets one key, say 123, and every source that has Customer ABC maps to 123.

Sync to LH_Keys is in Preview for Fabric environments: it has not yet been run on a tenant.

Data mesh > Design > Surrogate keys:

• New key table adds one, for example customer.
• Pick Keys workspace (or Change) sets where LH_Keys lives for the picked environment.
• The list shows each key table's key count, next key, and which tables use it. Select one to look up a business key, see its attributes, and copy the block that makes an entity use it.
• Sync to LH_Keys creates any missing key tables and writes the source system configuration.

An entity takes keys with a [[keys]] block. The first step of every run MERGEs business keys it hasn't seen into the key table (continuing from the highest key, never reusing one), then adds <name>_sk to every row. Rows with no business key get -1 (the Unknown member). A composite key is a list, stored as "a||b". Entities that share a key table can run at the same time, on one PC or several: each MERGE is committed against the version of the key table its new keys were worked out from, and is redone if another run committed first, so no key goes to two business keys. A run checks the key table for a key or business key that appears twice and stops, writing nothing, if it finds one.

[[keys]]
table = "customer"
business_key = "customer_id"     # or ["source_code", "customer_id"]

Keys in the platform#

With a platform in Fabric, an environment's keys come from SQL_Platform instead, one key system for every team. A project that already issued keys in LH_Keys moves them with Move keys on this tab: every key keeps its number, each row is checked, and only then does the environment switch (keys = "platform" in environments.toml).

In an environment whose keys are in the platform, the project doesn't make its own key tables. Use a key table picks one of the platform's (it still writes metadata/keys/<name>.toml, so the project says which keys it uses). Only platform admins add key tables: Request a new one sends a request to Platform > Requests, and the key table shows in the list once it's approved. A run that needs a key table the platform doesn't have stops and says to request it; Sync lists any that are missing.

9. Source systems and the variable library#

Data mesh > Design > Source systems keeps every source's fixed id, code (like CRM), name, type, owner and active flag, and its connection in Dev, Test and Prod: server, port, database, schema, base URL, file path, authentication, tenant, username, Key Vault URL and secret name, Fabric connection id, gateway and extra options. Switch environment with the Dev / Test / Prod buttons above the connection fields.

With Preview features on, the platform's list is the one list of source systems: Add a source this project reads picks a source from it with its id, code, name, owner and type, and the project keeps only its connection. Sources made before the platform's list existed show under "Only in this project" until Move sources to the platform copies them there with the same ids. See One list of source systems.

Passwords and keys never go here. Give the Key Vault URL and the secret's name; the notebook reads the secret when it runs.

Sync to LH_Keys writes them to config.source_system and config.source_connection, one row per source per environment, so notebooks and pipelines can look up connection values by code.

Variable library VL_Sources (Preview, on the same tab; not yet run on a tenant) is one Fabric variable library per environment workspace, with that environment's values active. Publish to Dev / Test / Prod creates or updates it; promote updates it in the target. Notebooks read the same names in Fabric and on your PC:

lib = notebookutils.variableLibrary.getLibrary("VL_Sources")
lib.CRM_server_host

10. Semantic model (Preview)#

Data mesh > Deploy > Semantic model builds a Direct Lake semantic model over a layer's tables (the last layer, usually Gold) in Test or Prod. It reads the layer's lakehouse through its SQL analytics endpoint, with no import and no refresh schedule.

• Each entity is a table; dimensions and facts are marked. Facts get a row count measure.
• Relationships come from the metadata: a fact's [[lookup]] names the dimension and key, and any other column named like a dimension's surrogate key joins to it.
• Surrogate keys, _run_id and _loaded_at are hidden.
• Create model makes SM_<Layer> in the environment's workspace (rename it if you like). Update model updates it in place and keeps every lineage tag, so reports built on it keep working.
• If a model with that name already exists (for example after a rename), Fabric Desktop by KratosBI asks first: the confirmation names the model and says its definition will be replaced, including measures, tables and relationships added to it in Fabric. Nothing is written until you confirm.

When Fabric can't be read. Fabric Desktop by KratosBI doesn't guess whether a model exists or what a table holds:

• If the workspace's semantic models can't be listed (no access, throttling, a sign-in that failed), the app shows the reason and the tab offers neither Create model nor Update model. Try again once Fabric answers.
• A table that hasn't been written in the environment yet is left out, and the result says so (Not written yet, so left out).
• A table that exists but can't be read stops the build with the table's name and the reason, and nothing is written, so the model never silently loses a table.

Dev on the PC has no Direct Lake model: switch the environment to Test or Prod.

11. Bring existing tables under management#

To manage tables that already exist in a lakehouse, use Adopt on the lakehouse in the Estate X-ray on Home (Preview). It writes an entity for each table with [source] table pointing at the existing table and a contract from its columns. Review the files, then run them.

12. Run on a schedule or a server#

• On this PC or in Fabric: Schedule at the top of Metadata, or Schedules on an environment card (Preview). Listed in Monitor > Schedules.
• On a server, unattended: the headless agent runs the same engine on Windows or Linux as a service principal or managed identity, with a durable queue and retries. It hasn't yet run against a live tenant.

Reference#

Entity file#

metadata/entities/<layer>.<table>.toml. An example with the common sections:

name = "silver.orders"
owner = "data-team@contoso.com"
description = "Orders, cleaned"

[source]
entity = "bronze.orders"          # or files = "landing/orders/*.csv", landing = "erp/orders/*.csv" (the
                                  # platform's landing zone), table = "<abfss uri>", or link = "<link name>"
rename = { "cust_id" = "customer_id" }

[load]
mode = "incremental"              # full | incremental | scd1 | scd2
watermark = "modified_at"

[contract]
keys = ["order_id"]
drift = "ignore_new"              # evolve | ignore_new | strict; overrides the layer

[contract.columns]
order_id = "int64"
customer_id = "int64"
order_date = "date"
amount = "float64"
status = "string"
modified_at = "date"
net_amount = "float64"

[[clean]]
column = "status"
trim = true
case = "lower"

[[clean]]
dedupe_keep = "newest:modified_at"

[[derive]]
column = "net_amount"
expr = "amount - coalesce(discount, 0)"

[[check]]
column = "order_id"
not_null = true
unique = true
on_fail = "stop"

[[keys]]
table = "order"
business_key = "order_id"

Every written row also gets _run_id and _loaded_at.

layers.toml#

An optional [rules] section (same_layer_reads = "dag" lets a table read its own layer when that makes no loop; "never" forbids it), then one [[layer]] per layer, in order, with name, description, drift, required_checks (key_not_null, key_unique), reads_from, require_keys, retention_days and workspace (a designation: ingest, product, serve or sandbox; see Data mesh). The Layers tab writes this file for you.

environments.toml#

One section per environment ([dev], [test], [prod]):

Changing this file doesn't change the metadata version, so it never forces a re-promote.

No two environments can use the same workspace, as their own workspace_id or as a designation's: each environment's lakehouses have the same names (LH_Meta, LH_Bronze, ...), so a run in one would write the other's tables. Saving such a change is refused, and if the file is edited by hand to share one, both environments refuse to read or write until it's fixed.

A job file run with fabric-desktop run job.toml can't write a table the metadata project manages, or anything in a workspace that Test or Prod uses: those change only through a framework run and promote.

Data mesh files#

project.toml (the project's domain and team), metadata/products/<name>.toml (data products) and metadata/links/<alias>.toml (links to other teams' products), plus an optional [catalog] block in an entity file. All are described in Data mesh.

Column lineage (Preview)#

Fabric Desktop by KratosBI works out where every column comes from by reading the project's files, without running anything. Each column is one of: read from the source (by the same name, or through rename), a [[derive]] expression and the columns it uses, a [[lookup]] of a dimension's key, a [[keys]] surrogate key, or added by the framework. Each is labelled declared when a file states the link (a rename, an expression, a lookup, a key) or inferred when it is assumed from a matching name. Columns your Python adds ([hooks] adds, or every column of a [step] custom) are opaque: they're never guessed. Columns a hook runs after are flagged, because a hook can change any column. For one column, lineage follows the steps back to the landing files and finds every column built from it further on. On an entity's page, Columns in the Lineage card lists every column with how it's made and its label; pick one to see where it comes from and what reads it.

Columns and tables the framework adds#