Data mesh

Fabric Desktop by KratosBI lets many teams build on one platform without stepping on each other. A central team runs the platform: one business key system and one landing zone for everyone. Bronze processing is decentralized: each team builds its own Bronze from the landing zone, and everything after it; domain teams own their projects, publish data products, and read each other's products through links. Everything a team builds stays in its own Git repository.

!!! note "Preview" The Domains tabs are tagged Preview: they're tested on the PC but not yet on a live tenant.

Who does what#

All of it happens on the Platform page (central team, and requests from everyone), Data mesh > Domains (a project's own products and links) and OneLake > Data products (finding other teams' products). See The platform.

The pieces#

Logic lives in Git and goes through review and promote. Catalog text lives in the platform, so anyone can describe a table without a pull request.

Join the mesh#

Data mesh > Domains > Domain and workspaces. Pick your domain (and subdomain, if any), your team and a description, then choose Join the mesh. That writes project.toml at the top of the project:

[project]
name = "crm"
domain = "sales"
subdomain = "sales-emea"
team = "CRM analytics"
format = 2

Nothing else in the project changes, and a project without project.toml works exactly as before. Domains come from the platform; platform admins create them on Platform > Domains (also listed in Your organization's domains on this tab).

Domains come from Fabric#

The mesh's domains and subdomains are the ones defined in Fabric (Admin portal > Domains), so a project's domain here is the same one its workspaces have in Fabric.

How the import works:

• A Fabric domain matches the catalog domain already linked to it, else one with the same name (domains created by hand before are kept and linked, so their projects don't change), else it's added with an id made from its name (Finance & Risk becomes finance-risk).
• Imported domains follow Fabric: rename or move one in Fabric, then import again; the catalog id stays, so projects keep their domain. Their name and parent can't be changed in Fabric Desktop by KratosBI. Owners and description are the platform's and stay as you set them.
• A domain deleted in Fabric is reported and kept in the catalog, because projects may still use it. Delete it on Platform > Domains once nothing uses it.
• Fabric allows one level of subdomains, like the mesh. Anything deeper is skipped and reported.
• New domain on Platform > Domains still works for organizations that don't use Fabric domains; those show as "Only here".
• Domains are created, imported and edited only on Platform > Domains. Data mesh > Domains lists them, read only, with Manage in Platform.

Workspaces by designation#

One environment can spread its lakehouses over several workspaces, so access follows purpose:

On Domain and workspaces, pick or create a workspace for each designation and environment (a new one is named like Sales Ingest - Test). On Design > Layers, set each layer's Workspace. A layer with no designation stays in the environment's own workspace, and LH_Meta always does.

Data products#

Data mesh > Domains > Data products > New data product. Choose the tables to share (each becomes a port, with the name consumers use), an owner, and who it's open to:

A product is a file you review like any other change:

name = "customer_360"
title = "Customer 360"
description = "One row per customer, from every source"
owner = "crm-team@contoso.com"
access = "domain"
status = "published"
version = 1

[[port]]
name = "customers"
entity = "gold.dim_customer"

Status is draft, published or deprecated (with deprecated_on, the date it stops). Promote publishes every non-draft product to the catalog with where its ports live in the target environment; Publish to the catalog does the same by hand.

The product gate. A product open to anyone or to the domain can't be published, and its project can't be promoted, until it has a description and an owner and every port has a table description and a description on every column. The entity's About card shows what's missing.

Breaking changes. On promote, Fabric Desktop by KratosBI compares each port's columns with what the catalog holds for that port in the target environment.

• What blocks the promote. A column dropped, or a type changed (other than widening, such as int32 to int64 or float32 to float64), while another project reads that environment. The message names the projects, for example port customers drops email, and 1 other project reads it in test (finance).
• What doesn't. Added columns pass. So does anything nobody else reads yet.
• How to make a breaking change on purpose. Raise version in the product's file. The promote then goes through and says what the new version breaks.
• If the platform can't be reached. The check says so and doesn't block.

The check knows which projects read a port, not which columns each one uses, so dropping any published column counts as breaking.

Links#

OneLake > Data products > Link, Data products > Other teams' products > Link, or Links > New link. Pick a product and port, a name for it, and which of their environments each of yours reads. Dev usually reads their Prod, so you build against real data without copying it. The catalog refuses a link to a draft, to a private product, or to a domain product from another domain, unless the product's owner let your project in: OneLake > Data products > Request access sends them a request, and once they approve it on Platform > Requests, Link works. Approving lets the link be made; the owner still gives read access in OneLake (a data access role on their Product workspace).

An entity reads a link like any source:

[source]
link = "crm_customers"

At run time the link resolves, through the catalog, to the producer's table in the environment you read, and is read where it lives. To pin an address instead (for example while the producer has no platform yet), add it to the link file:

[location.prod]
table = "abfss://<workspace id>@onelake.dfs.fabric.microsoft.com/<lakehouse id>/Tables/dbo/dim_customer"

The Links tab shows where each link points in the picked environment, and which entities use it. A link that an entity still reads can't be removed.

Catalog text (About)#

Every entity page has an About card: description, tags, classification (Public, Internal, Confidential, Restricted), glossary terms, steward, and a description and classification per column. Save writes them to the catalog; no Git change is needed.

A team can keep starter text with the code, and it shows until someone writes their own:

[catalog]
description = "One row per customer version"
tags = ["customer", "gold"]
classification = "internal"

[catalog.columns.email]
description = "Primary email address"
classification = "confidential"

Glossary (Platform > Glossary) holds shared terms like Active customer. Platform admins keep shared terms and a domain's owners keep their domain's. Tag a table or column with terms on the entity's About tab. Recent changes in the catalog, and who made them, are on Platform > Audit.

At scale#

• Hundreds of projects. Each project is its own repository and its own workspaces; the platform holds only small catalog rows and keys, and every write is one function call.
• Many teams, one key. Keys are issued by the platform for everyone, so Customer ABC has one key in every team's tables (see The platform).
• No copies. Links read the producer's table where it lives. Use OneLake data access roles on the Product workspace to control what each consumer may read.
• Safe change. A producer can't silently break consumers: ports are named, deprecations carry a date, and the catalog lists who reads each product.

Related#

• The platform
• Metadata framework guide
• Working across teams