Data Factory in KratosForge (plan)

Status: Preview. Chris said go on 2026-10-24 and approved the mockup on 2026-10-25. This page is for contributors. It extends the orchestration plan from pipelines to the rest of Fabric Data Factory: Dataflow Gen2, copy jobs, connections and gateways, and monitoring of Fabric runs.

Rules for every story#

• Local first: anything that can run on this PC does, and its output stays in the run's sandbox. Copy jobs whose source and destination are OneLake or local files reuse the pipeline Copy data engine (P5).
• Fabric-only items (Dataflow Gen2, copy jobs with external sources) run in Fabric. Starting one asks first and runs the risk check, because it uses capacity.
• Reading definitions, run history and connections is a read. It starts no Fabric job and uses no capacity.
• Connections are read-only in KratosForge. The app never stores, creates or changes a connection or its credentials. Changes happen on Fabric's own Manage connections page, opened inside the app.
• No new rail item. The Pipelines page grows into a Data Factory page with tabs: Pipelines, Copy jobs, Dataflows, Connections. Runs stay in Monitor > Runs.
• Each screen gets a mockup first, and new screens are tagged Preview.

Stories#

DF-1 engine: Fabric pipeline runs#

Two engine calls, both reads (listed in audit.READ_ONLY):

• fabric_pipeline_runs(refresh=False, all_workspaces=False, workspace_ids=None) lists the job history of every pipeline in the chosen workspaces, newest first. With no workspace_ids it uses the workspaces Monitor > Schedules lists (pinned workspaces, environment workspaces and notebooks' workspaces), or every workspace with all_workspaces. The scan runs in the background. It returns {"pending": true} until it's done, then serves the result from memory until refresh. It keeps up to 20 runs per pipeline and 300 in all. A pipeline whose runs can't be read is counted in missed and logged once per scan.
• fabric_pipeline_activity_runs(workspace_id, job_id, started, ended) reads one run's activities from Fabric's activity-run query when its row is opened, never in the background. Fabric needs a time window, so the query covers the run's start and end with five minutes either side (up to now while the run is going). Each activity's input, output and error pass through redact.secrets and are capped at 16,000 characters.

Run states use the names local pipeline runs use (pending, running, succeeded, failed, cancelled, plus skipped for activities), so Monitor > Runs can show both kinds the same way.

Not tested yet: a live tenant. The activity-run query follows Microsoft's pipeline REST API page.

DF-2 engine: pipelines from a workspace#

Two engine calls, both reads (listed in audit.READ_ONLY):

• workspace_pipelines(workspace_id) lists a workspace's pipelines by name: item id, name, description and folder.
• workspace_pipeline_definition(workspace_id, item_id, name) reads one pipeline with getDefinition. It returns text, the getDefinition result, which the existing pipeline calls take as they are (pipeline_inspect, pipeline_compatibility, pipeline_run_check, pipeline_run_start). It also returns pipeline, which is what pipeline_import says about it. A SecureString default shows as (hidden). A definition this version can't read comes back with ok: false and the reason, not an error.

Both calls take Fabric GUIDs only. Nothing is written to disk: the definition stays in memory, as a pasted one does.

DF-2 screen: In a Fabric workspace#

The Pipelines page's open dialog has an In a Fabric workspace section (Preview) under the repository list. Signed out, it offers sign-in. Signed in, it lists the workspaces; picking one lists its pipelines with workspace_pipelines, and clicking one reads it with workspace_pipeline_definition and opens its text on the Pipelines page, the same way a file opens. The page header then has Open in Fabric.

The definition text holds SecureString defaults, so the page keeps it in memory only: it is never written to localStorage or the console. Only the picked workspace id is remembered (fd.pipelines.workspace). The Copy jobs, Dataflows and Connections tabs from the mockup come with their stories (DF-5, DF-6, DF-7), so the page shows no empty tabs until then.

DF-5: Connections tab#

One engine call, a read (listed in audit.READ_ONLY):

• fabric_connections(refresh=False) lists the connections the signed-in person can use (Fabric's GET /connections, scope Connection.Read.All, asked for once) and reads the pipeline and copy job definitions in each environment's workspace (Dev, Test and Prod first). Without environments, it reads the workspaces Monitor > Schedules lists. Every externalReferences.connection in a definition, nested activities included, counts as a use. A connection id that's used but isn't in the person's list goes under missing, because that step fails when this person runs it. The scan runs in the background and returns {"pending": true} until it's done, then serves the result from memory until refresh. It reads up to 500 definitions per scan.

Nothing from a credential is read: Fabric's list has only the credential type, and the connection's path passes through redact.secrets. The tab sits beside Pipelines on the Pipelines page. Manage connections in Fabric opens Fabric's page.

Not in this slice: gateway names and status, which need Gateway.Read.All, a new scope that waits on Chris's OK. Dataflow connections are in their M code and come with DF-7. Checking that each target environment has the connections a deploy needs comes with DF-8's connection mapping.

DF-7 engine: Dataflow Gen2 from a workspace#

Three engine calls, all reads (listed in audit.READ_ONLY). None of them starts a refresh or uses capacity:

• workspace_dataflows(workspace_id) lists a workspace's Dataflow Gen2 items by name: item id, name, description and folder.
• workspace_dataflow_definition(workspace_id, item_id, name) reads one dataflow with getDefinition. It returns each Power Query (M) query from mashup.pq in document order, with its M code, whether it loads, whether it's hidden, and its destination. A destination has the kind (Lakehouse, Warehouse, Azure SQL database, KQL database), workspace, item, schema, table, update method (Replace or Append) and whether it makes a new table. These come from the query's DataDestinations attribute and the _DataDestination query that attribute names. The call also returns the connections from queryMetadata.json, with each DatasourceId, so they can be matched with the Connections tab, and the whole mashup.pq. A definition without mashup.pq (a Gen1 dataflow, say) comes back with ok: false and the reason, not an error.
• fabric_dataflow_refreshes(workspace_id, item_id, name) lists the dataflow's newest 50 refreshes in the shape Monitor > Runs uses for pipeline runs.

The M text is only read, never evaluated: there's no local M engine. It passes through redact.secrets, because a query can carry a key inline (a Web.Contents header, say). Nothing is written to disk. The parser lives in orchestration/mashup.py. It splits the section on top-level semicolons and skips strings, comments and nested brackets.

DF-7 screen: Dataflows tab#

The Pipelines page's Dataflows tab (Preview) sits between Pipelines and Connections. You pick a workspace (the same one the open dialog remembers, fd.pipelines.workspace) and its dataflows are listed with workspace_dataflows. Opening one reads workspace_dataflow_definition and fabric_dataflow_refreshes together. Its views are Queries (M), with hidden destination and helper queries folded away, Destinations (and the connections the dataflow names) and Refresh history. Open in Fabric opens the workspace, because the address of a dataflow's own page isn't confirmed yet. The mockup's "Changes since Git" view comes with the Git diff work (DF-8).

Refresh in Fabric comes next, behind a confirm and the risk check. No live refresh runs without Chris's OK.