Engines and connection profiles (Preview)

KratosForge is growing from "this PC or Fabric" into one place to develop for Microsoft Fabric, Azure Databricks, or both. This page describes the two foundations that work is built on, what they can do today, and what is planned. Nothing on this page has a screen in the app yet: it is the engine underneath, built and tested first so the screens can't promise more than it does.

• An engine is what runs a notebook cell: Python on this PC, DuckDB, SQLite, Polars, and later Fabric Spark, Databricks Connect and Databricks SQL warehouses.
• A connection profile says which platforms a piece of work uses, where, how it signs in, which engine it runs on by default, and whether it may start remote compute.

What each engine can do#

Every engine states each capability at one of five levels, with the evidence behind it. KratosForge refuses to register an engine that claims more than it has an adapter and tests for.

Compatibility matrix#

"Tested" means automated tests in this repository, on Linux and Windows CI, with no cloud. Nothing here has run against a Fabric tenant or a Databricks workspace.

Engines on this PC bill nothing themselves. Reading or writing OneLake from this PC still uses Fabric capacity, so "runs on this PC" never means "uses no capacity".

A session's database file (an existing DuckDB or SQLite file) is opened read-only. Each session is its own in-memory database or Python process: two notebooks never see each other's tables or variables, and closing a session frees everything it held.

Switching engines#

A notebook has one engine at a time. Switching to another engine starts a new session: cells and outputs stay, but variables and tables in the old session are not carried over. KratosForge asks first, with two choices: close the old session, or keep it in the background so switching back finds it (local engines only; a remote session is never left running).

If a cell can't run on the chosen engine, KratosForge says why and stops. It never moves the cell to another engine on its own, and above all never to a remote one.

Remote compute needs your say-so#

A remote engine can start compute that is billed. It opens only when:

• you confirm it for that engine and that workspace, cluster or warehouse (the confirmation lasts an hour), or
• the connection profile's execution policy allows that engine without asking.

A profile can also forbid remote compute altogether.

Usage figures#

Every run records what it used, labeled as one of: measured on this PC, billed by the platform (with where the bill came from), or estimated (with its assumptions). Fabric capacity unit seconds and Databricks DBUs are different units, priced differently: KratosForge never adds them together or converts one into the other. Comparing money needs your pricing, in Optimize.

Connection profiles#

A profile needs only a name and a kind: This PC only, Microsoft Fabric, Azure Databricks, or Fabric and Databricks. Everything else has a sensible default:

The app's current behaviour is the built-in profile This PC and Fabric.

Sign-in: references, never secrets#

A profile never holds a password, token or key. It holds a reference to one:

A service principal's secret is named as env:NAME (an environment variable) or credman:TARGET (Windows Credential Manager). KratosForge refuses to save or import a profile with anything that looks like a token, key or password in it, and it never repeats the value in its error.

For Fabric and Databricks together, each platform keeps its own sign-in, workspaces and deploy targets, and the two can't share a secret.

Databricks CLI profiles#

If you already use the Databricks CLI, KratosForge reads the profile names and workspace addresses in ~/.databrickscfg (or DATABRICKS_CONFIG_FILE) so you can pick one. It never reads out tokens or client secrets. A profile that keeps a personal access token in that file gets a warning, because databricks auth login keeps tokens out of plain text. Only Azure Databricks is supported for now.

Sharing a profile#

Export writes a profile without anything tied to your PC: secret references, the Databricks CLI profile name and the last check are removed. Whoever imports it adds their own sign-in; the imported profile says what's missing.

Switching profiles#

Switching closes every open engine session (they may hold data read with the old sign-in). Cells and outputs stay. A notebook with unsaved changes must be saved or discarded first.

Not yet#

• No screens: the notebook's engine picker and profile setup come next, after a mockup.
• No Databricks sign-in, workspace discovery or execution yet; those need the Databricks SDK and will be labeled Preview until tested against a live workspace.
• Databricks Connect will run in its own Python environment: it ships its own PySpark, which would otherwise change what spark means in every local notebook. Its version is checked before anything starts (Databricks Runtime 16.4 or later, or serverless, for this app's Python 3.12).