delta-engine

Declarative schema management for Delta Lake tables on Databricks. You declare the state a table should have; the engine reads the state it actually has, computes the difference, checks that the difference is safe to apply, and runs exactly the DDL needed to close the gap.

What it is for

Delta Engine turns a table’s physical schema and Unity Catalog metadata into a version-controlled contract. It is useful when tables are long-lived and a team wants the same columns, properties, comments, tags, and constraints across environments, with drift and unsafe changes visible before DDL runs.

It does not transform or load data. Existing PySpark jobs, declarative pipelines, dbt models, or other systems can keep producing the rows while Delta Engine reconciles the catalog state around them. A one-off DDL statement is usually simpler for a one-off change; the value here comes from keeping the declarations and re-running them in CI or deployment workflows.

from delta_engine.databricks import build_spark_engine
from delta_engine.schema import Column, DeltaTable, Integer, String

customers = DeltaTable(
    catalog="dev",
    schema="silver",
    name="customers",
    columns=[
        Column("id", Integer(), nullable=False),
        Column("name", String()),
    ],
)

engine = build_spark_engine(spark)
report = engine.sync(customers)  # creates the table, or no-ops if it already matches

There is no migration script to write and no DDL to hand-order. The declaration is the source of truth; sync reconciles the catalog to it, every run. It returns a structured report rather than printing output. The getting-started tutorial shows the report, diff, and exact planned SQL and explains where to keep the declaration.

Choose how to run it

Delta Engine runs where your release or data workflow calls it. See Ways to use Delta Engine for release-time reconciliation, ETL readiness checks, and restricted governance deployments.

Install

pip install delta-engine

Declaring schemas is pure Python. A sync can use either a Databricks Spark session or a Databricks SQL warehouse connection; the latter needs no PySpark. See Installation for the Databricks compute and [sql] paths.

What a sync does

The engine reads the current catalog state, diffs it against your declaration, validates that the drift is safe to fix in place, plans deterministic DDL actions, orders tables so foreign-key dependencies are created first, executes, and returns a per-table report. Unsafe changes — dropping data, narrowing a column’s type, repartitioning — fail validation with a named rule before any SQL runs. How a sync works walks through the phases; the safety model explains what gets blocked and why.

Backend support

delta-engine targets Delta Lake tables on Databricks with Unity Catalog today. Backends plug in as adapters that read catalog state and execute plans, and the planning core takes no backend imports — so a Delta-compatible backend, such as open-source Unity Catalog, can be added by implementing those adapters. A genuinely different table format, such as Iceberg, would also need Delta-specific policy lifted out of the application layer first. See Architecture for what is and isn’t backend-neutral, and how to implement an adapter for the ports.

Where to go next

You want to…	Read
Install the package and sync your first table	Installation, Getting started
Choose a release or workflow pattern	Ways to use Delta Engine
Run read-only schema plans in GitHub Actions	CLI reference
Understand what a sync does before running one	How a sync works
Check whether the engine supports something	Capabilities and limitations
Declare keys, properties, tags, or comments	The how-to guides in the sidebar
See why a change was rejected	Safe-change rules
Understand the internals or add a backend	Architecture
Understand the environment support policy	Runtime compatibility