delta-engine

Declarative schema management for Delta Lake tables on Databricks. You declare the state a table should have; the engine reads the state it actually has, computes the difference, checks that the difference is safe to apply, and runs exactly the DDL needed to close the gap.

from delta_engine.databricks import build_spark_engine
from delta_engine.schema import Column, DeltaTable, Integer, String

customers = DeltaTable(
    catalog="dev",
    schema="silver",
    name="customers",
    columns=[
        Column("id", Integer(), nullable=False),
        Column("name", String()),
    ],
)

engine = build_spark_engine(spark)
engine.sync(customers)  # creates the table, or no-ops if it already matches

There is no migration script to write and no DDL to hand-order. The declaration is the source of truth; sync reconciles the catalog to it, every run.

Install

pip install delta-engine

Declaring schemas is pure Python. A sync can use either a Databricks Spark session or a Databricks SQL warehouse connection; the latter needs no PySpark. See Installation for the [spark] and [sql] paths.

What a sync does

The engine reads the current catalog state, diffs it against your declaration, validates that the drift is safe to fix in place, plans deterministic DDL actions, orders tables so foreign-key dependencies are created first, executes, and returns a per-table report. Unsafe changes — dropping data, narrowing a column’s type, repartitioning — fail validation with a named rule before any SQL runs. How a sync works walks through the phases; the safety model explains what gets blocked and why.

Backend support

delta-engine targets Delta Lake tables on Databricks with Unity Catalog today. Backends plug in as adapters that read catalog state and execute plans, and the planning core takes no backend imports — so a Delta-compatible backend, such as open-source Unity Catalog, can be added by implementing those adapters. A genuinely different table format, such as Iceberg, would also need Delta-specific policy lifted out of the application layer first. See Architecture for what is and isn’t backend-neutral, and how to implement an adapter for the ports.

Where to go next

You want to…	Read
Install the package and sync your first table	Installation, Getting started
Run read-only schema plans in GitHub Actions	CLI reference
Understand what a sync does before running one	How a sync works
Check whether the engine supports something	Capabilities and limitations
Declare keys, properties, tags, or comments	The how-to guides in the sidebar
See why a change was rejected	Safe-change rules
Understand the internals or add a backend	Architecture