Ways to use Delta Engine¶

Delta Engine is a reconciliation library rather than a deployment system or background controller. A sync runs only when your application, workflow, or deployment process calls it with a collection of table declarations.

This makes Delta Engine flexible about where it fits. A team can reconcile table contracts as part of a release, make table readiness a prerequisite of an ETL workflow, or manage governance metadata around tables whose structure is owned elsewhere.

The patterns below are not mutually exclusive. A project might plan changes in CI, apply them through a dedicated release job, and use a separate restricted deployment for centrally managed annotations.

Choose a pattern¶

Pattern	Best suited to	Where Delta Engine runs
Pattern 1: Release table contracts	Controlled production releases and continuing table estates	A dedicated job after the application bundle is deployed
Pattern 2: Reconcile a target table	ETL applications that own a single target table	Application startup before transformations or writes
Pattern 3: Reuse declarations in ETL	ETL applications that use declared schema and row identity at write time	Transformation and write code
Pattern 4: Use declarations in tests	ETL applications that test schemas, required values, and relationships	Unit tests
Pattern 5: Add governance	Split ownership between data-product, platform, and governance teams	A separate restricted-scope deployment

Pattern 1: Release table contracts with the application¶

For most production applications, the clearest pattern is to treat table contracts as part of the release.

        flowchart LR
    PR[Pull request] --> Review[Plan and review changes]
    Review --> Merge[Merge]
    Merge --> Release[Deploy and reconcile tables]
    Release --> Data[Run data workloads]

The declarations, pipeline code, and deployment configuration are versioned and released together. The deployment creates or updates the Databricks resources, then runs a dedicated job that reconciles the table declarations before the new data-producing workload is allowed to run.

This keeps responsibilities clear:

the bundle deploys jobs, pipelines, libraries, and configuration;
the reconciliation job manages the declared table state;
the ETL workloads produce and write rows.

Keep an explicit table registry¶

Expose the tables owned by the release through one explicit, ordered collection:

# myproject/table_registry.py
from myproject.tables.customers import customers
from myproject.tables.order_items import order_items
from myproject.tables.orders import orders

ALL_TABLES = (
    customers,
    orders,
    order_items,
)

An explicit registry makes the release boundary visible in code review: adding a table to the collection is an intentional decision to include it in the deployment.

Add a reconciliation entry point¶

Package the reconciliation logic with the application:

# myproject/sync_tables.py
from delta_engine import render_report
from delta_engine.databricks import build_spark_engine
from pyspark.sql import SparkSession

from myproject.table_registry import ALL_TABLES


def main() -> None:
    """Reconcile every table owned by this release."""
    spark = SparkSession.builder.getOrCreate()

    engine = build_spark_engine(spark)
    report = engine.sync(*ALL_TABLES)

    print(render_report(report))

The job succeeds only when the sync succeeds. If reading, planning, dependency resolution, or execution fails, Delta Engine raises SyncFailedError, and the job should fail rather than allowing the release to continue silently.

Deploy and run it through a bundle¶

A Declarative Automation Bundle can deploy the application wheel and configure the reconciliation entry point as a dedicated Python wheel task.

The release pipeline can then validate the bundle, deploy it, and run the reconciliation job:

databricks bundle validate --target production
databricks bundle deploy --target production
databricks bundle run --target production reconcile_tables

Bundle deployment and table reconciliation are separate operations. If the bundle is deployed but the reconciliation job fails, the new job definitions and application artifacts may already exist in the workspace. The release process should stop, expose the failed SyncReport, and prevent the updated data workload from starting until the declaration or catalog problem is resolved.

Plan the same registry in CI¶

The same registry can be planned during a pull request through the read-only CLI:

delta-engine plan myproject.table_registry:ALL_TABLES

This reads live Unity Catalog state, validates the declarations, and prints the semantic differences and exact planned DDL without applying them.

A real sync does not replay the dry-run output as a saved plan. It reads the catalog again and derives a fresh plan from the state present at deployment time. See How to report schema plans in CI for a complete read-only workflow.

Pattern 2: Reconcile a target table before its ETL runs¶

When an ETL application owns a single target table, it can reconcile that table once at startup before performing transformations or writing any rows:

ETL job starts
        ↓
reconcile target table
        ↓
run transformations
        ↓
write rows

Keep the table declaration alongside the ETL code and import it into the application entry point:

# myproject/main.py
from delta_engine.databricks import build_spark_engine
from pyspark.sql import SparkSession

from myproject.pipeline import run_pipeline
from myproject.tables.customers import customers


def main() -> None:
    """Reconcile the target table, then run the ETL pipeline."""
    spark = SparkSession.builder.getOrCreate()

    engine = build_spark_engine(spark)
    engine.sync(customers)

    run_pipeline(spark)

If the table is missing, Delta Engine creates it. If it has safe, supported drift, Delta Engine reconciles it. If it already matches the declaration, the sync applies no DDL and the ETL continues normally.

If the declaration is invalid, the live table cannot be read, the proposed transition is unsafe, or execution fails, SyncFailedError propagates and the ETL stops before run_pipeline() begins.

This pattern gives the ETL application a clear startup precondition:

The target table must exist and match its declared contract before the application writes any data.

Synchronize dependencies together¶

An ETL that owns one independent target normally needs to synchronize only that table:

engine.sync(customers)

Pass multiple declarations when the target has cross-table dependencies that Delta Engine must validate and order. For example, if orders declares a foreign key to customers, synchronize both declarations in the same run:

engine.sync(customers, orders)

Delta Engine validates the relationship, places the parent before the dependent, and prevents orders from executing if customers cannot reach its desired state.

There is no need to synchronize unrelated tables together merely because they belong to the same repository. Each ETL can reconcile its own target, while related tables are grouped only where their declarations depend on one another.

When this pattern works well¶

Use this pattern when:

one ETL application clearly owns one target table;
the application should be able to start in a new environment where the table does not yet exist;
table validity should be checked before every ETL run;
the job runs at a frequency where the additional catalog read and planning step are acceptable;
the runtime identity is permitted both to reconcile the table and write its data.

It is particularly straightforward for batch applications because the table check becomes part of the application’s normal startup sequence.

Pattern 3: Reuse table declarations throughout ETL code¶

A DeltaTable declaration can also be imported throughout the ETL that produces it. Table names, schemas, keys, and column lists then stay in one place rather than being repeated across the pipeline.

Conform output to the declared schema¶

Use the declaration to cast and order the final DataFrame before writing it:

result = transform(source).to(to_spark_schema(customers))

Build merges from the declared primary key¶

Use the same primary key declared on the table to construct the Delta merge condition. This handles composite keys without maintaining a separate join predicate:

from functools import reduce
from operator import and_

from pyspark.sql import functions as F


condition = reduce(
    and_,
    (
        F.col(f"target.{column}") == F.col(f"source.{column}")
        for column in customers.primary_key
    ),
)

(
    target.alias("target")
    .merge(result.alias("source"), condition)
    .whenMatchedUpdateAll()
    .whenNotMatchedInsertAll()
    .execute()
)

Derive update columns from the declaration¶

When a merge should not update key columns, derive the mutable columns instead of maintaining another list:

update_columns = [
    column.name
    for column in customers.columns
    if column.name not in customers.primary_key
]

updates = {
    column: F.col(f"source.{column}")
    for column in update_columns
}

(
    target.alias("target")
    .merge(result.alias("source"), condition)
    .whenMatchedUpdate(set=updates)
    .whenNotMatchedInsertAll()
    .execute()
)

Reuse the key wherever row identity matters¶

The declared primary key can also drive deduplication, windowing, and other key-based ETL logic. For example, retain the latest event for each customer:

from pyspark.sql import Window, functions as F


latest = Window.partitionBy(*customers.primary_key).orderBy(
    F.col("event_timestamp").desc()
)

result = result.withColumn("_row_number", F.row_number().over(latest))

Pattern 4: Use table declarations in unit tests¶

The same DeltaTable used by production code can define the schemas, keys, and relationships expected by ETL tests. Tests can therefore work from the real table contract instead of maintaining separate schemas and column lists.

Create test DataFrames from declared schemas¶

Use the declared Spark schema when creating test fixtures:

source = spark.createDataFrame(
    [
        (1, "Alice"),
        (2, "Bob"),
    ],
    schema=to_spark_schema(customers),
)

Assert the ETL produces the declared schema¶

Compare the final DataFrame with the same table contract:

def test_produces_the_declared_schema(source) -> None:
    result = transform(source)

    assert result.schema == to_spark_schema(customers)

A schema change now changes the production contract and the test expectation in one place.

Test non-nullable columns¶

Use the declaration to derive which columns must always contain a value:

def test_produces_no_nulls_in_required_columns(source) -> None:
    result = transform(source)

    required_columns = [
        column.name
        for column in customers.columns
        if not column.nullable
    ]

    for column in required_columns:
        assert result.filter(F.col(column).isNull()).isEmpty()

Test primary-key uniqueness¶

The declared primary key can drive data-level key checks:

def test_produces_unique_primary_keys(source) -> None:
    result = transform(source)

    duplicates = (
        result
        .groupBy(*customers.primary_key)
        .count()
        .filter(F.col("count") > 1)
    )

    assert duplicates.isEmpty()

Test foreign-key relationships¶

Related table declarations can also drive referential-integrity tests. For a foreign key whose local and referenced columns have the same names:

def test_produces_valid_customer_references(
    orders_result,
    customers_result,
) -> None:
    missing_customers = (
        orders_result
        .join(
            customers_result,
            on=list(orders.foreign_keys[0].columns),
            how="left_anti",
        )
    )

    assert missing_customers.isEmpty()

Pattern 5: Add governance without taking over the pipeline¶

Table ownership is not always all-or-nothing. A data-product team may own the schema and row production while a platform or governance team owns comments, classifications, and ownership tags.

Restricted scopes let those responsibilities remain separate:

        flowchart TB
    Pipeline[Pipeline or dbt project] -->|owns schema and rows| Table[Unity Catalog table]
    Governance[Delta Engine governance deployment] -->|owns comments and tags| Table

For example, a pipeline can continue to define and populate an orders table while a separate declaration manages its annotations:

from delta_engine.schema import Column, DeltaTable, Long, String

orders_annotations = DeltaTable(
    catalog="production",
    schema="sales",
    name="orders",
    columns=[
        Column(
            "order_id",
            Long(),
            nullable=False,
            comment="Stable identifier for an order.",
            tags={"classification": "identifier"},
        ),
        Column(
            "customer_email",
            String(),
            comment="Email supplied when the order was placed.",
            tags={"classification": "personal"},
        ),
    ],
    comment="Customer orders accepted by the commerce platform.",
    tags={
        "domain": "sales",
        "owner": "commerce-platform",
    },
    scope="annotations",
)

With scope="annotations", Delta Engine can reconcile table and column comments and tags, but it cannot add, remove, or alter columns, properties, layouts, or constraints.

Other available scopes include:

Scope	Managed state
`full`	Complete table definition
`metadata`	Comments, tags, and key constraints
`annotations`	Comments and tags
`tags`	Tags only

See How to deploy metadata only for the exact scope semantics.

Mirror the table state you do not own¶

A restricted declaration still describes the live table accurately. Its columns, layout, and any keys must match the state owned by the producing system, even though Delta Engine is not allowed to alter those aspects.

If the pipeline changes the schema and the governance declaration is not updated, the next sync fails rather than applying metadata against a table it no longer understands. This makes the ownership boundary safe, but it also means governance declarations must follow intentional changes made by the structural owner.

Use restricted scopes for streaming tables¶

Databricks streaming tables are structurally owned by their defining pipeline. Delta Engine can manage their externally alterable comments and tags through scope="annotations" or scope="tags", but broader scopes are rejected.

Do not declare the same comment or tag in both the pipeline definition and Delta Engine. A pipeline refresh can reassert its own declaration, creating recurring drift between two competing owners.

Operational boundaries¶

Whichever pattern you use:

Keep one write-capable sync per table at a time.
Treat a dry run as a live preview, not as an immutable saved plan.
Do not allow another tool to evolve an aspect that Delta Engine manages.
Do not run reconciliation inside every streaming micro-batch.
Prefer explicit table registries over implicit package crawling.
Do not describe a multi-statement table plan or multi-table run as transactionally atomic.
Inspect the returned SyncReport; successful process startup is not a substitute for checking the reconciliation result.

See Getting started for a complete first sync, How a sync works for the execution model, and Capabilities and limitations before using write-capable synchronization in an unattended production workflow.