delta_engine¶

delta-engine: declarative schema management for Delta Lake tables.

This is the curated runtime entry point. The preferred user imports are:

from delta_engine.schema import DeltaTable, Column, Integer
from delta_engine.databricks import build_spark_engine
from delta_engine import Engine

Only backend-neutral runtime types live here. Schema declarations belong in delta_engine.schema and Databricks helpers belong in delta_engine.databricks.

Submodules¶

Attributes¶

Failure

Exceptions¶

`DuplicateTableDefinitionError`	Raised when one sync receives two definitions for the same table.
`SyncFailedError`	Raised when one or more tables failed during sync.

Classes¶

`Engine`	High-level orchestrator to plan and execute changes.
`ExecutionFailure`	Details about a statement that failed while executing.
`FailurePhase`	The sync phase that produced a failure. Ordered so the earliest wins.
`ForeignKeyFailure`	A foreign key constraint that could not be applied, failing its whole table.
`ReadFailure`	Failure reading current catalog state for a table.
`SyncReport`	Aggregate report for a run across all tables.
`TableChangeState`	What happened to one table's intended catalog change within a run.
`TableRun`	Frozen public record of one table's sync run.
`TableRunStatus`	High-level status of a table's sync run.
`ValidationFailure`	Description of a validation rule failure.

Functions¶

`render_diff`(→ str)	Render every table's planned changes as +/-/~ blocks, under a DIFF title.
`render_report`(→ str)	Render the run: title, optional dry-run banner, status grid, failures section, footer.

Package Contents¶

exception DuplicateTableDefinitionError(qualified_name: delta_engine.domain.model.QualifiedName)¶

Bases: ValueError

Raised when one sync receives two definitions for the same table.

qualified_name¶

class Engine(reader: delta_engine.application.ports.CatalogStateReader, executor: delta_engine.application.ports.PlanExecutor)¶

High-level orchestrator to plan and execute changes.

The engine coordinates reading current state from a catalog, computing a diff of desired vs observed state, accepting or rejecting that diff at the validated planning boundary, resolving FK dependencies with full failure context, and executing accepted plans using the provided adapters.

reader¶

executor¶

sync(*tables: delta_engine.application.ports.DesiredTableSource, dry_run: bool = False) → delta_engine.application.report.SyncReport¶

Synchronize all registered tables to their desired state.

Runs lower → resolve → plan → execute → report. Resolution orders the tables dependency-first with their static facts, the plan pass takes each table through its read-only steps (read, plan, compile) and freezes one complete TableRun, execution attaches attempted statement results to the tables it reaches, and assembly derives dependency blocking from the edges.

A table that fails an early phase carries that failure on its run and is skipped by execution; it is still included in the report.

Parameters:

*tables – The table specifications to synchronize. Duplicate qualified names raise DuplicateTableDefinitionError before any phase runs.
dry_run – When True, stop after the plan pass and attempt no statements (zero catalog mutations). No table retains attempted statement results, while its plan still records the actions compiled from the observed snapshot. Blocking is derived rather than executed, so a dependent of a failed table still reports BLOCKED_BY_FAILED_DEPENDENCY in the preview. The report is returned instead of raising SyncFailedError when a table fails.

Returns:

The aggregate SyncReport for the run.

Raises:

DuplicateTableDefinitionError – If two table specifications have the same qualified name. No phase has run when this is raised.
SyncFailedError – On a real run (dry_run=False), if any table fails to read, validate, resolve foreign keys, or execute. The report is available on the exception’s report attribute. A dry run never raises.

class ExecutionFailure¶

Details about a statement that failed while executing.

phase: ClassVar[FailurePhase]¶

statement_index: int¶

exception_type: str¶

message: str¶

statement: str¶

format_lines() → tuple[str, Ellipsis]¶

headline() → str¶

property statement_number: int¶

The failing statement’s one-based display position.

statement_index remains zero-based so it indexes the compiled statements. Only the reader-facing number shifts, making “statement 3” agree with a report-grid progress count of “2/3”.

type Failure = ReadFailure | ValidationFailure | ExecutionFailure | ForeignKeyFailure¶

class FailurePhase¶

Bases: enum.IntEnum

The sync phase that produced a failure. Ordered so the earliest wins.

FOREIGN_KEY = 1¶

READ = 2¶

PLANNING = 3¶

EXECUTION = 4¶

class ForeignKeyFailure¶

A foreign key constraint that could not be applied, failing its whole table.

phase: ClassVar[FailurePhase]¶

table: delta_engine.domain.model.QualifiedName¶

local_columns: delta_engine.domain.collection_types.ListOrTuple[str]¶

references: delta_engine.domain.model.QualifiedName¶

reason: ForeignKeyFailureReason¶

format_lines() → tuple[str, Ellipsis]¶

headline() → str¶

class ReadFailure¶

Failure reading current catalog state for a table.

phase: ClassVar[FailurePhase]¶

exception_type: str¶

message: str¶

format_lines() → tuple[str, Ellipsis]¶

headline() → str¶

exception SyncFailedError(report: delta_engine.application.report.SyncReport)¶

Bases: Exception

Raised when one or more tables failed during sync.

report¶

class SyncReport¶

Aggregate report for a run across all tables.

started_at: datetime.datetime¶

ended_at: datetime.datetime¶

table_runs: delta_engine.domain.collection_types.ListOrTuple[TableRun]¶

dry_run: bool = False¶

classmethod assemble(*, started_at: datetime.datetime, ended_at: datetime.datetime, table_runs: delta_engine.domain.collection_types.ListOrTuple[TableRun], dry_run: bool) → Self¶

Assemble the run report, deriving dependency blocking from the graph.

Folds the blocking rule over the runs in dependency order: a table that did not converge — own failures, or a dependency that did not — marks its name, and a sound table its resolution reports as blocked is replaced by a copy carrying those failures. The run recorded nothing about blocking; this projection is where the consequence becomes visible, dry and real runs alike.

property has_failures: bool¶: Return True if any table failed in the run.

property has_changes: bool¶

True if any table’s plan holds actions.

States facts about planned changes only: a table that failed validation contributes failures, not changes. The CI gate idiom is report.has_failures or report.has_changes.

property table_change_states: tuple[TableChangeState, Ellipsis]¶: Catalog change state for each table run, in run order.

property duration_seconds: float¶: Wall-clock seconds the run took, start to end.

property counts: RunCounts¶

Per-outcome table counts.

A table that failed counts as failed whatever else it planned: the planned changes were not applied, so reporting them as changes would overstate what the run achieved.

property planned_sql_statements: dict[str, tuple[str, Ellipsis]]¶: Dotted table name → the SQL its plan compiles to; no-op tables omitted.

property failures_by_table: dict[delta_engine.domain.model.QualifiedName, tuple[delta_engine.application.failures.Failure, Ellipsis]]¶: Mapping of qualified table name to its failures (if any).

render() → str¶: Render the run’s status, failures, and summary as human-readable text.

render_diff() → str¶: Render every table’s planned changes as human-readable text.

to_dict() → dict[str, Any]¶: Project the whole run as plain, JSON-serialisable data; tables in run order.

class TableChangeState¶

Bases: enum.StrEnum

What happened to one table’s intended catalog change within a run.

NOT_PLANNED = 'not planned'¶

DEFERRED = 'deferred'¶

UNCHANGED = 'unchanged'¶

PLANNED = 'planned'¶

NOT_APPLIED = 'not applied'¶

PARTIALLY_APPLIED = 'partially applied'¶

APPLIED = 'applied'¶

class TableRun¶

Frozen public record of one table’s sync run.

Born complete at the engine’s plan pass: everything the table can know alone — its resolution, read, planning outcome, and compiled SQL — is fixed at construction, in lifecycle field order, and the trailing fields default to their not-applicable state. The two facts that depend on other tables are attached afterwards as functional updates: execution by the execute walk, blocked_failures at assembly. A run without execution is a legitimate terminal state (a dry run, a blocked table, a no-op plan, a deferred absent table), not an unfinished one.

plan is None when reading or planning failed; a successfully planned no-op retains an empty, target-bearing plan. compiled is populated on dry and real runs so the accepted plan and its statements remain inspectable even when execution is skipped or blocked. blocked_failures is the derived consequence of other tables’ fates, baked in at assembly — a blocked table records no execution outcome of its own. diff is derived from the planning outcome, which retains the complete set of differences it planned from — actions and unresolvable differences alike — so a table whose plan was rejected can still show what drifted. It is None when the read failed, because planning never ran.

resolution: delta_engine.application.relationships.TableResolution¶

read: delta_engine.application.ports.ReadResult¶

planning: delta_engine.application.planning.PlanningResult | None = None¶

compiled: delta_engine.application.ports.CompiledPlan | None = None¶

execution: delta_engine.application.ports.ExecutionResult | None = None¶

blocked_failures: delta_engine.domain.collection_types.ListOrTuple[delta_engine.application.failures.ForeignKeyFailure] = ()¶

property plan: delta_engine.domain.plan.ActionPlan | None¶: The accepted plan, or None when reading or planning failed.

property diff: delta_engine.domain.plan.TableDiff | None¶: The diff the planning outcome retains, or None when the read failed.

property failures: tuple[delta_engine.application.failures.Failure, Ellipsis]¶

Flatten canonical phase outcomes for callers.

Lifecycle order: structural, read, planning, execution — derived blocking sits at the execution position, where the run it replaced would have been.

property desired: delta_engine.domain.model.DesiredTable¶: The declaration this run reconciled, as retained by its resolution.

property qualified_name: delta_engine.domain.model.QualifiedName¶: The table identity from the declaration retained by this report.

property status: TableRunStatus¶

Status of the earliest phase that failed; DEFERRED or SUCCESS when none did.

Failures dominate deferral: a deferred table that also failed resolution reports the failure, because that is what needs acting on.

property has_failures: bool¶: True if the table did not fully succeed.

property has_changes: bool¶: True when the plan holds actions — drift was found and validated.

property creates_table: bool¶: True when the plan brings the table into existence rather than altering it.

property statement_progress: StatementProgress | None¶: How far execution got, or None when it did not run.

to_dict() → dict[str, Any]¶

Project this table’s run as plain, JSON-serialisable data.

The field names are a public stability contract (see the run report reference doc); changing them is a breaking change.

class TableRunStatus¶

Bases: enum.StrEnum

High-level status of a table’s sync run.

SUCCESS = 'SUCCESS'¶

DEFERRED = 'DEFERRED'¶

READ_FAILED = 'READ_FAILED'¶

PLANNING_FAILED = 'PLANNING_FAILED'¶

FOREIGN_KEY_FAILED = 'FOREIGN_KEY_FAILED'¶

EXECUTION_FAILED = 'EXECUTION_FAILED'¶

class ValidationFailure¶

Description of a validation rule failure.

subject is what the failure is about — a column, property key, or aspect — used by the compact headline in the report grid. Rules that judge the table as a whole leave it empty. details are the individual differences behind a summary judgment, for a rule whose message names a whole aspect rather than one column. They are separate lines rather than newlines inside message so the report renderer owns their indentation, as it already does for the SQL line of an execution failure; a rule that embeds its own layout composes wrongly wherever the failure is nested.

phase: ClassVar[FailurePhase]¶

rule_name: str¶

message: str¶

subject: str = ''¶

details: delta_engine.domain.collection_types.ListOrTuple[str] = ()¶

format_lines() → tuple[str, Ellipsis]¶

headline() → str¶

render_diff(report: delta_engine.application.report.SyncReport) → str¶: Render every table’s planned changes as +/-/~ blocks, under a DIFF title.

render_report(report: delta_engine.application.report.SyncReport) → str¶: Render the run: title, optional dry-run banner, status grid, failures section, footer.