delta_engine ============ .. py:module:: delta_engine .. autoapi-nested-parse:: delta-engine: declarative schema management for Delta Lake tables. This is the curated runtime entry point. The preferred user imports are:: from delta_engine.schema import DeltaTable, Column, Integer from delta_engine.databricks import build_spark_engine from delta_engine import Engine Only backend-neutral runtime types live here. Schema declarations belong in ``delta_engine.schema`` and Databricks helpers belong in ``delta_engine.databricks``. Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/delta_engine/databricks/index /autoapi/delta_engine/schema/index Exceptions ---------- .. autoapisummary:: delta_engine.DuplicateTableDefinitionError delta_engine.SyncFailedError Classes ------- .. autoapisummary:: delta_engine.Engine delta_engine.ExecutionFailure delta_engine.Failure delta_engine.FailurePhase delta_engine.ForeignKeyFailure delta_engine.ReadFailure delta_engine.SyncReport delta_engine.TableRunReport delta_engine.TableRunStatus delta_engine.ValidationFailure Functions --------- .. autoapisummary:: delta_engine.render_diff delta_engine.render_report Package Contents ---------------- .. py:exception:: DuplicateTableDefinitionError(qualified_name: delta_engine.domain.model.QualifiedName) Bases: :py:obj:`ValueError` Raised when one sync receives two definitions for the same table. .. py:attribute:: qualified_name .. py:class:: Engine(reader: delta_engine.application.ports.CatalogStateReader, executor: delta_engine.application.ports.PlanExecutor) High-level orchestrator to plan and execute changes. The engine coordinates reading current state from a catalog, computing a diff of desired vs observed state, accepting or rejecting that diff at the validated planning boundary, resolving FK dependencies with full failure context, and executing accepted plans using the provided adapters. .. py:attribute:: reader .. py:attribute:: executor .. py:method:: sync(*tables: delta_engine.application.ports.DesiredTableSource, dry_run: bool = False) -> delta_engine.application.report.SyncReport Synchronize all registered tables to their desired state. Runs the phases as a chain, each transforming the per-table runs: read → diff → plan → compile → resolve → execute. Each ``TableRunReport`` is born in the read phase and accretes its diff, plan, compiled SQL, failures, and execution as later phases run. A table that fails an early phase carries that failure forward and is skipped by execution; its partial run is still included in the report. :param \*tables: The table specifications to synchronize. Duplicate qualified names raise ``DuplicateTableDefinitionError`` before any phase runs. :param dry_run: When True, run read → diff → plan → compile → resolve but skip execution (zero catalog mutations). Every run's ``execution`` stays ``None`` while its ``plan`` still records the actions compiled from the observed snapshot, and the report is returned instead of raising ``SyncFailedError`` when a pre-execution phase fails. :returns: The aggregate :class:`SyncReport` for the run. :raises DuplicateTableDefinitionError: If two table specifications have the same qualified name. No phase has run when this is raised. :raises SyncFailedError: On a real run (``dry_run=False``), if any table fails to read, validate, resolve foreign keys, or execute. The report is available on the exception's ``report`` attribute. A dry run never raises. .. py:class:: ExecutionFailure Bases: :py:obj:`Failure` Details about a statement that failed while executing. .. py:attribute:: phase :type: ClassVar[FailurePhase] .. py:attribute:: statement_index :type: int .. py:attribute:: exception_type :type: str .. py:attribute:: message :type: str .. py:attribute:: statement :type: str .. py:method:: format_lines() -> tuple[str, Ellipsis] Return one or more human-readable lines describing this failure. .. py:method:: headline() -> str Return a compact one-line summary without the detail message, for the report grid. .. py:class:: Failure Bases: :py:obj:`abc.ABC` A failure that can render itself as display lines, tagged with its phase. .. py:attribute:: phase :type: ClassVar[FailurePhase] .. py:method:: format_lines() -> tuple[str, Ellipsis] :abstractmethod: Return one or more human-readable lines describing this failure. .. py:method:: headline() -> str :abstractmethod: Return a compact one-line summary without the detail message, for the report grid. .. py:class:: FailurePhase Bases: :py:obj:`enum.IntEnum` The sync phase that produced a failure. Ordered so the earliest wins. .. py:attribute:: READ :value: 1 .. py:attribute:: PLANNING :value: 2 .. py:attribute:: FOREIGN_KEY :value: 3 .. py:attribute:: EXECUTION :value: 4 .. py:class:: ForeignKeyFailure Bases: :py:obj:`Failure` A foreign key constraint that could not be applied, failing its whole table. .. py:attribute:: phase :type: ClassVar[FailurePhase] .. py:attribute:: table :type: delta_engine.domain.model.QualifiedName .. py:attribute:: local_columns :type: tuple[str, Ellipsis] .. py:attribute:: references :type: delta_engine.domain.model.QualifiedName .. py:attribute:: reason :type: ForeignKeyFailureReason .. py:method:: format_lines() -> tuple[str, Ellipsis] Return one or more human-readable lines describing this failure. .. py:method:: headline() -> str Return a compact one-line summary without the detail message, for the report grid. .. py:class:: ReadFailure Bases: :py:obj:`Failure` Failure reading current catalog state for a table. .. py:attribute:: phase :type: ClassVar[FailurePhase] .. py:attribute:: exception_type :type: str .. py:attribute:: message :type: str .. py:method:: format_lines() -> tuple[str, Ellipsis] Return one or more human-readable lines describing this failure. .. py:method:: headline() -> str Return a compact one-line summary without the detail message, for the report grid. .. py:exception:: SyncFailedError(report: delta_engine.application.report.SyncReport) Bases: :py:obj:`Exception` Raised when one or more tables failed during sync. .. py:attribute:: report .. py:class:: SyncReport Aggregate report for a run across all tables. .. py:attribute:: started_at :type: datetime.datetime .. py:attribute:: ended_at :type: datetime.datetime .. py:attribute:: table_reports :type: tuple[TableRunReport, Ellipsis] .. py:attribute:: dry_run :type: bool :value: False .. py:property:: has_failures :type: bool Return True if any table failed in the run. .. py:property:: has_changes :type: bool True if any table's plan holds actions. States facts about *planned* changes only: a table that failed validation contributes failures, not changes. The CI gate idiom is ``report.has_failures or report.has_changes``. .. py:property:: planned_sql_statements :type: dict[str, tuple[str, Ellipsis]] Dotted table name → the SQL its plan compiles to; no-op tables omitted. .. py:property:: failures_by_table :type: dict[delta_engine.domain.model.QualifiedName, tuple[delta_engine.application.failures.Failure, Ellipsis]] Mapping of qualified table name to its failures (if any). .. py:method:: to_dict() -> dict[str, Any] Project the whole run as plain, JSON-serialisable data; tables in run order. .. py:class:: TableRunReport Per-table report with outcomes and a single phase-ordered failure stream. Carries the exact SQL statements its plan compiles to (``planned_sql_statements``), populated on every run — dry or real — so a dry run can preview the DDL. Planned is not executed: a table blocked after planning (for example by a foreign-key failure) still reports the SQL its plan compiles to. .. py:attribute:: qualified_name :type: delta_engine.domain.model.QualifiedName .. py:attribute:: desired :type: delta_engine.domain.model.DesiredTable .. py:attribute:: read :type: delta_engine.application.ports.CatalogState .. py:attribute:: plan :type: delta_engine.domain.plan.ActionPlan .. py:attribute:: planned_sql_statements :type: tuple[str, Ellipsis] :value: () .. py:attribute:: failures :type: tuple[delta_engine.application.failures.Failure, Ellipsis] :value: () .. py:attribute:: execution :type: delta_engine.application.ports.ExecutionSummary | None :value: None .. py:property:: status :type: TableRunStatus Status of the earliest phase that failed; SUCCESS when nothing failed. .. py:property:: has_failures :type: bool True if the table did not fully succeed. .. py:property:: has_changes :type: bool True when the plan holds actions — drift was found and validated. .. py:method:: to_dict() -> dict[str, Any] Project this table's run as plain, JSON-serialisable data. The field names are a public stability contract (see the run report reference doc); changing them is a breaking change. .. py:class:: TableRunStatus Bases: :py:obj:`enum.StrEnum` High-level status of a table's sync run. .. py:attribute:: SUCCESS :value: 'SUCCESS' .. py:attribute:: READ_FAILED :value: 'READ_FAILED' .. py:attribute:: PLANNING_FAILED :value: 'PLANNING_FAILED' .. py:attribute:: FOREIGN_KEY_FAILED :value: 'FOREIGN_KEY_FAILED' .. py:attribute:: EXECUTION_FAILED :value: 'EXECUTION_FAILED' .. py:class:: ValidationFailure Bases: :py:obj:`Failure` Description of a validation rule failure. .. py:attribute:: phase :type: ClassVar[FailurePhase] .. py:attribute:: rule_name :type: str .. py:attribute:: message :type: str .. py:method:: format_lines() -> tuple[str, Ellipsis] Return one or more human-readable lines describing this failure. .. py:method:: headline() -> str Return a compact one-line summary without the detail message, for the report grid. .. py:function:: render_diff(report: delta_engine.application.report.SyncReport) -> str Render every table's planned changes as +/-/~ blocks, under a DIFF title. .. py:function:: render_report(report: delta_engine.application.report.SyncReport) -> str Render the run: title, optional dry-run banner, status grid, failures section, footer.