How a sync works

You never tell delta-engine what to change — you tell it what a table should look like, and each engine.sync(...) works out the changes itself. This page explains what happens between calling sync and getting a report back. You do not need any of it to run your first sync, but it is the mental model behind every other page: what gets compared, where a change can be rejected, and why one table’s failure does not take down the rest of the run.

The phases

Every sync runs the same chain of phases over every table in the call:

lower → resolve → read → diff → plan → compile → execute → report

Phase

Question it answers

Outcome

Lower

What tables does the declaration set define?

Domain tables, deduplicated, in deterministic order

Resolve

Can each declared foreign key work, and which tables go first?

Tables ordered dependency-first, with structural FK verdicts and their dependency edges

Read

What does the table look like right now?

Present (with its observed state), absent, or a read failure

Diff

How does observed state differ from desired?

Direct actions and non-action differences — foreign-key existence included — or no drift

Plan

Is the complete diff accepted?

A validated action plan, named validation failures with no plan, or a deferral when the table is absent and the declaration cannot create it

Compile

What exact backend statements apply it?

The SQL exposed on the report and passed unchanged to execution

Execute

Apply the compiled statements

Attempted results — or nothing, for tables skipped as no-ops, blocked, or on a dry run

Resolution comes before read because it needs nothing from the catalog: it judges the declarations against each other, so every table starts its worldly phases already knowing its position, its dependency edges, and whether its relationships are sound.

The result is a SyncReport with one entry per table. On a real run, if any table failed, the engine raises SyncFailedError with that report attached; a dry run always returns the report without raising.

The rest of this page looks at the behaviours that fall out of this design.

Diff produces actions; planning decides safety

The diff phase produces rich, backend-neutral actions directly — for example, DropColumn carries the complete observed column — but never judges whether executing one is a good idea. Ambiguous or unsupported states remain domain differences rather than being labelled as blockers; the application default policy decides to reject them. The planning boundary judges that complete diff and returns one of three results: an accepted ActionPlan, validation failures with no plan, or a deferral. A table that does not exist and whose declaration cannot create it (any scope narrower than full) is deferred before validation runs — the engine logs a warning and the table reports DEFERRED, neither changed nor failed, until something else creates it. An accepted plan carries the qualified table target and relation kind needed for compilation as well as its ordered actions.

This separation is why rejections are precise: a failed sync names the exact rule that fired and the column or table it fired on, rather than a generic “cannot apply changes”. The full rule set is in safe-change rules, and the safety model explains the reasoning behind it.

No SQL runs until a table fully passes

Validation is inseparable from plan construction, and planning happens before execution, so a rejected table is untouched — there is no plan containing the safe subset. Within execution, statements run in a deterministic order and a failure stops that table’s remaining statements. Re-running sync after fixing the cause is always the recovery path: the engine re-reads live state and plans only the drift that still remains.

One writer at a time: what success means

A sync is not transactional across its phases. Every plan is computed from the snapshot the read phase observed, and execution applies the compiled statements without re-reading the table in between. DDL from another actor that lands between the read and a statement is invisible to the run: either it makes a statement fail — reported loudly as an execution failure — or the run never notices it.

The operating contract is single-writer: while a sync runs, nothing else should run DDL against the tables it manages. Under that contract, a table reported as SUCCESS means every planned statement executed. The engine does not re-read the table afterwards to verify that the desired state now holds — a verification read could narrow the race with a concurrent writer, but it could not prevent the next writer from changing the table a moment later. Drift from outside the contract is caught the way all drift is caught: the next sync reads the live table and plans whatever remains.

One table’s failure does not abort the run

Each table carries its own result through the phases. A table that fails an early phase — say, its read failed — keeps that failure in its report entry and is skipped by the later phases, while every other table proceeds normally. You always get a complete report of what happened to every table, then a single SyncFailedError at the end if anything failed.

The exception is foreign-key dependents, which are deliberately not independent — see the next section.

Foreign keys order the run — and propagate failure

A foreign key can only be created if the table it references already exists with its primary key in place. The resolve phase therefore orders tables so referenced tables are executed before the tables that reference them. The foreign keys a table needs set or dropped are a difference like any other, so the diff states them; resolve judges only what one table’s declaration means for another’s.

The same dependency edges propagate failure: if a table fails — at any phase, including mid-execution — every table whose foreign keys depend on it is blocked rather than executed, reporting FOREIGN_KEY_FAILED. The rule is uniform: if a dependency won’t reach its desired state this sync, its dependents don’t run either.

Blocking is not something the engine records as it goes; it is worked out from the dependency edges once the run’s other outcomes are known. So a blocked table carries no execution outcome at all, and it names every dependency that let it down, whichever phase each one failed in. The upshot for you is that a blocked table’s report tells you the whole story of why it was skipped. Fix the upstream tables and re-sync. Foreign keys covers the declaration side.

Dry runs execute nothing

sync(..., dry_run=True) runs every phase but attempts no statements: resolve, read, diff, accepted/rejected planning, and compilation all happen, then the run stops. Because blocking is derived from the dependency edges rather than recorded while executing, the preview still shows which tables a failure would block. You get every action and SQL statement planned from the catalog snapshot it read, plus any read, validation, foreign-key, or dependency-blocking failure the run found. It cannot predict execution-time Databricks errors. The run makes zero catalog mutations and never raises SyncFailedError — failures come back in the report instead. (Two tables sharing a qualified name still raise DuplicateTableDefinitionError before any phase runs, the same as a real run.) See how to preview changes.

Where to drill down