Capabilities and limitations

This page summarises what delta-engine can and cannot manage. Each row links to the page with the detail.

Platform

Requirement	Supported
Backend	Delta Lake tables on Databricks with Unity Catalog — the supported target today; the reader reads managed and external Delta tables, plus Delta streaming tables for tag-only management, and any other relation a registered name resolves to (view, materialized view, foreign table, non-Delta format) fails its read (architecture)
Python	3.12 or later
PySpark	Needed only for the Spark backend; the SQL warehouse backend needs none. Declaring and planning are pure Python either way (installation)
Reads (both backends)	Unity Catalog only — every read is one DESCRIBE TABLE EXTENDED … AS JSON plus information_schema for tags, primary and foreign keys, and inbound foreign keys; a hive_metastore or other non-UC table is not readable and surfaces as a read failure on both the Spark and SQL warehouse backends (installation)

Identifier handling

Identifiers are case-insensitive, matching the platform: Databricks resolves all identifiers case-insensitively (backticked or not), and Unity Catalog stores catalog, schema, and table names in lowercase. The engine therefore accepts declarations in any case and normalizes every identifier — object name parts, column names, nested struct field names, and constraint names — to lowercase. A name that is already lowercase is preserved verbatim, including Unicode names such as straße. Case never distinguishes two identifiers, a case-only difference is never drift, and engine-created columns display in lowercase in the catalog.

Declared catalog, schema, and table names are validated against Unity Catalog’s object-name rules at declaration time: at most 255 characters, and no periods, spaces, forward slashes, control characters, or DEL. Column names are exempt from those rules; column names that require special characters need column mapping as described in column mapping and dropping columns.

What a sync manages

Aspect	Managed	Notes
Create table	✓	Missing tables are created from the declaration
Add column	✓	Must be nullable on an existing table (rules)
Drop column	✓	Requires delta.columnMapping.mode='name' declared (properties)
Loosen nullability	✓	NOT NULL → nullable is applied
Tighten nullability	✗	Blocked — backfill first, then tighten (rules)
Change column type	Widening only	Safe widenings (e.g. Integer → Long) apply in place with delta.enableTypeWidening='true' declared; anything else is blocked (type widening)
Rename column	✓	Declare renamed_from on the new column; requires delta.columnMapping.mode='name'. Editing a name directly (without the hint) is a drop plus an add (renaming a column)
Rename partition column	✓	Column mapping preserves the partition column’s physical identity, so its layout metadata follows the rename (rules)
Table and column comments	✓	Always managed; an empty declaration clears the comment (comments)
Table properties	✓	Six managed delta.* keys; other keys are rejected at declaration (properties)
Table and column tags	✓	Full-state: undeclared tags are removed (tags)
Primary keys	✓	Declared at table level (primary keys)
Foreign keys	✓	Must target the referenced table’s primary key; orders the sync; names are engine-generated and cannot be chosen (foreign keys)
Partitioning	Create only	Fixed after creation; changes are blocked (rules)
Clustering	✓	Liquid clustering keys are reconciled in place, unlike partitioning (clustering)
Metadata-only scope	✓	scope="metadata" restricts a sync to comments, tags, and keys (guide)
Tag-only scope	✓	scope="tags" restricts a sync to table and column tags (tags)
Streaming tables	Tags only	Discovered at read time; only scope="tags" declarations may target one — schema, comments, and properties belong to the owning pipeline (guide)
Dry run	✓	Full plan and validation, zero mutations (guide)

Outside the model

These features are not modeled at all: the engine never reads, creates, changes, or drops them, and they produce no drift.

Not modeled	Meaning
CHECK constraints	Cannot be declared; a constraint that references a renamed column must be changed before the rename
Key constraint options (RELY, MATCH, ON UPDATE/ON DELETE)	Keys are created with Databricks defaults (NOT ENFORCED NORELY); option drift is invisible, and an out-of-band RELY is lost when a primary-key change drops and re-adds the key
UNIQUE constraints	Cannot be declared or used as a registered foreign-key target, even on Databricks versions that support them
Identity and generated columns	Generation expressions are invisible; one that references a renamed column must be changed before the rename
Views and materialized views	Unsupported; a registered name that resolves to a view or materialized view fails its read rather than being planned against (streaming tables, by contrast, are read for tag-only management)
External table creation	Existing external Delta tables are read and reconciled like managed ones, but the engine creates managed tables only: a location cannot be declared, and an absent table is created managed
Grants, row filters, column masks	Governance beyond comments and tags is out of scope
Data	The engine runs DDL only; it never reads, writes, or backfills rows

Type support

The full matrix is in data types. The limitations in brief:

Limitation	Behaviour
Unsupported Spark types	Any column whose type the engine cannot model fails that table’s read; columns are never skipped, so drift is never invisible
CHAR(n) / VARCHAR(n)	Treated as String; the length bound is not modeled and never altered
Struct fields	Structs change as a whole: any field change is a blocked column type change
Decimal precision	Maximum 38, enforced at declaration

Clustering limits

Liquid clustering (clustering) has its own set of declaration-time and execution-time limits, distinct from partitioning:

Limitation	Behaviour
Key count	At most four clustered_by keys per table, rejected at declaration
Mutual exclusivity	A table cannot declare both partitioned_by and clustered_by, rejected at declaration
Unsupported key types	Array, Map, Struct, and Variant columns cannot be clustering keys, rejected at declaration
Nested struct-field keys	Clustering by a field inside a Struct column is not supported by the declaration — only top-level columns can be named in clustered_by
Stats and column order	Databricks only collects the file statistics clustering relies on for a table’s first 32 columns; a clustering key outside that range gets no skipping benefit
Runtime compatibility	Liquid clustering requires Databricks Runtime 13.3 LTS or later; delta-engine does not preflight this — see runtime features

Concurrent catalog changes

A sync is not transactional across its read, plan, and execute phases. Table creation compiles as a plain CREATE TABLE: if another writer creates the same name after the reader observed it missing, that statement errors and the table is reported as an execution failure rather than a false success. The next sync reads the table that actually exists and reports any resulting drift. Avoid concurrent creators for the same qualified table name.

Runtime features

delta-engine does not preflight Databricks Runtime or Delta protocol versions. Declaring a feature the workspace or table protocol does not support — key constraints, tags, change data feed — fails at execution with the original Databricks error. See runtime and Delta feature compatibility.

Reading a table relies on DESCRIBE TABLE EXTENDED … AS JSON, which needs Databricks Runtime 16.2 or later, or any SQL warehouse. Primary and foreign keys are then read from information_schema; because key constraints are available from Databricks Runtime 13.3 (GA 15.2) — below the AS JSON read floor — any runtime new enough to read a table can also observe its keys. As with every other runtime feature, delta-engine documents this floor rather than preflighting it — an unsupported runtime surfaces as a read failure.