Capabilities and limitations

This page summarises what delta-engine can and cannot manage. Each row links to the page with the detail.

Platform

Requirement	Supported
Backend	Delta Lake tables on Databricks with Unity Catalog — the supported target today; the reader reads managed and external Delta tables, plus Delta streaming tables for comment and tag management, and any other relation a registered name resolves to (view, materialized view, foreign table, non-Delta format) fails its read (architecture)
Python	3.12 or later
PySpark	Supplied by Databricks Runtime for the Spark backend and not installed by delta-engine; the SQL warehouse backend needs none. Declaring and planning are pure Python either way (installation)
Reads (both backends)	Unity Catalog only — every existing-table read uses one DESCRIBE TABLE EXTENDED … AS JSON for table shape, properties, and supported features, plus information_schema for tags, primary and foreign keys, and inbound foreign keys; a hive_metastore or other non-UC table is not readable and surfaces as a read failure on both backends (installation)

Identifier handling

Catalog, schema, table, and ordinary column references are case-insensitive, whether backticked or not. Unity Catalog stores catalog, schema, and table names in lowercase, so object name parts are normalized to lowercase exactly as the catalog stores them. Constraint-name DDL has an exception described below.

Column names and nested struct field names keep their declared or observed spelling. Constraint names do not: Unity Catalog stores them in lowercase. Case never distinguishes two identifiers: names differing only in case are the same column and collide as duplicates within one schema.

Within one declaration, case is free. A partition, clustering, primary-key, or foreign-key reference may use any casing; once attached to a table it uses the referenced Column.name when the public declaration is lowered. Thus Column("requestId", ...) and primary_key=["REQUESTID"] are accepted together, while DeltaTable.primary_key returns ("requestId",).

Against the catalog, case is exact. A declaration naming an existing column with different casing fails validation with ColumnSpellingMustMatchCatalog, which reports both spellings; a foreign key that spells the referenced table’s key differently from that table’s own declaration fails resolution with REFERENCED_COLUMN_CASE_MISMATCH. Nothing is rewritten to compensate: the engine emits the spelling the declaration carries, and this rule is what makes that safe on the Databricks DDL paths that resolve case-sensitively. Columns the sync creates are spelled as declared — a table that does not yet exist has no catalog spelling to agree with.

Exactness holds at every scope. A scope="metadata", scope="annotations", or scope="tags" declaration still names the columns it keys, comments, or tags, so it is judged the same way; narrowing the scope does not narrow this rule. DESCRIBE TABLE shows the spelling to copy. Callers that relied on lowercase accessor values should apply their own presentation policy.

Constraint names occupy one case-insensitive namespace per schema, across tables and constraint kinds, and information_schema exposes their normalized lowercase spelling. Despite that case-insensitive collision rule, DROP CONSTRAINT requires the exact catalog spelling; with IF EXISTS, a case mismatch silently does nothing. Databricks generates a name when raw SQL omits one. Delta-engine likewise omits the name unless primary_key_name or ForeignKey(name=...) is supplied. Either form accepts any structurally matching observed constraint: an explicit name is a creation preference, not ongoing managed state. Changing only that preference does not rename an existing constraint; Databricks has no direct constraint-rename clause.

Declared catalog, schema, and table names are validated against Unity Catalog’s object-name rules at declaration time: at most 255 characters, and no periods, spaces, forward slashes, control characters, or DEL. Column names are exempt from those rules; column names that require special characters need column mapping as described in column mapping and dropping columns.

What a sync manages

Aspect	Managed	Notes
Create table	✓	Missing tables are created from the declaration
Add column	✓	Must be nullable on an existing table (rules)
Drop column	✓	Requires delta.columnMapping.mode='name' declared (properties)
Loosen nullability	✓	NOT NULL → nullable is applied to top-level columns; nested field changes are blocked as struct type changes
Tighten nullability	✗	Top-level columns are blocked — backfill first, then tighten; nested field changes are blocked as struct type changes (rules)
Change column type	Widening only	Safe widenings (e.g. Integer → Long) apply in place with delta.enableTypeWidening='true' declared; anything else is blocked (type widening)
Rename column	✓	Declare renamed_from on the new column; requires delta.columnMapping.mode='name'. Editing a name directly (without the hint) is a drop plus an add (renaming a column)
Rename partition column	✓	Column mapping preserves the partition column’s physical identity, so its layout metadata follows the rename (rules)
Table and column comments	✓	Always managed; an empty declaration clears the comment (comments)
Table properties	✓	Six managed delta.* keys; other keys are rejected at declaration (properties)
Table and column tags	✓	Full-state: undeclared tags are removed (tags)
Primary keys	✓	Declared at table level; creation names may be requested (primary keys)
Foreign keys	✓	Must target the referenced table’s primary key; orders the sync; creation names may be requested (foreign keys)
Partitioning	Create only	Fixed after creation; changes are blocked (rules)
Clustering	✓	Liquid clustering keys are reconciled in place, unlike partitioning (clustering)
Metadata-only scope	✓	scope="metadata" restricts a sync to comments, tags, and keys (guide)
Tag-only scope	✓	scope="tags" restricts a sync to table and column tags (tags)
Annotations scope	✓	scope="annotations" restricts a sync to comments and tags (annotations)
Streaming tables	Comments and tags	Discovered at read time; only scope="annotations" and scope="tags" declarations may target one — schema, properties, and keys belong to the owning pipeline and must be mirrored, not managed (guide)
Dry run	✓	Full plan and validation, zero mutations (guide)

Outside the model

These features are not modeled at all: the engine never reads, creates, changes, or drops them, and they produce no drift.

Not modeled	Meaning
CHECK constraints	Cannot be declared; a constraint that references a renamed column must be changed before the rename
Key constraint options (RELY, MATCH, ON UPDATE/ON DELETE)	Keys are created with Databricks defaults (NOT ENFORCED NORELY); option drift is invisible, and an out-of-band RELY is lost when a primary-key change drops and re-adds the key
UNIQUE constraints	Cannot be declared or used as a registered foreign-key target, even on Databricks versions that support them
Identity and generated columns	Generation expressions are invisible; one that references a renamed column must be changed before the rename
Views and materialized views	Unsupported; a registered name that resolves to a view or materialized view fails its read rather than being planned against (streaming tables, by contrast, are read for comment and tag management)
External table creation	Existing external Delta tables are read and reconciled like managed ones, but the engine creates managed tables only: a location cannot be declared, and an absent table is created managed
Grants, row filters, column masks	Governance beyond comments and tags is out of scope
Data	The engine runs DDL only; it never reads, writes, or backfills rows

A comment this engine sets on a streaming table can be reverted by the owning pipeline. CREATE OR REFRESH is fully declarative, so a refresh re-applies the pipeline’s own comment and deletes metadata the refresh does not specify. The engine cannot read the pipeline’s defining SQL, so it cannot warn: a contested comment re-drifts on every pipeline update and each sync sets it again. Manage a given comment from one place only.

Type support

The full matrix is in data types. The limitations in brief:

Limitation	Behaviour
Unsupported Spark types	Any column whose type the engine cannot model fails that table’s read; columns are never skipped, so drift is never invisible
CHAR(n) / VARCHAR(n)	Treated as String; the length bound is not modeled and never altered
Struct fields	Structs change as a whole: any field change is a blocked column type change
Decimal precision	Maximum 38, enforced at declaration

Clustering limits

Liquid clustering (clustering) has its own set of declaration-time and execution-time limits, distinct from partitioning:

Limitation	Behaviour
Key count	At most four clustered_by keys per table, rejected at declaration
Mutual exclusivity	A table cannot declare both partitioned_by and clustered_by, rejected at declaration
Unsupported key types	Array, Map, Struct, Variant, Boolean, and Binary columns cannot be clustering keys, rejected at declaration; Boolean and Binary remain valid partition columns
Nested struct-field keys	Clustering by a field inside a Struct column is not supported by the declaration — only top-level columns can be named in clustered_by
Statistics	Clustering keys need collected file statistics. Unity Catalog external tables default to the first 32 columns; managed tables with predictive optimization choose statistics automatically. Verify that every declared key has statistics.
Runtime compatibility	Liquid clustering is available in Databricks SQL and Databricks Runtime 13.3 LTS or later; delta-engine does not preflight this — see runtime features

Concurrent catalog changes

A sync is not transactional across its read, plan, and execute phases, and it never re-reads a table after executing its statements. The operating contract is single-writer: while a sync runs, no other actor should run DDL against the tables it manages. A successful table report means every planned statement executed, not that the desired state was verified afterwards — DDL from another actor between the read and a statement is invisible to the run unless it makes a statement fail (how a sync works).

Table creation compiles as a plain CREATE TABLE: if another writer creates the same name after the reader observed it missing, that statement errors and the table is reported as an execution failure rather than a false success. The next sync reads the table that actually exists and reports any resulting drift. Avoid concurrent creators for the same qualified table name.

Runtime features

delta-engine does not preflight Databricks Runtime or Delta protocol versions. Declaring a feature the workspace or table protocol does not support — key constraints, tags, change data feed — fails at execution with the original Databricks error. See runtime and Delta feature compatibility.

Reading a table relies on DESCRIBE TABLE EXTENDED … AS JSON, which needs Databricks Runtime 16.2 or later, or any SQL warehouse. Primary and foreign keys are then read from information_schema; because key constraints are available from Databricks Runtime 13.3 (GA 15.2) — below the AS JSON read floor — any runtime new enough to read a table can also observe its keys. As with every other runtime feature, delta-engine documents this floor rather than preflighting it — an unsupported runtime surfaces as a read failure.

This project does not yet have live Spark-backend evidence for a numbered Databricks Runtime. The recurring live suite exercises the SQL warehouse backend; validate the Spark path in your own production environment until Spark-runtime evidence is published. Databricks Runtime 16.2 is a technical floor, not a runtime allowlist, and compatible newer runtimes remain allowed. See the runtime compatibility policy.

The Spark backend does not currently work on Dedicated access-mode compute (data_security_mode value SINGLE_USER, formerly called single-user access mode). Treat this mode as unsupported.

Delta table features

Declaring a type that needs a Delta table feature — TimestampNtz or Variant anywhere in a column’s type tree — makes the engine plan the enablement itself when the table exists without it: an explicit SET TBLPROPERTIES ('delta.feature.…' = 'supported') statement, ordered before the dependent column change and always visible in a dry run. Creating a table needs no planned enablement; Databricks enables required features from the created schema.

Feature enablement is a permanent protocol upgrade — older Delta clients may lose access to the table. The engine never plans DROP FEATURE, and features it does not model — deletion vectors, row tracking, and every other platform-managed or property-gated feature — are left entirely to the platform and its properties.