Runtime compatibility

This page records how maintainers handle Python, Databricks Runtime, Spark, and Delta compatibility. The policy is intentionally small: the base wheel is pure Python, Databricks supplies the Spark and Delta pair, and delta-engine uses a small PySpark surface around SQL-driven catalog operations.

Supported floors and tested environments are different things. A floor is a requirement; a tested environment is evidence. The tested list must not become an allowlist that rejects compatible newer environments.

Current contract

Surface	Contract
Python	Python 3.12 or later, with no upper bound
Spark backend	Databricks Runtime 16.2 or later, Unity Catalog, and the PySpark and Delta libraries supplied by the runtime
SQL warehouse backend	A Unity Catalog-enabled SQL warehouse; no PySpark or numbered Databricks Runtime required
Dedicated access mode	Unsupported; the Spark backend does not currently work with data_security_mode=SINGLE_USER

Databricks Runtime 16.2 is the Spark backend’s technical floor because the reader requires DESCRIBE TABLE EXTENDED … AS JSON. It is not a claim that every later numbered runtime has been tested.

Python versions

Package metadata expresses the minimum supported Python version and deliberately has no upper bound. A speculative upper bound would reject likely-compatible new Python releases and can make installers select an older delta-engine release instead.

CI runs the full suite on:

• the minimum supported Python version; and

• the latest stable Python version.

Intermediate minor versions remain supported without a dedicated job unless they introduce a distinct compatibility path, such as version-specific code, dependency resolution, or a reproduced defect. The exact built wheel is smoke-tested once on the Python floor; repeating the same packaging check on every Python minor would not prove a different property.

When a new stable Python version appears:

1. add it as the latest CI endpoint;

2. keep the minimum endpoint;

3. fix or narrowly document any demonstrated incompatibility; and

4. test the Spark path separately when a Databricks Runtime adopts that interpreter.

Raise the Python floor only when production code needs newer language or standard-library functionality, a required dependency drops the old version, or maintaining the old version has a concrete cost that outweighs its user value. Do not raise it merely because a new Python version exists or an internal development tool wants a newer interpreter.

A floor increase is a deliberate compatibility change. Announce it, update requires-python, classifiers, CI, the lock, and documentation together, and identify the final delta-engine release that works on the retired Python version.

Databricks Runtime, Spark, and Delta

Do not publish, install, or select pyspark or delta-spark for Databricks users. Databricks owns the compatible Spark and Delta combination in each runtime; shadowing either package from a notebook can create the mismatch the runtime is designed to avoid.

delta-engine therefore does not maintain a Spark-by-Delta dependency matrix, one wheel per runtime, a runtime allowlist, or a maximum Databricks Runtime. The local Spark suite checks delta-engine against the repository’s locked development environment, but it is not evidence that the Spark adapter works on a numbered Databricks Runtime.

When a new Databricks Runtime appears:

1. make no package-metadata change by default;

2. test the existing exact wheel on that runtime before claiming it as tested;

3. if it passes, record the environment as evidence without requiring a new delta-engine release; and

4. if it fails, reproduce the narrow boundary, then fix it or document that concrete incompatibility.

Any published test evidence should identify the exact delta-engine artifact, Databricks Runtime, and access mode. A runtime absent from that evidence remains allowed unless a concrete incompatibility is known.

New Databricks or Delta features

A new platform feature does not change delta-engine’s contract until the project intentionally adopts it. When it does:

• document the feature’s own Databricks Runtime or Delta protocol requirement;

• let the user choose a runtime that provides the feature;

• keep baseline operations available on older supported runtimes; and

• prefer the original Databricks error unless a targeted preflight materially improves safety or avoids partial execution.

An optional feature must not raise the package-wide runtime floor. Raise the global floor only when a baseline operation genuinely requires a newer platform and retaining the old path is impractical.

What each test proves

Evidence	What it supports
Minimum/latest Python CI	Pure-Python and locked integration behaviour at the supported interpreter endpoints
Installed-wheel smoke	The built wheel installs and its base public surface works on the Python floor
Local Spark suite	Delta-engine behaviour against the locked development Spark/Delta pair
Live SQL warehouse suite	Shared SQL/catalog behaviour and the SQL warehouse adapter
Live Spark notebook smoke	The exact wheel and Spark adapter work on the named runtime and access mode

Only the final row establishes evidence for a numbered Databricks Runtime. Until that coverage exists, describe 16.2 as the technical floor and avoid calling any numbered runtime live-tested.

See Installation for the consumer setup and Capabilities and limitations for current user-visible restrictions.