delta-explain

Make Delta Lake file pruning visible.

You run a query with a filter. The engine reads some files. But how many were actually eliminated, and why? delta-explain answers that — from metadata alone, without running a query engine.

Given a Delta table and a WHERE predicate, it reads the transaction log directly (via delta-kernel-rs, never the parquet data) and shows, step by step, which files partition pruning and data skipping would eliminate, and which survive.

$ delta-explain ./users -w "country = 'DE' AND age > 40"

Predicate Analysis:
  partition-safe: country = 'DE'
  stats-safe:     age > 40
  confidence:     conservative

Phase 1: Partition pruning [exact]
  files remaining: 2  (-4, 67% pruned)
Phase 2: Data skipping (min/max statistics) [conservative]
  files remaining: 1  (-1, 50% pruned)

Total reduction: 6 -> 1 files (83% pruned)

What it is for

• Understand pruning. See how a predicate splits across the two elimination mechanisms, and — with --explain-why — get told why a fragment did not prune and what to change.
• Guard it in CI. --min-pruning and --assert-stats turn a silent layout regression (a rewrite that drops partitioning, a query that scans everything) into a failed build. See Gating pruning in CI.
• Build on it. A versioned JSON contract, a Python wrapper, and a self-contained report viewer.

What it is not

delta-explain is not a query engine and not an optimizer. It never reads data files, never predicts runtime, and never overstates: a file any engine needs is never reported as pruned. When it cannot reason about a fragment it keeps files and says so, rather than guessing. The full contract — guarantees, degradation rules, exit codes — is in What delta-explain guarantees.

Positioning, meant literally

Production-usable as a conservative Delta metadata diagnostic and CI guardrail; not yet a fully production-grade general-purpose Delta observability product. The roadmap tracks where that line moves next.

Install

Pick whichever fits your environment. All routes ship the same binary; the Python wheel bundles it plus a thin API.

Homebrew (macOS, Linux)

brew tap cdelmonte-zg/tap
brew install delta-explain

Scoop (Windows)

scoop bucket add cdelmonte-zg https://github.com/cdelmonte-zg/scoop-bucket
scoop install delta-explain

PyPI (no Rust needed)

pip install delta-explain

The wheel ships the compiled binary (the delta-explain command works from the same environment) plus a thin Python API — see From Python.

Cargo (crates.io)

cargo install delta-explain

Docker (amd64 + arm64)

docker run --rm -v /path/to/table:/data ghcr.io/cdelmonte-zg/delta-explain \
  /data -w "col > 10"

For pipelines, pin a release tag or a digest; :latest is for local exploration only.

Pre-built binaries and .deb

Every release attaches archives for six targets (Linux glibc/musl/ARM64, macOS Intel/Apple Silicon, Windows) each with a .sha256, plus .deb packages for amd64 and arm64. Grab them from the latest release. The musl build is statically linked and runs on any Linux distribution.

Quickstart

Three minutes, no cloud account. The repo ships a runnable demo and a small real table.

On a real table (no setup)

The repo includes fixtures/taxi-nyc, a small Delta table written from public NYC TLC yellow-taxi data, partitioned by pickup date:

# date is the partition column: directory-level pruning, exact
delta-explain fixtures/taxi-nyc -w "pickup_date = '2024-01-03'"

# date prunes partitions, fare prunes on min/max stats within them
delta-explain fixtures/taxi-nyc -w "pickup_date = '2024-01-03' AND fare_amount > 50"

# a predicate that does NOT prune - and the tool tells you why
delta-explain fixtures/taxi-nyc -w "PULocationID = 132" --explain-why

That last command is the point of the tool: the pickup-zone column is not clustered, so data skipping cannot help, and --explain-why says so with a fix. See Why isn't it pruning?.

The full demo (Docker)

examples/quickstart/ brings up a MinIO + demo-table stack and walks the gate story end to end (a healthy table, a layout regression the gate catches, the JSON contract). Follow its README:

cd examples/quickstart
docker compose up -d
# then the commands in examples/quickstart/README.md

Next

• Reading a report — what every line means.
• Gating pruning in CI — turn it into a build check.

Reading a report

A text report has four parts. Take:

$ delta-explain ./users -w "country = 'DE' AND age > 40"

Predicate Analysis:
  partition-safe: country = 'DE'
  stats-safe:     age > 40
  unsplittable:   -
  confidence:     conservative

Files in snapshot: 6

Phase 1: Partition pruning [exact]
  predicate:       country = 'DE'
  files remaining: 2  (-4, 67% pruned)
Phase 2: Data skipping (min/max statistics) [conservative]
  predicate:       age > 40
  files remaining: 1  (-1, 50% pruned)

Total reduction: 6 -> 1 files (83% pruned)

Predicate Analysis

Each top-level AND conjunct is routed to the mechanism that can eliminate files on it:

• partition-safe — references partition columns only; prunes at the directory level, exactly.
• partition-exact — outside the kernel's language (an any-shape LIKE) but all its columns are partition columns, so it is evaluated directly against the literal partition values.
• stats-safe — references non-partition columns; prunes on per-file min/max statistics.
• unsplittable — mixes both axes, or contains a construct the tool cannot reason about; applied conservatively (keeps all files) with a diagnostic.

Confidence

Labels how precisely the elimination can be explained, never whether it is correct (it always is, conservatively):

• exact — partition values compared directly; every dropped file provably has no match.
• conservative — min/max ranges can overlap a bound without a match, so files may be kept in excess, never dropped in excess.
• incomplete — part of the predicate could not be attributed to one phase; the totals stay sound, the attribution does not.

Phases

The pruning is decomposed into chained phases — partition pruning first, then data skipping — each showing the predicate fragment it honored and the files remaining. It is a decomposition of what the metadata makes possible, not an execution trace.

Per-file detail

Add --verbose to see each file kept or dropped and why; in JSON mode it emits a machine-readable files[] array. Cap large listings with --limit.

The precise guarantees behind every label are in What delta-explain guarantees.

Why isn't my Delta query pruning files?

Your query filters on a column, but the engine still reads the whole table — the scan is slow, or the cloud bill is bigger than it should be. This page is a diagnostic playbook: find the symptom, get the cause, apply the fix. Every step is something delta-explain measures from the transaction log, without running a query.

First: measure what is actually happening

Point delta-explain at the table with your predicate and read the total pruning:

delta-explain ./your-table -w "your = 'predicate' AND ..."

If the total reduction is near 0%, the query really is scanning everything. Now ask why:

delta-explain ./your-table -w "..." --explain-why

The Why: section names the exact reason — one of the four cases below. Each is a deterministic reading of your table's metadata, not a guess. (More on the flag: The --explain-why advisor.)

Symptom: partition pruning never runs

[NO_PARTITION_FILTER] The table is partitioned by pickup_date, but the predicate
  filters on none of those columns, so partition pruning cannot run.

Cause. Partition pruning eliminates files at the directory level, but only on the partition columns. If your WHERE clause doesn't reference one, the directory structure can't help — every partition is a candidate.

Fix. Filter on a partition column when you can. If your queries genuinely filter on a different column than the table is partitioned by, the table is partitioned for the wrong workload — see the wrong-column trap below.

Symptom: data skipping eliminates nothing

Two different causes wear the same symptom — the tool distinguishes them.

The statistics are there, but the ranges overlap

[WEAK_DATA_SKIPPING] Data skipping eliminated no files for 'fare_amount > 60':
  the per-file min/max ranges all overlap the predicate's bound.

Cause. Data skipping works by comparing your bound against each file's recorded min/max. If every file spans the whole range of the column — because the rows are not ordered by it — no file can be excluded. This is the classic "stats exist but are useless" case.

Fix. Order the data so each file covers a narrow range of the column: sort or cluster the data — for example with Z-order or liquid clustering where available. Then a fare_amount > 60 query touches only the high-fare files.

There are no statistics to skip on

[STATS_ABSENT] The table carries no file statistics, so data skipping cannot
  prune on 'fare_amount > 60'.

Cause. No min/max were recorded for that column — the writer collected none, or the column is past the delta.dataSkippingNumIndexedCols budget (Delta indexes only the first N leaf columns, 32 by default).

Fix. Ensure the writer records statistics for the columns you filter on. If the column is past the indexed-columns budget, reorder the schema or raise delta.dataSkippingNumIndexedCols so it is covered.

Symptom: part of my predicate is "unsupported"

[UNSUPPORTED_FRAGMENT] The fragment 'UPPER(name) = 'X'' is outside the pruning
  language and was applied conservatively, keeping all files.

Cause. Function calls, arithmetic on columns, subqueries, and column-to- column comparisons cannot prune on file metadata — there is no min/max to compare a computed value against. A mixed partition OR data fragment also can't be split cleanly across the two mechanisms.

Fix. For a mixed OR, rewrite it so the partition and data filters are independent conjuncts (AND) where the logic allows. For a function on a column, Delta file metadata cannot prune the computed expression directly — filter on the raw column instead, or precompute the value into a stored, indexed column.

Symptom: I partitioned it, and it got worse

A frequent mistake: partition by a high-cardinality column (a user id, a zone id) and end up with thousands of tiny files — the small-files problem — and still no pruning for the queries that matter.

Measured on real NYC-taxi data, the same trips laid out four ways against a date filter, a fare filter, and both:

Fix. Partition by a column with the right cardinality for how you actually query (a date, not a 260-value zone id), and get file-level skipping by sorting or clustering within the partitions. The full walkthrough is in Tuning a table layout.

Then: keep it from regressing

Once a layout prunes, a later change — a rewrite that drops partitioning, a query that loses its filter — can silently undo it. Gate the pruning in CI so the build fails instead of the cloud bill rising:

delta-explain ./table -w "..." --min-pruning 80

See Gating pruning in CI.

The --explain-why advisor

Counting files tells you that a predicate did not prune. --explain-why tells you why, and what to change. (For a symptom-first playbook, see Why isn't my Delta query pruning files?.)

$ delta-explain fixtures/taxi-nyc -w "PULocationID = 132" --explain-why
  ...
Why:
  [NO_PARTITION_FILTER] The table is partitioned by pickup_date, but the
    predicate filters on none of those columns, so partition pruning cannot run.
    -> Filter on a partition column (pickup_date) to eliminate whole directories
       before data skipping.
  [WEAK_DATA_SKIPPING] Data skipping eliminated no files for 'PULocationID = 132':
    the per-file min/max ranges all overlap the predicate's bound.
    -> Ranges this wide usually mean the data is not sorted or clustered by that
       column; ordering by it may enable skipping.

Deterministic, not a model

Each diagnosis is a function of the report the tool already computed — the classification, the stats coverage, the partition columns, the per-phase pruning. Nothing is predicted. That is deliberate: the why must be as trustworthy as the numbers it explains, and reproducible enough to gate on in CI. (An LLM, if you want prose, is a consumer of the JSON outside the tool, never bundled — see ADR 0007.)

The diagnosis codes

Stable identifiers; new codes are additive.

WEAK_DATA_SKIPPING fires only when every file has statistics: on a partial-stats table the survivors may be kept for missing stats, not overlapping ranges, and the tool will not assert a cause it cannot prove.

In JSON

With the flag, the JSON document gains an additive explain array of {code, severity, message, suggestion}. It is present only with --explain-why, so existing consumers are unaffected. See The JSON report.

Tuning a table layout

A table layout is a bet on which queries matter. delta-explain makes the bet measurable — before you run a query — so you can iterate on partitioning and file layout with numbers instead of guesses.

The repo ships an executable notebook, examples/taxi-optimization/, that walks a real optimization on NYC taxi data. Every number in it is produced by delta-explain on a real Delta table the notebook writes.

The arc

Four layouts of the same trips, measured against three queries (a date filter, a fare filter, both):

The lessons the tool surfaces at each step:

1. The baseline is hopeless — every file spans the whole week, nothing prunes.
2. Partitioning by the wrong (high-cardinality) column makes it worse — 240 tiny files, and the date query still does not prune.
3. The right partition column wins the date axis, and the tool shows exactly which queries it does not help.
4. Partitioning plus ordering prunes both axes — at a file-count cost the tool makes explicit instead of leaving to guesswork.

And the tool predicts step 4 from step 3: running --explain-why on the date-partitioned layout with a fare query returns WEAK_DATA_SKIPPING — "order by that column" — which is precisely the next move.

This is the shift from file counter to pruning advisor: for the queries you care about, how many files does this layout eliminate, and what would improve it?

Gating pruning in CI

The failure mode delta-explain was built to catch: a change quietly breaks pruning — a table gets rewritten without partitioning, a query loses its partition filter — nothing errors, and every downstream scan silently reads the whole table. A gate turns that into a failed build.

The flags

• --min-pruning <PCT> — exit 1 if total pruning is below the threshold.
• --assert-stats — exit 1 if any file is missing statistics.

delta-explain s3://lake/events -w "region = 'eu' AND ts > '2026-06-01'" \
  --min-pruning 80

On failure the report still prints (with result: "fail") and stderr carries ASSERTION FAILED: .... The exit-code contract is precise and stable — see the table in What delta-explain guarantees.

In a pipeline (JSON)

delta-explain ./table -w "..." --min-pruning 80 --format json \
  | jq -e '.result == "pass"'

stdout is always a complete report or empty, so a downstream jq never parses a partial document.

GitHub Action

A composite action wraps the CLI with matching inputs. Pin the release tag:

- uses: cdelmonte-zg/delta-explain@v0.6.0
  with:
    table: s3://lake/events
    where: "region = 'eu'"
    min-pruning: "80"

Attach a report artifact

Generate a verbose JSON report and render it with the report viewer into a self-contained report.html, uploaded as a run artifact — so a reviewer of a failed gate sees which phase did not prune and which files survived, instead of an exit code.

Cloud storage

delta-explain reads s3://, az://, and gs:// tables directly. Credentials come in by environment, by profile, or by explicit options.

Credentials

• On cloud infrastructure (EC2/ECS, EKS, AKS, GKE): with no explicit credentials the storage layer uses the provider's ambient chain (instance profile, Managed Identity, Workload Identity). Add --env-creds when the credentials live in environment variables instead.
• On a laptop (AWS): --profile <name> resolves static keys, session token, and region from ~/.aws/credentials / ~/.aws/config. Profiles that rely on SSO, credential_process, or role assumption are not resolved — export them first and use --env-creds:

  eval $(aws configure export-credentials --profile corp --format env)
  delta-explain --env-creds s3://bucket/table -w "..."
  

• Static keys (MinIO, local dev): pass them via repeated --option KEY=VALUE, expanding from environment variables to keep secrets out of argv.

Examples

# S3 with credentials from the environment
delta-explain --env-creds s3://bucket/path/to/table -w "date = '2024-01-01'"

# S3 public bucket
delta-explain --region us-east-1 --public s3://my-public-bucket/table -w "id > 100"

# Azure
delta-explain --env-creds az://container/table -w "region = 'eu-west-1'"

# GCS (Workload Identity on GKE, or service-account JSON via env)
delta-explain gs://bucket/table -w "day >= DATE '2026-01-01'"

The end-to-end walkthroughs for MinIO/S3 and GCS live in examples/.

Valid --option keys pass through to the object_store builders; see upstream for the per-backend list.

The report viewer

A single self-contained HTML page that renders one JSON report: the pruning funnel, the predicate analysis, gates, the --explain-why diagnoses, and a filterable per-file table that stays usable at hundreds of thousands of files (the text listing does not). No dependencies, no network requests — it works air-gapped and as a CI artifact.

It is a client of the versioned JSON contract, exactly like the Python wrapper: it adds no analysis of its own and renders any saved report.

Use

delta-explain ./table -w "country = 'DE' AND age > 40" \
  --format json --verbose > report.json

Then drop report.json onto viewer/report-viewer.html, or pick it with the button.

For a one-file artifact (e.g. to attach to a CI run), inject the report into the page so it renders itself — the recipe is in viewer/README.md. The killer workflow: after a failed --min-pruning gate in CI, upload the report.html — the reviewer sees which phase did not prune and which files survived, instead of an exit code.

Scale

A 200,000-file verbose report (~51 MB JSON) loads and renders in about a second, because the table only ever mounts the rows in view. Toward a million files the browser's JSON parse dominates — generate the report with --limit there.

From Python

pip install delta-explain ships the compiled binary plus a thin wrapper. It is a client of the CLI-and-JSON contract, not a second API: it shells out and returns the report.

from delta_explain import explain

report = explain("s3://warehouse/events",
                 where="country = 'DE' AND age > 40",
                 min_pruning=80, env_creds=True)

report.passed              # False also makes the CLI exit 1 in CI
report.total_pruning_pct
report["analysis"]["confidence"]   # the report is also a Mapping

Keyword arguments mirror the CLI flags one to one (verbose, limit, explain_why, at_version, profile, region, public, options, ...).

Outcomes

• Gate failure (--min-pruning / --assert-stats) is not an error: it comes back as a Report with passed == False, mirroring the CLI's exit-code contract.
• Runtime errors (unreadable table, bad predicate, storage) raise DeltaExplainError with the CLI's message.

Typed accessors

Report is a read-only Mapping (so report["analysis"]["confidence"] works) plus convenience properties: schema_version, total_files, final_files, total_pruning_pct, result, passed, files (with verbose=True), and explain (with explain_why=True).

Semantics: what delta-explain guarantees

This is the contract. Everything else in the documentation is commentary on these statements.

delta-explain reports metadata-level file pruning over a Delta snapshot.

• It does not simulate a query planner.
• It does not predict runtime.
• It does not compensate for deletion vectors, column mapping, or clustering: it detects and declares them.
• It reports conservative survivor sets: a file any engine needs is never reported as pruned.
• When it cannot reason about a predicate fragment, it keeps files and emits a diagnostic, never a silent wrong number.

The pipeline

Every invocation follows the same linear sequence; there is no hidden concurrency and no adaptive behavior:

1. Baseline scan — one kernel log replay (JSON commits + checkpoint Parquet) enumerates the snapshot's files and their statistics.
2. Parse and normalize — the --where predicate is parsed once into an owned AST, then normalized: negations push down to the leaves (De Morgan), a literal-prefix LIKE on a string column rewrites to a lexicographic range (country LIKE 'D%' becomes country >= 'D' AND country < 'E', in the binary code-point order that partition values and min/max statistics compare with; on any other column type LIKE matches the value cast to a string, which a range cannot express, so the fragment routes to the evaluator or degrades), and conjuncts common to every OR branch factor out of the OR. All rewrites preserve SQL three-valued semantics, so they can change attribution and confidence but never the survivor set.
3. Classify — each top-level AND conjunct is routed to one bucket: partition_safe (references partition columns only), partition_exact (outside the kernel's predicate language — any-shape LIKE — but with fully known semantics and every column a partition column), stats_safe (references no partition column), or unsplittable (mixes both, or contains an opaque construct).
4. Phase scans — the partition-safe fragment runs as its own kernel scan, and the partition-exact fragment is evaluated per file against the literal partition values; their intersection is Phase 1. The full predicate, conservatively stripped of unsupported fragments and intersected with the partition-exact verdict, runs as the final scan.
5. Attribution — the survivor-set difference between phases is attributed: directory-level partition pruning first, min/max data skipping second.
6. Gates and rendering — --min-pruning / --assert-stats evaluate, and the report renders as text or JSON.

Soundness

The one hard guarantee: the survivor set is a superset of the files that contain matching rows. This is a contract verified on the current support surface (the constructs and table shapes in the test matrix and the differential harness), not a theorem about arbitrary future features: when the tool meets something outside that surface, it degrades or refuses loudly rather than extend the claim silently. Pruning may be less aggressive than an engine's (and usually is not), but a pruned file never contains a match.

This is validated continuously by the differential harness (examples/differential): Spark computes, per predicate, the files that actually contain matching rows, and the harness asserts the survivor set covers them, on a matrix that includes rewritten forms (NOT over mixed OR, factored OR-of-ANDs, prefix LIKE ranges), partition-literal evaluation (non-prefix LIKE on partition columns) and null-safe comparisons.

Confidence

Confidence labels how precisely the elimination can be explained, never whether it is correct (it always is, in the conservative direction):

• exact — the phase's elimination is precise: partition values are compared directly, every dropped file provably contains no match.
• conservative — sound but possibly loose: min/max ranges can overlap a predicate's bound without the file containing a matching row. Files may be kept in excess, never dropped in excess.
• incomplete — part of the predicate could not be attributed to a single phase (mixed OR, or an unsupported construct). The totals are still sound; what is lost is the clean attribution, and a diagnostic note says why.

The global confidence is the least informative label across fragments.

Diagnostics (--explain-why)

--explain-why turns the report into advice: a set of diagnoses, each a deterministic function of the report the tool already computed (the classification, the stats coverage, the partition columns, the per-phase pruning). No prediction, no model - the why is as trustworthy as the numbers it explains, and reproducible enough to gate on. Each diagnosis carries a stable code, a severity, a message, and a suggestion where one applies; the stable codes are NO_PARTITION_FILTER, WEAK_DATA_SKIPPING, STATS_ABSENT, and UNSUPPORTED_FRAGMENT. In JSON the diagnoses appear as an additive explain array, present only with the flag. Natural-language phrasing of a diagnosis, if wanted, belongs to a consumer of that JSON (an LLM the user brings), never bundled in the tool.

Degradation rules

Constructs outside the pruning language — function calls, arithmetic, LIKE in any non-prefix shape (leading or embedded wildcards, _, NOT LIKE, ESCAPE), subqueries, column-to-column comparisons — do not abort the run:

• under a top-level AND, the unsupported fragment is dropped from the scan predicate (keeping more files, never fewer) and the sibling conjuncts still prune;
• an OR or NOT touching an unsupported fragment is dropped whole, because its truth value cannot be bounded;
• the fragment is reported as unsplittable, confidence degrades to incomplete, and an UNSUPPORTED_EXPRESSION note carries the reason.

The partition axis has one carve-out. Partition values are exact literals per file, so a fragment whose semantics the tool fully knows (any-shape LIKE today) and whose columns are all partition columns does not degrade: it is evaluated per file against those literals (partition_exact), it prunes in Phase 1, and confidence stays exact. The fragment is constant across a file's rows and a row is selected only when the predicate is TRUE, so a file whose values make it FALSE or NULL is dropped exactly. When the evaluator must abstain instead of deciding (e.g. LIKE over a timestamp column, whose cast-to-string format is engine-dependent), the affected files are kept, confidence downgrades to conservative, and a PARTITION_EVAL_GAP note counts them. Opaque constructs — function calls, arithmetic, subqueries, whose semantics the tool does not model — always degrade by the rules above, partition columns or not.

Malformed SQL is different: it is a user error and fails the run.

Table features: declared, not compensated

Protocol features that distort or reframe the numbers are detected and declared in table_features, with warnings, but the numbers are not adjusted:

• Deletion vectors — record counts include soft-deleted rows on files that carry a vector (DELETION_VECTORS warns with the file count; enabled-but-unused stays silent because the numbers are correct).
• Column mapping — the log stores physical column names; verbose statistics may display them (COLUMN_MAPPING). Kernel pruning itself resolves the mapping.
• Liquid clustering — declared with the clustering columns (LIQUID_CLUSTERING); layout is managed by clustering, not directory partitions, and data skipping still applies. Undetectable on fully checkpointed logs (no public kernel accessor for system metadata domains).

Engines may prune less than this report shows

The report reflects what the metadata makes possible, computed with the strongest sound techniques in production use. Engines make different choices; the known divergences are documented in the README's Current limitations (notably IN-list strategies). The report never overstates correctness — only, potentially, the pruning a specific engine will realize.

Exit codes and the error contract

Two properties are load-bearing for CI:

• stdout is either a complete report or empty. A failure never emits a partial document, so delta-explain ... | jq ... cannot parse garbage.
• Panics are a bug by policy, enforced by compiler lints (clippy::unwrap_used and friends are denied in production code): any defect must surface as exit 1 with a clean message, not exit 101 with a backtrace.
• A consumer that stops reading does not crash the run. When stdout closes mid-report (delta-explain ... | head, a crashed jq), the remaining output is skipped, stderr stays clean, and the exit code still reflects the gates.

Scope: table protocol, not file format

delta-explain explains Delta-level file elimination only: partition pruning and file-level data skipping. Parquet row-group predicate pushdown (filtering inside surviving files on row-group footer statistics) is a file-format concern, deliberately out of scope; it may appear later as a separate mode.

Known blind spots

• Statistics exist only for the first delta.dataSkippingNumIndexedCols leaf columns; predicates past the budget classify as stats-safe but cannot prune, and stats.mode reflects table coverage, not per-predicate reachability.
• On a fully checkpointed log (no JSON commits): an empty partitioned table's partition columns are undetectable, and liquid clustering is undetectable.

The JSON report, field by field

The goal of this page: a consumer can trust the JSON without reading the code. The formal schema lives at schemas/report-v0.4.schema.json and the integration suite validates every emitted document against it; this page explains what the fields mean.

Versioning

The document carries schema_version, versioned independently of the tool: additive changes bump the minor (and ship a new schema file), breaking changes bump the major. Consumers should:

• branch on stable field names and assertions[].name, not on positions;
• tolerate unknown fields (a newer minor may add some);
• check schema_version if they depend on a specific shape.

Always present

Only with --verbose

Only with --explain-why

Per file: path, size_bytes, partition_values (string map), num_records (nullable; includes soft-deleted rows under deletion vectors), has_stats, kept, pruned_by.

• kept: true means the file survives every phase. The set of kept files is the survivor set: guaranteed to be a superset of the files any engine needs for this predicate (see semantics).
• pruned_by names the first phase that eliminated the file (one of the phases[].name values), or null when kept. Phases chain, so "first phase whose survivor set misses the file" is exactly the phase that eliminated it.

confidence

Appears globally (analysis.confidence) and per phase. It labels how precisely the elimination can be explained, never whether it is correct:

• exact — provably precise elimination (partition values compared directly);
• conservative — sound but possibly loose (min/max ranges can overlap a bound without the file containing a match);
• incomplete — part of the predicate could not be attributed (mixed OR, unsupported construct); totals remain sound, a note says why.

Stable note codes

Notes are {code, message} pairs. The message text may be reworded at any time; the code values are stable identifiers, and new codes are additive:

assertions[]

Discriminated by name:

• {"name": "min_pruning", "threshold": N, "actual": N, "result": "pass"|"fail"}
• {"name": "stats_complete", "missing_count": N, "result": "pass"|"fail"}

result at the top level aggregates them; a fail also makes the process exit 1.

Error contract

On any runtime error stdout is empty and the message goes to stderr, so delta-explain ... --format json | jq ... never parses a partial document. The full exit-code table is in semantics.

Consumer patterns

# gate on the overall result
delta-explain ./t -w "..." --min-pruning 80 --format json | jq -e '.result == "pass"'

# survivor set as a file list
delta-explain ./t -w "..." --format json --verbose | jq -r '.files[] | select(.kept) | .path'

# fail a pipeline when deletion vectors appear
delta-explain ./t --format json | jq -e '.table_features.deletion_vectors.files_with_deletion_vectors == 0'

What delta-explain is validated against

The claim "works on real Delta tables" is only as good as the tables it is exercised on. This page lists them, and just as deliberately, what is not covered yet.

Real writers

Two independent Delta implementations write the tables in the test suite:

• delta-rs (the deltalake Python package) generates the canonical fixtures: partitioned, non-partitioned, empty, partial statistics, nested struct columns, statistics budgets, temporal/decimal/narrow-int layouts (fixtures/create_test_table.py).
• Apache Spark 4.1 + Delta 4.3 writes the exotic checkpoint fixtures: classic multi-part checkpoints and V2 UUID-named checkpoints with parquet sidecars (fixtures/create_exotic_checkpoints.py), plus the table behind the differential harness.

The checkpoint-only fixtures (JSON commits removed, the shape log-retention cleanup produces) are derived from generated tables; one has its checkpoint hand-rewritten to carry only structured stats_parsed, a layout deltalake does not emit.

Log shapes

The integration matrix covers: JSON-commit logs, checkpoint-only logs (both stats layouts), multi-part checkpoints, V2 UUID-named checkpoints with sidecars, log compaction (compacted files coexisting with their original commits: no double counting, identical pruning, time travel into the range), and multi-commit logs with time travel at every version.

Protocol features

Synthesized logs (the suite's LogBuilder) exercise detect-and-declare on: deletion vectors (present and enabled-but-unused), column mapping by name with per-field physical names, liquid clustering domain metadata, in-commit timestamps, unknown writer features, and the catalog-managed refusal path.

The differential oracle

examples/differential runs Spark as ground truth over MinIO (S3 API), on two tables - a synthetic users table and a taxi table written by Spark from real NYC TLC data: for each of 29 predicates, Spark computes which files actually contain matching rows, and the harness asserts delta-explain's survivor set covers them. The matrix includes normalized forms (De Morgan pushdown, factored ORs), LIKE in both its rewritten shape (prefix ranges) and its partition-evaluated shapes (non-prefix, NOT LIKE, _) - the latter checked against Spark's own LIKE on the real taxi partition column - and null-safe comparisons. It reruns on every change to predicate semantics and weekly in CI (the Validation workflow); results to date: sound on all, exact on that layout.

Scale

Measured on synthetic logs at 200k files in three shapes: a single JSON commit, 2000 JSON commits, and 2000 commits consolidated by a real kernel-written parquet checkpoint. Numbers in the README's Performance notes; anyone can reproduce them with cargo run --release --example gen_scale_log (see its docstring for the three invocations). The automated regression ceiling is a 1000-file smoke.

Not covered yet, on purpose

• Databricks/Unity-Catalog managed tables: detected and refused with an explanation (their commits live in the catalog, so a filesystem-only analysis cannot be trusted). No support, by declaration.
• Parts of the real-cloud auth surface: the weekly Validation workflow now runs, besides the Spark differential oracle over MinIO and the Azurite az:// smoke, a real-cloud-smoke leg against real S3, Azure Blob, and GCS demo tables with --env-creds (real authentication, endpoints, and regions: the class of the two 0.4.0 cloud bugs, which no emulator caught). Still manual: --profile against real AWS, abfss:///ADLS Gen2, and every auth mechanism the smoke's credentials do not exercise (instance metadata, SSO, workload identity).
• Tables written by engines other than delta-rs and delta-spark (e.g. Trino, Flink): the protocol is the contract and the kernel does the reading, but no fixture in the suite comes from them.
• Unknown writer features: not silently absorbed; declared with an UNRECOGNIZED_TABLE_FEATURE warning.

Architecture

The pipeline is a thin, linear sequence — no hidden concurrency, no adaptive behavior:

1. Baseline scan — one kernel log replay (JSON commits + checkpoint Parquet) enumerates the snapshot's files and their statistics.
2. Parse and normalize — the --where predicate is parsed once into an owned AST, then normalized (De Morgan pushdown, the string-column prefix-LIKE range rewrite, OR factoring). Every rewrite preserves SQL three-valued semantics, so it can change attribution but never the survivor set.
3. Classify — each top-level AND conjunct is routed to partition_safe, partition_exact, stats_safe, or unsplittable.
4. Phase scans — the partition fragments run as a kernel scan intersected with the partition-literal evaluator; the full predicate, stripped of unsupported fragments, runs as the final scan.
5. Attribution — the survivor-set differences become chained, labeled phases.
6. Diagnose, gate, render — --explain-why diagnoses, gates evaluate, and the report renders as text or JSON.

One parse, three interpreters

The owned predicate AST (Pred) sits between sqlparser and everything else, so there is one parse and three independent interpreters that can never disagree:

• the analyzer produces the classification the user reads;
• the kernel bridge produces the predicate the kernel executes;
• the partition-literal evaluator decides what a file's partition values decide (for constructs the kernel cannot lower, like an any-shape LIKE).

Because all three read the same tree, "what the kernel sees", "what the user reads", and "what gets evaluated" cannot drift. Growing the pruning language means widening the AST, never adding a second parser — the decision recorded in ADR 0005 and generalized in ADR 0006.

Diagnostics (--explain-why) are a separate advisor layer over the finished report, not a fourth AST interpreter: a deterministic rules engine, by deliberate design (ADR 0007).

Why metadata-only is a feature

The tool never reads data files. That is what makes it fast (cost scales with the number of add actions, not data volume), sound (it reports what the metadata makes possible), and honest under maintenance operations: VACUUM deletes data files that are already absent from the current snapshot, so it does not affect what the tool reports.

The module-by-module map and the working conventions live in the repo's CLAUDE.md.

Architecture decisions

The why behind the module boundaries and the contracts. Division of labor: this book's Reference states what the tool guarantees; an ADR records why a load-bearing decision was made, what it rejected, and what it constrains.

A decision earns an ADR only when it crosses module boundaries or constrains future work and it rejected a plausible alternative. Records are immutable once accepted; a change of mind produces a new record that supersedes the old.

The full records live under docs/adr/:

The recurring theme: the value is a sound, explainable answer, so every decision protects that — one parse so interpreters cannot drift (0001), loud degradation over silent guessing (0002), compile-time gaps over runtime surprises (0003), a versioned contract as the only stable surface (0004), and a diagnostic layer that is deterministic rather than predictive (0007).

Vision

delta-explain is a metadata-level diagnostic tool for Delta Lake. It makes file elimination visible without executing queries.

The positioning, stated once and meant literally: delta-explain is production-usable as a conservative Delta metadata diagnostic and CI guardrail, not yet a fully production-grade general-purpose Delta observability product. The "not yet" items below are the long-term roadmap; a capability moves across that line through a release note and updated docs, never silently.

This document outlines the planned evolution.

Principles

These apply across all releases:

• Not an optimizer. No query planner simulation, no execution time prediction.
• Metadata only. Reads the transaction log and file statistics. Never touches data files.
• Value comes from the semantic model, not from feature count. Each release should deepen understanding of why pruning works or fails, not just count files.

v0.1: Soft launch (shipped April 2026)

The tool works end-to-end for the common case: given a predicate and a Delta table, it shows how many files are eliminated by partition pruning and data skipping.

What works:

• Partition pruning and data skipping phases, reported separately
• SQL predicates: comparisons, AND/OR/NOT, IN, BETWEEN, IS [NOT] NULL
• Per-file verbose output showing kept/dropped with reason
• JSON output for programmatic consumption
• CI assertions (--min-pruning, --assert-stats)
• Local and cloud storage (S3, Azure, GCS)

v0.2: Confidence and classification (shipped May 2026)

Released as the v0.2.0 → v0.2.3 series. The tool now explains why pruning worked or failed, not just the file counts.

• Confidence model: each result tagged as exact, conservative, or incomplete depending on stats completeness and predicate separability
• Predicate classification: each clause explicitly labeled as partition_safe, stats_safe, or unsplittable, with coded notes explaining why
• Stable JSON schema (v0.1.0): versioned, documented, with analysis/notes/assertions blocks
• Expanded test fixtures: tables with partial stats; OR-mixed predicates classified as unsplittable and covered by the existing canonical fixture
• Distribution: crates.io, Docker (multi-arch), pre-built binaries for six targets, .deb packages, Homebrew tap, Scoop bucket

The architecture and design rationale behind v0.2 are written up in detail in the companion deep-dive: delta-explain: Making Delta Lake Pruning Visible.

v0.3: Checkpoint support and type hardening (shipped July 2026)

Works reliably on production tables that have been checkpointed and vacuumed. The whole analysis rides on the kernel's log replay, which reads checkpoint Parquet natively: one source of truth for counts, per-file statistics, and assertions.

• Checkpoint coverage: per-file statistics come from the stats payload the kernel carries on each scan row, JSON commits and checkpoint Parquet alike, in both layouts (add.stats JSON and structured stats_parsed). No more [no stats] on long-lived tables, no more --assert-stats false positives.
• Partition columns on checkpoint-only logs: the metaData action in the JSON commits is the primary source; when log cleanup has removed it, partition columns fall back to the kernel-replayed partitionValues keys, preserving the two-phase attribution.
• delta-kernel-rs 0.24, object_store 0.13.
• Internal architecture: lib/bin split; scan, attribution, gates, render, and error modules; the attribution arithmetic and the CI gates are pure and unit-tested.

Shipped in the same v0.3.0 release: type hardening and the production sprint. Date and timestamp coercion was the gate to real tables: most production tables are partitioned or clustered by date, and a predicate that cannot be typed cannot prune.

• Type coercions: DATE, TIMESTAMP (UTC-normalized), TIMESTAMP_NTZ (wall-clock, offsets rejected as ambiguous), DECIMAL (exact scale), and narrow integers, resolved against the Delta schema; DATE '...' / TIMESTAMP '...' literal forms accepted
• Time travel: --at-version <N>
• Alongside: --profile (AWS shared config), the composite GitHub Action, and the first differential harness

v0.4: Predicate analysis, production tables, the public contract (shipped July 2026)

One release for the whole cycle. Goal: reduce false negatives for common patterns without becoming an optimizer, be honest on tables that look like production, and make the contract externally verifiable. The substrate comes first: an owned minimal predicate AST, produced by a small converter from sqlparser (one parse, two interpreters: kernel lowering and classification). Owning SQL lexing forever is the wrong trade, so sqlparser stays as the front end; the converter is the only module coupled to it, and everything outside the pruning language collapses into an explainable Unsupported leaf at that boundary. The rewrites below operate on the owned AST.

• Light normalization: flatten nested ANDs, push negations down to the leaves (De Morgan, three-valued-logic safe)
• OR factoring: factor conjuncts common to every OR branch out of the OR, so (col = 'A' AND x) OR (col = 'A' AND y) exposes col = 'A' as a partition-safe top-level conjunct (a single-column OR like col = 'A' OR col = 'B' already classifies as partition-safe)
• Unsplittable explanations: for each unsplittable fragment, explain why it couldn't be classified (mixed columns, function calls, etc.), and degrade unsupported expressions to a conservative keep-all with a diagnostic warning instead of a fatal error

That is the complexity ceiling for the predicate analyzer: anything beyond crosses into optimizer territory. Also in v0.4: IS [NOT] DISTINCT FROM (the kernel's native null-safe comparison), and NULL literals typed from the column.

Production-table honesty (formerly the v0.5 plan, delivered in the same cycle):

• Detect and declare protocol features: deletion vectors (record counts overcount and the report says so), column mapping (physical vs logical names), liquid clustering. Declared in a table_features JSON block with warnings; the numbers are never silently compensated.
• Per-file machine-readable output: --verbose --format json emits a files[] array (kept, pruned_by, sizes, partition values), capped by --limit; the differential harness consumes it instead of scraping text.
• Scale, measured: 200k files in ~1.6 s / ~320 MB, linear; numbers and the ~1M-file memory ceiling documented in the README.
• Exotic log shapes, proven: log compaction, classic multi-part checkpoints, and V2 UUID-named checkpoints with sidecars are covered by the test matrix (the kernel handled them; now that is a guarantee, not an assumption).

And the public contract, stabilized:

• docs/semantics.md: what the tool guarantees, what it deliberately does not do, degradation rules, exit codes.
• A formal JSON Schema (schemas/report-v0.2.schema.json) validated by the integration suite against every emitted document shape, plus the field-by-field reference in docs/json-schema.md.
• RELEASE.md and a repeatable three-minute quickstart (examples/quickstart/).

The road that led here

Every remaining professionalization item has landed: the "validated against" story (docs/validation.md, including the declared managed-table boundary), CI test jobs on macOS and Windows behind the required gate, reproducible benchmark tooling (examples/gen_scale_log.rs, three log shapes), supply-chain checks (dependabot plus cargo-audit, which found and fixed three real advisories on its first run), and the example notebooks re-verified end to end against MinIO and real GCS.

After 0.4.0 the plan is deliberate: a stabilization period of bug fixing and validation only, no new surface. The roadmap below resumes after that.

v0.5 - v0.6: Diagnostic layer and the report viewer (shipped July 2026)

Goal, delivered: shift from "file counter" to "pruning advisor", and make a report readable without a terminal.

• --explain-why: a deterministic diagnostic engine (ADR 0007) that turns the report into advice - which fragment blocked pruning and what to change - with stable codes (NO_PARTITION_FILTER, WEAK_DATA_SKIPPING, STATS_ABSENT, UNSUPPORTED_FRAGMENT). No ML, nothing predicted: the why is as trustworthy as the numbers it explains, and gate-able in CI. JSON gains an additive explain array (schema_version 0.4.0).
• The report viewer (viewer/): a self-contained HTML page that renders a --format json --verbose report - pruning funnel, predicate analysis, gates, diagnoses, and a filterable per-file table that stays usable at 200k files. A client of the JSON contract (no new analysis, no CLI change), air-gapped and CI-artifact safe.

Still planned in this track:

• --engine-profile: emulate a specific engine's pruning strength (max | datafusion | spark | kernel) so a CI gate asserts what your engine will do, not the metadata's theoretical best. The known divergences (IN-list strategies) are documented in the README today; this makes them selectable.

Later: Fidelity and coverage (planned)

Moving declared boundaries across the line, one release note at a time:

• Deletion vectors: compensate, not just declare. The DV descriptor carries cardinality; numRecords - cardinality is the live record count. Cheap, but it changes a reported number's meaning, so it ships with a schema note.
• Batch mode (--predicates <file>): one log replay, N reports; also the natural place for task-level parallelism, which stays out of the single-invocation core by design.
• One log replay per invocation: today a predicate costs up to three metadata scans; on remote tables that is the real latency. Redundancy elimination before any parallelism.
• LIKE: shipped in v0.5.0, ahead of this list - prefix patterns rewrite to a lexicographic range and prune on both axes, and any-shape LIKE on partition columns is evaluated exactly against the literal partition values (partition_exact).
• JSON schema 1.0: declared only after 0.2 has soaked unchanged under real consumers for a few months.

Adoption track

• AWS shared-config profiles: shipped in v0.3.0 (--profile; SSO and credential_process produce an actionable error pointing at aws configure export-credentials)
• GitHub Action: shipped in v0.3.0 as a composite action at the repo root, one-line uses: with CLI-mirroring inputs and step outputs
• Next: full SSO resolution if demand shows up, and richer Action outputs

Trust track

• Differential testing: the same predicates over the same tables through a reference engine, asserting the survivor set covers every file with matching rows. The harness in examples/differential (MinIO + Spark 4.1) runs a 29-predicate matrix over two tables - a synthetic users table and a taxi table written by Spark from real NYC TLC data - including normalized forms (De Morgan, factored ORs), LIKE in both its rewritten and partition-evaluated shapes (the latter checked against Spark's own LIKE on real partition values), and null-safe comparisons: sound on all, exact on many. It consumes the JSON files[] contract instead of scraping text (the "validated against" documentation shipped in docs/validation.md). A scheduled weekly Validation workflow now runs the harness and an Azurite leg for the az:// path (a GCS emulator leg is not possible today: object_store has no emulator support for gs://, so GCS coverage stays a manual verification against a real bucket). Next: fixtures written by third-party engines (Trino, Flink).

Ongoing: the kernel track

delta-explain builds on delta-kernel-rs, so some improvements arrive by adopting a newer kernel rather than writing code here:

• Void schema type: tables containing a void column are unreadable today; support landed on kernel main after 0.24 and ships with the next release.
• thrift advisory (issue #9): drops out of the dependency tree once a kernel release moves to parquet 59+.
• Public partition-columns accessor: if the kernel exposes one, it replaces the checkpoint-only fallback and also covers the empty-table edge.
• Public accessor for system metadata domains: delta.clustering is read from the JSON commits today because the kernel rejects system domains on its public API; an accessor would also close the checkpoint-only clustering blind spot.
• Data skipping on the kernel's binary In: today it evaluates conservatively; if a kernel release adds skipping, our IN-as-OR expansion becomes an optimization choice instead of a necessity.

Future: Compare mode

• Same predicate across two tables (flat vs partitioned, before vs after compaction)
• Side-by-side output with delta highlighted

Library surface: the Python half is shipped

pip install delta-explain ships platform wheels containing the compiled CLI plus a thin Python module (explain() -> Report): a pruning check as a function call in a notebook or a pytest fixture, with gate failures as outcomes and runtime errors as exceptions - the CLI's exit-code contract in Python types. Deliberately a client of the CLI-and-JSON contract, not a second API: in-process PyO3 bindings (the polars model) and a stable Rust facade remain future work, taken up only if demand shows, and the explain() signature is designed to survive that swap unchanged.

Changelog

All notable changes to delta-explain are documented here.

The format is based on Keep a Changelog, and from v0.2.0 onwards the project follows Semantic Versioning. The JSON output carries an explicit schema_version field whose changes also follow SemVer relative to that field.

[Unreleased]

Nothing yet.

[0.6.0] — 2026-07-05

The release that turns the explainer into an advisor, and makes a report readable without a terminal. --explain-why adds a deterministic diagnostic layer — why a fragment did not prune and what to change — built as a rules engine over data the tool already computes, never a model, so the answer stays as trustworthy as the numbers and gate-able in CI. A self-contained report viewer renders a JSON report as a navigable page (pruning funnel, analysis, diagnoses, a per-file table usable at 200k files), and a new documentation site gives the now-larger project a single navigable home. The JSON contract grows additively (schema_version 0.3.0 → 0.4.0, a flag-gated explain array). The survivor-set oracle now runs on real NYC-taxi data in addition to synthetic tables, and the architecture decisions behind the diagnostic layer are recorded in docs/adr/0007.

Added

• --explain-why: a deterministic diagnostic layer that turns the report into advice (ADR 0007). Each diagnosis is a function of data the tool already computes - no ML, no prediction, so the why is as trustworthy as the numbers and reproducible enough to gate on. v1 codes: NO_PARTITION_FILTER (predicate misses the partition column), WEAK_DATA_SKIPPING (stats present but the min/max ranges overlap the bound, so nothing is eliminated; the fix - sort/cluster by the column - is a recommendation, not a proven-layout claim), STATS_ABSENT, UNSUPPORTED_FRAGMENT. Additive JSON change: an explain array present only with the flag, schema_version 0.3.0 -> 0.4.0, new schemas/report-v0.4.schema.json. The report viewer and the Python wrapper render the new field.

• A report viewer (viewer/): one self-contained HTML page that renders a --format json --verbose report - pruning funnel, analysis buckets, gates, warnings, diagnoses, and a filterable per-file table that stays usable at 200k files. A pure client of the versioned JSON schema (#89): no new analysis, no CLI surface change, works on any saved report and as an air-gapped CI artifact.

• A documentation site (book/, mdBook -> GitHub Pages at https://cdelmonte-zg.github.io/delta-explain/): a navigable, searchable home with guides, reference, and architecture. The reference and project pages are symlinks into the canonical docs/, VISION.md, and CHANGELOG.md, so nothing is duplicated.

• Real-data examples: fixtures/taxi-nyc, a small Delta table written by a real writer (deltalake) from public NYC TLC data, and an executable examples/taxi-optimization notebook that uses delta-explain to drive a layout optimization. The differential harness gained a taxi table so the Spark survivor-set oracle runs on real data too.

[0.5.0] — 2026-07-05

The LIKE release, and the release where the predicate pipeline became three interpreters over one owned AST. Prefix LIKE patterns rewrite to lexicographic ranges and prune on both axes; every other LIKE shape on partition columns is evaluated exactly, per file, against the literal partition values - a new interpreter (partition_eval) whose four-valued logic drops on SQL NULL as surely as on FALSE and keeps, loudly, on its own ignorance. The JSON contract grows additively (analysis gains partition_exact, schema_version 0.2.0 -> 0.3.0), degraded text phase lines stopped printing stripped fragments as if they contributed, and the architecture decisions behind all of it are now records in docs/adr/, including the one the implementation falsified and refined. The Spark differential matrix grew to twenty predicates and stayed green.

Added

• Prefix LIKE patterns now prune instead of degrading (#72): normalization rewrites col LIKE 'abc%' into the lexicographic range col >= 'abc' AND col < 'abd' (and a wildcard-free pattern into plain equality), so the fragment classifies and prunes through the ordinary column rules on both the partition and the stats axis. The rewrite is exact under SQL three-valued semantics over binary code-point order, and applies to string columns only: on any other type LIKE matches the value cast to a string, which a comparison range cannot express. Non-prefix patterns, NOT LIKE, and ESCAPE forms keep the conservative degradation path. No JSON schema change: the fragment strings simply show the rewritten form.
• Partition-only fragments outside the kernel's predicate language now evaluate exactly against the literal partition values instead of degrading (#75): country LIKE '%E' on a partitioned table prunes in Phase 1 with confidence: exact, in any pattern shape including NOT LIKE. A file whose partition values make the fragment FALSE or NULL is dropped exactly (the fragment is constant across the file's rows, and SQL selects a row only on TRUE); when the evaluator must abstain (e.g. LIKE over a timestamp column, whose cast-to-string format is engine-dependent) the files are kept and confidence downgrades with a new PARTITION_EVAL_GAP note. Opaque constructs (functions, arithmetic, subqueries) still degrade, partition columns or not. Additive JSON change: the analysis block gains partition_exact, schema_version moves to 0.3.0, and the formal schema ships as schemas/report-v0.3.schema.json.

Changed

• Degraded phase lines in text output no longer print a stripped fragment as if it contributed (#76): the line shows what actually reached the kernel scan plus an annotation, e.g. predicate: age > 40 (+1 unsupported fragment, keeps all files). Mixed-axis fragments that the kernel does scan with (an unsplittable OR) keep the full display: only attribution is lost there, not pruning. Text only; the JSON phases[].predicate field is unchanged.

Fixed

• A consumer closing stdout mid-report (delta-explain ... | head, a crashed jq) made the process panic with a backtrace (exit 101), violating the error contract. The write error is now handled: output stops, stderr stays clean, and the exit code still reflects the gate verdict. Text rendering moved to fallible writes and JSON rendering is now separated from printing along the way.

[0.4.0] — 2026-07-03

This cycle added no single big feature; it gave the tool a public contract, and then verified it. The predicate pipeline was rebuilt on an owned AST with normalization and diagnostic degradation; protocol features are detected and declared instead of silently distorting numbers; the JSON output is now a formal, CI-enforced schema (schema_version 0.1.0 -> 0.2.0, additive); the semantics, the validation story, and the release process are written down; scale is measured and reproducible; the test suite runs on Linux, macOS, and Windows with dependency audits; a weekly Validation workflow runs the Spark differential oracle and an Azure emulator smoke; and pip install delta-explain ships the CLI inside platform wheels with a thin Python API. Real-cloud verification on AWS, GCS, and Azure found and fixed two release-blocking bugs before any user could (az:// paths and --env-creds were both broken); the negative-path and edge-case sweeps found three more. Positioning, meant literally: production-usable as a conservative Delta metadata diagnostic and CI guardrail, not yet a fully production-grade general-purpose Delta observability product.

Added

• pip install delta-explain. Platform wheels ship the compiled CLI binary plus a thin Python module wrapping the JSON contract: explain() returns a Report (mapping plus typed accessors), gate failures come back as passed == False, runtime errors raise DeltaExplainError with the CLI's message - the exit-code contract of docs/semantics.md, in Python types. No Rust toolchain involved; the module is a client of the CLI-and-schema contract, not a second API surface.
• Scheduled validation workflow. The checks too slow for per-PR CI now run weekly and on demand: the Spark differential harness in fresh mode, and an Azurite end-to-end smoke of the az:// path (the automated version of the manual check that caught the az:// regression).

Fixed

• Azure tables work again. The direct log reader derived its path by re-parsing the table URL into a second object store, which on az:// requires the account credentials it does not have: every Azure run failed with "Account must be specified". The prefix now derives from the URL path alone (the real store, already scoped and authenticated, was in hand the whole time). Verified end to end against Azurite, including gates, normalization, and per-file JSON; S3 (differential harness) and real GCS re-verified unaffected.
• --env-creds reads the environment now. The flag passed an allow_env option that no layer ever consumed: object_store's parse path silently drops unknown keys and never consults the environment, so the flag was a no-op that fell through to the instance-metadata chain - accidentally working on EC2, hanging toward IMDS anywhere else. The well-known variables (AWS key/secret/session token/region/ endpoint, Azure account name/key, Google service-account paths) are now read by the CLI itself and injected as explicit store options, layered under --profile, flags, and --option. This also makes the GitHub Action's env-creds input work outside AWS runners with instance roles. Verified against real S3 (both auth chains, gates and diagnostics).

Added

• The managed-table boundary, declared. Catalog-managed tables (catalogManaged / catalogOwned-preview reader features) now fail before the kernel scan with a plain explanation: their latest commits live in the catalog, not the filesystem log, so a filesystem-only analysis cannot be trusted. In-commit timestamps are declared in table_features.in_commit_timestamps (no warning: nothing the tool reports depends on them), and writer features outside the known-benign set earn an UNRECOGNIZED_TABLE_FEATURE warning listing them, so a table from a newer writer is flagged instead of silently half-read. Additive schema change within 0.2.0 (two new table_features fields).

• The public contract, written down and enforced. docs/semantics.md states what delta-explain guarantees (conservative survivor sets, degradation with diagnostics, confidence meaning, exit-code table) and what it does not do (no planner simulation, no runtime prediction, no feature compensation). schemas/report-v0.2.schema.json is a formal JSON Schema of the report, strict on purpose, and the integration suite validates every emitted document shape against it; docs/json-schema.md explains each field, the verbose-only fields, and the stable note codes. The README gains a documentation pointer and a measured "Performance notes" section (200k files: ~1.6 s, ~320 MB, linear).

• Exotic log shapes proven. The test matrix now covers log compaction (synthesized <start>.<end>.compacted.json files: no double counting, identical pruning, time travel through the compacted range), classic multi-part checkpoints, and V2 UUID-named checkpoints with parquet sidecars (two new Spark-written fixtures, fixtures/create_exotic_checkpoints.py regenerates them). The kernel handled all of them already; the matrix makes that a tested guarantee instead of an assumption.

• Detect-and-declare for protocol features. The report now declares the table features that distort or reframe its numbers, without changing how pruning is computed: deletion vectors (enabled flag and count of files actually carrying vectors; when present, a DELETION_VECTORS warning says record counts include soft-deleted rows), column mapping (COLUMN_MAPPING: the log stores physical column names, verbose statistics may display them), and liquid clustering (LIQUID_CLUSTERING with the clustering columns, read from the delta.clustering domain; delta-kernel 0.24 exposes no public accessor for system domains, so on a fully checkpointed log clustering goes undetected, the same blind spot as partition columns). JSON gains an additive table_features block; table warnings share the text "Warnings!" section with analyzer notes and now print even when no predicate is given.

• Per-file detail in JSON (--verbose) and --limit. With --verbose, the JSON document gains a files array (per file: path, size_bytes, partition_values, num_records, has_stats, kept, and pruned_by, the phase that eliminated it) plus a files_truncated flag; --limit N caps per-file listings in both the JSON array and the text phase listings, with a ... and N more files tail note. Without --verbose no files array is emitted and the existing summary fields are unchanged, while schema_version moves to 0.2.0 because the verbose schema gained additive fields. This is what makes the output usable on large tables (a 200k-file verbose listing is 25 MB of text) and gives machine consumers the survivor set without parsing text: the differential harness now reads files[].kept instead of scraping [KEPT] lines.

• Predicate normalization. Predicates are normalized before classification: negations push down to the leaves (De Morgan, three-valued-logic safe), and conjuncts common to every OR branch factor out of the OR. NOT (country = 'DE' OR age > 30) now splits into a partition-pruning phase and a data-skipping phase instead of being one unsplittable fragment, and (country = 'DE' AND x) OR (country = 'DE' AND y) exposes country = 'DE' as partition-safe. Rewrites preserve SQL semantics, so survivor sets do not change; what improves is attribution and confidence.

• IS [NOT] DISTINCT FROM (null-safe comparison). Maps to the kernel's native Distinct predicate, which is evaluated exactly over partition values; file statistics cannot prune on it, so on data columns it keeps files conservatively. This closes the last gap between the SQL surface and the predicate language of delta-kernel 0.24.

• Diagnostic degradation for unsupported expressions. Constructs outside the pruning language (function calls, arithmetic, LIKE, subqueries, column-to-column comparisons) no longer abort the command. The fragment is reported as unsplittable with an UNSUPPORTED_EXPRESSION warning explaining why, it conservatively keeps all files, and the remaining conjuncts still prune. Malformed SQL still fails. JSON change is additive (a new note code), schema_version unchanged.

Fixed

• --format rejects unknown values. A typo like --format jsn silently fell back to text output; in a pipeline that means jq parsing a text report. It is now a usage error listing the accepted values.
• --min-pruning without --where is a usage error. It previously ran and failed as "total pruning 0.0% is below threshold", hiding the actual mistake; the help text always documented the requirement, now the parser enforces it.
• Stats payloads that are valid JSON but not an object count as missing. A stats blob like "some string" or [1,2] produced a present-but-empty entry: the file passed --assert-stats and stats.mode stayed exact while contributing nothing to skipping. Such payloads now count as missing statistics, like unparseable JSON already did.

Changed

• The predicate pipeline is rebuilt around an owned minimal AST: one parse, two interpreters (kernel lowering with schema-driven coercion, and classification). Reported fragments now render from the normalized AST, so cosmetic differences can appear (!= prints as <>, flipped comparisons normalize to column-on-the-left, negations appear pushed down). Behavior parity was validated against the differential harness: survivor sets are identical to v0.3.0 on the full predicate matrix.

[0.3.0] — 2026-07-02

One release for the whole cycle: the kernel-backed statistics pipeline plus the production-readiness sprint. The JSON schema_version is unchanged (0.1.0).

Added

• --at-version <N> (time travel). Analyze the table at a historical version instead of the latest: the kernel replays the log only up to that version, so pruning is computed against the historical layout. Enables before/after comparisons around OPTIMIZE or a careless rewrite.

• Temporal and narrow-type literal coercion. Predicates on DATE, TIMESTAMP, TIMESTAMP_NTZ, DECIMAL, SHORT, and BYTE columns now resolve their literals against the Delta schema, so event_date = '2026-07-02' prunes date partitions instead of being kept conservatively as a string-vs-date mismatch. Quoted strings coerce by column type; the explicit DATE '...' / TIMESTAMP '...' forms are also accepted (for TIMESTAMP, RFC 3339 offsets normalize to UTC and bare dates mean midnight UTC; TIMESTAMP_NTZ is wall-clock and rejects offset-bearing literals as ambiguous). Decimal literals must fit the column scale; rounding a predicate bound silently would change its meaning. New test-table-temporal fixture: date-partitioned, with timestamp, decimal, and narrow-int per-file ranges.

• --profile <name> (AWS shared config). Resolves static credentials, session token, and region from ~/.aws/credentials and ~/.aws/config, the same files the AWS CLI reads. Closes the laptop gap where --env-creds fell through to instance-only providers. SSO, credential_process, and role-assumption profiles are not resolved; the error points at aws configure export-credentials.

• GitHub Action. The repo doubles as a composite action: uses: cdelmonte-zg/delta-explain@<ref> installs a released binary and runs the gate with CLI-mirroring inputs, failing the job when an assertion fails and exposing pruning-pct, final-files, and result as step outputs. An action-smoke workflow exercises pass and fail paths against the committed fixtures.

• Differential harness (examples/differential): MinIO + Spark stack where Spark computes, per predicate, the files that actually contain matching rows, and the harness asserts delta-explain's survivor set covers them. Sound on the full ten-predicate matrix (equality, ranges, AND, the unsplittable OR, IN, BETWEEN, NOT, float bounds, empty result), and on the reference layout exact: every kept file contains a match.

• test-table-checkpointed and test-table-checkpointed-struct fixtures: three appends, a checkpoint at v2, every JSON commit removed — the shape delta.logRetentionDuration cleanup produces. The struct variant carries stats only as the stats_parsed column (hand-rewritten checkpoint, since deltalake always writes JSON stats). Integration tests lock in stats display, pruning, and --assert-stats behavior on both layouts.

Changed

• Per-file statistics are sourced from the kernel scan. The verbose view reads the stats payload the kernel carries on each scan row, produced by the same log replay that drives the pruning counts, checkpoint Parquet included. On long-lived tables after log cleanup, files referenced only by a checkpoint now show real min/max statistics instead of [no stats]. Both checkpoint layouts are covered: add.stats JSON and structured stats_parsed (delta.checkpoint.writeStatsAsJson=false); the scan requests the parsed stats schema so the kernel falls back to COALESCE(add.stats, ToJson(add.stats_parsed)). A malformed stats payload now counts as missing statistics instead of passing as an empty entry.
• --assert-stats no longer false-positives on checkpointed logs. A file is flagged only when its add action genuinely carries no statistics. On tables whose older commits were consolidated into a checkpoint, the assertion previously failed even though statistics existed; it now passes. If a pipeline relied on that failure as a proxy for "the log was checkpointed", it will now (correctly) pass.
• delta-kernel-rs 0.20 → 0.24 (and object_store 0.12 → 0.13). No behavioral change beyond the stats sourcing above; the full test suite passes unchanged on the new kernel.

Fixed

• Partition columns on fully checkpointed logs. Partition columns are identified from the metaData action in the JSON commits, which no longer exists after log cleanup consolidates everything into a checkpoint; the two-phase attribution collapsed into a single data-skipping phase (totals stayed correct). When no metaData is found, partition columns now fall back to the partitionValues keys the kernel replays out of the checkpoint: protocol data per add action, not an inference from paths. New test-table-checkpointed-part fixture locks the two-phase attribution in on a checkpoint-only partitioned log.

[0.2.3] — 2026-06-20

Data skipping on nested (struct) columns addressed by dotted paths (profile.age). The JSON schema_version is unchanged (0.1.0): per-column statistics are surfaced only in verbose text output, so this is additive.

Added

• Nested stats display. Per-file statistics for struct columns are now flattened to dotted leaf keys (profile.age: 25..35, profile.score: 75.3..92) in --verbose output, instead of the raw struct object blob. Works at any nesting depth (profile.geo.zip). New test-table-nested fixture exercises the path; test-table-stats-budget documents the ceiling — each nested leaf counts toward dataSkippingNumIndexedCols, so a wide struct can starve later columns of statistics.
• Runnable cloud examples. examples/minio-s3/ and examples/gcs/ provide end-to-end notebooks and Docker Compose stacks so contributors can reproduce the S3 and GCS paths locally without a cloud account.

Fixed

• Type coercion on nested columns. A predicate comparing a nested double/long/float leaf to an integer literal (e.g. profile.score > 90) no longer aborts with Invalid comparison operation: Float64 > Int32; the literal now coerces to the nested leaf type resolved from the schema.
• S3/GCS bucket-prefix resolution. delta-explain now reads a Delta table at any bucket sub-prefix, not only at the bucket root. Previously, tables located under a prefix (e.g. s3://my-bucket/warehouse/sales/) were never resolved and the tool exited with a storage error. Fixes #3.

[0.2.2] — 2026-05-09

This release adds binary distribution channels and aligns the README with the companion deep-dive article. The binary itself is unchanged from v0.2.1; the JSON schema_version remains 0.1.0.

Added

• --version / -V flag on the CLI. Reports the binary version (derived from CARGO_PKG_VERSION at build time). Useful for support reports and for verifying which release a CI runner picked up.
• Pre-built binaries on every tagged release for six targets: Linux x86_64 (glibc and musl), Linux aarch64, macOS x86_64 and aarch64, Windows x86_64. Each archive ships with an SHA256 checksum.
• Debian/Ubuntu .deb packages (amd64 and arm64) built via cargo-deb and uploaded as Release assets. Install with sudo dpkg -i delta-explain_<version>_<arch>.deb.
• Homebrew tap at cdelmonte-zg/homebrew-tap. Install with brew install cdelmonte-zg/tap/delta-explain. The formula is regenerated automatically with fresh SHA256s on every tagged release.
• Scoop bucket at cdelmonte-zg/scoop-bucket. Install with scoop bucket add cdelmonte-zg https://github.com/cdelmonte-zg/scoop-bucket && scoop install delta-explain. Manifests are auto-updated by Scoop's Excavator workflow (every three hours).
• MSRV declared: rust-version = "1.88" in Cargo.toml (let-chains).
• LICENSE file (MIT) at the repository root, required by cargo-deb and other distribution channels.

Documentation

• README rewritten for clarity and alignment with the deep-dive article:
  • Install section reorganised: Homebrew, Scoop, .deb, pre-built binaries listed alongside crates.io, Git, and Docker.
  • Current limitations rewritten as headline + paragraph. Corrected the previous slip on stats.mode (per-table coverage, not per-predicate reachability). Clarified that older JSON commits are removed by Delta's log cleanup (governed by delta.logRetentionDuration), independently from VACUUM.
  • "How it works" step 3 now scans the original predicate, covering the unsplittable case correctly. The analyzer's vocabulary is threaded through to the scan steps.
  • conservative confidence now mentions overlapping ranges, not just missing stats.
  • JSON output framed as pre-1.0 (additive minor, breaking major), not "stable contract from v0.2.0".
  • New [!IMPORTANT] callout in Cloud storage explaining that --env-creds does not currently pick up AWS_* env vars or AWS_PROFILE on a developer laptop. On EC2/ECS/EKS/GKE/AKS the default credential chain works as expected.

[0.2.1] — 2026-05-08

Documentation

• README refreshed for the v0.2.0 schema: the main example now shows the Predicate Analysis block and per-phase [confidence] tags, the JSON output section documents the v0.1.0 schema fields (analysis, stats, assertions, result, version metadata), and the OR-mixed limitation is rephrased to reflect that those predicates are now explicitly classified as unsplittable rather than silently downgraded.

[0.2.0] — 2026-05-08

This release closes the FASE 0 / FASE 1 set in the roadmap and freezes the JSON output as a stable contract. It is breaking for anyone parsing the previous JSON shape.

Breaking

• JSON schema v0.1.0. The output now carries top-level schema_version ("0.1.0"), tool_version, elapsed_ms, an analysis block, a stats block (with categorical mode ∈ {exact, partial, absent}), an assertions array, and a result field (pass / fail / null).
• stats_coverage is gone — replaced by stats, which includes the new mode field.
• phases[].files[] (the per-file detail array) has been removed from the JSON output. Per-file information remains available in the text output via --verbose. The stable JSON schema is summary-only.
• Assertions are now evaluated before the JSON is printed, so the document always reflects their outcome and the top-level result.

Added

• Predicate analyzer (src/predicate_analyzer.rs). Splits top-level AND predicates into partition_safe, stats_safe, and unsplittable fragments and emits coded notes (e.g. UNSPLITTABLE_OR) when fragments cannot be separated.
• Confidence model. Each run reports an overall confidence (exact / conservative / incomplete) plus a per-phase confidence tag, derived from which buckets the predicate populated.
• Predicate Analysis section in the text output, plus a Warnings! section listing analyzer notes by code.
• test-table-partial-stats fixture for exercising the stats.mode = "partial" code path (4 files; 2 with stats, 2 without).
• Semantic regression suite (tests/semantic_regression.rs) — six canonical tests, one per minimum case in the roadmap.
• CLAUDE.md documenting project conventions for contributors and AI assistants.

Fixed

• stats::read_stats_async was inserting an empty FileStats for every Add action regardless of whether the action carried a stats field, so stats_coverage reported every file as having stats. Files without log-level stats are now skipped at read time.

[0.1.1] — 2026-04-13

Fixed

• partitionColumns are now read from the Delta log metaData action instead of being inferred from file paths. Adds an empty-table fixture and 16 regression tests.