- [Delta Table Specification](#delta-table-specification) - [File Types](#file-types) - [Data Files](#data-files) - [Deletion Vector Files](#deletion-vector-files) - [Change Data Files](#change-data-files) - [Delta Log Entries](#delta-log-entries) - [Checkpoints](#checkpoints) - [Sidecar Files](#sidecar-files) - [Log Compaction Files](#log-compaction-files) - [Last Checkpoint File](#last-checkpoint-file) - [Version Checksum File](#version-checksum-file) - [File Size Histogram Schema](#file-size-histogram-schema) - [Actions](#actions) - [Change Metadata](#change-metadata) - [Format Specification](#format-specification) - [Add File and Remove File](#add-file-and-remove-file) - [Add CDC File](#add-cdc-file) - [Writer Requirements for AddCDCFile](#writer-requirements-for-addcdcfile) - [Reader Requirements for AddCDCFile](#reader-requirements-for-addcdcfile) - [Transaction Identifiers](#transaction-identifiers) - [Protocol Evolution](#protocol-evolution) - [Commit Provenance Information](#commit-provenance-information) - [Domain Metadata](#domain-metadata) - [Reader Requirements for Domain Metadata](#reader-requirements-for-domain-metadata) - [Writer Requirements for Domain Metadata](#writer-requirements-for-domain-metadata) - [Sidecar File Information](#sidecar-file-information) - [Checkpoint Metadata](#checkpoint-metadata) - [Action Reconciliation](#action-reconciliation) - [Table Features](#table-features) - [Table Features for New and Existing Tables](#table-features-for-new-and-existing-tables) - [Supported Features](#supported-features) - [Active Features](#active-features) - [Column Mapping](#column-mapping) - [Writer Requirements for Column Mapping](#writer-requirements-for-column-mapping) - [Reader Requirements for Column Mapping](#reader-requirements-for-column-mapping) - [Deletion Vectors](#deletion-vectors) - [Deletion Vector Descriptor Schema](#deletion-vector-descriptor-schema) - [Derived Fields](#derived-fields) - [JSON Example 1 — On Disk with Relative Path (with Random Prefix)](#json-example-1--on-disk-with-relative-path-with-random-prefix) - [JSON Example 2 — On Disk with Absolute Path](#json-example-2--on-disk-with-absolute-path) - [JSON Example 3 — Inline](#json-example-3--inline) - [Reader Requirements for Deletion Vectors](#reader-requirements-for-deletion-vectors) - [Writer Requirement for Deletion Vectors](#writer-requirement-for-deletion-vectors) - [Iceberg Compatibility V1](#iceberg-compatibility-v1) - [Writer Requirements for IcebergCompatV1](#writer-requirements-for-icebergcompatv1) - [Iceberg Compatibility V2](#iceberg-compatibility-v2) - [Writer Requirement for IcebergCompatV2](#iceberg-compatibility-v2) - [Timestamp without timezone (TimestampNtz)](#timestamp-without-timezone-timestampntz) - [V2 Checkpoint Table Feature](#v2-checkpoint-table-feature) - [Row Tracking](#row-tracking) - [Row IDs](#row-ids) - [Row Commit Versions](#row-commit-versions) - [Reader Requirements for Row Tracking](#reader-requirements-for-row-tracking) - [Writer Requirements for Row Tracking](#writer-requirements-for-row-tracking) - [VACUUM Protocol Check](#vacuum-protocol-check) - [Writer Requirements for Vacuum Protocol Check](#writer-requirements-for-vacuum-protocol-check) - [Reader Requirements for Vacuum Protocol Check](#reader-requirements-for-vacuum-protocol-check) - [Clustered Table](#clustered-table) - [Writer Requirements for Clustered Table](#writer-requirements-for-clustered-table) - [Variant Data Type](#variant-data-type) - [Variant data in Parquet](#variant-data-in-parquet) - [Writer Requirements for Variant Type](#writer-requirements-for-variant-type) - [Reader Requirements for Variant Data Type](#reader-requirements-for-variant-data-type) - [Compatibility with other Delta Features](#compatibility-with-other-delta-features) - [Variant Shredding](#variant-shredding) - [Shredded Variant data in Parquet](#shredded-variant-data-in-parquet) - [Writer Requirements for Variant Shredding](#writer-requirements-for-variant-shredding) - [Reader Requirements for Variant Shredding](#reader-requirements-for-variant-shredding) - [Catalog-managed tables](#catalog-managed-tables) - [Terminology: Commits](#terminology-commits) - [Terminology: Delta Client](#terminology-delta-client) - [Terminology: Catalogs](#terminology-catalogs) - [Catalog Responsibilities](#catalog-responsibilities) - [Reading Catalog-managed Tables](#reading-catalog-managed-tables) - [Commit Protocol](#commit-protocol) - [Getting Ratified Commits from the Catalog](#getting-ratified-commits-from-the-catalog) - [Publishing Commits](#publishing-commits) - [Maintenance Operations on Catalog-managed Tables](#maintenance-operations-on-catalog-managed-tables) - [Creating and Dropping Catalog-managed Tables](#creating-and-dropping-catalog-managed-tables) - [Catalog-managed Table Enablement](#catalog-managed-table-enablement) - [Writer Requirements for Catalog-managed tables](#writer-requirements-for-catalog-managed-tables) - [Reader Requirements for Catalog-managed tables](#reader-requirements-for-catalog-managed-tables) - [Table Discovery](#table-discovery) - [Sample Catalog Client API](#sample-catalog-client-api) - [Additional Requirements for Writers](#additional-requirements-for-writers) - [Creation of New Log Entries](#creation-of-new-log-entries) - [Consistency Between Table Metadata and Data Files](#consistency-between-table-metadata-and-data-files) - [Delta Log Entries](#delta-log-entries-1) - [Checkpoints](#checkpoints-1) - [Checkpoint Specs](#checkpoint-specs) - [V2 Spec](#v2-spec) - [V1 Spec](#v1-spec) - [Checkpoint Naming Scheme](#checkpoint-naming-scheme) - [UUID-named checkpoint](#uuid-named-checkpoint) - [Classic checkpoint](#classic-checkpoint) - [Multi-part checkpoint](#multi-part-checkpoint) - [Problems with multi-part checkpoints](#problems-with-multi-part-checkpoints) - [Handling Backward compatibility while moving to UUID-named v2 Checkpoints](#handling-backward-compatibility-while-moving-to-uuid-named-v2-checkpoints) - [Allowed combinations for `checkpoint spec` `checkpoint file naming`](#allowed-combinations-for-checkpoint-spec---checkpoint-file-naming) - [Metadata Cleanup](#metadata-cleanup) - [Data Files](#data-files-1) - [Append-only Tables](#append-only-tables) - [Column Invariants](#column-invariants) - [CHECK Constraints](#check-constraints) - [Generated Columns](#generated-columns) - [Default Columns](#default-columns) - [Identity Columns](#identity-columns) - [Writer Version Requirements](#writer-version-requirements) - [Requirements for Readers](#requirements-for-readers) - [Reader Version Requirements](#reader-version-requirements) - [Table Properties](#table-properties) - [Appendix](#appendix) - [Valid Feature Names in Table Features](#valid-feature-names-in-table-features) - [Deletion Vector Format](#deletion-vector-format) - [Deletion Vector File Storage Format](#deletion-vector-file-storage-format) - [Per-file Statistics](#per-file-statistics) - [Statistics for Variant Columns](#statistics-for-variant-columns) - [Partition Value Serialization](#partition-value-serialization) - [Schema Serialization Format](#schema-serialization-format) - [Primitive Types](#primitive-types) - [Struct Type](#struct-type) - [Struct Field](#struct-field) - [Array Type](#array-type) - [Map Type](#map-type) - [Variant Type](#variant-type) - [Column Metadata](#column-metadata) - [Example](#example) - [Checkpoint Schema](#checkpoint-schema) - [Last Checkpoint File Schema](#last-checkpoint-file-schema) - [JSON checksum](#json-checksum) - [How to URL encode keys and string values](#how-to-url-encode-keys-and-string-values) - [Delta Data Type to Parquet Type Mappings](#delta-data-type-to-parquet-type-mappings) # Overview This document is a specification for the Delta Transaction Protocol, which brings [ACID](https://en.wikipedia.org/wiki/ACID) properties to large collections of data, stored as files, in a distributed file system or object store. The protocol was designed with the following goals in mind: - **Serializable ACID Writes** - multiple writers can concurrently modify a Delta table while maintaining ACID semantics. - **Snapshot Isolation for Reads** - readers can read a consistent snapshot of a Delta table, even in the face of concurrent writes. - **Scalability to billions of partitions or files** - queries against a Delta table can be planned on a single machine or in parallel. - **Self describing** - all metadata for a Delta table is stored alongside the data. This design eliminates the need to maintain a separate metastore just to read the data and also allows static tables to be copied or moved using standard filesystem tools. - **Support for incremental processing** - readers can tail the Delta log to determine what data has been added in a given period of time, allowing for efficient streaming. Delta's transactions are implemented using multi-version concurrency control (MVCC). As a table changes, Delta's MVCC algorithm keeps multiple copies of the data around rather than immediately replacing files that contain records that are being updated or removed. Readers of the table ensure that they only see one consistent _snapshot_ of a table at time by using the _transaction log_ to selectively choose which _data files_ to process. Writers modify the table in two phases: First, they optimistically write out new data files or updated copies of existing ones. Then, they _commit_, creating the latest _atomic version_ of the table by adding a new entry to the log. In this log entry they record which data files to logically add and remove, along with changes to other metadata about the table. Data files that are no longer present in the latest version of the table can be lazily deleted by the vacuum command after a user-specified retention period (default 7 days). # Delta Table Specification A table has a single serial history of atomic versions, which are named using contiguous, monotonically-increasing integers. The state of a table at a given version is called a _snapshot_ and is defined by the following properties: - **Delta log protocol** consists of two **protocol versions**, and if applicable, corresponding **table features**, that are required to correctly read or write the table - **Reader features** only exists when Reader Version is 3 - **Writer features** only exists when Writer Version is 7 - **Metadata** of the table (e.g., the schema, a unique identifier, partition columns, and other configuration properties) - **Set of files** present in the table, along with metadata about those files - **Set of tombstones** for files that were recently deleted - **Set of applications-specific transactions** that have been successfully committed to the table ## File Types A Delta table is stored within a directory and is composed of the following different types of files. Here is an example of a Delta table with four entries in the commit log, stored in the directory `mytable`. ``` /mytable/_delta_log/00000000000000000042.json /mytable/_delta_log/00000000000000000042.checkpoint.parquet /mytable/_delta_log/00000000000000000043.json /mytable/_delta_log/00000000000000000044.json /mytable/_delta_log/00000000000000000045.json /mytable/_delta_log/_last_checkpoint /mytable/_change_data/cdc-00000-924d9ac7-21a9-4121-b067-a0a6517aa8ed.c000.snappy.parquet /mytable/part-00000-3935a07c-416b-4344-ad97-2a38342ee2fc.c000.snappy.parquet /mytable/deletion_vector-0c6cbaaf-5e04-4c9d-8959-1088814f58ef.bin ``` This example represents a table after [metadata cleanup](#metadata-cleanup) has removed older log entries. The checkpoint at version 42 contains the complete table state, while versions 43-45 are subsequent commits. Each file type is described in the sections below. ### Data Files Data files can be stored in the root directory of the table or in any non-hidden subdirectory (i.e., one whose name does not start with an `_`). By default, the reference implementation stores data files in directories that are named based on the partition values for data in that file (i.e. `part1=value1/part2=value2/...`). This directory format is only used to follow existing conventions and is not required by the protocol. Actual partition values for a file must be read from the transaction log. ### Deletion Vector Files Deletion Vector (DV) files are stored in the root directory of the table alongside the data files. A DV file contains one or more serialised DV, each describing the set of *invalidated* (or "soft deleted") rows for a particular data file it is associated with. For data with partition values, DV files are *not* kept in the same directory hierarchy as data files, as each one can contain DVs for files from multiple partitions. DV files store DVs in a [binary format](#deletion-vector-format). ### Change Data Files Change data files are stored in a directory at the root of the table named `_change_data`, and represent the changes for the table version they are in. For data with partition values, it is recommended that the change data files are stored within the `_change_data` directory in their respective partitions (i.e. `_change_data/part1=value1/...`). Writers can _optionally_ produce these change data files as a consequence of operations that change underlying data, like `UPDATE`, `DELETE`, and `MERGE` operations to a Delta Lake table. If an operation only adds new data or removes existing data without updating any existing rows, a writer can write only data files and commit them in `add` or `remove` actions without duplicating the data into change data files. When available, change data readers should use the change data files instead of computing changes from the underlying data files. In addition to the data columns, change data files contain additional columns that identify the type of change event: Field Name | Data Type | Description -|-|- _change_type|`String`| `insert`, `update_preimage` , `update_postimage`, `delete` __(1)__ __(1)__ `preimage` is the value before the update, `postimage` is the value after the update. ### Delta Log Entries Delta Log Entries, also known as Delta files, are JSON files stored in the `_delta_log` directory at the root of the table. Together with checkpoints, they make up the log of all changes that have occurred to a table. Delta files are the unit of atomicity for a table, and are named using the next available version number, zero-padded to 20 digits. For example: ``` ./_delta_log/00000000000000000000.json ``` Delta files use newline-delimited JSON format, where every action is stored as a single-line JSON document. A Delta file, corresponding to version `v`, contains an atomic set of [_actions_](#actions) that should be applied to the previous table state corresponding to version `v-1`, in order to construct the `v`th snapshot of the table. An action changes one aspect of the table's state, for example, adding or removing a file. **Note:** If the [catalogManaged table feature](#catalog-managed-tables) is enabled on the table, recently [ratified commits](#ratified-commit) may not yet be published to the `_delta_log` directory as normal Delta files - they may be stored directly by the catalog or reside in the `_delta_log/_staged_commits` directory. Delta clients must contact the table's managing catalog in order to find the information about these [ratified, potentially-unpublished commits](#publishing-commits). The `_delta_log/_staged_commits` directory is the staging area for [staged](#staged-commit) commits. Delta files in this directory have a UUID embedded into them and follow the pattern ` . .json`, where the version corresponds to the proposed commit version, zero-padded to 20 digits. For example: ``` ./_delta_log/_staged_commits/00000000000000000000.3a0d65cd-4056-49b8-937b-95f9e3ee90e5.json ./_delta_log/_staged_commits/00000000000000000001.7d17ac10-5cc3-401b-bd1a-9c82dd2ea032.json ./_delta_log/_staged_commits/00000000000000000001.016ae953-37a9-438e-8683-9a9a4a79a395.json ./_delta_log/_staged_commits/00000000000000000002.3ae45b72-24e1-865a-a211-34987ae02f2a.json ``` NOTE: The (proposed) version number of a staged commit is authoritative - file `00000000000000000100. .json` always corresponds to a commit attempt for version 100. Besides simplifying implementations, it also acknowledges the fact that commit files cannot safely be reused for multiple commit attempts. For example, resolving conflicts in a table with [row tracking](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#row-tracking) enabled requires rewriting all file actions to update their `baseRowId` field. The [catalog](#terminology-catalogs) is the source of truth about which staged commit files in the `_delta_log/_staged_commits` directory correspond to ratified versions, and Delta clients should not attempt to directly interpret the contents of that directory. Refer to [catalog-managed tables](#catalog-managed-tables) for more details. ### Checkpoints Checkpoints are also stored in the `_delta_log` directory, and can be created at any time, for any committed version of the table. For performance reasons, readers should prefer to use the newest complete checkpoint possible. For time travel, the checkpoint used must not be newer than the time travel version. A checkpoint contains the complete replay of all actions, up to and including the checkpointed table version, with invalid actions removed. Invalid actions are those that have been canceled out by subsequent ones (for example removing a file that has been added), using the [rules for reconciliation](#Action-Reconciliation). In addition to above, checkpoint also contains the [_remove tombstones_](#add-file-and-remove-file) until they are expired. Checkpoints allow readers to short-cut the cost of reading the log up-to a given point in order to reconstruct a snapshot, and they also allow [Metadata cleanup](#metadata-cleanup) to delete expired JSON Delta log entries. Readers SHOULD NOT make any assumptions about the existence or frequency of checkpoints, with one exception: [Metadata cleanup](#metadata-cleanup) MUST provide a checkpoint for the oldest kept table version, to cover all deleted [Delta log entries](#delta-log-entries). That said, writers are encouraged to checkpoint reasonably frequently, so that readers do not pay excessive log replay costs due to reading large numbers of delta files. The checkpoint file name is based on the version of the table that the checkpoint contains. Delta supports three kinds of checkpoints: 1. UUID-named Checkpoints: These follow [V2 spec](#v2-spec) which uses the following file name: `n.checkpoint.u.{json/parquet}`, where `u` is a UUID and `n` is the snapshot version that this checkpoint represents. Here `n` must be zero padded to have length 20. The UUID-named V2 Checkpoint may be in json or parquet format, and references zero or more checkpoint sidecars in the `_delta_log/_sidecars` directory. A checkpoint sidecar is a uniquely-named parquet file: `{unique}.parquet` where `unique` is some unique string such as a UUID. For example: ``` 00000000000000000010.checkpoint.80a083e8-7026-4e79-81be-64bd76c43a11.json _sidecars/3a0d65cd-4056-49b8-937b-95f9e3ee90e5.parquet _sidecars/016ae953-37a9-438e-8683-9a9a4a79a395.parquet _sidecars/7d17ac10-5cc3-401b-bd1a-9c82dd2ea032.parquet ``` 2. A [classic checkpoint](#classic-checkpoint) for version `n` of the table consists of a file named `n.checkpoint.parquet`. Here `n` must be zero padded to have length 20. These could follow either [V1 spec](#v1-spec) or [V2 spec](#v2-spec). For example: ``` 00000000000000000010.checkpoint.parquet ``` 3. A [multi-part checkpoint](#multi-part-checkpoint) for version `n` consists of `p` "part" files (`p > 1`), where part `o` of `p` is named `n.checkpoint.o.p.parquet`. Here `n` must be zero padded to have length 20, while `o` and `p` must be zero padded to have length 10. These are always [V1 checkpoints](#v1-spec). For example: ``` 00000000000000000010.checkpoint.0000000001.0000000003.parquet 00000000000000000010.checkpoint.0000000002.0000000003.parquet 00000000000000000010.checkpoint.0000000003.0000000003.parquet ``` A writer can choose to write checkpoints with following constraints: - Writers are always allowed create a [classic checkpoint](#classic-checkpoint) following [v1 spec](#v1-spec). - Writers are forbidden to create [multi-part checkpoints](#multi-part-checkpoint) if [v2 checkpoints](#v2-checkpoint-table-feature) are enabled. - Writers are allowed to create v2 spec checkpoints (either [classic](#classic-checkpoint) or [uuid-named](#uuid-named-checkpoint)) if [v2 checkpoint table feature](#v2-checkpoint-table-feature) is enabled. Multi-part checkpoints are [deprecated](#problems-with-multi-part-checkpoints), and writers should avoid creating them. Use uuid-named [V2 spec](#v2-spec) checkpoints instead of these. Multiple checkpoints could exist for the same table version, e.g. if two clients race to create checkpoints at the same time, but with different formats. In such cases, a client can choose which checkpoint to use. Because a multi-part checkpoint cannot be created atomically (e.g. vulnerable to slow and/or failed writes), readers must ignore multi-part checkpoints with missing parts. Checkpoints for a given version must only be created after the associated delta file has been successfully written. #### Sidecar Files A sidecar file contains file actions. These files are in parquet format and they must have unique names. These are then [linked](#sidecar-file-information) to checkpoints. Refer to [V2 checkpoint spec](#v2-spec) for more detail. The sidecar files can have only [add file and remove file](#Add-File-and-Remove-File) entries as of now. The add and remove file actions are stored as their individual columns in parquet as struct fields. These files reside in the `_delta_log/_sidecars` directory. ### Log Compaction Files Log compaction files reside in the `_delta_log` directory. A log compaction file from a start version `x` to an end version `y` (`y` must be _greater_ than `x`) will have the following name: ` . .compacted.json`. This contains the aggregated actions for commit range `[x, y]`. Similar to commits, each row in the log compaction file represents an [action](#actions). The commit files for a given range are created by doing [Action Reconciliation](#action-reconciliation) of the corresponding commits. Instead of reading the individual commit files in range `[x, y]`, an implementation could choose to read the log compaction file ` . .compacted.json` to speed up the snapshot construction. Example: Suppose we have `00000000000000000004.json` as: ``` {"commitInfo":{...}} {"add":{"path":"f2",...}} {"remove":{"path":"f1",...}} ``` `00000000000000000005.json` as: ``` {"commitInfo":{...}} {"add":{"path":"f3",...}} {"add":{"path":"f4",...}} {"txn":{"appId":"3ae45b72-24e1-865a-a211-34987ae02f2a","version":4389}} ``` `00000000000000000006.json` as: ``` {"commitInfo":{...}} {"remove":{"path":"f3",...}} {"txn":{"appId":"3ae45b72-24e1-865a-a211-34987ae02f2a","version":4390}} ``` Then `00000000000000000004.00000000000000000006.compacted.json` will have the following content: ``` {"add":{"path":"f2",...}} {"add":{"path":"f4",...}} {"remove":{"path":"f1",...}} {"remove":{"path":"f3",...}} {"txn":{"appId":"3ae45b72-24e1-865a-a211-34987ae02f2a","version":4390}} ``` Writers: - Can optionally produce log compactions for any given commit range Readers: - Can optionally consume log compactions, if available - The compaction replaces the corresponding commits during action reconciliation ### Last Checkpoint File The Delta transaction log will often contain many (e.g. 10,000+) files. Listing such a large directory can be prohibitively expensive. The last checkpoint file can help reduce the cost of constructing the latest snapshot of the table by providing a pointer to near the end of the log. Rather than list the entire directory, readers can locate a recent checkpoint by looking at the `_delta_log/_last_checkpoint` file. Due to the zero-padded encoding of the files in the log, the version id of this recent checkpoint can be used on storage systems that support lexicographically-sorted, paginated directory listing to enumerate any delta files or newer checkpoints that comprise more recent versions of the table. ### Version Checksum File The Delta transaction log must remain an append-only log. To enable the detection of non-compliant modifications to Delta files, writers can optionally emit an auxiliary file with every commit, which contains important information about the state of the table as of that version. This file is referred to as the **Version Checksum** and can be used to validate the integrity of the table. ### Version Checksum File Schema A Version Checksum file must have the following properties: - Be named `{version}.crc` where `version` is zero-padded to 20 digits (e.g., `00000000000000000001.crc`) - Be stored directly in the `_delta_log` directory alongside Delta log files - Contain exactly one JSON object with the following schema: Field Name | Data Type | Description | optional/required -|-|-|- txnId | String | A unique identifier for the transaction that produced this commit. | optional tableSizeBytes | Long | Total size of the table in bytes, calculated as the sum of the `size` field of all live `add` actions. | required numFiles | Long | Number of live `add` actions in this table version after Action Reconciliation. | required numMetadata | Long | Number of `metaData` actions. Must be 1. | required numProtocol | Long | Number of `protocol` actions. Must be 1. | required inCommitTimestampOpt | Long | The in-commit timestamp of this version. Present if and only if [In-Commit Timestamps](#in-commit-timestamps) are enabled. | optional setTransactions | Array[`txn`] | Live [Transaction Identifier](#transaction-identifiers) actions at this version. | optional domainMetadata | Array[`domainMetadata`] | Live [Domain Metadata](#domain-metadata) actions at this version, excluding tombstones. | optional metadata | Metadata | The table [metadata](#change-metadata) at this version. | required protocol | Protocol | The table [protocol](#protocol-evolution) at this version. | required fileSizeHistogram | FileSizeHistogram | Size distribution information of files remaining after [Action Reconciliation](#action-reconciliation). See [FileSizeHistogram](#file-size-histogram-schema) for more details. | optional allFiles | Array[`add`] | All live [Add File](#add-file-and-remove-file) actions at this version. | optional numDeletedRecordsOpt | Long | Number of records deleted through Deletion Vectors in this table version. | optional numDeletionVectorsOpt | Long | Number of Deletion Vectors active in this table version. | optional deletedRecordCountsHistogramOpt | DeletedRecordCountsHistogram | Distribution of deleted record counts across files. See [this](#deleted-record-counts-histogram-schema) section for more details. | optional ##### File Size Histogram Schema The `FileSizeHistogram` object represents a histogram tracking file counts and total bytes across different size ranges. It has the following schema: Field Name | Data Type | Description | optional/required -|-|-|- sortedBinBoundaries | Array[Long] | A sorted array of bin boundaries where each element represents the start of a bin (inclusive) and the next element represents the end of the bin (exclusive). The first element must be 0. | required fileCounts | Array[Long] | Count of files in each bin. Length must match `sortedBinBoundaries`. | required totalBytes | Array[Long] | Total bytes of files in each bin. Length must match `sortedBinBoundaries`. | required Each index `i` in these arrays corresponds to a size range from `sortedBinBoundaries[i]` (inclusive) up to but not including `sortedBinBoundaries[i+1]`. The last bin ends at positive infinity. For example, given boundaries `[0, 1024, 4096]`: - Bin 0 contains files of size [0, 1024) bytes - Bin 1 contains files of size [1024, 4096) bytes - Bin 2 contains files of size [4096, ∞) bytes The arrays `fileCounts` and `totalBytes` store the number of files and their total size respectively that fall into each bin. This data structure enables efficient analysis of file size distributions in Delta tables. ### Deleted Record Counts Histogram Schema The `DeletedRecordCountsHistogram` object represents a histogram tracking the distribution of deleted record counts across files in the table. Each bin in the histogram represents a range of deletion counts and stores the number of files having that many deleted records. Field Name | Data Type | Description | optional/required -|-|-|- deletedRecordCounts | Array[Long] | Array of size 10 where each element represents the count of files falling into a specific deletion count range. | required The histogram bins correspond to the following ranges: - Bin 0: [0, 0] (files with no deletions) - Bin 1: [1, 9] (files with 1-9 deleted records) - Bin 2: [10, 99] (files with 10-99 deleted records) - Bin 3: [100, 999] (files with 100-999 deleted records) - Bin 4: [1000, 9999] (files with 1,000-9,999 deleted records) - Bin 5: [10000, 99999] (files with 10,000-99,999 deleted records) - Bin 6: [100000, 999999] (files with 100,000-999,999 deleted records) - Bin 7: [1000000, 9999999] (files with 1,000,000-9,999,999 deleted records) - Bin 8: [10000000, 2147483646] (files with 10,000,000 to 2147483646 (i.e. Int.MaxValue-1 in Java) deleted records) - Bin 9: [2147483647, ∞) (files with 2147483647 or more deleted records) This histogram allows analyzing the distribution of deleted records across files in a Delta table, which can be useful for monitoring and optimizing deletion patterns. #### State Validation Readers can validate table state integrity at a particular version by: 1. Reading the Version Checksum file for that version 2. Independently computing the same metrics by performing [Action Reconciliation](#action-reconciliation) on the table state 3. Comparing the computed values against those recorded in the Version Checksum If any discrepancy is found between computed and recorded values, the table state at that version should be considered potentially corrupted. ### Writer Requirements - Writers SHOULD produce a Version Checksum file for each commit - Writers MUST ensure all metrics in the Version Checksum accurately reflect table state after Action Reconciliation - Writers MUST write the Version Checksum file only after successfully writing the corresponding Delta log entry - Writers MUST NOT overwrite existing Version Checksum files ### Reader Requirements - Readers MAY use Version Checksums to validate table state integrity - If performing validation, readers SHOULD verify all required fields match computed values - If validation fails, readers SHOULD surface the discrepancy to users via error messaging - Readers MUST continue functioning if Version Checksum files are missing ## Actions Actions modify the state of the table and they are stored both in delta files and in checkpoints. This section lists the space of available actions as well as their schema. ### Change Metadata The `metaData` action changes the current metadata of the table. The first version of a table must contain a `metaData` action. Subsequent` metaData` actions completely overwrite the current metadata of the table. There can be at most one metadata action in a given version of the table. Every metadata action **must** include required fields at a minimum. The schema of the `metaData` action is as follows: Field Name | Data Type | Description | optional/required -|-|-|- id|`GUID`|Unique identifier for this table | required name|`String`| User-provided identifier for this table | optional description|`String`| User-provided description for this table | optional format|[Format Struct](#Format-Specification)| Specification of the encoding for the files stored in the table | required schemaString|[Schema Struct](#Schema-Serialization-Format)| Schema of the table | required partitionColumns|`Array[String]`| An array containing the names of columns by which the data should be partitioned | required createdTime|`Option[Long]`| The time when this metadata action is created, in milliseconds since the Unix epoch | optional configuration|`Map[String, String]`| A map containing configuration options for the metadata action | required #### Format Specification Field Name | Data Type | Description -|-|- provider|`String`|Name of the encoding for files in this table options|`Map[String, String]`|A map containing configuration options for the format In the reference implementation, the provider field is used to instantiate a Spark SQL [`FileFormat`](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala). As of Spark 2.4.3 there is built-in `FileFormat` support for `parquet`, `csv`, `orc`, `json`, and `text`. As of Delta Lake 0.3.0, user-facing APIs only allow the creation of tables where `format = 'parquet'` and `options = {}`. Support for reading other formats is present both for legacy reasons and to enable possible support for other formats in the future (See [#87](https://github.com/delta-io/delta/issues/87)). The following is an example `metaData` action: ```json { "metaData":{ "id":"af23c9d7-fff1-4a5a-a2c8-55c59bd782aa", "format":{"provider":"parquet","options":{}}, "schemaString":"...", "partitionColumns":[], "configuration":{ "appendOnly": "true" } } } ``` ### Add File and Remove File The `add` and `remove` actions are used to modify the data in a table by adding or removing individual _logical files_ respectively. Every _logical file_ of the table is represented by a path to a data file, combined with an optional Deletion Vector (DV) that indicates which rows of the data file are no longer in the table. Deletion Vectors are an optional feature, see their [reader requirements](#deletion-vectors) for details. When an `add` action is encountered for a logical file that is already present in the table, statistics and other information from the latest version should replace that from any previous version. The primary key for the entry of a logical file in the set of files is a tuple of the data file's `path` and a unique id describing the DV. If no DV is part of this logical file, then its primary key is `(path, NULL)` instead. The `remove` action includes a timestamp that indicates when the removal occurred. Physical deletion of physical files can happen lazily after some user-specified expiration time threshold. This delay allows concurrent readers to continue to execute against a stale snapshot of the data. A `remove` action should remain in the state of the table as a _tombstone_ until it has expired. A tombstone expires when *current time* (according to the node performing the cleanup) exceeds the expiration threshold added to the `remove` action timestamp. In the following statements, `dvId` can refer to either the unique id of a specific Deletion Vector (`deletionVector.uniqueId`) or to `NULL`, indicating that no rows are invalidated. Since actions within a given Delta commit are not guaranteed to be applied in order, a **valid** version is restricted to contain at most one file action *of the same type* (i.e. `add`/`remove`) for any one combination of `path` and `dvId`. Moreover, for simplicity it is required that there is at most one file action of the same type for any `path` (regardless of `dvId`). That means specifically that for any commit… - it is **legal** for the same `path` to occur in an `add` action and a `remove` action, but with two different `dvId`s. - it is **legal** for the same `path` to be added and/or removed and also occur in a `cdc` action. - it is **illegal** for the same `path` to occur twice with different `dvId`s within each set of `add` or `remove` actions. - it is **illegal** for a `path` to occur in an `add` action that already occurs with a different `dvId` in the list of `add` actions from the snapshot of the version immediately preceeding the commit, unless the commit also contains a remove for the later combination. - it is **legal** to commit an existing `path` and `dvId` combination again (this allows metadata updates). The `dataChange` flag on either an `add` or a `remove` can be set to `false` to indicate that an action when combined with other actions in the same atomic version only rearranges existing data or adds new statistics. For example, streaming queries that are tailing the transaction log can use this flag to skip actions that would not affect the final results. The schema of the `add` action is as follows: Field Name | Data Type | Description | optional/required -|-|-|- path| String | A relative path to a data file from the root of the table or an absolute path to a file that should be added to the table. The path is a URI as specified by [RFC 2396 URI Generic Syntax](https://www.ietf.org/rfc/rfc2396.txt), which needs to be decoded to get the data file path. | required partitionValues| Map[String, String] | A map from partition column to value for this logical file. See also [Partition Value Serialization](#Partition-Value-Serialization) | required size| Long | The size of this data file in bytes | required modificationTime | Long | The time this logical file was created, as milliseconds since the epoch | required dataChange | Boolean | When `false` the logical file must already be present in the table or the records in the added file must be contained in one or more `remove` actions in the same version | required stats | [Statistics Struct](#Per-file-Statistics) | Contains statistics (e.g., count, min/max values for columns) about the data in this logical file | optional tags | Map[String, String] | Map containing metadata about this logical file | optional deletionVector | [DeletionVectorDescriptor Struct](#Deletion-Vectors) | Either null (or absent in JSON) when no DV is associated with this data file, or a struct (described below) that contains necessary information about the DV that is part of this logical file. | optional baseRowId | Long | Default generated Row ID of the first row in the file. The default generated Row IDs of the other rows in the file can be reconstructed by adding the physical index of the row within the file to the base Row ID. See also [Row IDs](#row-ids) | optional defaultRowCommitVersion | Long | First commit version in which an `add` action with the same `path` was committed to the table. | optional clusteringProvider | String | The name of the clustering implementation. See also [Clustered Table](#clustered-table)| optional The following is an example `add` action for a partitioned table: ```json { "add": { "path": "date=2017-12-10/part-000...c000.gz.parquet", "partitionValues": {"date": "2017-12-10"}, "size": 841454, "modificationTime": 1512909768000, "dataChange": true, "baseRowId": 4071, "defaultRowCommitVersion": 41, "stats": "{\"numRecords\":1,\"minValues\":{\"val..." } } ``` The following is an example `add` action for a clustered table: ```json { "add": { "path": "date=2017-12-10/part-000...c000.gz.parquet", "partitionValues": {}, "size": 841454, "modificationTime": 1512909768000, "dataChange": true, "baseRowId": 4071, "defaultRowCommitVersion": 41, "clusteringProvider": "liquid", "stats": "{\"numRecords\":1,\"minValues\":{\"val..." } } ``` The schema of the `remove` action is as follows: Field Name | Data Type | Description | optional/required -|-|-|- path| String | A relative path to a file from the root of the table or an absolute path to a file that should be removed from the table. The path is a URI as specified by [RFC 2396 URI Generic Syntax](https://www.ietf.org/rfc/rfc2396.txt), which needs to be decoded to get the data file path. | required deletionTimestamp | Option[Long] | The time the deletion occurred, represented as milliseconds since the epoch | optional dataChange | Boolean | When `false` the records in the removed file must be contained in one or more `add` file actions in the same version | required extendedFileMetadata | Boolean | When `true` the fields `partitionValues`, `size`, and `tags` are present | optional partitionValues| Map[String, String] | A map from partition column to value for this file. See also [Partition Value Serialization](#Partition-Value-Serialization) | optional size| Long | The size of this data file in bytes | optional stats | [Statistics Struct](#Per-file-Statistics) | Contains statistics (e.g., count, min/max values for columns) about the data in this logical file | optional tags | Map[String, String] | Map containing metadata about this file | optional deletionVector | [DeletionVectorDescriptor Struct](#Deletion-Vectors) | Either null (or absent in JSON) when no DV is associated with this data file, or a struct (described below) that contains necessary information about the DV that is part of this logical file. | optional baseRowId | Long | Default generated Row ID of the first row in the file. The default generated Row IDs of the other rows in the file can be reconstructed by adding the physical index of the row within the file to the base Row ID. See also [Row IDs](#row-ids) | optional defaultRowCommitVersion | Long | First commit version in which an `add` action with the same `path` was committed to the table | optional The following is an example `remove` action. ```json { "remove": { "path": "part-00001-9…..snappy.parquet", "deletionTimestamp": 1515488792485, "baseRowId": 4071, "defaultRowCommitVersion": 41, "dataChange": true } } ``` ### Add CDC File The `cdc` action is used to add a [file](#change-data-files) containing only the data that was changed as part of the transaction. The `cdc` action is allowed to add a [Data File](#data-files) that is also added by an `add` action, when it does not contain any copied rows and the `_change_type` column is filled for all rows. When change data readers encounter a `cdc` action in a particular Delta table version, they must read the changes made in that version exclusively using the `cdc` files. If a version has no `cdc` action, then the data in `add` and `remove` actions are read as inserted and deleted rows, respectively. The schema of the `cdc` action is as follows: Field Name | Data Type | Description -|-|- path| String | A relative path to a change data file from the root of the table or an absolute path to a change data file that should be added to the table. The path is a URI as specified by [RFC 2396 URI Generic Syntax](https://www.ietf.org/rfc/rfc2396.txt), which needs to be decoded to get the file path. partitionValues| Map[String, String] | A map from partition column to value for this file. See also [Partition Value Serialization](#Partition-Value-Serialization) size| Long | The size of this file in bytes dataChange | Boolean | Should always be set to `false` for `cdc` actions because they _do not_ change the underlying data of the table tags | Map[String, String] | Map containing metadata about this file The following is an example of `cdc` action. ```json { "cdc": { "path": "_change_data/cdc-00001-c…..snappy.parquet", "partitionValues": {}, "size": 1213, "dataChange": false } } ``` #### Writer Requirements for AddCDCFile For [Writer Versions 4 up to 6](#Writer-Version-Requirements), all writers must respect the `delta.enableChangeDataFeed` configuration flag in the metadata of the table. When `delta.enableChangeDataFeed` is `true`, writers must produce the relevant `AddCDCFile`'s for any operation that changes data, as specified in [Change Data Files](#change-data-files). For Writer Version 7, all writers must respect the `delta.enableChangeDataFeed` configuration flag in the metadata of the table only if the feature `changeDataFeed` exists in the table `protocol`'s `writerFeatures`. #### Reader Requirements for AddCDCFile When available, change data readers should use the `cdc` actions in a given table version instead of computing changes from the underlying data files referenced by the `add` and `remove` actions. Specifically, to read the row-level changes made in a version, the following strategy should be used: 1. If there are `cdc` actions in this version, then read only those to get the row-level changes, and skip the remaining `add` and `remove` actions in this version. 2. Otherwise, if there are no `cdc` actions in this version, read and treat all the rows in the `add` and `remove` actions as inserted and deleted rows, respectively. 3. Change data readers should return the following extra columns: Field Name | Data Type | Description -|-|- _commit_version|`Long`| The table version containing the change. This can be derived from the name of the Delta log file that contains actions. _commit_timestamp|`Timestamp`| The timestamp associated when the commit was created. Depending on whether [In-Commit Timestamps](#in-commit-timestamps) are enabled, this is derived from either the `inCommitTimestamp` field of the `commitInfo` action of the version's Delta log file, or from the Delta log file's modification time. ##### Note for non-change data readers In a table with Change Data Feed enabled, the data Parquet files referenced by `add` and `remove` actions are allowed to contain an extra column `_change_type`. This column is not present in the table's schema. When accessing these files, readers should disregard this column and only process columns defined within the table's schema. ### Transaction Identifiers Incremental processing systems (e.g., streaming systems) that track progress using their own application-specific versions need to record what progress has been made, in order to avoid duplicating data in the face of failures and retries during a write. Transaction identifiers allow this information to be recorded atomically in the transaction log of a delta table along with the other actions that modify the contents of the table. Transaction identifiers are stored in the form of `appId` `version` pairs, where `appId` is a unique identifier for the process that is modifying the table and `version` is an indication of how much progress has been made by that application. The atomic recording of this information along with modifications to the table enables these external system to make their writes into a Delta table _idempotent_. For example, the [Delta Sink for Apache Spark's Structured Streaming](https://github.com/delta-io/delta/blob/master/core/src/main/scala/org/apache/spark/sql/delta/sources/DeltaSink.scala) ensures exactly-once semantics when writing a stream into a table using the following process: 1. Record in a write-ahead-log the data that will be written, along with a monotonically increasing identifier for this batch. 2. Check the current version of the transaction with `appId = streamId` in the target table. If this value is greater than or equal to the batch being written, then this data has already been added to the table and processing can skip to the next batch. 3. Write the data optimistically into the table. 4. Attempt to commit the transaction containing both the addition of the data written out and an updated `appId` `version` pair. The semantics of the application-specific `version` are left up to the external system. Delta only ensures that the latest `version` for a given `appId` is available in the table snapshot. The Delta transaction protocol does not, for example, assume monotonicity of the `version` and it would be valid for the `version` to decrease, possibly representing a "rollback" of an earlier transaction. The schema of the `txn` action is as follows: Field Name | Data Type | Description | optional/required -|-|-|- appId | String | A unique identifier for the application performing the transaction | required version | Long | An application-specific numeric identifier for this transaction | required lastUpdated | Option[Long] | The time when this transaction action is created, in milliseconds since the Unix epoch | optional The following is an example `txn` action: ```json { "txn": { "appId":"3ba13872-2d47-4e17-86a0-21afd2a22395", "version":364475 } } ``` ### Protocol Evolution The `protocol` action is used to increase the version of the Delta protocol that is required to read or write a given table. Protocol versioning allows a newer client to exclude older readers and/or writers that are missing features required to correctly interpret the transaction log. The _protocol version_ will be increased whenever non-forward-compatible changes are made to this specification. In the case where a client is running an invalid protocol version, an error should be thrown instructing the user to upgrade to a newer protocol version of their Delta client library. Since breaking changes must be accompanied by an increase in the protocol version recorded in a table or by the addition of a table feature, clients can assume that unrecognized actions, fields, and/or metadata domains are never required in order to correctly interpret the transaction log. Clients must ignore such unrecognized fields, and should not produce an error when reading a table that contains unrecognized fields. Reader Version 3 and Writer Version 7 add two lists of table features to the protocol action. The capability for readers and writers to operate on such a table is not only dependent on their supported protocol versions, but also on whether they support all features listed in `readerFeatures` and `writerFeatures`. See [Table Features](#table-features) section for more information. The schema of the `protocol` action is as follows: Field Name | Data Type | Description | optional/required -|-|-|- minReaderVersion | Int | The minimum version of the Delta read protocol that a client must implement in order to correctly *read* this table | required minWriterVersion | Int | The minimum version of the Delta write protocol that a client must implement in order to correctly *write* this table | required readerFeatures | Array[String] | A collection of features that a client must implement in order to correctly read this table (exist only when `minReaderVersion` is set to `3`) | optional writerFeatures | Array[String] | A collection of features that a client must implement in order to correctly write this table (exist only when `minWriterVersion` is set to `7`) | optional Some example Delta protocols: ```json { "protocol":{ "minReaderVersion":1, "minWriterVersion":2 } } ``` A table that is using table features only for writers: ```json { "protocol":{ "readerVersion":2, "writerVersion":7, "writerFeatures":["columnMapping","identityColumns"] } } ``` Reader version 2 in the above example does not support listing reader features but supports Column Mapping. This example is equivalent to the next one, where Column Mapping is represented as a reader table feature. A table that is using table features for both readers and writers: ```json { "protocol": { "readerVersion":3, "writerVersion":7, "readerFeatures":["columnMapping"], "writerFeatures":["columnMapping","identityColumns"] } } ``` ### Commit Provenance Information A delta file can optionally contain additional provenance information about what higher-level operation was being performed as well as who executed it. When the `catalogManaged` table feature is enabled, the `commitInfo` action must have a field `txnId` that stores a unique transaction identifier string. Implementations are free to store any valid JSON-formatted data via the `commitInfo` action. When [In-Commit Timestamps](#in-commit-timestamps) are enabled, writers are required to include a `commitInfo` action with every commit, which must include the `inCommitTimestamp` field. Also, the `commitInfo` action must be first action in the commit. An example of storing provenance information related to an `INSERT` operation: ```json { "commitInfo":{ "timestamp":1515491537026, "userId":"100121", "userName":"michael@databricks.com", "operation":"INSERT", "operationParameters":{"mode":"Append","partitionBy":"[]"}, "notebook":{ "notebookId":"4443029", "notebookPath":"Users/michael@databricks.com/actions"}, "clusterId":"1027-202406-pooh991" } } ``` ### Domain Metadata The domain metadata action contains a configuration (string) for a named metadata domain. Two overlapping transactions conflict if they both contain a domain metadata action for the same metadata domain. There are two types of metadata domains: 1. **User-controlled metadata domains** have names that start with anything other than the `delta.` prefix. Any Delta client implementation or user application can modify these metadata domains, and can allow users to modify them arbitrarily. Delta clients and user applications are encouraged to use a naming convention designed to avoid conflicts with other clients' or users' metadata domains (e.g. `com.databricks.*` or `org.apache.*`). 2. **System-controlled metadata domains** have names that start with the `delta.` prefix. This prefix is reserved for metadata domains defined by the Delta spec, and Delta client implementations must not allow users to modify the metadata for system-controlled domains. A Delta client implementation should only update metadata for system-controlled domains that it knows about and understands. System-controlled metadata domains are used by various table features and each table feature may impose additional semantics on the metadata domains it uses. The schema of the `domainMetadata` action is as follows: Field Name | Data Type | Description -|-|- domain | String | Identifier for this domain (system- or user-provided) configuration | String | String containing configuration for the metadata domain removed | Boolean | When `true`, the action serves as a tombstone to logically delete a metadata domain. Writers should preserve an accurate pre-image of the configuration. To support this feature: - The table must be on Writer Version 7. - A feature name `domainMetadata` must exist in the table's `writerFeatures`. #### Reader Requirements for Domain Metadata - Readers are not required to support domain metadata. - Readers who choose not to support domain metadata should ignore metadata domain actions as unrecognized (see [Protocol Evolution](#protocol-evolution)) and snapshots should not include any metadata domains. - Readers who choose to support domain metadata must apply [Action Reconciliation](#action-reconciliation) to all metadata domains and snapshots must include them -- even if the reader does not understand them. - Any system-controlled domain that imposes any requirements on readers is a [breaking change](#protocol-evolution), and must be part of a reader-writer table feature that specifies the desired behavior. #### Writer Requirements for Domain Metadata - Writers must preserve all domains even if they don't understand them. - Writers must not allow users to modify or delete system-controlled domains. - Writers must only modify or delete system-controlled domains they understand. - Any system-controlled domain that imposes additional requirements on the writer is a [breaking change](#protocol-evolution), and must be part of a writer table feature that specifies the desired behavior. The following is an example `domainMetadata` action: ```json { "domainMetadata": { "domain": "delta.deltaTableFeatureX", "configuration": "{\"key1\":\"value1\"}", "removed": false } } ``` ### Sidecar File Information The `sidecar` action references a [sidecar file](#sidecar-files) which provides some of the checkpoint's file actions. This action is only allowed in checkpoints following [V2 spec](#v2-spec). The schema of `sidecar` action is as follows: Field Name | Data Type | Description | optional/required -|-|-|- path | String | URI-encoded path to the sidecar file. Because sidecar files must always reside in the table's own _delta_log/_sidecars directory, implementations are encouraged to store only the file's name (without scheme or parent directories). | required sizeInBytes | Long | Size of the sidecar file. | required modificationTime | Long | The time this logical file was created, as milliseconds since the epoch. | required tags|`Map[String, String]`|Map containing any additional metadata about the checkpoint sidecar file. | optional The following is an example `sidecar` action: ```json { "sidecar":{ "path": "016ae953-37a9-438e-8683-9a9a4a79a395.parquet", "sizeInBytes": 2304522, "modificationTime": 1512909768000, "tags": {} } } ``` #### Checkpoint Metadata This action is only allowed in checkpoints following [V2 spec](#v2-spec). It describes the details about the checkpoint. It has the following schema: Field Name | Data Type | Description | optional/required -|-|-|- version|`Long`|The checkpoint version.| required tags|`Map[String, String]`|Map containing any additional metadata about the v2 spec checkpoint.| optional ##### Checkpoint Metadata Tags The following tag keys may be present in the `tags` map. All are optional, so there is no requirement for writers to produce these and readers cannot assume their presence. Tag Key | Value Type | Description -|-|- sidecarNumActions|`String` (parseable as `Long`)|The total number of actions stored across all [sidecar files](#sidecar-files) in this checkpoint. sidecarSizeInBytes|`String` (parseable as `Long`)|The total size in bytes across all [sidecar files](#sidecar-files) in this checkpoint. numOfAddFiles|`String` (parseable as `Long`)|The number of `add` file actions in this checkpoint. sidecarFileSchema|`String` (JSON-encoded `StructType`)|The schema of the [sidecar files](#sidecar-files) in this checkpoint. The value is the JSON serialization of the sidecar file's Parquet schema. Readers can use this to avoid reading the Parquet footer of sidecar files to determine their schema. E.g. ```json { "checkpointMetadata":{ "version":1, "tags":{ "sidecarNumActions":"1234", "sidecarSizeInBytes":"5678", "numOfAddFiles":"42", "sidecarFileSchema":"{\"type\":\"struct\",\"fields\":[{\"name\":\"add\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"path\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"remove\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"path\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}}]}" } } } ``` # Action Reconciliation A given snapshot of the table can be computed by replaying the events committed to the table in ascending order by commit version. A given snapshot of a Delta table consists of: - A single `protocol` action - A single `metaData` action - A collection of `txn` actions with unique `appId`s - A collection of `domainMetadata` actions with unique `domain`s, excluding tombstones (i.e. actions with `removed=true`). - A collection of `add` actions with unique path keys, corresponding to the newest (path, deletionVector.uniqueId) pair encountered for each path. - A collection of `remove` actions with unique `(path, deletionVector.uniqueId)` keys. The intersection of the primary keys in the `add` collection and `remove` collection must be empty. That means a logical file cannot exist in both the `remove` and `add` collections at the same time; however, the same *data file* can exist with *different* DVs in the `remove` collection, as logically they represent different content. The `remove` actions act as _tombstones_, and only exist for the benefit of the VACUUM command. Snapshot reads only return `add` actions on the read path. To achieve the requirements above, related actions from different delta files need to be reconciled with each other: - The latest `protocol` action seen wins - The latest `metaData` action seen wins - For `txn` actions, the latest `version` seen for a given `appId` wins - For `domainMetadata`, the latest `domainMetadata` seen for a given `domain` wins. The actions with `removed=true` act as tombstones to suppress earlier versions. Snapshot reads do _not_ return removed `domainMetadata` actions. - For `commitInfo` actions, only the `commitInfo` from the commit at the snapshot version is included in the snapshot. [Checkpoints](#checkpoints) and [log compaction files](#log-compaction-files) do not preserve `commitInfo` actions, so this information must be read from the JSON commit file at the snapshot version. - Logical files in a table are identified by their `(path, deletionVector.uniqueId)` primary key. File actions (`add` or `remove`) reference logical files, and a log can contain any number of references to a single file. - To replay the log, scan all file actions and keep only the newest reference for each logical file. - `add` actions in the result identify logical files currently present in the table (for queries). `remove` actions in the result identify tombstones of logical files no longer present in the table (for VACUUM). - [v2 checkpoint spec](#v2-spec) actions are not allowed in normal commit files, and do not participate in log replay. # Table Features Table features must only exist on tables that have a supported protocol version. When the table's Reader Version is 3, `readerFeatures` must exist in the `protocol` action, and when the Writer Version is 7, `writerFeatures` must exist in the `protocol` action. `readerFeatures` and `writerFeatures` define the features that readers and writers must implement in order to read and write this table. Readers and writers must not ignore table features when they are present: - to read a table, readers must implement and respect all features listed in `readerFeatures`; - to write a table, writers must implement and respect all features listed in `writerFeatures`. Because writers have to read the table (or only the Delta log) before write, they must implement and respect all reader features as well. ## Table Features for New and Existing Tables It is possible to create a new table or upgrade an existing table to the protocol versions that supports the use of table features. A table must support either the use of writer features or both reader and writer features. It is illegal to support reader but not writer features. For new tables, when a new table is created with a Reader Version up to 2 and Writer Version 7, its `protocol` action must only contain `writerFeatures`. When a new table is created with Reader Version 3 and Writer Version 7, its `protocol` action must contain both `readerFeatures` and `writerFeatures`. Creating a table with a Reader Version 3 and Writer Version less than 7 is not allowed. When upgrading an existing table to Reader Version 3 and/or Writer Version 7, the client should, on a best effort basis, determine which features supported by the original protocol version are used in any historical version of the table, and add only used features to reader and/or writer feature sets. The client must assume a feature has been used, unless it can prove that the feature is *definitely* not used in any historical version of the table that is reachable by time travel. For example, given a table on Reader Version 1 and Writer Version 4, along with four versions: 1. Table property change: set `delta.enableChangeDataFeed` to `true`. 2. Data change: three rows updated. 3. Table property change: unset `delta.enableChangeDataFeed`. 4. Table protocol change: upgrade protocol to Reader Version 3 and Writer Version 7. To produce Version 4, a writer could look at only Version 3 and discover that Change Data Feed has not been used. But in fact, this feature has been used and the table does contain some Change Data Files for Version 2. This means that, to determine all features that have ever been used by the table, a writer must either scan the whole history (which is very time-consuming) or assume the worst case: all features supported by protocol `(1, 4)` has been used. ## Supported Features A feature is supported by a table when its name is in the `protocol` action’s `readerFeatures` and/or `writerFeatures`. Subsequent read and/or write operations on this table must respect the feature. Clients must not remove the feature from the `protocol` action. Writers are allowed to add support of a feature to the table by adding its name to `readerFeatures` or `writerFeatures`. Reader features should be listed in both `readerFeatures` and `writerFeatures` simultaneously, while writer features should be listed only in `writerFeatures`. It is not allowed to list a feature only in `readerFeatures` but not in `writerFeatures`. A feature being supported does not imply that it is active. For example, a table may have the [Append-only Tables](#append-only-tables) feature (feature name `appendOnly`) listed in `writerFeatures`, but it does not have a table property `delta.appendOnly` that is set to `true`. In such a case the table is not append-only, and writers are allowed to change, remove, and rearrange data. However, writers must know that the table property `delta.appendOnly` should be checked before writing the table. ## Active Features A feature is active on a table when it is supported *and* its metadata requirements are satisfied. Each feature defines its own metadata requirements, as stated in the corresponding sections of this document. For example, the Append-only feature is active when the `appendOnly` feature name is present in a `protocol`'s `writerFeatures` *and* a table property `delta.appendOnly` set to `true`. # Column Mapping Delta can use column mapping to avoid any column naming restrictions, and to support the renaming and dropping of columns without having to rewrite all the data. There are two modes of column mapping, by `name` and by `id`. In both modes, every column - nested or leaf - is assigned a _physical_ name, and a unique 32-bit integer as an id. The physical name is stored as part of the column metadata with the key `delta.columnMapping.physicalName`. The column id is stored within the metadata with the key `delta.columnMapping.id`. ## Field Path A _field path_ is the path from the schema root to a [struct field](#struct-field), formed by the ordered sequence of field names along that path. When the path traverses an [Array Type](#array-type) element, or a [Map Type](#map-type) key or value, the path component is `element`, `key`, or `value`, respectively. A _physical field path_ exists only when Column Mapping mode is `id` or `name`; it is formed by replacing each struct field name in a field path with that struct field's physical name. In these modes, a physical field path must be unique across all versions of the table. This supports cheap column deletions in `name` mode. The column mapping is governed by the table property `delta.columnMapping.mode` being one of `none`, `id`, and `name`. The table property should only be honored if the table's protocol has reader and writer versions and/or table features that support the `columnMapping` table feature. For readers this is Reader Version 2, or Reader Version 3 with the `columnMapping` table feature listed as supported. For writers this is Writer Version 5 or 6, or Writer Version 7 with the `columnMapping` table feature supported. The following is an example for the column definition of a table that leverages column mapping. See the [appendix](#schema-serialization-format) for a more complete schema definition. ```json { "name" : "e", "type" : { "type" : "array", "elementType" : { "type" : "struct", "fields" : [ { "name" : "d", "type" : "integer", "nullable" : false, "metadata" : { "delta.columnMapping.id": 5, "delta.columnMapping.physicalName": "col-a7f4159c-53be-4cb0-b81a-f7e5240cfc49" } } ] }, "containsNull" : true }, "nullable" : true, "metadata" : { "delta.columnMapping.id": 4, "delta.columnMapping.physicalName": "col-5f422f40-de70-45b2-88ab-1d5c90e94db1" } } ``` In this example, the field path of the nested field `d` is `["e", "element", "d"]`, and its physical field path is `["col-5f422f40-de70-45b2-88ab-1d5c90e94db1", "element", "col-a7f4159c-53be-4cb0-b81a-f7e5240cfc49"]`. ## Writer Requirements for Column Mapping In order to support column mapping, writers must: - Write `protocol` and `metaData` actions when Column Mapping is turned on for the first time: - If the table is on Writer Version 5 or 6: write a `metaData` action to add the `delta.columnMapping.mode` table property; - If the table is on Writer Version 7: - write a `protocol` action to add the feature `columnMapping` to both `readerFeatures` and `writerFeatures`, and - write a `metaData` action to add the `delta.columnMapping.mode` table property. - Write data files by using the _physical name_ that is chosen for each column. The physical name of the column is static and can be different than the _display name_ of the column, which is changeable. - Write the 32 bit integer column identifier as part of the `field_id` field of the `SchemaElement` struct in the [Parquet Thrift specification](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift). - Track partition values, column level statistics, and [clustering column](#clustered-table) names with the physical name of the column in the transaction log. - Assign a physical name for each new column that is added to the schema, and ensure the physical field path of the new column is unique across all versions of the table. In addition, column identifiers need to be assigned to each column. The maximum id that is assigned to a column is tracked as the table property `delta.columnMapping.maxColumnId`. This is an internal table property that cannot be configured by users. This value must increase monotonically as new columns are introduced and committed to the table alongside the introduction of the new columns to the schema. ## Reader Requirements for Column Mapping If the table is on Reader Version 2, or if the table is on Reader Version 3 and the feature `columnMapping` is present in `readerFeatures`, readers and writers must read the table property `delta.columnMapping.mode` and do one of the following. In `none` mode, or if the table property is not present, readers must read the parquet files by using the display names (the `name` field of the column definition) of the columns in the schema. In `id ` mode, readers must resolve columns by using the `field_id` in the parquet metadata for each file, as given by the column metadata property `delta.columnMapping.id` in the Delta schema. Partition values and column level statistics must be resolved by their *physical names* for each `add` entry in the transaction log. If a data file does not contain field ids, readers must refuse to read that file or return nulls for each column. For ids that cannot be found in a file, readers must return `null` values for those columns. In `name` mode, readers must resolve columns in the data files by their physical names as given by the column metadata property `delta.columnMapping.physicalName` in the Delta schema. Partition values and column level statistics will also be resolved by their physical names. For columns that are not found in the files, `null`s need to be returned. Column ids are not used in this mode for resolution purposes. # Deletion Vectors To support this feature: - To support Deletion Vectors, a table must have Reader Version 3 and Writer Version 7. A feature name `deletionVectors` must exist in the table's `readerFeatures` and `writerFeatures`. When supported: - A table may have a metadata property `delta.enableDeletionVectors` in the Delta schema set to `true`. Writers must only write new Deletion Vectors (DVs) when this property is set to `true`. - A table's `add` and `remove` actions can optionally include a DV that provides information about logically deleted rows, that are however still physically present in the underlying data file and must thus be skipped during processing. Readers must read the table considering the existence of DVs, even when the `delta.enableDeletionVectors` table property is not set. DVs can be stored and accessed in different ways, indicated by the `storageType` field. The Delta protocol currently supports inline or on-disk storage, where the latter can be accessed either by a relative path derived from a UUID or an absolute path. ## Deletion Vector Descriptor Schema The schema of the `DeletionVectorDescriptor` struct is as follows: Field Name | Data Type | Description -|-|- storageType | String | A single character to indicate how to access the DV. Legal options are: `['u', 'i', 'p']`. pathOrInlineDv | String | Three format options are currently proposed: If `storageType = 'u'` then ` `: The deletion vector is stored in a file with a path relative to the data directory of this Delta table, and the file name can be reconstructed from the UUID. See Derived Fields for how to reconstruct the file name. The random prefix is recovered as the extra characters before the (20 characters fixed length) uuid. If `storageType = 'i'` then ` `: The deletion vector is stored inline in the log. The format used is the `RoaringBitmapArray` format also used when the DV is stored on disk and described in [Deletion Vector Format](#Deletion-Vector-Format). If `storageType = 'p'` then ` `: The DV is stored in a file with an absolute path given by this path, which has the same format as the `path` field in the `add`/`remove` actions. offset | Option[Int] | Start of the data for this DV in number of bytes from the beginning of the file it is stored in. Always `None` (absent in JSON) when `storageType = 'i'`. Interpret as `0` if absent for other `storageType`s. sizeInBytes | Int | Size of the serialized DV in bytes (raw data size, i.e. before base85 encoding, if inline). cardinality | Long | Number of rows the given DV logically removes from the file. The concrete Base85 variant used is [Z85](https://rfc.zeromq.org/spec/32/), because it is JSON-friendly. ### Derived Fields Some fields that are necessary to use the DV are not stored explicitly but can be derived in code from the stored fields. Field Name | Data Type | Description | Computed As -|-|-|- uniqueId | String | Uniquely identifies a DV for a given file. This is used for snapshot reconstruction to differentiate the same file with different DVs in successive versions. | If `offset` is `None` then ` `. Otherwise ` @ `. absolutePath | String/URI/Path | The absolute path of the DV file. Can be calculated for relative path DVs by providing a parent directory path. | If `storageType='p'`, just use the already absolute path. If `storageType='u'`, the DV is stored at ` / /deletion_vector_ .bin`. This is not a legal field if `storageType='i'`, as an inline DV has no absolute path. ### JSON Example 1 — On Disk with Relative Path (with Random Prefix) ```json { "storageType" : "u", "pathOrInlineDv" : "ab^-aqEH.-t@S}K{vb[*k^", "offset" : 4, "sizeInBytes" : 40, "cardinality" : 6 } ``` Assuming that this DV is stored relative to an `s3://mytable/` directory, the absolute path to be resolved here would be: `s3://mytable/ab/deletion_vector_d2c639aa-8816-431a-aaf6-d3fe2512ff61.bin`. ### JSON Example 2 — On Disk with Absolute Path ```json { "storageType" : "p", "pathOrInlineDv" : "s3://mytable/deletion_vector_d2c639aa-8816-431a-aaf6-d3fe2512ff61.bin", "offset" : 4, "sizeInBytes" : 40, "cardinality" : 6 } ``` ### JSON Example 3 — Inline ```json { "storageType" : "i", "pathOrInlineDv" : "wi5b=000010000siXQKl0rr91000f55c8Xg0@@D72lkbi5=-{L", "sizeInBytes" : 40, "cardinality" : 6 } ``` The row indexes encoded in this DV are: 3, 4, 7, 11, 18, 29. ## Reader Requirements for Deletion Vectors If a snapshot contains logical files with records that are invalidated by a DV, then these records *must not* be returned in the output. ## Writer Requirement for Deletion Vectors When adding a logical file with a deletion vector, then that logical file must have correct `numRecords` information for the data file in the `stats` field. # Catalog-managed tables With this feature enabled, the [catalog](#terminology-catalogs) that manages the table becomes the source of truth for whether a given commit attempt succeeded. The table feature defines the parts of the [commit protocol](#commit-protocol) that directly impact the Delta table (e.g. atomicity requirements, publishing, etc). The Delta client and catalog together are responsible for implementing the Delta-specific aspects of commit as defined by this spec, but are otherwise free to define their own APIs and protocols for communication with each other. **NOTE**: Filesystem-based access to catalog-managed tables is not supported. Delta clients are expected to discover and access catalog-managed tables through the managing catalog, not by direct listing in the filesystem. This feature is primarily designed to warn filesystem-based readers that might attempt to access a catalog-managed table's storage location without going through the catalog first, and to block filesystem-based writers who could otherwise corrupt both the table and the catalog by failing to commit through the catalog. Before we can go into details of this protocol feature, we must first align our terminology. ## Terminology: Commits A commit is a set of [actions](#actions) that transform a Delta table from version `v - 1` to `v`. It contains the same kind of content as is stored in a [Delta file](#delta-log-entries). A commit may be stored in the file system as a Delta file - either _published_ or _staged_ - or stored _inline_ in the managing catalog, using whatever format the catalog prefers. There are several types of commits: 1. **Proposed commit**: A commit that a Delta client has proposed for the next version of the table. It could be _staged_ or _inline_. It will either become _ratified_ or be rejected. 2. **Staged commit** : A commit that is written to disk at `_delta_log/_staged_commits/ . .json`. It has the same content and format as a published Delta file. - Here, the `uuid` is a random UUID that is generated for each commit and `v` is the version which is proposed to be committed, zero-padded to 20 digits. - The mere existence of a staged commit does not mean that the file has been ratified or even proposed. It might correspond to a failed or in-progress commit attempt. - The catalog is the source of truth around which staged commits are ratified. - The catalog stores only the location, not the content, of a staged (and ratified) commit. 3. **Inline commit** : A proposed commit that is not written to disk but rather has its content sent to the catalog for the catalog to store directly. 4. **Ratified commit** : A proposed commit that a catalog has determined has won the commit at the desired version of the table. - The catalog must store ratified commits (that is, the staged commit's location or the inline commit's content) until they are published to the `_delta_log` directory. - A ratified commit may or may not yet be published. - A ratified commit may or may not even be stored by the catalog at all - the catalog may have just atomically published it to the filesystem directly, relying on PUT-if-absent primitives to facilitate the ratification and publication all in one step. 5. **Published commit** : A ratified commit that has been copied into the `_delta_log` as a normal Delta file, i.e. `_delta_log/ .json`. - Here, the `v` is the version which is being committed, zero-padded to 20 digits. - The existence of a ` .json` file proves that the corresponding version `v` is ratified, regardless of whether the table is catalog-managed or filesystem-based. The catalog is allowed to return information about published commits, but Delta clients can also use filesystem listing operations to directly discover them. - Published commits do not need to be stored by the catalog. ## Terminology: Delta Client This is the component that implements support for reading and writing Delta tables, and implements the logic required by the `catalogManaged` table feature. Among other things, it - triggers the filesystem listing, if needed, … [truncated]