TES Format Specification
TES is the custom archive format written by the Archive outbound when format is custom. Files use the .tes suffix.
The design goal is crash safety without a write-ahead log: an append-only, copy-on-write file with a double super block. A commit never mutates committed data in place, so an interrupted commit always leaves the previous committed state intact.
All integers are big-endian.
File Layout
A TES file is a sequence of three regions:
| Offset | Size | Region |
|---|---|---|
0 |
4096 | Super Block A |
4096 |
4096 | Super Block B |
8192 |
X Bytes | Append region |
The append region holds data blocks and index blocks, written in append order and never overwritten within their committed range.
Super Block
Each super block occupies a fixed 4096-byte page. Two slots exist; a commit writes the newer super block to the slot not used by the previous commit (alternating by transaction parity). On open, both slots are read and the valid slot with the highest transaction id wins.
| Offset | Size | Field | Description |
|---|---|---|---|
0 |
4 | Magic | TES1 |
4 |
2 | Version | Format version, currently 1 |
8 |
8 | Transaction | Transaction id, incremented per commit |
16 |
8 | Root Offset | File offset of the index block |
24 |
8 | Root Length | Byte length of the index block |
32 |
8 | File Size | Committed high-water mark |
40 |
8 | Entry Count | Number of records in the index |
48 |
8 | Created At | Commit time, Unix seconds |
4092 |
4 | Checksum | CRC32 (IEEE) of bytes 0 to 4092 |
A super block is valid when its magic matches and its checksum verifies. A torn write to a super block fails the checksum, and the reader falls back to the other slot.
Index Block
The index block is the central directory: the full set of records, rewritten on every commit and pointed to by Root Offset. Bytes past File Size (an interrupted append) are ignored.
| Offset | Size | Field | Description |
|---|---|---|---|
0 |
4 | Record Count | Number of records that follow |
4 |
X Bytes | Records | Record Count records |
Record
Each record describes one version of one path from one inbound. Variable-length fields are length-prefixed strings: a 2-byte length followed by the bytes.
| Size | Field | Description |
|---|---|---|
| 2 + M | Path | Path relative to the inbound root |
| 2 + M | Inbound | Tag of the inbound that produced the entry |
| 1 | Type | Entry type (see below) |
| 1 | Tombstone | 1 if the path was deleted, else 0 |
| 4 | Mode | Permission bits |
| 8 | ModTime | Modification time, Unix seconds |
| 8 | Size | Content length in bytes, before compression |
| 4 | User ID | Owner user id |
| 4 | Group ID | Owner group id |
| 2 + M | User Name | Owner user name |
| 2 + M | Group Name | Owner group name |
| 2 + M | Link Target | Symlink or hardlink target |
| 4 | Device Major | Device major number |
| 4 | Device Minor | Device minor number |
| 8 | Version | Transaction id that wrote this record |
| 8 | Commit Time | Commit time of that transaction, Unix seconds |
| 8 | Data Offset | File offset of the stored content |
| 8 | Data Length | Stored length at Data Offset, after compression |
| 32 | Checksum | SHA-256 of the content before compression |
| 32 | Source Checksum | SHA-256 of the source content as read by the inbound, zero when not computed |
| 1 | Codec | Content codec (see below) |
| 1 | Enc | Content encryption (see below) |
| 2 | Xattr Count | Number of extended attributes that follow |
| ... | Xattrs | Xattr Count extended attributes |
Xattr
Each extended attribute is a length-prefixed name followed by a length-prefixed value. The value is raw bytes and may be binary (for example a POSIX ACL).
| Size | Field | Description |
|---|---|---|
| 2 + M | Name | Attribute name |
| 4 | Length | Value length in bytes |
| Length | Value | Attribute value |
Type
| Value | Type |
|---|---|
0 |
Regular file |
1 |
Directory |
2 |
Symbolic link |
3 |
Hard link |
4 |
Named pipe |
5 |
Character device |
6 |
Block device |
7 |
Socket |
Codec
The codec applied to the stored content. Data Length counts the stored (post-codec) bytes; Size and Checksum describe the original content, so the checksum is stable across codecs and drives deduplication.
Source Checksum differs from Checksum when wrapper outbounds transformed the content before it reached the archive: Checksum then hashes the transformed bytes, while Source Checksum still hashes the source content, which is what compare rules with checksum enabled consult.
| Value | Codec |
|---|---|
0 |
None (stored) |
1 |
gzip |
2 |
zstd |
3 |
lz4 |
Enc
The encryption applied to the stored content, after compression. Each encrypted blob carries its own key-wrapping header (salt) and AEAD-chunked ciphertext, so the reader needs only the password; the algorithm is taken from this field. Data Length counts the encrypted (and, if applicable, compressed) bytes.
| Value | Encryption |
|---|---|
0 |
None |
1 |
aes-256-gcm |
2 |
chacha20 |
Data Block
A data block is the stored content of a regular file, written to the append region at Data Offset with Data Length bytes. When a codec and/or encryption is set, the content is compressed first and then encrypted, so the block holds encrypt(compress(content)); reading reverses the order. It carries no header of its own; the record locates and bounds it. Directories, links, and special files have Data Length 0.
Deduplication
Before appending a data block, the writer compares the content's SHA-256 (of the original, pre-codec bytes) against the blocks already written, including those from earlier transactions. On a match the record reuses the existing block's Data Offset, Data Length, and Codec instead of storing a second copy; the bytes speculatively appended for the write are left as an ignored tail past File Size. Identical content is therefore stored once regardless of path or transaction.
Compaction
Compaction rewrites a file into a fresh one holding only the latest non-tombstone record for each inbound and path, discarding superseded versions, tombstones, and the data blocks they alone referenced. Data blocks are relocated verbatim — codec and checksum preserved, and re-deduplicated during the copy — so no content is decoded or re-encoded. The rewrite is atomic (a sibling temp file is committed, then renamed over the original) and restarts Version history at 1, trading point-in-time history for reclaimed space.
Commit Protocol
A commit writes one transaction:
- Append all data blocks for the transaction.
- Append the index block containing every record, old and new.
fsync.- Write the new super block to the older slot.
fsync.
If a crash occurs before step 4 completes, the older super block still points at the previous index, and the appended data and index of the interrupted transaction lie past the previous File Size, where they are ignored and later overwritten. If a crash tears the super block write in step 4, its checksum fails and the reader uses the other slot. A committed transaction is therefore all-or-nothing.
Versioning
Writing the same path from the same inbound in a later transaction appends a new record with a higher Version. The reader selects, per inbound and path, the record with the highest Version, or the highest Version at or before a requested point in time using Commit Time. The same path from two different inbounds is kept as two distinct entries, not versions of one. A record with Tombstone set marks the path as deleted at that version and is not returned.
Reading
To read the current state:
- Read both super blocks, select the valid one with the highest Transaction.
- Read Root Length bytes at Root Offset and parse the index.
- For each inbound and path, keep the record with the highest Version, skipping tombstones.
- Read each record's content from Data Offset for Data Length bytes.