Tombstones Spec changes

[zimeon]

From use case Flagging File Corruption #14, Supporting Physical File Level Deletion #42, and Collapsing Object Versions #46. See also #670 for implementation notes changes.

[zimeon]

Summary of proposed specification changes based on 2025-08-18 Extended Editors Meeting

Tombstones

Handling file loss, file corruption, and version collapse all require a change in the OCFL v1 assumption of object version immutability. Even in a notionally immutable system, corruption can still occur, and so these facilities may be used in a system that otherwise adheres to version immutability.

OCFL v2 introduces the notion of tombstones to record information about files that are either no longer present in prior versions of an object, or have become corrupted. The inventories of prior versions are not changed, but the meaning of object validity is: an object with tombstones may be valid even if prior versions are invalid because of missing content of mismatching digests. In a valid object the latest version is always valid. It may no longer be possible to extract the state of all prior object versions.

Because the Storage Root conformance declaration must be for a specification version at least as high as all objects contained (no change), tombstones may appear only in OCFL v2 or higher Storage Roots. The use of this and other v2 features is declared in the ocfl_features.json file.

To record that a content file within an object has been deleted, vanished, or became corrupted, a new version is created that adds tombstones and adjusts the manifest. It is recommended that no other changes be made to the object at the same time as the recording of the deletion or corruption. The creation of a new version provides an opportunity to write a new version message with user information, time, and any other human-readable information about the changes being recorded.

The following table summarizes different situations where content is removed, lost or changed, and how they can be handled with OCFL v1 and v2 features:

Operation	Does the file still exist on storage?	Is the file still recorded in the manifest?	Is the file referenced in the state of the current version?	Is the file referenced in tombstones (with original digest)?	Do you add a tombstone to a child object that links to the file?
Purge object (delete complete object including all content files) (OCFL v1+)	No	n/a	n/a	n/a	Yes (or re-write child object as well)
Purge file by rewriting object (OCFL v1+)	No	No	No	No	Yes (or re-write child object as well)
Exclude file from version state (OCFL v1+ "soft delete/unlink")	Yes	Yes	No	No	No (no impact on child object)
Tombstone - no file (OCFL v2)	No (lost or deleted)	No	No	Yes	No (inherit tombstone from parent)
Tombstone - file with changed digest (OCFL v2)	Yes (corrupted file is there but the checksum doesn't match the original checksum)	Yes (with current/new digest)	No	Yes	No (inherit tombstone from parent)
Tombstone - file with no digest (OCFL v2)	Yes (file present or listed but unreadable)	Yes (but no digest,  recorded as blank in manifest)	No	Yes	No (inherit tombstone from parent)

Specification Changes for OCFL Objects

The manifest block records all content files present within the object. An empty digest in the manifest indicates that a file with the given content path exists on the filesystem but cannot be read or cannot be read consistently.

• The value of the manifest key MUST be a JSON object, and each key MUST be either an empty string, or else correspond to a digest value key found in one or more state blocks of the current and/or previous version blocks of the OCFL Object.
• (No changes to specification of manifest values and the construction of content paths.)

The tombstones block, if present, records all content files no-longer present within the object with the given digest. This includes both files that have been deleted or otherwise no longer exist, and files that have been changed or corrupted and no longer have the given digest. The digests in the tombstones block MUST appear in one or more prior version states.
A tombstones block MAY be present in OCFL v2.0 and high inventories, and MUST NOT be present inventories for earlier OCFL version inventories.

• The value of the tombstones key MUST be a JSON object, and each key MUST correspond to a digest value key found in one or more state blocks of previous version blocks of the OCFL Object.
• The value for each digest in the tombstones block MUST be a JSON list.
  • In the case of content in the object, the previous content path or paths of the content are given in the list.
  • In the case of digests that were linked content, the value is an empty list and the digest does not appear in the links block.

As with OCFL v1.x, the state block of the latest version MUST refer only to digests that appear in the manifest, and MUST NOT contain an empty digest. However, the state blocks for prior versions MAY refer to digests that appear in the tombstones block. It may be impossible to extract the state of prior versions.

A change from v1.1 is that it is now possible to have a digest in the manifest block that does not appear in any version state. However, there MUST be an entry in the tombstones block for the corresponding content path with a digest that does appear in a version state.

Example of object with single file that is removed entirely

{
  "digestAlgorithm": "sha512",
  "head": "v2",
  "id": "http://example.org/tombstone_delete_single_file",
  "manifest": { },
  "tombstones": {
    "7545b8...f67": [ "v1/content/file.txt" ]
  },
  "type": "https://ocfl.io/2.0/spec/#inventory",
  "versions": {
    "v1": {
      "created": "2020-10-12T01:00:00Z",
      "message": "One file",
      "state": {
        "7545b8...f67": [ "file.txt" ]
      },
      "user": {
        "address": "mailto:alice@example.org",
        "name": "Alice"
      }
    },
   "v2": {
      "created": "2021-01-00T02:00:00Z",
      "message": "The one file had to be deleted entirely for legal reasons",
      "state": { },
      "user": {
        "address": "mailto:alice@example.org",
        "name": "Alice"
      }
    }
  }
}

Example of marking file corruption and file than cannot be read

{  
  "digestAlgorithm": "sha512",
  "head": "v2",
  "id": "http://example.org/tombstone_unreadable_file_and_corrupt_file",
  "manifest": {
    "": [ "v1/content/file1.txt" ],
    "aaa143...79a": [ "v1/content/file2.txt" ]
  },
  "tombstones": {
    "7545b8...f67": [ "v1/content/file1.txt" ],
    "fe4512...e47": [ "v1/content/file2.txt" ]
  },
  "type": "https://ocfl.io/2.0/spec/#inventory",
  "versions": {
    "v1": {
      "created": "2023-10-02T12:00:00Z",
      "message": "Two files",
      "state": {
        "7545b8...f67": [ "file1.txt" ],
        "fe4512...e47": [ "file2.txt" ]
      },
      "user": {
        "address": "mailto:alice@example.org",
        "name": "Alice"
      }
    },
   "v2": {
      "created": "2024-09-20T10:09:00Z",
      "message": "File 1 cannot be read reliably, remove from state. File 2 is corrupted with a different checksum, change name but retain in state",
      "state": {
        "aaa143...79a": [ "file2_corrupted.txt" ]
      },
      "user": {
        "address": "mailto:eve@example.org",
        "name": "Eve"
      }
    }
  }
}

Example of retaining only uncorrupted file of two original files

In this example we choose not to retain the corrupted file in the latest version state. This means that there is a (corrupted) file on disk and hence in the manifest that does not appear in any version state.

{
  "digestAlgorithm": "sha512",
  "head": "v2",
  "id": "http://example.org/tombstone_retain_uncorrupted_file",
  "manifest": {
    "7545b8...f67": [ "v1/content/file1.txt" ],
    "aaa143...79a": [ "v1/content/file2.txt" ]
  },
  "tombstones": {
    "fe4512...e47": [ "v1/content/file2.txt" ]
  },
  "type": "https://ocfl.io/2.0/spec/#inventory",
  "versions": {
    "v1": {
      "created": "2023-10-02T12:00:00Z",
      "message": "Two files",
      "state": {
        "7545b8...f67": [ "file1.txt" ],
        "fe4512...e47": [ "file2.txt" ]
      },
      "user": {
        "address": "mailto:alice@example.org",
        "name": "Alice"
      }
    },
   "v2": {
      "created": "2024-09-20T10:09:00Z",
      "message": "file2.txt was corrupted, retain only file1.txt in state",
      "state": {
        "7545b8...f67": [ "file1.txt" ]
      },
      "user": {
        "address": "mailto:eve@example.org",
        "name": "Eve"
      }
    }
  }
}

Specification Changes for OCFL Storage Root

The ocfl_features.json file at the Storage Root level MUST be present for OCFL v2 and higher storage roots, and it MUST indicate that tombstones are used if they are used within any object in the Storage Root. A Storage Root is invalid if it contains an object using tombstones yet the use of tombstones is not declared in ocfl_features.json.

Relationship between packaging and tombstones (see also separate issue on packaging)

Packages are designed to be transparent to the inventory - one usually simply refers to files inside the packages

• If a per-version package is corrupted or lost then we can record the loss of files that were in it in the inventory of a new object version. Whether all the files in a version are lost as a result of loss of one of a multipart package set is dependent on the packaging method.
• We imagine allowing packages.json to also have a tombstones block to indicate package files that are corrupt. A new version will have to be written that indicates file changes in the inventory. Motivation for recording old (good) digest is that we might want to use it identify an uncorrupt copy on a replica storage system

Example of corrupted package

Difference between digests for v3/mypackage.v3.z02 indicates that it is corrupt

{
  "digestAlgorithm": "sha512",
  "type": "https://ocfl.io/1.1/spec/#packages",
  "manifest": { 
    "aaa..999": [ "v3/mypackage.v3.z02" ],
    "": [ "v3/mypackage.v3.z03" ],
    "ces..229": [ "v3/mypackage.v3.zip" ]   # this one little part is good!
  },
  "tombstones": { 
    "cde..123": [ "v3/mypackage.v3.z01" ],  # deleted/lost
    "ade..789": [ "v3/mypackage.v3.z02" ],  # corrupted
    "afb..786": [ "v3/mypackage.v3.z03" ]   # unreadable
  }
  "format": "zip", 
  "formatVersion": "6.3.10"
 }

[srerickson]

I'm struggling to understand the "file present or listed but unreadable" scenario, in which the the digest is recorded with an empty string.

An empty digest in the manifest indicates that a file with the given content path exists on the filesystem but cannot be read or cannot be read consistently.

Certainly the underlying storage layer for an OCFL repository might be degraded or unreliable, but is it really necessary, or even useful, to represent that fact at the OCFL level (especially considering that degradation might be temporary)? I'm concerned this risks blurring abstractions.

[zimeon]

Thanks for your comment @srerickson . I agree that an empty digest is a somewhat ugly edge case but it does give a validator a clear message to not throw and error for something that exists but isn't otherwise listed, and to not try to check the digest. Other options seem unappealing:

• Do not list in manifest, include in tombstones - the manifest no longer lists everything on disk, the rule has to be that anything listed in tombstones may or may not still be present on disk with an undefined digest
• List in manifest (with what digest?) and include in tombstones - This then changes the expectation of being able to check the digest of everything listed in the manifest

Do you have other ideas for how we could support the case of something on disk for which we cannot create a digest (pr a consistent digest)?

[srerickson]

My thinking is that "the case of something on disk for which we cannot create a [consistent] digest" is a failure mode that OCFL does not need to support or account for. (Edit: or it already accounts for it in the exactly the way it should: through a validation error). If the storage layer is found to be broken such that files can't be reliably read, what is the use of adding new object versions describing this fact?

I'm still working on articulating a general principle behind my objection, but I think it might be this: An OCFL storage root should always be transferable to different storage media without needing to create new versions for objects in the root. This, of course, assumes that the storage root exists on reliable digital media. The empty digest proposal doesn't satisfy this test because, presumably, you wouldn't be able to recreate the broken file on a new, working file system.

[zimeon]

Based on @srerickson's #669 (comment) above, it seems possible that (assuming we keep all the current tombstones features) some repositories might use some affordances (maybe recording lost/deleted and corrupted) but not others (empty digest for unreadable). We could allow this to be expressed in the Storage Root Feature Flags (#671) as either separate features or additional information in the feature object, such as:

{
  "tombstones" : { "used": true , "deletion": true, "corrupted": true, "unreadable": false},
  "links" : { },
  "version_packaging":{ "used": false }
}