stat returns 404 error for directories

[skaldapp]

Unfortunately, another serious issue has been identified in the S3 background operations.

The core of the problem lies in the unconventional design of the folder system in Amazon S3. In Amazon S3, folders do not exist in the conventional sense of a file system. S3 is a "flat" object storage service, wherein all files (objects) reside at the same level within a bucket. What appears as folders in the management console or via file managers is merely a visual abstraction based on object key names. Consequently, folders in S3 can be categorized into two types:

1. Explicitly created folders: These are generated either by executing a command such as
   aws s3api put-object --bucket bucket_name --key folder_name/ (the trailing slash is mandatory)
   or by clicking the "Create Folder" button in the AWS Console. In such cases, S3 creates an empty object of 0 bytes whose key ends with a forward slash (e.g., folder_name/). This construct exists solely to enable the interface to display an "empty directory." The presence of such a directory can be verified using the headObject API call, which will return a successful response (200 OK) along with metadata for this special empty object:

   • ContentLength: 0
   • ContentType: typically application/x-directory (set by the Console)
   • ETag and LastModified: present, as with any standard object

2. Implicit folders: These arise automatically when uploading files with a hierarchical key path. For instance, uploading a file with the key photos/vacation/sea.jpg will cause the prefixes photos/ and photos/vacation/ to appear as folders.

In the majority of use cases, folders in S3 are of the second type. Unfortunately, it is precisely these implicit folders that introduce significant complications. For example, invoking headObject for the key photos/ will return a 404 Not Found error. From S3's perspective, no object with that exact key exists; only objects whose keys begin with that prefix are present. Furthermore, when all objects within such implicit "folders" are deleted, the folder structure disappears automatically.

This behavior is documented in the AWS guide: Organizing objects in the Amazon S3 console by using folders. Additionally, a comprehensive article on this topic is available: The Strange Case of Amazon S3 Bucket Folders.

Returning to the zenfs background operations for S3: during read, write, and similar operations, zenfs performs stat calls on the directories containing the target files. As expected, it receives a 404 Not Found response for implicit folders, which subsequently leads to failures in file read/write operations. This issue stems directly from the aforementioned peculiarity of S3's object model.

I have endeavored to describe the problem in as much detail as possible. Should you require further clarification, I would be glad to address any questions.

[skaldapp]

Further to my previous message. In Amazon S3, determining the existence or absence of an arbitrary object, whether it constitutes a file or a directory, is accomplished through a rather non-intuitive methodology. Specifically, this is achieved by verifying whether at least one object exists under the designated prefix. Although it may appear counterintuitive, this approach remains fully effective when applied to individual files as well. Below is a representative implementation of an object existence verification function (returning true if the object exists, false otherwise):

const isExists = async (Bucket: string, Prefix: string) => {
  const MaxKeys = 1,
    { KeyCount } = await s3.send(
      new ListObjectsV2Command({ Bucket, MaxKeys, Prefix }),
    );
  return !!KeyCount;
};

This implementation reliably verifies the existence of both files and directories, irrespective of their creation method. Illustrative usage examples are provided below:

isExists("my-bucket", "test.md");
isExists("my-bucket", "folder/test.md");
isExists("my-bucket", "folder/folder/test.md");
isExists("my-bucket", "folder");
isExists("my-bucket", "folder/");
isExists("my-bucket", "folder/folder");
isExists("my-bucket", "folder/folder/");

[james-pre]

Hey @skaldapp,

Thanks for reporting this. ZenFS keeps track of directories but by any implicit rules but through an explicit object.

Every file (whether that be a symlink, directory, regular file, FIFO, etc.) is stored as a object. These object have associated metadata. This metadata is what determines file type. For example, directories' have a mode that satisfies mode & S_IFMT == S_IFDIR.

While this is different from how you are describing directory determination, it's important to realize that ZenFS tries to implement a complete file system that can't work with more basic rules. For example, symlinks wouldn't fit into the system you described.

What I can do is look into a potential migration path that will attempt to add the missing metadata to S3 buckets that weren't created using ZenFS. That might look something like this:

const s3 = await resolveMountConfig({
    backend: S3Bucket,
    // ... 
}); 

// example migration
await s3.repairMissingMetadata(); 

await configure(...);

However that feature is outside the scope of this bug report so I'd handle it in a dedicated issue.

I apologize for the inconvenience. For now you should be able to work on this by re-creating the file system using ZenFS' API so that metadata is correctly added.

[james-pre]

Just had a thought that maybe I can look into adding a trailing slash after directory objects since that would improve compatibility with the S3 console, however that requires some more investigation on my part.