Skip to content

CIP-4.md

Latest commit

 

History

History
131 lines (104 loc) · 8.08 KB

CIP-4.md

File metadata and controls

131 lines (104 loc) · 8.08 KB
CIP Title Author Status Type Category Created
4 Upload-File EldenYang Draft Standards Core 23/03/22

Abstract

The document upload function introduced in this CIP refers to the process that users who have purchased storage space store their data on the CESS network. It is designed to address the situation where the three parties, namely the user, storage node, and consensus node, do not trust each other, and create a more comprehensive file upload process. This enables the three parties to authenticate whether the file has been correctly stored or to determine if their respective tasks have been performed correctly.

Motivation

  1. The current file upload process has certain flaws that cannot guarantee the interests of the three parties, namely the user, storage node, and consensus node. To ensure that storage nodes truly and effectively store user data, a more comprehensive design is needed.
  2. Improve the user experience and enhance the speed of file storage from the user's perspective.
  3. Reduce the computational pressure on consensus nodes and distribute tag calculation tasks.

Specification

Overall Process

The upload process is divided into two stages. In the first stage, the user slices and duplicates the file according to the data specification, generates an upload declaration, and assigns storage nodes based on the chain. The user then transfers the specified fragments to the designated storage nodes. After the storage nodes receive the correct file, they call the transaction to report the transfer result. When all storage nodes have completed the report, the first stage of the upload is completed. The chain will create file metadata, set the file status to Calculate, and the file can be downloaded in this state but cannot be deleted and is not within the scope of challenge. In the second stage, a sufficient time-based task will be initiated. Within the specified time, the storage nodes need to find a consensus node to calculate the tag for the fragments on their own. After the time expires, the file status will change to Active, and the order will be completed. At the same time, regardless of whether the storage node has completed the task, the file will be included in the challenge.

Blockchain Node Interface

Data Structure

Order metadata structure includes the following fields:

pub struct DealInfo<T: Config> {
	pub(super) stage: u8,
	pub(super) segment_list: SegmentList<T>,
	pub(super) needed_list: SegmentList<T>,
	pub(super) user: UserBrief<T>,
	pub(super) assigned_miner: MinerTaskList<T>,
	pub(super) share_info: BoundedVec<SegmentInfo<T>, T::SegmentCount>,
	pub(super) complete_list: BoundedVec<AccountOf<T>, T::FragmentCount>,
}

stage: The stage of the order (first stage or second stage).

segment_list: A list of segments for each file after it has been sliced and redundantly stored, and each segment corresponds to a list of fragments.

needed_list: A list of segments that need to be stored for this order, including the fragments corresponding to each segment.

user: The declarer of the order.

assigned_miner: A list of storage node tasks that specifies the fragments each miner needs to store.

share_info: Information about segments in the order that do not need to be uploaded and can be shared.

complete_list: A list of storage nodes that have completed the task.

File metadata structure includes the following fields:

pub struct FileInfo<T: Config> {
	pub(super) completion: BlockNumberOf<T>,
	pub(super) stat: FileState,
	pub(super) segment_list: BoundedVec<SegmentInfo<T>, T::SegmentCount>,
	pub(super) owner: BoundedVec<UserBrief<T>, T::OwnerLimit>,
}

completion: The block height at which the file metadata was created.

stat: The status of the file.

segment_list: A list of data segments for the file, as well as information about the data blocks after being sliced and redundantly stored.

owner: A list of owners of the file.

Data segment information includes the following fields:

pub struct SegmentInfo<T: Config> {
	pub(super) hash: Hash,
	pub(super) fragment_list: BoundedVec<FragmentInfo<T>, T::FragmentCount>,
}

hash: The hash of the data segment.

fragment_list: A list of information about the data blocks after being sliced and redundantly stored.

Data block information includes the following fields:

pub struct FragmentInfo<T: Config> {
	pub(super) hash: Hash,
	pub(super) avail: bool,
	pub(super) miner: AccountOf<T>,
}

hash: The hash of the data block.

avail: A flag indicating whether the data block is available or not.

miner: The account of the storage node that is storing the data block.

Order Declaration

Overview After the user processes the file according to the specification, the user can call the transaction to declare an order. After the chain checks that the file is correct, one of the following three situations will occur:

  1. The file already exists on the chain, and the declarer becomes one of the holders of the file.
  2. Some of the data segments of the file already exist on the chain and can be shared, so there is no need to upload them, or there are no shared data segments, and all the segments need to be uploaded. Based on the check of the sharing status, the storage nodes are assigned, and the order information is generated.
  3. All data segments of the file already exist on the chain, and there is no need to upload them. A new file meta-information will be created directly.

interface define

Param Type Description
origin OriginFor Signature Source
file_hash Hash File hash
deal_info SegmentList Data Segment Redundancy Information List
user_brief UserBrief User Summary Information

Transfer Report

Function Overview: After receiving the specified fragment assigned in the order, the storage node needs to call the transaction to report the transfer result. When the last storage node in the order completes the report, it means that the first stage of the order is completed and the file is successfully uploaded. The chain will generate file metadata and start the second stage timing task.

Under time-permitting conditions, a storage node can accumulate up to a maximum of five order transfer results to report together. The chain will check the allocation situation in the order information one by one to see if the storage node exists and make corresponding processing. Orders with abnormal situations will be recorded separately in the failure list and thrown through events.

Interface Definition.

Param Type Description
origin OriginFor Signature Source
deal_hash Vec Order hash list

First Stage Timing Task - Reassign Storage Nodes

Overview When an order is generated, a timing task is started. If the assigned storage nodes fail to report the transmission result within the specified time, the system will reassign storage nodes for this order. The maximum number of reassignments is 5. If the limit is reached, the file upload will fail and the order information will be deleted. Interface Definition Automatically called by the chain using Root permission.

Second Stage Timing Task - Tag Calculation Completion

Overview When the last storage node of an order completes the transmission result report, the order enters the second stage. At the same time, the chain starts a timing task with sufficient time, and after the expiration, the file status will be changed to active. Also, the storage space for all storage nodes in this order will be calculated, and included in the scope of rewards and challenges. Interface Definition Automatically called by the chain using Root permission.

Copyright

Copyright and related rights waived via CC0.