Giter Site home page Giter Site logo

documents's People

Contributors

jing-git avatar streetycat avatar

Stargazers

avatar

Watchers

avatar avatar

Forkers

streetycat jing-git scatteredcode

documents's Issues

1.2 Introduction CYFS for Developers: Named data and data property rights

Named data and data property rights

Named data

  1. Data name

Any Data is named with it's hash, any modification will change its name. This is a key design, it will ensure that the data you reference has not been tampered with.

  1. Named Data

Any data named in this way is called Named Data.

  1. Named Object

Data is divided into structured data and unstructured data, and structured data is called Object, Any Object named in this way is called Named Object.

  1. Immutable and mutable

In principle, Object is immutable, if you need to change it, create a new object.

But for convenience, there is a design to modify it:

The content of Object is split into two parts:

  • Desc: immutable, the ObjectId is calculate with this part.the size is limmited at 64KBytes.
  • Body: mutable, It has nothing to do with ObjectId. We can update the body without modify the ObjectId. Therefore, developers must consider how to ensure that the Body is the version we need.

ObjectId structure

  1. Area

Every object should belong to a different Area, and accept the legal supervision of the corresponding Area.

  • country: 9bits
  • carrier: 4bits
  • city: 13bits
  • inner: 8bits // set by the creator.
  1. ObjectType

Any Object requires a field that explicitly specifies its type, it's ObjectType.

All types are divided into 4 categories:

  • Standuard: It's the basic types defined in cyfs-base. Collected in the module cyfs-base.
  • Core: Objects with general semantics gradually formed during the development of CYFS. Collected in the module cyfs-core.
  • DecAppObject: Objects defined by an application developer to solve a specific problem in a specific application.
  • Data: Short immediate value object.
  1. ObjectId structure

The name of a Named Object is ObjectId.

The main component of ObjectId is the SHA256 of ObjectDesc, It also contains the type and area information.The total length is 32Bytes(256bits).

The structure is as follow:

ObjectId = Catgory(2bits) + DataContent(38bits) | CommonObjectId(38bits)

Catgory = Standuard(1) | Core(2) | DecAppObject(3) | Data(0)

if Catgory = Data(0) {
    DataContent = 0b_000000 + ImmediateValue(248bits)
} else {
    CommonObjectId = TypeDetail(4bits) + Area(34bits) + SHA256(`ObjectDesc`)[5..]
    TypeDetail = ObjectTypeCode | FieldFlags
    if Catgory = Standuard(1) {
        ObjectTypeCode = Device(1) | People(2) | Group(3) | AppGroup(5) | UnionAccount(6) | Chunk(7) | File(8) | Dir(9) | Diff(10) |
            ProofOfService(11) | Tx(12) | Action(13) | ObjectMap(14) | Contract(15)
    } else {
        FieldFlags = is_area_exist(1bit) + is_single_exist(1bit) + is_mult_key_exist(1bit) + is_owner_exist(1bit)
        is_area_exist = 1 | 0
        is_single_exist = 1 | 0
        is_mult_key_exist = 1 | 0
        is_owner_exist = 1 | 0
    }
    Area = country(9bits) + carrier(4bits) + city(13bits) + inner(8bits)
    * if the Area is not set the bits shoud be set to 0.
}

Data property rights

  1. Entitled object

An object is said to be entitled object if it can provide proof of the authenticity of another object.It can be the owner of an object to own it.A common method of proof is the signature, so, most entitled object contains a key-pair,for example:People and Device.but a Group contains many People as the members, Its proof requires the signature of a majority of members.

  1. Owned object

An object is said to be owned object if it's created with a owner, most public object is owned object,Any object of external origin without owner and signature is untrusted.

Owner is contained in the ObjectDesc, it's immutable field, it's always the creator.

  1. Right to profit

The Owner is immutable, but there are many tradable objects(for example: NFT). the updated property is the Right to profit, that usually occurs on the chain. The immutable Owner strongly protects the copyright of the objects.

5.2 Introduction CYFS for Developers: R Link

R Link

It's like below:

cyfs://$target_zone.r/$dec_id/$inner_path[?mode=json&mime=txt]

The R Link read the specified field mounted on the RootState of a DECApp. it's usually used to access the mutable Objects.

  • $target_zone: As the domain for http, It represents the zone who will receive and responce the request.
  • $dec_id: The DecApp(like service of http) which will maintain the state, and responce the request.
  • $inner_path: The path of the Object saved on RootState.

Different from O link, the content specified by R link can be modified by the owner, It behaves much like http, and we can always use it instead of the http.

Domain for R Link

Different from O link, the domain of R link is depend on the DecApp. So, there are cross domain issues and defense. We must open the permissions to the target DecApp to allow it's access.

1.1 Introduction CYFS for Developers: Hierarchy and topology of CYFS protocol

Hierarchy and topology of CYFS protocol

Let's look at the readme in official repository first.

Hierarchy and topology of CYFS protocol

The above is the overall architecture diagram of CYFS, which can help to establish a macro understanding of the design of CYFS. The core design of CYFS can be decomposed into the following parts

  • Design of NamedObject and double-section construction of cyfs://o/$ownerid/$objid (Object Link).
  • The BDT protocol improves TCP/IP: the network protocol evolves from address-oriented to trusted identity-oriented.
  • MetaChain upgrades DNS, realizes decentralized addressing, and realizes Data is Account.
  • The Owner deploys its own server OOD (Owner Online Device) on the network, which supports the decentralized accessibility of cyfs://
  • Design proofs of storage and proofs of service based on the game-based consensus theory of dispute before going on the chain, and realize the decentralized high reliability and high availability of cyfs://
  • The off-chain delivery of computing resources is separated from on-chain matching, and an evolvable economic model is used to build decentralized shared cloud computing
  • Data property rights analysis based on the input and results of trusted computing, to realize the evolution from on-chain smart contracts to off-chain Data Exchage Contract (DEC).
  • The Service of Web3 DEC App is installed on everyone's OOD, realizing the decentralization of application services

There are also two auxiliary connectivity devices that are not mentioned:

  • SN: Super Node.It's service that assists devices deployed in NAT to communicate with each other. In CYFS, anyone can deploy a SN server, and anyone can freely choose the appropriate SN server.Both parties complete the transaction through SN service contract and service certificate.

  • PN: Proxy Node. They are used as proxies to transfer data when both communication devices cannot complete a direct connection. As the SN service, anyone can deploy a PN server, and anyone can freely choose the appropriate PN server.Both parties complete the transaction through PN service contract and service certificate.

Explain some new words:

  • Zone: The devices owned by the same owner. It can be represented by its owner. Unowned devices are allowed, it will form a zone that include it only. All devices can connect each other directly.

  • OOD: Owner Online Device, The central device of a zone, it builds a star network with other devices, it assists other devices in the same zone to communicate with the other zones. And it is the centralized storage of data in the zone, running as a service(gateway) in http. It's the most important device for a zone in CYFS, it's the personal server for the user.

  • MetaChain: It is a blockchain system that maintains the consensus of each node in the CYFS network.Its implementation is not yet complete, just a conceptual version?

1.3 Introduction CYFS for Developers: Core components and functional modules

Core components and functional modules

I will introduce the distribution of project source code files, and then introduce the main core modules one by one.

Directory specification

The CYFS project directory is expanded as follows:

/--CYFS
    |--doc  // formal design documents
    |--scripts // some scripts for build/publish...
    |--src // the source code, We will specifically introduce the source code directory
    |--... // other files for solution: .gitignore, .eslintrc, package.json, .etc

All modules are divided into several categories and collected into corresponding directories in src:

  1. 3rd
    The main code comes from an external project, but with a few custom modifications.

  2. component
    Basic functional modules, most of which belong to this type, each implements its own functions, is referenced by other runnable instances, and has interdependence internally.

    If your module will be depended on by other modules, it should be collect in component. If part of your module will be depended on, you should split the part into a separate module.

    As examples:
    cyfs-base: It's the most basic module,Almost all modules depend on it.
    cyfs-core: It defines all normalized objects
    cyfs-util: Some auxiliary modules

    I will introduce them in other issues.

  3. meta
    The modules for MetaChain, it's just a conceptual version?
    I haven't dabbled in this part yet, I will introduce it in future.

  4. service
    Services in CYFS system, They run as independent processes on various CYFS devices.

    I will introduce them in other issues.

  5. tests

    Some test cases and sample programs. We can help to supplement and improve, it's a good entrance to enter the CYFS world, and contribute to the cause of Web3.

  6. tools

    Some tools to help users/developers to use the CYFS. We can design more useful tools.

  7. misc

    Others.

Services

  1. Devices

As introduced here, there are 2 types of devices in CYFS for users and developers. They are OOD(gateway) and runtime.The relationship is as follow:

Zone A <--> Zone Other
runtime A1 <--> OOD(gateway) A OOD(gateway) O <--> runtime O1
runtime A2 <--> <--> runtime O2
runtime ... <--> <--> runtime ...
runtime An <-->
  1. Services

In a zone, several services is running on the OOD always. These services are running depend on the gateway, the gateway is also a service, It is equivalent to nginx for http, receive and forward the requests to the target services.

gateway <--> app-manager
<--> chunk-manager
<--> file-manager
<--> ...
  • app-manager: It manages the DecApps running on CYFS, install,uninstall,update,start,restart,stop,deamon.
  • chunk-manager: The chunk-manager binary is the first experimental NDN implementation (completed in April 2020), which is quite different from the current NDN implementation. Currently this component is only used for Service and DecApp package download.
  • file-manager: Similar to chunk-manager, it is the first experimental NON implementation (completed in March 2020), and is currently only used for Service and DecApp package downloads.
  • ood-deamon: It will restart the gateway when the process is dead. It is an independent service that does not depend on gateway.
  • ood-control: It is not a standalone binary service and is used by programs like ood-daemon or ood-installer. The purpose is to complete stack activation-related functions via the http interface.
  1. DECApp

By the way, the server of a DECApp is also running as the services on OOD. They listen to requests from clients through the gateway and respond.

Thanks @weiqiushi for the correction

cyfs-base

This is the most basic component, and almost all other components depend on it.
It implement the follow functions:

  1. The basic types or structures:

  • AccessString: Structures for ACL.It refers to the design of file system permission in linux.It divides users into 6 groups:

    • CurrentDevice
    • CurrentZone
    • FriendZone
    • OthersZone
    • OwnerDec
    • OthersDec

    These groups can be combined arbitrarily to get the target group that the user wants to configure permissions.

    And we can configure 3 permissions:

    • Call
    • Write
    • Read

  • Base36: Transform the BLOB with A human-readable string ignoring case. The string will be used case-insensitively, .eg:

    • Address Bar in browser
    • FileSystem of windows

  • channel: This name is used in many occasions. Here it defines the operating environment of the CYFS system. Currently, the CYFS system has 3 operating environments:

    • nightly: The integrated environment for developers.
    • beta: The public test environment for users.
    • stable: The formal version release environment.
      Different environments are independent of each other, they are connected to different MetaChains and SNs, and cannot be interconnected.

  • endpoint: Encapsulation of IpAddr, It's used to connect into the network.

  • BuckyError: Define the Results for CYFS.

  • Name: As the domain for http, we can register a readable nick name for any object.

  • paths: The req-path(similar as the URI for http) or the path in RootState

  • ports: The listening ports when the CYFS running.

  • protocol_fields: The HTTP header fields, HTTP is used at RPC between the stack(runtime/gateway) and the other services or DECApp.

  • time: Defines various standard timestamps, and the methods to convert between them

  1. The framework for encode/decode:
  • RawEncode/RawDecode: Encode with binary.
  • Protobuf: Encode with protobuf
  • JSON: Encode with JSON
  1. Encapsulation of the crypto library:
  • AES
  • Hash: Only for SHA256?
  • KeyPair(PrivateKey/PublicKey): RSA1024/2048/3072, and SECP256K1.
  • Signature/Verify: Signature with the KeyPair
  1. The 14 standard objects:
  • Action: It seems a empty object, Maybe the design draft may not be finalized yet.
  • AppGroup: It seems a empty object, Maybe the design draft may not be finalized yet.
  • Chunk: The piece in any binary data. but there is only the Hash of the content is included in the Object system without the content itself.
  • Device: The object describing a device,it includes all the parameters used to connect each other:
    _ Endpoint
    _ SN
    _ PN
    _ Protocol version * And we can give it a readable name.
    All the fields is mutable, so, they are saved in Body.The Device will be publish on the MetaChain, Therefore,we should query the latest version from MetaChain when we realized that it was no longer effective.
  • Dir: The object describing a directory in filesystem.It includes some files in different sub-path, the file(even chunk) list will be included in the desc if the size is little enough, otherwise the content(file/chunk list) will be included in the body or in another file.If the content is not saved in desc, there should be a hash to verify the authenticity of the content.
  • File: The object describing a file in filesystem.It includes 3 fields:
    • File length, immutable, saved in desc
    • File hash, immutable, saved in desc
    • The contained chunks, immutable, saved in body, So,we must verify the file hash instead of the chunkids.
  • Diff: The delta between 2 file, it's usefull to describe different versions of the same file.
  • Group: It's used to describe an organization,The main fields is the administrators and members, they are all mutable,we should update it through the MetaChain or other synchronization methods.There is a issue for it.
  • People: Describing a natural person, There are only 3 fields currently:
    • name
    • icon
    • ood_list
      All the fields is mutable,and the new version will be pushed to the MetaChain.
  • Relation: It seems a empty object, Maybe the design draft may not be finalized yet.
  • Tx: It is the concept of tx in the blockchain, describing a transaction itself, the main field includes:
    • caller
    • message: transaction details
    • gas
  • UnionAccount: Joint account of both parties(Provider and Consumer) to the transaction.They transfer the initial coin and set their respective shares,As the contract progresses, the share is revised.All the updates must be signed by both, and executed on the MetaChain.Both parties are free to withdraw their share of coins.The UionAccount object only describes the account itself, not the current state of the account. It has 3 fields only:
    • left: party to the transaction
    • right: another party to the transaction
    • service type: type of the transation
      How to update/query the state of the account?Maybe we can find it on the MetaChain
  • Contract: Describe the commitment of both parties to the transaction (price, quality, etc.).
  • ProofOfService: Describe the confirmation of the final transaction quantity and quality.Provider and consumer must sign on the contract and receipt.
    When there is a dispute in the transaction, one party presents the ProofOfService signed by the other party and submits it to MetaChain for arbitration, and the transaction can be completed according to the contract.
    The design of the specific service contract is complicated and different from each other, so I need to elaborate on it in another document.By the way,The contract seems a temporary design.
  • ObjectMap: The container for objects, The ObjectId is calculate with all items, it will change when any item is updated, and we can regard the two versions before and after modification as two different objects.The ObjectMap can express three structures:
    • Map<Key, ObjectId>
    • Set
    • Diff: The delta between 2 ObjectMap, it can be stored in a file.
  1. Other structures for object
  • Area: Every object should belong to a different Area, and accept the legal supervision of the corresponding Area. The Area field is set when the object is generate, and is encoded in header 40 bits of the ObjectId.
  • NamedObject: The basic trait for any Object.
  • TypelessObject: It's a shell for any Object.It collects the common fields,and it includes the object content in the form of buffer.
  • AnyNamedObject: It's a abstract shell for any Object. It can be generate from any object, and we can decode it to get the original object.It is often used with TypelessObject.
  • NDNObject: It's a shell for all unstructured data(Dir,File,Diff)
  • ObjectLink: **I don't understand its exact design intent.**Maybe it's a reference to another Object.
  • ObjectSigns: It describes the signature information of an object, and the current signature independently signs Desc and Body respectively.I think there will be some risk,You are welcome to join the discussion.
  • UniqueId: Binary with a length of 16 bytes, We can fill the ObjectDesc with the random UniqueId when we need to distinguish the object with same fields in ObjectDesc:
    • No fields in ObjectDesc
    • The fields in ObjectDesc is same, but the ObjectBody will be update in different time.They are 2 different objects.
  1. cyfs-base-derive

Some usefull macro is implement in this crate:

  • RawEncode
  • RawDecode
  • ProtobufEncode
  • ProtobufDecode
  1. cyfs-base-meta

The basic structures for MetaChain.

cyfs-core

I mentioned 4 object categories in previous part. The Core object is collected in the crate cyfs-core.Many object types have been collected, most of those support the functions of the current CYFS system.

  1. common

  • Text

    It's a common Object to describe a String.You can save a short(<=31Bytes) string in the ObjectId immediately building with the ObjectIdDataBuilder.Otherwise, you can use the Text object.

    It contains 3 fields:

    • id: immutable
    • header: immutable
    • value mutable
      I can't tell the design intent from the field name, maybe it need to rename or redesign?
      You can flexibly use these three fields according to your needs, for example:
    • Make a mutable Text,fill the id with key, and file the value with the content, header for other immutable annotation.
    • Make a immutable simple Text,fill the id with the content, header for immutable annotation, and value for mutable annotation.
    • Make a immutable large(>=64KBytes,Actually less for the header of Desc) Text, fill the value with the content, and fill the id with the hash of content to verify the value, header for immutable annotation.
    • Other usage

  • Storage

    It's a common Object to describe a binary buffer(Vec),it can describe both the immutable and mutable buffer.
    It contains 3 fields:

    • id: immutable,set by the creator.
    • hash: immutable,the hash of the buffer,set None if the buffer is mutable to avoid the ObjectId is changed when the buffer is changed.
    • value: mutable, the content of the buffer.

  1. app

    It supports the running of the system service AppManager,Various related entity objects and behavior objects are defined here.

  • DecApp: Describe App Details, the main fields as follow:
    • id: immutable, UniqueId?
    • source: mutable, what?
    • icon: mutable
    • desc: mutable, the description for the App
    • source_desc: mutable, what?
    • tags: tags for the App
  • AppCmd: Describe the control commands for APP, it has 2 fields:
    • app id
    • command code:
      • Add: add the app to the app list.
      • Remove: remove the app from the app list
      • Install: install the app in my zone
      • Uninstall: uninstall the app from my zone
      • Start: start the app
      • Stop
      • SetPermission: update the permission for the app
      • SetQuota: config the limit amout of resource for the app
      • SetAutoUpdate
  • AppCmdList: Use this object when we need to execute multiple commands sequentially.
  • AppList: APP list organized by category, the main fields as follow:
    • category: immutable, the name of the APP category
    • id: immutable, set by user to manage the APP category when the AppList is created, What's the different with category field? It's unique?
    • list of applications: mutable,The latest version is maintained by OOD in the zone.
  • AppLocalList: What does the Local mean?
  • AppLocalStatus: What does the Local mean?
  • AppManagerAction: Is it related to the implementation logic of AppManager?
  • AppSetting: Configuration for the app.
  • AppStatus
    • id: immutable,DecAppId.
    • version: mutable
    • status: mutable, Why is it a boolean?
      The latest version is maintained by OOD in the zone.
  • DefaultAppList
    APPs that are automatically installed after the CYFS system is installed.
  1. group

There is a issue for it.The core objects for group is implemented in cyfs-core.

We use the blockchain structure to record the state evolution process of the Group.The consensus algorithm is Hotstuff.

  • GroupProposal
    The object that prompts the infomation of group to update.It will answer the DECApp 3 questions:
    • Who want to update?
    • What does he want to update?
    • How to update?
      The fields is follow:
    • rpath: immutable, rpath the blockchain is running on.there is a blockchain is running for each rpath of the group.
    • method: immutable,the method provided by the DECApp,it will be called with the proposal.
    • params: immutable, the parameters for the method, it's defined and parsed by the DECApp.
    • meta_block_id: immutable,the highest block on MetaChain currently,Indicates that this proposal was made after the block on MetaChain.
    • effective_begining: immutable, when the proposal will effective.
    • effective_ending: immutable, when the proposal will invalid, The proposal should be discarded if it has not reached consensus after effective_ending.
    • payload: mutable, A buffer reserved for DECApp, the DECApp guarantees its correctness. It's usefull when the params is large.
  • GroupQuorumCertificate
    The certificate to the blocks for Hotstuff.It has 2 types:
    • QC: the quorum certificate for the previous block.
      • block_id: immutable,current blockid.
      • prev_block_id: immutable,the previous block will be proven.
      • round: immutable,the round of the previous block.
      • votes: immutable,the set of signatures for the QC.
        • voter: the signer
        • signature: the signature
    • TC:the timeout certificate for last round.
      • round: immutable,the timeout round number
      • votes: immutable,the set of signatures for the round
        • voter: the signer
        • high_qc_round: the max round of the block which has collect the QC.
        • signature: the signature
  • GroupConsensusBlock
    GroupConsensusBlock is used to describe the content of each block.
    The fields is follow:
    • rpath: immutable, rpath the blockchain is running on.there is a blockchain is running for each rpath of the group.
    • body_hash: immutable, the hash of GroupConsensusBlock, is used to verify the body.
    • result_state_id: immutable, the ObjectId of the result calculated by the DECApp,
    • height: immutable, the height of the block.
    • meta_block_id: immutable,the highest block on MetaChain currently,Indicates that this block was constructed after the block on MetaChain.
    • round: immutable,the round to vote for the block,used by Hotstuff.
    • group_chunk_id: immutable,the version of Group the block is depend on.
    • proposals: immutable,save in body, verify by body_hash,the set of the proposals(proposal/result/receipt/context) collected by the block.
    • qc: immutable,save in body, verify by body_hash,the quorum certificate for the previous block.
    • tc: immutable,save in body, verify by body_hash,the timeout certificate for last round.

  1. nft

    NFT is a popular application scenario in recent years.CYFS collect and defined its own NFT objects.In theory,any named data is NFT in CYFS,but most NFTs are files in fact.So NFT is limited to file types.

    • NFTList
      Is a file list:
    • nft_list: immutable, file list.

  2. trans

I don't know this part?

  • TransContext
  1. zone
    It's the manager of the devices belonging to the same owner.
  • Zone
    The field is follow:
    • owner: immutable
    • ood_list: mutable
    • known_device_list: mutable
      How to maintain the latest version of the Body?

CyfsStack(cyfs-stack/cyfs-stack-loader/cyfs-lib)

CyfsStack integrates all functional modules provided by the CYFS system.Any service or DECApp will call the CyfsStack to use the CYFS.
There are 2 solutions:

  • All services and DECApp are executed as plugins in the same process as CyfsStack.
    This has several disadvantages:
    • A dynamically loaded plug-in system, the developers have to spend more to learn it.
    • All services and DECApps will be affected by exceptions thrown by any DECApp, that is a disaster.
    • A DECApp can easily access the memory of another.
  • Every service or DECApp is executed in a new process, they call the CyfsStack through the RPC.
    This solution is selected, and it's more friendly:
    • The developers can develop DECApp with their familiar technology.
    • DECApps can run independently in separate processes without interfering with each other.They can even run in separate dockers to keep resources isolated.
      It also has several disadvantages:
    • RPC in different programming languages is needed.but we can use it immediatly while anyone has complete it.
    • Performance will be lost in RPC.

RPC

  1. Protocol for RPC
  • The communication protocol of RPC adopts HTTP based on WebSocket or BDT.
  • There is some common headers defined in cyfs-base/src/base/protocol_fields.rs.
  1. Framework of RPC

There are client and service for RPC, the service is CyfsStack, and the client is SharedCyfsStack.

client service
Implement in SharedCyfsStack Implement in CyfsStack
Multiple instances Only one instance for one device
Open in any DECApp process Start with gateway(for OOD) or cyfs-runtime(for other devices)
Public for developers of DECApp Invisible for developers of DECApp
Module named by cyfs-${function-name}-lib Module named by cyfs-${function-name}
  1. Data flow diagram

The data will flow from the DECApp to the processor in service.

  • It will go through an indeterminate count of forwarding processes..
  • The forwarding target may be other devices, or different modules on the current device.
  • Each forwarding process has a input to receive the data, and a output to send the data except the initiator with only output and the ending processor with only input.
    • The inputs are usually named in the form of ${function-name}InputProcessor,and the parameter with data is named in ${function-name}InputRequest.
    • The outputs are usually named in the form of ${function-name}OutputProcessor,and the parameter with data is named in ${function-name}OutputRequest.
    • The forwarding process are usually named in the form of ${function-name}InputTransformer or ${function-name}OutputTransformer.

It's flow as follow:

graph TB
    subgraph DECApp
        User-->Interface[SharedCyfsStack: function with FunctionOutputRequest]
    end

    Interface-->FindTheTarget

    subgraph SharedCyfsStack
        FindTheTarget[find target device]-->RPC.send[RPC.send]
    end

    RPC.send.->RPC.recv
    RPC.send.->OtherDevice

    subgraph CyfsStack
        RPC.recv-->ObjectHttpListener
        ObjectHttpListener-->FunctionRequestHandlerEndpoint[FunctionRequestHandlerEndpoint.call]
        FunctionRequestHandlerEndpoint-->FunctionInputProcessor[FunctionInputProcessor.call_function]
        FunctionInputProcessor-->FindNextProcessor[output_processor = get_forward]
        FindNextProcessor-->InputOfOtherModule[input_processor]
        InputOfOtherModule-->NextN[next...]
        NextN-->Ending[ending_input_processor.call_function]
    end

    FindNextProcessor.->OtherDevice

    subgraph OtherDevice.CyfsStack
        OtherDevice[Other CyfsStack in different device]
    end

    OtherDevice.CyfsStack-.eq.-CyfsStack

We can integrate our own modules into the CYFS protocol stack according to the above framework.and you can also do it follow the issue step by step.

The crates

  1. cyfs-stack

It implements the CyfsStack and all processors to route the data for each module.It's inaccessable immediatly for DECApp, so it will never be depended on by any DECApp.There are some directories named in ${function-name} or ${function-name}_api.They are implemented with the function module:

  • ${function-name}: implement some basic structure, it's accessable for other modules.
  • ${function-name}_api: it is the specific call routing logic of the function module.
  1. cyfs-lib

It implements the SharedCyfsStack and some interface for the default function modules.It's the entry interface for DECApp to call the CyfsStack.So,most DECApp will depend on this crate.

If you are extending new functionality to CYFS,It's suggested to make a new crate named in cyfs-${function-name}-lib to instead of implement in cyfs-lib,you can export a interface to initialize your module with the SharedCyfsStack.

  1. cyfs-stack-loader

There are so many parameters to initialize the CyfsStack, You are suggested to initialize the CyfsStack with cyfs-stack-loader if you want to initialize yourself, for example:

  • Debuging your extended functions for CYFS.
  • Make a tool like the zone-simulator.
    But you should only open the CyfsSharedStack if you are making a DECApp based on CYFS. it will connect to your CyfsStack initialized by cyfs-runtime or gateway.

0.1 Introduction CYFS for Developers: Objectives and technical vision of CYFS protocol

Objectives and technical vision of CYFS protocol

Our goal is to become http 3.0 (to be discussed).

  • It's a basic upgrade to the http protocol, like this:
http://192.168.100.188/path => cyfs://$wallet_addr/path
  • It will reduce the cost of building trusted URLs and realize a decentralized trusted infrastructure:
  1. Solve 404 issues. You can save your data in CYFS anytime if you want.
    It's not mean that you can open the data anytime even the data is removed.
    It's meaning that, If you want, you can easily choose to save the data permanently, such as storing it in the blockchain or saving it off-site in the backup system.
  2. Build immutable URLs. You can refer to any URL without worrying about it being tampered with by the source website.

5.1 Introduction CYFS for Developers: O Link

cyfs://

Now, we know some basic concepts in the CYFS protocol, how to exchange information between the client and server, and some internal implementation principles. But there is still a little difference from http://, that is how to use cyfs:// in browsers and html pages.

In CYFS, all information is an object, and cyfs:// provides a variety of ways to manipulate objects:

  • O link: by ObjectId.
  • R link: path by RootState.
  • A link: access the specified version of DECApp.

O Link

It's like below:

cyfs://o/$target_zone/$obj_id[?mode=json&mime=txt]
cyfs://o/$target_zone/$dir_id/$inner_path[?mode=json&mime=txt]

This protocol is the simplest and most used cyfs protocol. Since the obj_id is included in the request, the result must be credible. Since it is used the most, default parameters are used in most times to construct requests, and the FastResp mechanism of BDT is used as much as possible.

  • $target_zone: As the domain for http, It represents the zone who will receive and responce the request.
  • $obj_id: The object which we want to get.
  • $dir_id: The ObjectId of the directory which we want to read.
  • $inner_path: The relative path which the file in the directory specified by the $dir_id.

If the mode is not specified, there are different default Response Body for different types of objid. The Body of FileObject is the most special, which is a continuous binary file.

Accelerate for O Link

Batch: When requesting objB, objC..., if you know that the request is initiated after requesting objA to the same zone (Learning the source grouping of HTTP requests), unless the relevant URL has special configuration, you can ignore objB, objC O Link target_zone, but uniformly initiate batch obj requests to target_zone_A.

Client load balancing: There are CacheNodes configuration in the target_zone, the client will request CacheA, CacheB, OOD in order of distance from itself to try to get the Object.

Domain for O Link

For browsers, the second segment of the URL is the domain, so all O Links are in the same domain.It means there are no cross domain issues and defense. It's a static content, and the defense is not needed.

Permissions for O Link

  1. If someone leak the ObjectId, the Object will be at risk of being accessed, So we should not tell the ObjectId to others if it's confidential.
  2. In default, A object can be accessed by the owner and his friends, The others must access it from the shared friends.
  3. The owner can set the ACL to open permissions for others.

Simplify O Link with name

cyfs://eips/index.html

We can register a name for Object to simplify the access link to Object, and we can publish a static website like above.

Cross domain security

  • There is no cross-domain problem with static resources.
  • Cross-domain is the earliest execution sandbox, and some domains allow access to code running in other domains.
  • Script code S in domain A requests resources from domain B, which may cause problems.
    • Potential attack on B's server.
    • Easy to make mock up stations.
    • Get cookies.

2.2 Introduction CYFS for Developers: DECApp debugging and deployment

DECApp debugging and deployment

zone-simulator

The zone-simulator is a useful tool in coding. It can simulate two zones in a single process, with two built-in oods, and each ood has two devices. It is very convenient to use this tool to run multiple DECApp instances on the local machine at the same time during the development and debugging phase. Just configure different RPC ports to connect to different devices when each instance opens SharedCyfsStack to access the emulator environment.

Zone1 <--> Zone2
People1 People2
Device1 <--> ood1 ood2 <--> Device1
Device2 <--> <--> Device2

The usage is as follows(Excerpted from a previous document on the official website):

  1. Mnemonic
  • By default, zone-simulator derives all people and device information from mnemonics through the bip protocol, so the same set of mnemonics generates the same zone configuration information every time.
  • A random mnemonic will be generated by default when zone-simulator is started first time, and saved in {cyfs-root}/etc/zone-simulator/mnemonic; this configuration will be loaded each startup, if you need to create new zone desc information, then delete this configuration file.
  • the debug version is slow to derive the key from the mnemonic, and it usually takes one to two minutes to complete the execution.
  1. Zone desc

zone contains follow infomation：

  • 2 people desc, representing 2 zones respectively.
  • 2 ood desc, representing the ood in 2 zones respectively.
  • The 4 device desc represent 2 devices in each zone, and the ood in same zone constitutes an independent zone.

When the zone-simulator is started, the above information will be written into the {cyfs-root}/etc/zone-simulator/desc_list directory, which corresponds to the mnemonic one-to-one, and the desc list generated by the same set of mnemonics will are the same, and development and testing can directly rely on this information.

For example, the mnemonic is below:

million used drastic castle during top category wood street visa steak bachelor

The generated desc list is as follows:

zone1 as follows:
people:5r4MYfFFQetBPDyeuBuDxPT7zowk9497SbpMQsZK2G19
ood:5aSixgLq9LVWbPByjRYgyjGicXtCN5S1nF1DMKZygLRE
device1:5aSixgPFg7R48VSo2b9PXcSHofK3TUFUcHnHbtmB7fJr
device2:5aSixgRaonSm1qWeCh8kNNrPGaB6pbETsqoN4Zb5Ev3H

zone2 as follows:
people:5r4MYfFGppKJXEjLvsgbKsibmTVLFZhL5cuZdEcXSr68
ood:5aSixgM12wHBfJBW1Q9n4xv1c5cu5g8hSxbrkhAQoojq
device1:5aSixgPdH5stj9pHrRVGXPNKiyrPsy8932BEr3zwFhXF
device2:5aSixgRx45jDU4Fg4ijYo3Yfo6RajtKSHgnGCVJUnk65

The above device_id and people_id can be directly used as the target of router and other interfaces to interact with the corresponding protocol stack.

For example, put_object to ood1:

const ood1 = DeviceId.from_base_58('5aSixgLq9LVWbPByjRYgyjGicXtCN5S1nF1DMKZygLRE').unwrap();
const target = ood1.object_id;
stack.put_object({
    ...,
    target
})

You can directly use the people_id of zone1 as the target:

const people1 = PeopleId.from_base_58('5r4MYfFFQetBPDyeuBuDxPT7zowk9497SbpMQsZK2G19').unwrap();
const target = people1.object_id;
stack.put_object({
    ...,
    target
})
  1. Ports for RPC

2 zones, each zone contains 1 ood + 2 devices, so there are six stacks in total. In order to facilitate external sdk calls, fixed ports are used, and the allocation is as follows:

device bdt-port http-port ws-port
zone1-ood1 20001 21000 21001
zone1-device1 20002 21002 21003
zone1-device2 20003 21004 21005
zone2-ood1 20010 21010 21011
zone2-device1 20011 21012 21013
zone2-device2 20012 21014 21015

The http-port and ws-port can be used as the service interface of the local SharedObjectStack, we can open SharedObjectStack directly with them, and pass it to the corresponding interface.
For example, if you want to use the protocol stack of zone1-device2, you can open SharedCyfsStack as follow:

const param = SharedObjectStackParam.new_with_ws_event_ports(21004, 21005).unwrap();
const stack = SharedObjectStack.open(param);
await stack.online();
  1. Scene simulation

The above two zones can simulate most of the interaction scenarios in cyfs:

  • Interaction between two devices in same zone, you can select device1 and device2 in the same zone.
  • Interaction between the device and ood in same zone, you can choose any device+ood in the same zone.
  • Cross-zone interaction, select 1 device in each zone(zone1 or zone2).
  • Cross-zone service, use zone1 as client and zone2 as server, then use zone1-device1+ood2, and dec-service can be connected to ood2.
  1. Command line

There are 2 commands, and some auxiliary information can be generated:

  • random: Generate a random set of mnemonic words。
shell> zone-simulator.exe --random
random mnemonic as follows:
side act dilemma super open tonight shallow wrong brother various potato punch
  • dump: Dump the desc+sec information of people/ood/device generated by the current mnemonic, and save it in the default directory {cyfs}/etc/zone-simulator/, user1 indicates the information related to zone1, and user2 indicates the information related to zone2. These information can be placed under {cyfs}/etc/desc for direct use by gateway, runtime or zone-stack etc.
    Note: The generated information is not on the chain. If you rely on the chaining logic, you must use the desc-tool to be on the chain.。

Deploy your project

After solving most of the errors in the zone-simulator environment, We can deploy the project to the test environment and start testing:

  1. There are 2 environments currently used for testing, they are nightly and beta, and the stable environment has not yet been launched. The target environment for deployment depends on the environment of cyfs-tool, cyfs-runtime and OOD, they must be the same environment, either nightly or beta.
  2. The cyfs deploy command is used to deploy a project, It should run in the same directory as the project file cyfs.config.json, it will take a long time. Here we introduce the contents of the cyfs.config.json file in detail:
{
    "app_name": "your project name",
    "version": "1.0.0", // version with 3 fields
    "description": "descript your project",
    "icon": "url for your icon that has published.cyfs://",
    "service": { // configuration for service
        "pack": ["directory1", "directory2", "..."], // A list of directories for source code in server-side, which will be packaged together using same relative paths
        "type": "node", // type of your project: nodejs or native
        "dist_targets": ["x86_64-pc-windows-msvc", "x86_64-unknown-linux-gnu"], // target os
        "app_config": {
            "default": "service_package.cfg" // configuration for service, introduce later
        },
        "app_dependent_config": {
            "default": "dependent.cfg" // Dependent configuration, which may depend on other `DECApp`
        },
        "app_acl_config": {
            "default": "acl.cfg" // The permissions required by `DECApp`, and the permissions configuration opened to others
        }
    },
    "web": { // configuration for web
        "folder": "src/www/dist", // Directory of the source code for web
        "entry": "index.html" // Entry webpage
    },
    "dist": "dist", // The directory for the packaged resources
    "owner_config": ".cyfs\\owner.json", // Owner of `DECApp`, which will be generated when using `cyfs-tool` to build the project, usually configures the identity file exported by the developer environment. It can be modified with the `cyfs modify` command, but it will become a new project after the change, and all information including `DecAppId` and historical versions will be changed, pay attention to synchronously modify related configurations.
    "ext_info": {
        "cyfs-app-store": { // Configuration displayed on `AppStore`
            "tag": ["tag1", "tag2", "..."], // Add some tags
            "community": ["your community name"],
            "releasedate": {
                "1.2.20": "2022-11-14" // the version history
            }
        }
    },
    "app_id": "9tGpLNna8UVtPYCfV1LbRN2Bqa5G9vRBKhDhZxxxxxxx", // your `DecAppId`
    "config_version": 1 // version of the format used by the config file？
}

Fields of service_package.cfg as follow：

{
	"id": "your readable project name", // unique?
	"version": "1.0.0", // verison with 3 fields
	"install": ["cmd1", "cmd2", "..."], // Commands in order for install the `DECApp`
	"start": "command", // Command to start the `DECApp`
	"stop": "command", // Command to stop the `DECApp`
	"status": "command", // Command to query the status the `DECApp`
	"executable": [] // ?
}

You can execute cyfs deploy to publish your DECApp after the above information is configured correctly. The URL to install DECApp will show in the last line. Now others can also get your latest version. **If you are not ready for others, you can separate the development People from the publish People, and use the People environment to deploy your DECApp. **

  1. Open the AppStore on the browser, and install the DECApp with the URL generated by cyfs deploy. Now you can use and check that your DECApp works correctly.

4.4 Introduction CYFS for Developers: Concurrent computing and throughput optimization

Concurrent computing and throughput optimization

In the personal property rights environment, all calculations are completed on the user's personal server (OOD), it is still a centralized system, and we can optimize concurrency and throughput like the central server of Web2. However, personal servers are only used to process self information. and the workload in central server of Web2 has been distributed actually to many personal servers of Web3, and the problems have been naturally alleviated.

Shared property rights are based on blockchain technology, and the process of blockchain consensus is serial; it is difficult to meet the high concurrency requirements in many Web2 application scenarios.

We can use multiple chains to improve the concurrent processing capability of shared property rights. In other words, we can optimize the design to disperse information in multiple rpath.

For example: a DAO organization manages the information of all its users.

Its structure based on RootState should be as follows:

/--users
    |--user1
    |   |--...
    |--user2
    |   |--...
    |--...
    |--userN

All information of every user must be updated serially if there is only one chain. In fact, users are not related to each other in most time, we can update B without waiting for A.

Then, we try to start a chain for each user, manage the branch of each subpath with a consensus chain (rpath):

/users/user1
/users/user2
...
/users/userN

There is a small problem here. We also need to manage the user list jointly. In fact, the status of each user has reached a consensus, the rpath list maintained by this organization can express the current user list in most time. However, there is no strict consensus chain to maintain it, the rpath list in each node may be inconsistent for some abnormalities, and there is no historical record of updates. If your application is very sensitive to these issues, you need to design another rpath(/users/list) to maintain the user list, and you need to design carefully for the consistency between its content and the rpath list.

Now, does it feel familiar to server engineers in the Web2 era? Yes, it is actually similar to the divide-table and divide-database that are often used in Web2, but the form has changed. I will not go into specific details for lack of experience.

3.1 Introduction CYFS for Developers: CYFS permission system

CYFS permission system

Every one needs to protect secret information from being accessed, and CYFS is an open platform, and anyone has the opportunity to read the information stored on it. The best way to protect privacy is not to deal with other people, and sensitive information is not shown to others. In CYFS, it is necessary to protect the ObjectId of sensitive information. Once someone else gets your ObjectId, it is possible to get the object content itself. This is very important, and it should be regarded as a principle for using CYFS. Information can be leaked in many ways:

  • Send to others actively.
  • Malicious or poor quality DECApps may deliver it on purpose or accidentally. Therefore, We should try our best to install DECApp from a reliable source, and even read the source code and build DECApp by ourselves.

It's too difficult to distinguish and prevent dangerous behaviors by users themselves. Therefore, CYFS provides a permission system to help users manage private data. The user can prohibit all DECApp except the system DECApp from accessing an object, and can also prohibit any other user from actively accessing an object. No one can read the object content without permissions even if the ObjectId is leaked. As a basic system, CYFS provides a flexible methods to config the permissions, which can only open partial access permissions for specific trusted users or DECApp; of course, the author cannot prevent Object from continuing to propagate after it has been displayed to others, It's also unpreventable in real world.

Let me emphasize again that users should try their best to abide by the confidentiality principle mentioned above. The system can only provide auxiliary management capabilities and cannot be used as the final guarantee.

I will describe the design of the CYFS permissions (hereafter referred to as ACL) system as follow.

Default Permissions

If no permissions are configured for an Object, the default permissions are effective as follows:

  • If the Owner of Object is not itself, anyone has the full permissions.

  • If the request is from the current Zone or a friend Zone, and the source DECApp is same as the DECApp that constructs Object, the request will be granted.

    It's important to add friends cautiously!

  • Otherwise the request will be prevented.

Level

Currently, CYFS already supports several ways to configure ACL:

  • Configured by the req_path field of the request:
    • It can be used to prevent the requests to the service provided by DECApp.
    • It can be used to restrict the access rights of the RootState subtree, and its lower-level subtrees also apply the same permission configuration, unless the lower-level subtrees are configured otherwise.
  • Configure objects that meet certain rules (rmeta).
  • Configure for a single object.

The rules are hit one by one in above order, and the permission configuration with the highest matching degree is found and applied. If you still remember the architectural design of CyfsStack, the ACL system is a Processor in it, and the implementation code is in src\cyfs-stack\src\non_api\acl.

Static configuration as file

ACL can be statically configured by app-manager to read the acl configuration in the installation package when DECApp is installed. For the specific configuration method, refer to the app_acl_config field of the cyfs.config.json file.

The file is used to configure the permissions that the app needs and awarded to others by the app. This file will be read by app-manager when the app is installed, and the corresponding permissions will be registered with the system according to the configured values.

A complete permission configuration file is shown as follows:

[self]

[self.access]   // Grant permissions to your own data by path
// /test3 by each alone group
"/test3" = [{group = "OthersDec", access = "-wx"}, {group = "CurrentDevice", access = "---"}]
// complete string, set/clear each bits for each group
"/test2" = "rwxrwxrwx---rwx---"
"/test1" = "rwxrwxrwx---rwx--x"

[self.specified]    // Authorize your own data to the specified DECApp or user
"/test3" = {access = "--x", dec_id = "9tGpLNnDpa8deXEk2NaWGccEu4yFQ2DrTZJPLYLTxxxx"} // The specified  DECApp running on any zone can call the specified path.
"/test2" = {access = "--x", zone_category = "current-zone", dec_id = "9tGpLNnDpa8deXEk2NaWGccEu4yFQ2DrTZJPLYLTxxxx"} // The specified  DECApp running on the specified zone can call the specified path.
"/test1" = {access = "--x", zone = "5aSixgLwnWbmcDKwBtTBd7p9U4bmqwNU2C6h6SCvxxxx"} // Any DECApp running on the specified zone can call the specified path.

// Request authorization from a specified DECApp(DECID_A)
[DECID_A.specified]
// The `dec_id` field in SpecifiedGroup will never be set, otherwise the rule will be discarded.
"/test3" = {access = "--x"} // Apply for the permission to call the specified path.
"/test2" = {access = "--x", zone_category = "current-zone"} // Apply for the permission to call the specified path limited in the specified zone("current-zone").
"/test1" = {access = "--x", zone = "5aSixgLwnWbmcDKwBtTBd7p9U4bmqwNU2C6h6SCvxxxx"} // Apply for the permission to call the specified path limited in the specified zone("5aSixgLwnWbmcDKwBtTBd7p9U4bmqwNU2C6h6SCvxxxx").

[DECID_A.config]    // **None currently?**

  1. This file is structured in toml format, where each section represents authorization in the same direction, and the section name rules are as follows:

    ${SectionName} = ${dec_id}[.access|specified|config]

  • ${dec_id}: the DecAppId that needs to provide permissions. If it is the current DecAppId (can be replaced by self), it means that the permission is open to others, otherwise, it means that the current DECApp needs the permissions from the specified DECApp.
  • access: A more general permission configuration method divided by group,
  • specified: The way to configure permissions for specific targets (DECApp, Zone, etc.), key is the request path, which can be the path of RootState, or the value of req_path in the request. It can manage permissions more granularly than access
  • config: None?

  1. The content of each line is as follows:

    ${SectionRow} = ${key} = ${AccessString} | ${SpecifiedGroup}

  • ${key}: request path, which can be the path of RootState, or the value of req_path in the request.

  • AccessString: Indicates the complete permission corresponding to a path. Refer to the file system permission in linux. A set of specific permissions is represented by 3 bits. A total of 6 groups of 18 bits represent a complete permission, and each bit represents the authorization of the corresponding group.

    There are 3 permissions currently: Read/Write/Call:

    pub enum AccessPermission {
    Call = 0b001, // 用`x`表示
    Write = 0b010, // 用`w`表示
    Read = 0b100, // 用`r`表示
}

    And there are 6 groups divided according to device and DECApp:

    pub enum AccessGroup {
    CurrentDevice = 0,
    CurrentZone = 3,
    FriendZone = 6,
    OthersZone = 9,

    OwnerDec = 12,
    OthersDec = 15,
}

    Now, there are 2 methods to represent an AccessString in a configuration file:

    • A complete string, use an 18-chats string to represent a complete permission, and use linux "rwx-" in the group to represent the permission of each bit. Groups can be separated by spaces or underscores

      Example: Full permissions for OwnerDec in CurrentDevice | CurrentZone, read | write permissions for OwnerDec in FriendZone, and read permission for OthersDec in OthersZone. We can express it as "rwxrwxrw-r--rwxr--"; it is equivalent to "rwx rwx rw- r-- rwx r--" or "rwx_rwx_rw-_r--_rwx_r--".

    • Grant permissions for specified groups separately based on the default permissions, expressed as an array formed as {group, access}, group is the enumeration name of AccessGroup, and access is a 3-chats "rwx-" string.
      • Default AccessString permission: "rwxrwxrwx---rwx"
      • Still take the above permissions as an example, expressed as [{group = "FriendZone", access = "rw-"}, {group = "OthersZone", access = "r--"}, {group = "OthersDec" , access = "r--"}]

  • SpecifiedGroup: The permissions of a specific DECApp, Zone, or zone_category, the difference from AccessString is that AccessString can only configure permissions for each group, and SpecifiedGroup can limit permissions into a specific DECApp or Zone.

    There are 4 fields:

    • access: Permission configuration, which is a 3-chats "rwx-" string

    • zone: ZoneId of the Zone which the permission is applied to. Here, it is usually filled with the PeopleId of Zone. You can also fill in a DeviceId, indicating that it is limited to a specific Device. We can remove the field to match any device.

    • zone_category: The group of zone this permission will be applied to, if not filled, it means for any devices. Can take one of the following values:

      • "current-device"
      • "current-zone"
      • "friend-zone"
      • "other-zone"

      I found that the zone_category is words with hyphen, it's different with it in AccessGroup for AccessString. can we use the same style?

    • dec_id: DECApp which this permission applies to, if not filled, it means for any DECApps.

    Among the 4 fields, access is required, and zone, zone_category, and dec_id must be filled in at least one. The permission will be passed when all the three conditions of zone, zone_category, and dec_id are matched.

Dynamic configuration interface

We can config the ACL statically by app-manager to read the acl configuration in the installation package when DECApp is installed. And we can also config it dynamically by calling SharedCyfsStack when the DECApp is running. The entry point of the dynamic configuration interface is:

let meta: GlobalStateMetaStub = SharedCyfsStack.root_state_meta_stub();

GlobalStateMetaStub provides different interfaces for various configuration methods. I will introduce as follow.

  1. req_path constraints
  • add_access: Add a permission constraint on req_path, the core structure is designed as follows:
pub struct GlobalStatePathAccessItem {
    // GlobalState path, must end with /
    pub path: String,

    // Access value, it's a `AccessString` as uint32, or a SpecifiedGroup
    pub access: GlobalStatePathGroupAccess,
}
  • remove_access: Use path as key to delete the parameter specified permission.
  1. rmeta constraints
  • add_object_meta: Add a matching rule, the core structure is designed as follows:
pub struct GlobalStateObjectMetaItem {
    // Object dynamic selector: `${key} ${operator} ${value}`
    pub selector: String,

    // Access value
    pub access: GlobalStatePathGroupAccess,

    // Object referer's depth, default is 1
    pub depth: Option<u8>,
}

We can use the following operators in selector:

`==`, `!=`, `<=`, `>=`, `<`, `>`, `&&`, `||`, `&`, `^`, `|`, `!`.

The key and corresponding value are as follows:

key type value
obj_type_code u16 ObjectTypeCode enum int values
object_category string ObjectCategory enum values
obj_type u16 value of obj_type
object.dec_id string ObjectId
object.author string ObjectId
object.owner string ObjectId
object.create_time u64 bucky time
object.update_time u64 bucky time
object.expired_time u64 bucky time
insert_time u64 bucky time
update_time u64 bucky time
  • remove_object_meta: Use selector as key, delete the permission specified by the parameter.
  1. object_id constraints
  • Just specify the access parameter directly when put_object is called, if you want to change it, call put_object again.

Principles of select

How to choose a method to config the permission, I suggest as follow:

  • You should write it clearly in the static configuration file if you want to grant permissions to the others (other DECApp or Zone); dynamic configuration in the program is more difficult to maintain. You can use dynamic configuration if the static configuration file can not do it.

  • You must write it in this configuration file and let app-manager register during installation, if you want to request permissions from other DECApp. There is no way to register dynamically in code.

  • You must use the specified section to request permissions from other DECApp, anything in the access field is ignored.

  • There is no way to request permissions from other zone.

4.1 Introduction CYFS for Developers: Sharing Property Rights

Sharing Property Rights

There are many entities in the world other than people,family,club,company,country,etc.They are organizations made up of many people,the people are related, or they have the same purpose or hobby.They have common assets and often have to make resolutions on matters.They also have personal positions and interests.

There are many solutions for web2:

  • Information system for enterprise
  • Community in Github
  • Server in Discord
  • etc.

The data stored in the IT system by these organizations should be owned by themself. Members of these organizations share the property rights of these data, and they jointly manage the data status (addition, update, deletion and query).

In the web3 era, we need a decentralized solution to make sure the users to obtain more autonomy and more certain rights and interests. Blockchain provides us with a good idea.We can make these applications run on a Consortium Blockchain.

0.2 Introduction CYFS for Developers: Role and contribution of CYFS protocol in the decentralized network field

Role and Contribution of CYFS Protocol in Decentralized Network Field

  1. By upgrading the http:// protocol, traditional web application developers can develop decentralized applications in a familiar way. This significantly lowers the entry barrier for Web3 developers, allowing more developers to enter the Web3 field and providing a solid foundation for the emergence of large-scale killer Web3 applications.
  2. The distributed network architecture enables developers to not worry about large-scale concurrency issues in most situations.
  3. Large-scale off-chain data exchange greatly reduces the number of on-chain transactions, effectively reducing the usage cost of Web3 applications.

5.3 Introduction CYFS for Developers: A Link

A Link

It's like below:

cyfs://$version_name.$dec_id.a/$inner_path[?mode=json&mime=txt]

We can send a request to the specified DECApp with specified version in current Zone.

  • $version_name: It's the name(no . included) registered on MetaChain for a product, It's also can be a hash.
  • $dec_id: The DecApp(like service of http) which will be called by the request.
  • $inner_path: The request path defined and responded by DecApp, similar to http.

Domain for R Link

The domain of A Link is related to $version_name and $dec_id, similarly as R Link, there are cross domain issues and defense, we need to open relevant permissions to the clients of DECApp.

4.2 Introduction CYFS for Developers: Proposal and consensus achievement process

Proposal and consensus achievement process

From the previous introduction to the scenario of Shared Property Rights, it is not difficult to see that the key issues is:

  • How to build a Consortium Blockchain simply and cheaply?
  1. Simply

  • Define standard Objects for CYFS users to ensure they representing in the same way.

  • Use the BDT protocol to ensure the interconnection between CYFS users, even users under NAT, and protect privacy through protocol encryption.

  • Provide a basic consensus framework to ensure that all proposals are received, executed and verified in a unified order.The DECApp developers only need to define the implementation and verification methods of proposals according to their needs,it's a Smart Contract.

  • For the ending user,They only need to complete the operation following the guidance of DECApp:

    • Initialize the Group.
    • Create the proposal.
    • Submit the proposal.
    • Check and sign the proposal as a vote.
    • The decide will be formed automatically by DECApp.

The follow sequence diagram will show a common flow for a proposal.

%% 时序图例子,-> 直线，-->虚线，->>实线箭头

sequenceDiagram

participant User
participant CYFS
participant DECApp

User->>CYFS: post(proposal)
CYFS->>DECApp: on_execute(proposal,last_result)
DECApp->>DECApp: result=execute(proposal,last_result)
DECApp-->>CYFS: result
CYFS->>CYFS: broadcast(proposal,result) to other members
CYFS->>DECApp: on_verify(proposal,last_result)
DECApp->>DECApp: is_ok=verify(proposal,last_result)
DECApp-->>CYFS: is_ok
CYFS-->>User: finish(result,is_ok)

A vote on a proposal is a proposal that depends on the original proposal,and its execution process is exactly the same as the general proposal.

A proposal that requires voting, from the original proposal to the formation of the final resolution, the implementation process is as follows.

%% 时序图例子,-> 直线，-->虚线，->>实线箭头

sequenceDiagram

participant User
participant CYFS
participant DECApp

User-->>DECApp: post(proposal)
DECApp->>DECApp: result_proposal_list=add(proposal,last_proposal_list)
DECApp-->>CYFS: result_proposal_list
User->>CYFS: proposal_list = get_voting_proposals()
CYFS-->>User: proposal_list
User->>User: vote=make_vote(proposal_list.select(),voter=self)
User-->>DECApp: post(vote)
DECApp->>DECApp: all_votes=collect_votes(vote)
DECApp->>DECApp: is_enough=check(all_votes)
DECApp->>DECApp: if is_enough {result=decide()}
DECApp->>DECApp: if !is_enough {result=add(vote,last_vote_list)}
DECApp-->>CYFS: result

We can find that the working mode for users and developers hasn't changed substantially against Web2, there are only some differences in form:

  • In the Web2 era, the database is directly updated when the proposal is executed after the user has been authenticated.
  • In CYFS, there is an additional verification stage before the proposal execution results take effect. Usually, each node executes it once to compare whether the respective calculation results are the same.

And, all cyfs:// protocols will support the Group as a Zone,the same interface will be provided as People,so,it will work in the same way for People zone.

  1. Cheaply
    The entire process is almost completed locally by the Group members, and no additional on-chain fees are required.

Hotstuff

I will introduce Hotstuff briefly. Please refer to the professional literature if you want to learn more about it.

Fault tolerance

Hotstuff is a type of BFT consensus algorithm:

  1. Assume that the number of malicious nodes is f;
  2. The total number of all nodes is N.
  3. To reach a correct consensus:
    • The total number of votes is v;
    • The number of votes for normal nodes is n, in the worst case, all malicious nodes also participated in the vote, n > f;
    • The number of unvoted nodes is N - v, n > N - v;

$$
\begin{cases}
\ v >= n + f; => min(v) = n + f => min(v) = min(n) + f \
\ n > f; => min(n) = f + 1 \
\ n > N - v; => max(N) = n + v - 1 \
\end{cases}

=>

\begin{cases}
\ min(v) >= 2f + 1; \
\ n + v > N; => min(n) + min(v) > N \
\end{cases}

=> N < f + 1 + 2f + 1 = 3f + 2; \
=> N <= 3f + 1

$$

From the above calculations, we can prevent malicious nodes accounting for up to 1/3(excluding) of the total when we collect signatures from more than 2/3(excluding) nodes.

Consensus process

  • PBFT

  1. PBFT is the first available BFT algorithm, and its basic process is:

    • Sort all nodes in a certain order;
    • Select one node in order as the master node, responsible for sorting, executing, packaging, blocking for all proposals;
    • The master node broadcasts the packaged block to other nodes;
    • All nodes verify the information described in the block they received, sign the vote, and broadcast the vote to all other nodes again;
    • Each node conducts 2nd signature votes when 2f+1 signatures are collected, and broadcasts the vote to all other nodes again;
    • Each node update the local result and return it to the client when 2f+1 2nd signatures are collected;
    • The client confirms that the request is processed correctly after 2f+1 2nd signatures are collected.

    ** Here, it is required to collect 2f+1 signatures twice, because each node must ensure that 2f+1 nodes have received votes, so that they can recover in the event of a failure. **

  2. View change:

    There should be a mechanism to change the master when the master node does evil or fails, otherwise the activity of the system cannot be guaranteed.

    The current running state of all nodes is a view, and the process of changing the master node is view change:

    • A node finds that the master node is faulty, selects the next node as the new master node, and initiates a view change request;
    • Other nodes sign the vote if they also agree to view change, and attach the block vote with the highest signature height of 2f+1; then broadcast to other nodes;
    • Same as the previous consensus process, all nodes arrive a same state on view change again;
    • Each node synchronizes the local chain to the highest level according to the voting information attached to the voting during the consensus process.

  • Hotstuff

    Hotstuff optimizes PBFT in several pionts:

    1. Pipelining: the vote for each block is also the confirmation of the next state of the previous block, and the whole process only has two broadcasts per block on average (2*n, block broadcast and voting);

    2. Simplify the state machine: the view change process is subtly integrated into the block consensus process, and the view change process only changes the node responsible for collecting votes for the next block;

1.1 Multiple encoding methods: Type Encoding Format

Multiple encoding methods

All information based on CYFS can choose one of the following three encoding methods. Both Protobuf and JSON have their standard encoding specifications. I won’t go into details here. We only introduce the binary encoding format.

  1. Binary

    Binary format encoding, usually used for standardized fixed encoding, the most space-saving, but not easy to expand.

    When binary encoding is selected, the type designer needs to fill all fields into a Buffer one by one in a certain order.

  2. Protobuf

    Using Protobuf encoding, while taking into account certain space efficiency and scalability, this is the encoding method selected by most complex types.

    When using Protobuf encoding, the type designer only needs to assign each field of the object to the corresponding Protobuf type, and then use the Protobuf method to encode and decode. Compatibility processing needs to refer to the Protobuf specification.

    ** Note: The Protobuf specification we chose is Proto3, considering backward compatibility, the Protobuf version is unlikely to be upgraded. Of course, other versions can be selected for the type of new design, but different versions need to be carefully introduced. **

  3. JSON

    Encoded in JSON, which is less space efficient but more readable.

    Because of its low space efficiency, the JSON method is usually not selected to encode and decode information. JSON encoding usually appears in the scene of human-computer interaction.

** No matter which encoding method you choose, you must pay attention to the stability of the encoding when you use it for object encoding (for example: Set and Map should pay attention to the order of each element), otherwise the ObjectId obtained by calculating the same content multiple times may be different. **

0 Type Encoding Format: Contents

Table of contents

I want to collect and organize various information encoding formats in the CYFS protocol in this document. By reading this article, you can gain the ability to:

  1. Correctly encode various information to store or exchange with other peers.
  2. Correctly restore the encoded information obtained from the storage system or other peers.
  3. When designing a new composite type, understand the data encoding specification, and be able to correctly select the appropriate basic type for encoding and decoding.

I. Basic type

II. Object encoding format

4.3 Introduction CYFS for Developers: Status verification

Verify status

In the web2 era, any information is read from a central server, you can trust any information from it as you trust the source server. It is very simple.

But in the web3 era, for decentralization, all information comes from servers provided by someone. For single owned information, we can trust it with the signature from owner as proof of accountability. As for shared property rights information, we can only obtain information from one of the members, and these members are likely to falsify information for their interests. Therefore, we need a verification method for the shared property rights information.

As mentioned in the previous article, we use Hotstuff as the consensus algorithm for sharing property rights. Naturally, we also adopt the corresponding verification strategy with 2f+1 signature rate.

  1. Verify with block

Every block on the chain can be verified by any node. We use the BFT consensus algorithm. Each valid block holds the voting signatures of 2f+1 nodes naturally, so all the fields contained in the block are recognized, the field result_state_id is the ObjectId of the state of property rights information.

  1. Verify the sub-fields
  • Verify based on ObjectId

We known earlier that result_state_id is a tree structure object implemented by ObjectMap; result_state_id is obtained by computing Hash from all branches and leaf node values in this tree.

Therefore, all branch and leaf node values of result_state_id are trusted.

However, we must read the full Desc (immutable part) of the Object to verify if an Object is matched with the ObjectId, in other words: we must read all elements of ObjectMap and compare their Hash with result_state_id to verify that one of the elements is correct.

  • Which fields will be read to verify the sub-field with the specified path?

Now, it's clear that we must verify ObjectMapId of a branch in each level to verify the sub-field in a tree specified by a path. A flowchart is as follow:

flowchart TB

    Init(path, state_id) --> IsEmptyPath{path is empty}
    IsEmptyPath --no--> Read["obj_map = get(state_id)"]
    IsEmptyPath --yes--> Found(return state_id)
    Read --> Verify["state_id = ObjectId(obj_map)"]
    Verify --> NextPath["sub_paths=split(path,'/');\nnext_path=sub_paths[0];\npath=sub_paths[1..].join('/');"]
    NextPath --> NextState["state_id = obj_map.get(next_path)"]
    NextState --> IsExistState{state_id == None}
    IsExistState --yes--> None(return None)
    IsExistState --no--> IsEmptyPath

We can see that we must read the full content of the first-level subdirectory contained in each level of directory to verify a child node specified by the path in the tree:

flowchart TB

    Root --> P_1_1[p.1.1]
    Root --> P_1_2[p.1.2]
    Root --> P_1_3[p.1.3]
    Root --> P_1_x[...]
    Root --> P_1_f1[folder.1]
    Root --> P_1_x1[...]
    P_1_f1 --> P_2_1[p.2.1]
    P_1_f1 --> P_2_x[...]
    P_1_f1 --> P_2_f2[folder.2]
    P_1_f1 --> p_2_x[...]
    P_2_f2 --> P_x_x[...]
    P_x_x --> P_n_fn[folder.n]
  1. Performance

We stated a complete state verification scheme above, but there is a problem that we must read a large amount of redundant data for a specified field of a shared property right information. It is an impossible task in a scenario with a large data.

I think there are several solutions:

  • Design each rpath (representing a consensus unit in a shared property rights) carefully so that its capacity is small, and divide the large container structures into segments for management (for example: every 16 is a segment).

  • Add 2f+1 signatures to the result of each request. The server returns only the specified field(0 redundancy) for each status request, but servers owned by members of the organization must sign on the field. Of course, the burden on the service node will be increased for signatures, the more queries, the more signatures.

Documents: Introduction CYFS for Developers

I want to finish the documents to introduct CYFS for developers. you can develop your own DECApp, or join the team to improve the CYFS protocol.

I have create a repository for it. Anyone can raise an issue in the repository to help me to complete it.

Following is the table of contents:

Table of Contents

Introduction

I. Network Architecture and Core Components

II. DECApp Development

III. Security and Authentication

IV. Data Sharing Property Rights

  • Consensus algorithm adopted by CYFS protocol
  • Process of validation and consensus achievement
  • Concurrent computing and throughput optimization

V. cyfs:// Protocol

  • cyfs://o/
  • cyfs://r/
  • cyfs://a/

VI. Communication and Data Transmission

  • Peer-to-peer communication protocol BDT
  • Data transmission and encryption
  • Network optimization and congestion control strategies

VII. MetaChain

Documents: Introduction CYFS for Developers

I want to finish the documents to introduct CYFS for developers. you can develop your own DECApp, or join the team to improve the CYFS protocol.

I have create a repository for it. Anyone can raise an issue in the repository to help me to complete it.

Following is the table of contents:

Table of Contents

Introduction

I. Network Architecture and Core Components

II. DECApp Development

III. Security and Authentication

IV. Sharing Property Rights

V. cyfs:// Protocol

VI. Communication and Data Transmission

VII. MetaChain

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.