Docs > Object Subsystem (ObjectNode)

Object Subsystem (ObjectNode)

The object subsystem provides S3-compatible object storage interface. It makes possible for ChubaoFS becomes a fusion storage that expose two type interface (POSIX and S3-compatible). So that user can operate files stored in ChubaoFS by using native Amazon S3 SDKs.

Structure

../_images/cfs-object-subsystem-structure.png

The ObjectNode is a functional subsystem node. It fetch volume view (volume topology) on demand from Resource Manager (Master). Each ObjectNode communicate with metadata subsystem (MetaNode) and data subsystem (DataNode) directly.

ObjectNode is stateless design with high scalability and it have ability to operate all files stored in the ChubaoFS cluster which it belong to directly without any volume-mount operation.

Features

  • Provides object storage interface compatible with native Amazon S3 SDKs.
  • Fusion storage expose two type interface (POSIX and S3-compatible).
  • Stateless and high scalability

Semantic Transform

Based on our POSIX-compatible design. Every file operate request comes from object storage interface need to be made semantic transform to POSIX.

POSIX Object Storage
Volume Bucket
Path Key

Example:

../_images/cfs-object-subsystem-semantic.png

Put object ‘example/a/b.txt’ will be create and write data to file ‘/a/b.txt’ in volume ‘example’.

Authentication

The signature validation algorithm in object storage interface is fully compatible with native Amazon S3 service. The authentication consisting of AccessKey and SecretKey generated by Resource Manager (Master) with volume creation. The AccessKey is a 16-character string unique in the entire ChubaoFS cluster.

Authentication keys owned by volume and stored with volume view (volume topology) by Resource Manager (Master). User can fetch it by using administration API, see Get Volume Information at Volume

Invisible Temporary Data

In order to make write operation in object storage interface atomically. Every write operation will create and write data to an invisible temporary. The volume operator in ObjectNode puts file data into temporary which only have ‘inode’ without ‘dentry’ in metadata. When all the file data stored successfully, the volume operator create or update ‘dentry’ in metadata makes it visible to users.

Object Key Conflict (Important)

The POSIX and object storage are two different types of storage product, and the object storage is a Key-Value pair storage service. So that the object with key ‘a/b/c’ and the object with key ‘a/b’ are different object without any conflict.

But ChubaoFS is based on POSIX design. According to semantic transformation rule, the ‘b’ part in key ‘a/b/c’ transform to folder ‘b’ under the folder ‘a’ , and in key ‘a/b’ transform to file ‘b’ under the folder ‘a’.

The object key like this is conflict in ChuBaoFS.

Supported S3-Compatible APIs

Supported SDKs

Object Node provides S3-compatible object storage interface, so that you can operate files by using native Amazon S3 SDKs.

Name Language Link
AWS SDK for Java Java https://aws.amazon.com/sdk-for-java/
AWS SDK for JavaScript JavaScript https://aws.amazon.com/sdk-for-browser/
AWS SDK for JavaScript in Node.js JavaScript https://aws.amazon.com/sdk-for-node-js/
AWS SDK for Go Go https://docs.aws.amazon.com/sdk-for-go/
AWS SDK for PHP PHP https://aws.amazon.com/sdk-for-php/
AWS SDK for Ruby Ruby https://aws.amazon.com/sdk-for-ruby/
AWS SDK for .NET .NET https://aws.amazon.com/sdk-for-net/
AWS SDK for C++ C++ https://aws.amazon.com/sdk-for-cpp/
Boto3 Python http://boto.cloudhackers.com