ctripcorp/ceph-sync

Sync tool between LOCAL file system and REMOTE object storage.

ceph-sync

Tool to sync contents between local file system and remote object storage.

This tool can achieve synchronizations as:

• directory A → container B
• container B → directory A
• container B → container C

Here, directory is located in local file system and made up of files and sub directories, while container (also called bucket according to AWS S3) is vessel in remote CEPH storage where objects saved.

Table of Contents

• Get Started
• Connection Config
• API

Links

• CHANGE LOG
• Homepage

Get Started

In CLI:

# Command "ceph-sync", "cehsync" and "ossync" will be generated.
npm install -g ceph-sync

# Show help info.
ceph-sync -h

# Run sync task.
ceph-sync --source /path/of/container --target /path/of/conn.json

As API:

const fs2ceph = require('ceph-sync/fs2ceph');

const progress = fs2ceph(
	/* source */ '/path/of/container',
	/* target */ connConfig );

progress.on('error', (err) => {
	// ...
});

progress.on('end', (meta) => {
	// Sychronization successfully finished.
});

Connection Config

The connection configuration is a JSON object required by the dependent package ceph. To describe an accessible (readable and writable) CEPH container, following properties are required:

• endPoint
• subuser
• key
• container

Here is a dummy example:

{
	"endPoint"   : "http://storage.example.com/",
	"subuser"    : "userName:subUserName",
	"key"        : "380289ba59473a368c593c1f1de6efb0380289ba5", 
	"container"  : "containerName"
}

For CLI usage, CEPH connection config should be stored in a JSON file.

CLI

When installed globally, ceph-sync will create a homonymous global command. Run ceph-sync -h in terminal to print the man page.

ceph-sync will occupy a hidden directory named .ceph-sync in home directory of current user.

API

ceph-sync offers three functions to achieve different tasks:

• jinang/Progress ceph2ceph(object sourceConn, object targetConn, object options)
• jinang/Progress ceph2fs(object sourceConn, string targetDir, object options)
• jinang/Progress fs2ceph(string sourceDir, object targetConn, object options)

1. Here 2 is a homophone of to.
2. sourceConn and targetConn may be an object containing CEPH storage connection configuration, or an instance of swift Connection.
3. The functions accept similar options argument, see section Parameter options for details.
4. The functions are all asynchronous and will return an instance of jinang/Progress. Via the returned value, we may learn about and control the sync progress. See section Get Into Sync Progress for details.

Each function may be required solely:

const cephSync  = require('ceph-sync');

const ceph2ceph = require('ceph-sync/ceph2ceph');
const ceph2fs   = require('ceph-sync/ceph2fs');
const fs2ceph   = require('ceph-sync/fs2ceph');

// E.g., next two functions are equivalent.
cephSync.ceph2ceph
ceph2ceph

Parameter options

• string[] options.names
  Object names to be synchronised.

• Function options.mapper Object name mapper.

• Function options.filter Object name filter.

• Function options.dualMetaFilter Filter with paramenter (stat, meta).

• number options.maxCreated
  Maximum creation allowed (then the progress will be terminated).

• number options.maxCreating
  Maximum cocurrent creating operation allowed.

• number options.maxErrors
  Maximum exceptions allowed (then the progress will be terminated).

• number options.retry
  Maximum retry times on exception for each object or object list.

Get Into Sync Progress

Via the returned instance of jinang/Progress, we may learn about what happened and then control the sync progress.

• progress.on(string eventName, Function listener)
  See section Events During Sync Progress for aviable events and their accompanied arguments.

• progress.abort()
  Terminate the progress as soon as possible.

• progress.quit()
  Quit the progress gracefully.

Events During Sync Progress

Event: 'created'

• Object meta

Event: 'moveon'

• string mark

Event: 'ignored'

• Object meta

Event: 'skipped'

• Object meta

Event: 'warning'

• Error error

Event: 'error'

• Error error

Event: 'end'

• Object meta

  {
  	errors /* number */,
  	created /* number */, 
  	ignored /* number */,
  }