peilingfeng/obs-tutorial

Introduction

Object Storage Tutorial.

Basic Concept

SNIA Tutorials on Object Storage:

• Object Storage 101
• Object Storage 201
• Object Storage Technology.

The Storage Networking Industry Association (SNIA) is a not–for–profit global organization, made up of member companies spanning the global storage market.

Preparation

Environment

Git and Github (Recommended)

Git tutorial https://github.com/cs-course/git-tutorial.

For those who want unlimited private repositories, try bitbucket.

How to establish Python Environment

• Python Distributions:
  • Option 1: Anaconda
    • Download from https://repo.continuum.io/archive/Anaconda3-5.1.0-Windows-x86_64.exe & install
  • Option 2: WinPython
    • Download from https://sourceforge.net/projects/winpython/files/WinPython_3.6/3.6.3.0/ & install
• Fast deployment by docker:
  • Option 3: Python Docker https://github.com/Zhan2012/python-lab
    • docker login daocloud.io && docker pull daocloud.io/zhan2016/python-lab:master-31a932d

How to establish Java Environment

Ongoing course: Java Programming, 2017-2018 2nd semester, just follow your teacher's guide.

Installation helper scripts https://github.com/Zhan2012/java-bundle (For adventurers).

How to use Linux in Windows or MacOS (Optional)

Virtual Machine: Virtualbox, VMWare ...

For those who want to run mock-s3 and s3proxy without trouble, Linux is a must, refer to https://github.com/cs-course/vagrant-tutorial.

How to run servers within docker container (Optional)

For those who want to run Openstack Swift or Ceph in docker, refer to Docker tutorial https://github.com/cs-course/docker-tutorial.

Object Storage Server

• Object Storage for Beginners:
  • Option 1: Minio, latest version https://minio.io/downloads.html.
• Experimental Mock Servers:
  • Option 2: mock-s3, a Python clone of fake-s3. Requires Python 2.7.
    • Option 2.1: Python 3 migrating, maybe BUGGY https://github.com/Zhan2012/mock-s3.
  • Option 3: s3proxy, access other storage via the S3 API. Binary bundles here
    • Option 3.1: Use Java/Maven to build mvn package -Dmaven.test.skip=true.
  • Option 4: fake-s3, a lightweight server clone of Amazon S3. Depends on Ruby, the Origin project.
  • Option 5: s3mock, S3 mock library for Java/Scala. Java/SBT Building is required.
  • Option 6: S3Mock, S3 mock as Docker image or JUnit rule. Use Java/Maven to build, use docker to run, contributed by Adobe (c).
• Industry Level Projects:
  • Option 7: Openstack Swift, fast deployment by All-in-one container: https://github.com/cs-course/openstack-swift-docker.
  • Option 8: Ceph, docker files and images to run Ceph in containers: https://github.com/ceph/ceph-container.

Besides Option 1, Option 2, 3 offer compile-free executable.

Object Storage Client

• Standalone Utilities:
  • Option 1: Minio Client https://docs.minio.io/docs/minio-client-quickstart-guide
  • Option 2: s3cmd https://github.com/s3tools/s3cmd
    • run pip install s3cmd in python environment
    • Configure for Minio https://docs.minio.io/docs/s3cmd-with-minio
  • Option 3: aws-shell https://github.com/awslabs/aws-shell
    • run pip install aws-shell in python environment
    • Configure for Minio https://docs.minio.io/docs/aws-cli-with-minio
    • Official Manual https://docs.aws.amazon.com/cli/latest/userguide/using-s3-commands.html
• APIs:
  • Option 4: aws-sdk-java
  • Option 5: boto

Option 2 & 3 are more general and versatile, both are widely used for various object storage services.

Object Storage Benchmark

• COSBench https://github.com/intel-cloud/cosbench
  • User Guide https://github.com/intel-cloud/cosbench/raw/master/COSBenchUserGuide.pdf.
  • Example workload https://github.com/Zhan2012/obs-tutorial/raw/master/workload-example.xml.
  • Other examples https://github.com/open-io/dockerfiles/tree/master/cosbench-openio/examples.

References

• COSBench: cloud object storage benchmark https://dl.acm.org/citation.cfm?doid=2479871.2479900
• COSBench: A Benchmark Tool for Cloud Object Storage Services http://www.cs.cmu.edu/~qingzhen/files/cosbench_cloud12.pdf
• COSBench: A benchmark tool for Cloud Storage https://www.snia.org/sites/default/files/files2/files2/SDC2013/presentations/Cloud/YaguangWang__COSBench_Final.pdf

Basic Functionality

In computer programming, create, read, update, and delete (as an acronym CRUD) are the four basic functions of persistent storage.

Operation	SQL	HTTP
Create	INSERT	PUT / POST
Read (Retrieve)	SELECT	GET
Update (Modify)	UPDATE	PUT / POST / PATCH
Delete (Destroy)	DELETE	DELETE

Evaluation

Throughput, Latency under different object size, concurrency, server total.

Suggested topics:

• How object size affects performance?
  • for a particular application, is there a best way to fit into OBS?
• The main factors behind I/O latency?
  • Get latency distribution first.
• What will happen when clients are crowded?
• Why tests 'fail'? (not terminate)
• The outcome of scaling out (putting more servers into system)?

More insights are encouraged.

Further thoughts

• How to do these experiments on your own codes (besides COSBench)?
• Using Python as Lab Platform
  • Jupyter Notebook Tutorial https://github.com/cs-course/jupyter-tutorial

Future Readings

Recent SNIA blog posts on Object Storage http://sniablog.org/category/object-storage/.

Enterprise level Object Store comparison.

Zhan.Shi @ 2018