HSDS is a web service that implements a REST-based web service for HDF5 data stores. Data can be stored in either a POSIX files system, or using object based storage such as AWS S3, Azure Blob Storage, or OpenIO <openio.io>. HSDS can be run a single machine using Docker or on a cluster using Kubernetes (or AKS on Microsoft Azure) The commercial offering based on this code is known as Kita™. More info at: https://www.hdfgroup.org/solutions/hdf-kita/.
- Main website: https://www.hdfgroup.org/kita
- Source code: https://github.com/HDFGroup/hsds
- Mailing list: [email protected] [email protected]
- Documentation: http://h5serv.readthedocs.org (For REST API)
- RESTful HDF5 White Paper: https://www.hdfgroup.org/pubs/papers/RESTful_HDF5.pdf
- SciPy17 Presentation: http://s3.amazonaws.com/hdfgroup/docs/hdf_data_services_scipy2017.pdf
- HDF5 For the Web: https://hdfgroup.org/wp/2015/04/hdf5-for-the-web-hdf-server
- HSDS Security: https://hdfgroup.org/wp/2015/12/serve-protect-web-security-hdf5
- HSDS with Jupyter: https://www.slideshare.net/HDFEOS/hdf-kita-lab-jupyterlab-hdf-service
- AWS Big Data Blog: https://aws.amazon.com/blogs/big-data/power-from-wind-open-data-on-aws/
Make sure you have Python 3, docker, docker-compose installed, then:
- Setup password file:
$ cp admin/config/passwd.template admin/config/passswd.txt
- Start server:
$ ./runall.sh
- To run test suite, set an environment variable for the admin password:
$ export ADMIN_PASSWORD=admin
- Run the test suite:
$ ./testall.sh
To shut down the server, run: $ ./stopall.sh
Note: passwords can (and should for production use) be modified by changing values in hsds/admin/config/password.txt and rebuilding the docker image.
On AWS
See: docs/docker_install_aws.md for complete install instructions.
See: docs/kubernetes_install.md for setup on Kubernetes.
On Azure (BETA)
For complete instructions to install on a single Azure VM:
For complete instructions to install on Azure Kubernetes Service (AKS):
On DCOS (BETA)
For complete instructions to install on DCOS:
On Prem (POSIX-based storage) (BETA)
For complete instructions to install on a desktop or local server:
As a REST service, clients be developed using almost any programming language. The test programs under: hsds/test/integ illustrate some of the methods for performing different operations using Python and HSDS REST API (using the requests package).
The related project: https://github.com/HDFGroup/h5pyd provides a (mostly) h5py-compatible interface to the server for Python clients.
For C/C++ clients, the HDF REST VOL is a HDF5 library plugin that enables the HDF5 API to read and write data using HSDS. See: https://github.com/HDFGroup/vol-rest.
HSDS only modifies the storage location that it is configured to use, so to uninstall just remove source files, Docker images, and S3 bucket/Azure Container/directory files.
Create new issues at http://github.com/HDFGroup/hsds/issues for any problems you find.
For general questions/feedback, please use the Kita™ forum: https://forum.hdfgroup.org/c/kita.
HSDS is licensed under an APACHE 2.0 license. See LICENSE in this directory.
The HDF Group provides access to an HSDS instance that is integrated with JupyterLab: Kita™ Lab. Kita™ Lab is a hosted Jupyter environment with these features:
- Connection to a HSDS instance
- Dedicated Xeon core per user
- 10 GB Posix Disk
- 100 GB S3 storage for HDF data
- Sample programs and data files
Sign up for Kita™ Lab here: https://www.hdfgroup.org/hdfkitalab/.
The HDF Group provides an AWS Marketplace product, Kita™ Server, which provides simple installation of HSDS and related AWS resources. Kita™ offers these features:
- Stores usernames and passwords in a secure DynamoDB Table
- Creates a AWS CloudWatch dashboard for service monitoring
- Aggregates container logs to AWS CloudWatch
Kita™ Server for AWS Marketplace can be found here: https://aws.amazon.com/marketplace/pp/B07K2MWS1G.