cloudmesh/cloudmesh-nist

NIST Big Data Reference Architecture Interfaces

Document URL

• https://cloudmesh.github.io/cloudmesh-nist/nistvol8-2.epub
• https://cloudmesh.github.io/cloudmesh-nist/nistvol8-2.pdf

pandoc-codeblock-include is broken, a fixed version can be installed from

pip install --upgrade --force git+https://github.com/laszewsk/pandoc-codeblock-include

The working draft of the specification is in

• https://github.com/cloudmesh-community/nist/blob/main/docs/nistvol8-2.md

The snapshot of the volume is available in various formats that should not be modified but can be used for reviewes at

• https://github.com/cloudmesh-community/nist/blob/main/docs/nistvol8-2.md
• https://github.com/cloudmesh-community/nist/blob/main/docs/nistvol8-2.epub
• https://github.com/cloudmesh-community/nist/blob/main/docs/nistvol8-2.pdf
• https://github.com/cloudmesh-community/nist/blob/main/docs/nistvol8-2.docx

It is easy to contribute to the document with the help of pull requests that allow eddits that will be addressed by the editor of this Volume (Gregor von Laszewski). Pull request can even be created via the GUI features of Github. You will need a github.com account and login into github to make use of this.

With Pandoc a version in MS Word can be created and formating changes can be applied. However all editorial non look and feel changes must not be conducted in the docx version but in the markdown version so we have a record of the changes in the github history. The look and feel, can be managed by a professional editor.

Once finalized, the Spec Draft will be hosted here in word format

https://cloudmesh-community.github.io/nist/spec/

and uploaded to NIST.

The document workflow is as follows:

1. contributor gets github account
2. contributor logs into github
3. contributor eidts the md document and creates a pull request
4. pull request is reviewed by Gregor von Laszewski and others while discussing it in the working group meetings if needed
5. pull request is either rejected, accepted, or accepted with changes
6. Once a number of changes are accepted a new version is created, however the most up to date version is always avalaible as markdown document
7. The version number of the document is increased.

Super old:

• Outdated document: https://github.com/cloudmesh/nist-volume8
• On Gregors Computer ~/Desktop/Documents/OneDrive/nist-volume-8

New Python function to Microservice generator

Please note that we have developed a convenient python function to microservices toolkit at

• https://github.com/cloudmesh/cloudmesh-openapi/

This tool just takes a function or class defined in python and generates the OpenAPI as well as the server code automatically. It can also be used as a starting point for customizing the API specification.

Requirements

We need more things which we have not yet written down here

$ npm install -g spectacle-docs
$ brew install ghc cabal-install
$ cabal update
$ cabal install pandoc-include-code
$ cabal install pandoc-include

Get the code

git clone https://github.com/cloudmesh-community/nist.git

cd nist/cloudmesh/openapi/service

We assume all terminals are cd'ed to the directory nist/cloudmesh/openapi/service

Running the test program

make service
make start # in one window
make test # in the other window

Creating the spec in HTML

make doc

viewing the doc

make view # or open public/index.html in your browser

Publish the spec

make publish

Create a new service

cm-openapi service init SERVICE

Example:

./bin/cm-openapi service init health
services/example -> services/health

A Service Example

This example demonstrates two points. First, it adds security support of http basic auth and apikey/secret auth. Second it provides the concept of the virtualdirectory as a demonstration specification to showcase an example usage of a service with authentication, as well as showcasing the idea of virtualdirectory.

The source for this example is located in

• https://github.com/cloudmesh-community/nist/tree/main/services/virtualdirectory

Start mongo

$ cd mongo
$ make start

To run the demo:

1. Make sure the mongo container is running as required by the previous examples;

2. In the top level of nist git repo say

   $ git clone git clone https://github.com/cloudmesh-community/nist.git
   $ cd nist
   $ make all
   

3. Then say

   $ cd service; make all
   

This will run our new combined service virtualdirectory

In another terminal window, go also the the nist git repo directory. The auth credential is configured in the file:

services/secconf.yaml

---

⚠️ Please make sure to set your own passwords. We will move the file in future to

~/.cloudmesh/services/seccont.yaml

However we have not yet integrated this into the code. Thus be careful that when creating pull requests not to check in your version of secconf.yaml.

---

Then say

$ cd services/virtualdirectory/test
$ ./test_virtualdirectory.sh

This will add an ftp endpoint as an entry of the virtualdirectory. (currently ftp is the only supported protocol for the virtualdirectory).

Then run these from a terminal to test the two service endpoints.

$ curl -u admin:secret http://localhost:8080/cloudmesh/virtualdirectories
$ curl -u admin:secret http://localhost:8080/cloudmesh/virtualdirectory/UbuntuFTP

One line Convenient Demo Script

We assume that you have docker installed on your machine. This demo works at this time only on macOS

To execute the demo do the following

  $ git clone git clone https://github.com/cloudmesh-community/nist.git
  $ cd nist
  $ make demo

In case you do not have an macOS machine we have shot a convenient video that explains how simple it is to run our services.

• https://www.youtube.com/watch?v=fl-4PGPzwsM

Please note that the service here is build from an open API specification that is includes in

• https://github.com/cloudmesh-community/nist/blob/main/services/virtualdirectory/virtualdirectory.yaml

The code for the service backend is located at

• https://github.com/cloudmesh-community/nist/blob/main/services/virtualdirectory/virtualdirectory_controller.py

The password configuration file is located at

• https://github.com/cloudmesh-community/nist/blob/main/services/secconf.yaml

As this is only a demonstration, we have used the most simple security possible. We assume the server runs only on the local host. Please make sure to properly secure it.

cloudmesh-nist's Issues

Test new templates for md -> docx conversion

Laurie will provide docx templates to apply specific formatting to the pandoc docx output (when converting from the nistvol8-2.md document. These templates should be tested to determine if the following desired formatting is applied:

• Bullet list should be styled with the "List Bullet" style (and not "Normal" style)
• The alignment in the "Normal" style should be Left and not Justified
• Verify correct styling of all headers, through level 6,

Spec table eidts

• For the table included with {include=./specstable.md}: The last column header has a typo (“Refernce” --> “Reference”)
• Can we add a title for this table? (e.g., Table 1: List of Specifications)

add a better futur work section

deal with pull request #27

#27

reformulated design by example

reformulated design by example

including workflow how to get the spec to create a service

this shoudl be done through a suplementary document

Section 4 review comments

ISSUES COULD NOT BE FOUND OR ARE A WORD ISSUE

• 1. References come in as APA style instead of the IEEE style used in the other documents. Last year, we changed from APA previously used to IEEE to be in line with Volume 8.

  issue in word template as epub shows IEEE style publications

• 4. Acknowledgements: The following text might be better in section 1.3, report production. In other volumes we list the editors for each version (not the major changes to the text for each version)

• 44. Format Acronyms

  This is a word template issue and needs to be fixed in word. epub shows it correctly.

• 43. Change Acronyms and Bibliography from sections to Appendices (CAN WE DO THIS IN GITHUB? OR JUST LEAVE IT FOR THE WORD MODIFICATIONS?)

  This is not easyly done, and best done in final version of the word document

• 46. Bibliography: Internet2. eduPerson Object Class Specification citation has a line return after "March" but it should not.

  shows up correctly in epub. word issue

UNCLEAR WHAT NEEDS TO BE DONE:

• 47. Correct citations within document and recreate Bibliography. Citations were added using Mendeley. However, the change should be made in GitHub so that the correct reference list is used.

TODO

• 7. Section 4.2.1, Authentication: Change: • The service that acts on behalf of the user needs to have access…” Also, add period at the end of this bullet. (CHANGE IN GITHUB – I didn’t see the identity.yaml file in cloudmesh-nist/spec/)

• 8. Section 4.2.1, Authentication: Change “An example for a configuration file is provided in the GitHub repository [insert reference].” . (CHANGE IN GITHUB – I didn’t see the identity.yaml file in cloudmesh-nist/spec/)

• change all TODO's in spec dir

  • 29. Section 4.6.5, Nic: removed the following text “TODO: assign and improve”
  • 32. Section 4.8.1, Microservice: Removed the following text “TODO: assign and improve”
  • 37. Section 4.10.1, Reservation: Removed the following text “TODO: assign and improve”
  • 38. Section 4.11.1, Stream: Removed the following text “TODO: assign and improve”
  • 39. Section 4.11.2, Filter: Removed the following text “TODO: assign for review and improvement”
  • 41. Section 4.12.1, Hadoop (MapReduce): Removed the following text “TODO: assign for review and improvement”
  • 42. Section 4.13.1, Deployment: Removed the following text “TODO: assign for review and improvement”

• Acronyms: We should add descriptions for container and hybrid cloud in addition to the reference to the NIST SP800-180 document

FIXED OR NOT FOUND

⚠️ Comments are based on a version that was older than the one in GITHUB

• 16. Section 4.3.3.2.2.1, GET /cloudmesh/variables/{name}: Change “Returns variables”

• 17. Section 4.3.4.1, Properties key table, description column: “tha”  “the”

• 22. Section 4.6.1.1, Properties Image: Change “…they are passed along as space separated strings”

• 18. Section 4.3.4, Keyvaluestore: Keyvaluestore is a service to store key, value, and type information, all of which are stored as Strings. However, the type could be used to transform it into a non-string type.

• 13. Section 4.3.2, Alias: A user may desire to create an alias for a resource. An alias may be used to more easily remember a resource. A User can deploy a service in which such aliases are stored. The aliases could naturally be shared with other users. A resource could have one or more aliases. The reason for an alias is that a resource may have a complex name but a user may want to refer to the resource using a name that is suitable for the user’s application.

  This is an outdated text and a new text was already checked into github

• 2. In the text, GitHub should have G and H capitalized. Change Github  GitHub

• 3. All document: Replace “cleansing” with “cleaning”; “cleansed” with “cleaned” (one instance Section 2.2.4.2 Preparation)
  “The following milestones were achieved in the versions listed below:

• a. Version 2.1: A previous volume used just the definition of the schema based on examples. It was easier to read but only included the definition of the resources and not the interaction with the resources. This volume was in place until June 2018.

• b. Version 2.2: This version was significantly changed and now uses OpenAPI to specify the Interfaces between the various services and components.

• c. Version 3.1.0: The version includes the significant improvements of the object specifications.

• 5. Section 3.4, Version Management: Change the following: “Previous work that shaped the current version of this volume are documented in the prior version of Volume 8 [insert reference] and Cloudmesh in support of the NIST Big Data Architecture Framework [insert reference].”

• 6. Section 4.1: Delete the extra column in the Table of Specifications

• 9. Section 4.2.2, Organization: Change “Such groups and rolls can be used to specify access rights to services.”

• 10. Section 4.2.2.2.4.1: GET /cloudmesh/organization/users/{username}: Returns the specific user of that organization

• 11. Section 4.2.3, User: Change “A user can be added to a named list of users. A group associated with the user can be used to augment users to be part of one or more groups. The following terminology is distinguished .”

• 12. Section 4.2.3.1, Properties user: in description column, change “firstanme”  “firstname”

• 14. Section 4.3.3.2.1.1, GET /cloudmesh/variables: Change “Returns all variables”

• 15. Section 4.3.3.2.1.2, PUT /cloudmesh/variables: change “Create a new variables” to “Create new variables”

• 45. Acronyms: Change “Virtual File System”

• 48. In the slurmjob.yaml make sure we change “gonna” to “going to”

• 23. Section 4.6.2.1, Properties Flavor: Change “A description for the compute node”. Change “specifies whether the flavor features an ephemeral disk”

• 24. Section 4.6.2, Flavors: The flavor specifies elementary information about a compute node. This information includes name, id, label, ram size, swap size, disk space, availability of ephemeral disk, available bandwidth, price value, cloud name, and date and time of last update. Flavors and the corresponding information are essential to size a virtual cluster appropriately.

• 25. Section 4.6.3.2.1.1, GET /vm: Change “specify the cloud from which we list, if omitted all clouds are returned.”

• 26. Section 4.6.4, Secgroup: A security group defines the incoming and outgoing security rules which can then be assigned to a node when a node is being created. Once the node is up, the connection to and from the node will be decided by the security group rules, in addition to any other possible rules applied on network devices or from the instance’s firewall settings. A security group may have one or multiple rules and a node may be associated with one or more security groups.

• 27. Section4.6.4.3.3.1, GET /cloudmesh/secgroup/{secgroup}/rule: Get the defined security group rules for a specified secgroup

• 28. Section 4.6.4.3.4.1, GET /cloudmesh/secgroup/{secgroup}/rule/{ruleid} and Section 4.6.4.3.4.2 DELETE /cloudmesh/secgroup/{secgroup}/rule/{ruleid}: Change “The name of the secgroup from which the rule will be deleted”

• 30. Section 4.7.1.2.1.2, PUT /cloudmesh/container: Change “Create new containers”

• 31. Section 4.7.1.2.2.1, GET /cloudmesh/container/{name}: Change “Returns containers”

• 33. Section 4.8.1.2.2.1, GET /cloudmesh/microservice/{name}: Change “Get microservice info by name”

• 34. Section 4.9.1.2.1.1, GET /vm: Change “specify the cloud from which we list, if omitted all clouds are returned.”

• 35. Section 4.9.2, Slurmjob: Removed the following text: “⭕ compare with batchjob”. The ⭕ is a red circle in the online version. Line in the .md file is “:o: compare with batchjob”

• 36. Section 4.9.2.2.1.1, GET /vm: Change “specify the cloud from which we list, if omitted all clouds are returned.”

• 40. Section 4.12.1, Hadoop: there was a comment to rename the section to MapReduce (“⭕ rename to mapreduce”). I removed the comment and changed Hadoop to MapReduce throughout section 4.12.1.

• In the specifications, sometimes the names are separated and sometimes not. For example, “Virtual Cluster” is listed as two words and “Virtualdirectory” as one. Should all two word specifications be separated? Does this need to be done in the yaml files also?

• 19. Section 4.4.4.3.3.1, GET /cloudmesh/virtualdirectory: Change “Returns all virtual directories”

• 20. Section 4.5.1, Virtual Cluster: “A Virtual Cluster is modeled as one Front-end, or manager node, and one or multiple compute nodes. The manager node usually served as a login node and can be accessed from outside via a public IP. The compute nodes are connected to the manager node via a private, usually high-performance (high-throughput and low-latency), network and thus accessible only from the manager node. To use the virtual cluster, login to the manager node, and from there login to any compute node, or submit a job to run on the compute nodes.” This should be revised to remove “you”

• 21. Section 4.5.1.4.3.1, GET /cloudmesh/virtualcluster/virtualcluster/{virtualclustername}/manager and Section 4.5.1.4.4.1, GET /cloudmesh/virtualcluster/virtualcluster/{virtualclustername}/{nodename}: Change “name of the virtual cluster”

• Figure 2: The text in the Provider View figure is a bit small. It seems that we could take advantage of the vertical space a bit more by maybe wrapping the text on some of the levels.

• Section 4.2.3, User: What is the difference between the two terms listed (group and organization). The descriptions do not make the distinction clear. Both seem to be collections that a user can belong to.

• Section 4.9.1, Batchjob: The text says “VM is a service to manage virtual machines.” But doesn’t mention batchjob. In addition, the subsections are about vm. Could this be pulling from the vm yaml instead of the batchjob yaml? Same with Section 4.9.2, Slurmjob.

• In the specifications, sometimes the names are separated and sometimes not. For example, “Virtual Cluster” is listed as two words and “Virtualdirectory” as one. Should all two word specifications be separated? Does this need to be done in the yaml files also?

• Section 4.12.1.1, Properties MapReduce: The table mentions ansible. Is this ok?

Various comments

• Figure 2: The text in the Provider View figure is a bit small. It seems that we could take advantage of the vertical space a bit more by maybe wrapping the text on some of the levels.
• Acronyms: We should add descriptions for container and hybrid cloud in addition to the reference to the NIST SP800-180 document
• Section 4.9.1, Batchjob: The text says “VM is a service to manage virtual machines.” But doesn’t mention batchjob. In addition, the subsections are about vm. Could this be pulling from the vm yaml instead of the batchjob yaml? Same with Section 4.9.2, Slurmjob.
• Bibliography: It seems that the document generator is pulling the wrong bibliography file. There were only two references cited within the document, both of which are in references.bib file. However, the Bibliography in the document contains the references from the bib.md file. Also, we need to add the NBDIF references to the references.bib file as well.
• In the specifications, sometimes the names are separated and sometimes not. For example, “Virtual Cluster” is listed as two words and “Virtualdirectory” as one. Should all two word specifications be separated? Does this need to be done in the yaml files also?
• In the slurmjob.yaml make sure we change “gonna” to “going to”
• Section 4.12.1.1, Properties MapReduce: The table mentions ansible. Is this ok?

Add references to list of Volumes

Please add a reference after each Volume in the bullet list in the Executive Summary and in the Introduction/Background Section (Section 1.1). These should reference version 2 of each document (except Volume 5 which only has version 1).

cm-openapi-gregor

find out the difference between the old version and the new version and see how to integrate them.
Gregor did not want to overwrite the new version so he checked his version in with -gregor as postfix

Remove reference to future work

We are removing references to future work on the documents. The paragraph in Section 1.3, Report Production might need to be removed.

This document is expected to grow with the help of contributions from
the community to achieve a comprehensive set of interfaces that will be
usable for the implementation of Big Data Architectures. To achieve
technical and high-quality document content, this document will go
through a public comment period along with NIST internal review.

update to python 3.7 right now it uses 3.6.4

• update the virtual directory example to 3.7.0
• see if it works
• update all to 3.7.0

Add new navigation diagram

The new navigation diagram (NIST Big Data Interoperability Framework Documents Navigation Diagram) should be added after the new paragraph introducing the figure (at the end of section 1.4 Report Structure).
NBDIFDocNavigationFig_2019-05-28_LA.pptx

Database: import and/or add

The database has an import and an add statement. It is important to dicusss if both are needed or just one.

Also the database.yaml file need sto be reviewed