motine/ohouse

Ohouse

The Clearinghouse

This is/shall a Clearinghouse for all the projects mentioned.

Moved!

Since I am not part of the development any more, this project has been renamed to [C-BAS] and moved here.

Installation

Environment

This software is based on AMsoil, please refer to the installation guide there.
Please follow at least the "Base" and "Plugins" steps.

Dependencies

Please install Swig (needed for the M2Crypto Python package). Ohouse currently relies on a MongoDB database running on the local host (and default port).

Python dependencies can then be installed using pip install -r requirements.txt.

Run

Fire up the server via python src/main.py and install the packages which are not found.

Note that sometimes the output is only given to the log in log/amsoil.log.

Additional

• Copy the following files and adust the entries as required

  • The deploy/config.json.example to deploy/config.json
  • The deploy/registry.json.example to deploy/registry.json
  • The deploy/supplementary_fields.json.example to deploy/supplimentary_fields.json

• Either

  • Install trust root certificates to deploy/trust (as pem) and a admin cert (admin-key.pem and admin-cert.pem) to admin.
  • Or create test certificates and credentials and copy them to the respective places (see test/creds/TODO.md).

• Run the server with python src/main.py

• In a new cosole run the config client python admin/config_client.py --interactive and make change according to your setup

Test drive

You may run the xx_tests.py scripts in test/unit to make sure everything works fine. The test scripts assume that there are test certificates and credentials in test/creds (for creating them please see test/creds/TODO.md).

Architectural decisions

• Please see the ofed - plugin.py for considerations on how to protect information regarding authZ.
• Ohouse will support both v1 and v2 of the Uniform Clearinghouse API. This is realised through different service endpoints for each version.

ohouse's Issues

Fix Travis CI

Travis had a problem after the merges, please investigate.

Document gen-certs.py

Remove authentication from nginx config

Several URNs For Matching During Slice LookUp

In the current Ohouse implementation, only lookup with one matching URN is supported; however, in the specification several URNs are supported during lookup.

Test with jProbe

Use pyPElib to enforce policies

See github page for more details.

Also feed this back into AMsoil.

Configurable MongoDB instance

Add functionality to change MongoDB IP, port, database name etc. in Ohouse.

Add info about setting up virtualenvwrapper

Add a reference and short intro how to setup virtualenvwrapper (possibly in AMsoil), so developers can choose to go this way.

Determine AuthZ levels

The Federation Service API suggests the following user levels: LEAD, ADMIN, MEMBER, OPERATOR, AUDITOR.

Consider also the existing OFELIA privileges.

A way to decouple the privelege levels from Ohouse is desired. This way, each Ohouse deployment can determine an individual set AuthZ levels and the corresponding actions possible with this.

Test with Omni GENI client

As Ohouse implements the Federation Service API, an Omni GENI client should be able to talk to it. This needs to be confirmed.

See webpage for more details: http://trac.gpolab.bbn.com/gcf/wiki/OmniOverview

Tests For Create and Update

Adding more checks for Create and Update methods. For example, comparing the values to the ones in Config.Json to check if the expiry and creation dates are valid or not.

Validate slice names

When creating slices, make sure that the name is legal (hyphen is legal, underscore is not). Please see the Standard: NB: SLICE_NAME must adhere to the restrictions for slice names in the Aggregate Manager (AM) API, namely that it must be <= 19 characters, only alphanumeric plus hyphen, no leading hyphen..

Also, document it in the docs/README.md.

Implement 'get_credentials' method

get_credentials method is available for SLICE and MEMBER objects.

Documentation from the API:

Provide list of credentials for the caller relative to the given slice. 
If the invocation is in a speaks-for context, the credentials will be for the 
‘spoken-for’ member, not the invoking tool.

For example, this call may return a standard SFA Slice Credential and some 
ABAC credentials indicating the role of the member with respect to the slice.

Note: When creating an SFA-style Slice Credential, the following roles 
typically allow users to operate at known GENI-compatible 
aggregates: "*" (asterisk)  or the list of "refresh", "embed", 
   "bind", "control" "info".

Arguments:
  slice_urn: URN of slice for which to get member’s credentials
  options: Potentially contains ‘speaking_for’ key indicating a speaks-for 
     invocation (with certificate of the accountable member 
     in the credentials argument)
 Return:
  List of credential in “CREDENTIALS” format, i.e. a list of credentials with 
  type information suitable for passing to aggregates speaking AM API V3.

Documentation Platform

Implement and use certificate and credential checks

No authorisation takes place to determine whether a user can perform a particular operation. This includes the access to particular fields of data.

Improve error message if mongodb is not running

Catch the exception and give a reasonable error message with what to do to fix it, instead of this:

Traceback (most recent call last):
  File "src/main.py", line 42, in <module>
    main()
  File "src/main.py", line 21, in main
    pm.init(config.PLUGINS_PATH)
  File "/Users/motine/Documents/AMsoil/dev/Ohouse/src/amsoil/core/pluginmanager.py", line 261, in init
    pluginInfo.setup()
  File "/Users/motine/Documents/AMsoil/dev/Ohouse/src/amsoil/core/pluginmanager.py", line 146, in setup
    self._plugin_module.setup()
  File "/Users/motine/Documents/AMsoil/dev/Ohouse/src/plugins/mongodb/plugin.py", line 5, in setup
    mongo_database = MongoDB()
  File "/Users/motine/Documents/AMsoil/dev/Ohouse/src/plugins/mongodb/mongodatabase.py", line 25, in __init__
    client = pymongo.MongoClient()
  File "/Users/motine/.virtualenvs/ohouse/lib/python2.7/site-packages/pymongo/mongo_client.py", line 352, in __init__
    raise ConnectionFailure(str(e))
pymongo.errors.ConnectionFailure: could not connect to localhost:27017: [Errno 61] Connection refused

Add an 'additional_info' return value to the 'extract_certificate_info' method

Implement 'speaking_for' invocations

Documentation from the API:

Best practices dictate that individuals should speak as themselves: that is, the entity on
the other side of an SSL connection is the one referred to by the certificate on the
connection. Obviously, people typically use tools or software interfaces to create these
connections. When a tool is acting directly on a user’s desktop using the user’s key
and cert with the user’s explicit permission, it may be acceptable to consider the tool
as speaking as the user. But for many tools, the tool is acting on behalf of the user in
invoking Federation or AM API calls. In this case, it is important for the tool to not
speak as the user but to speak for the user, and to have the service to whom the tool is
speaking handle the authorization and accountability of this request accordingly.

Accordingly, a Federation Registry and associated Authorities should support speaks-for
API transactions. These API transactions use the same signatures as the calls described in
this document, with these enhancements:

- A 'speaking_for' option containing the URN of the user being spoken for

- A speaks-for credential in the list of credentials: a statement signed by the user
indicating that the tool has the right to speak for the user, possibly limited to a
particular scope (e.g. slice, project, API call, time window).

The service call is then required to determine if the call is being made in a speaks-for
context or not (that is, the ‘speaking_for’ option provided). If so, the call must
determine if the tool is allowed to speak for the user by checking for the presence of a
valid speaks-for credential and the spoken-for user’s cert. If so, the call should
validate if the user is authorized to take the proposed API action. If so, the action is
taken and accounted to the user, with identity of the speaking_for tool logged. If the
call is ‘speaks-for’ but any of these additional criteria are not met, the call should
fail with an authorization error. If the call is not a ‘speaks-for’, then the normal
authorization is performed based on the identity (certificate) provided with the SSL
connection.

Aggregates are also encouraged to support speaks-for authentication and authorization, but
this is an aggregate-internal policy and implementation decision, and outside the scope of
this document.

Consider deployment specific configuration

For example, check that a PROJECT or SLICE object cannot be extended beyond a certain time limit (such as no extensions beyond a month from the current time).

These restrictions such be decoupled into a well-defined and separate JSON configuration file.

Add detailed documentation on field's format of mongodb plugin

E.g. I don't know what format a match should have in a lookup call.
Maybe it is sufficient to add links to the pymongo doc.

Please think about common use cases (e.g. not only finding one entry but multiple).

Refactor Ohouse config

As discussed on the 8th of April:

• defaults.json moves to the plugin folder
• supplementary fields get their own file in deploy
• general config (e.g. #19 or #16) gets its own file in deploy
• we keep the configdb and configrpc for AMsoil/flask config
• the registry database gets its own file

Did I forget anything?

Fix typo in delegatetools.py

_type should be type_

Document example Ohouse workflow

Documentation Platform

Deciding which documentation platform (Sphinx, Doxygen, Read The Docs) to use for documenting the Ohouse project.

Add more slice name tests

test for the all cases individually here

    for slice_name in ['invalid_slicename', '-invalid', 'asdasd!!!']:
        # ... testing ...

Implement registration portal

A GUI for new users to register with Ohouse (thus creating a MEMBER object).

As new member creation is outside of the scope of the Federation Service API, this interface will be provided out-of-band. It is still in the scope of Ohouse however.

Consider a Flask or Django for a web registration portal. Also consider xml-signer.

Add @servicemethod decorators in the trust plugin

Fix error when checking out Ohouse and running the serve for the first time

Traceback (most recent call last):
  File "src/main.py", line 42, in <module>
    main()
  File "src/main.py", line 21, in main
    pm.init(config.PLUGINS_PATH)
  File "/Users/motine/Documents/AMsoil/dev/Ohouse/src/amsoil/core/pluginmanager.py", line 261, in init
    pluginInfo.setup()
  File "/Users/motine/Documents/AMsoil/dev/Ohouse/src/amsoil/core/pluginmanager.py", line 140, in setup
    raise PluginBootstrapModuleNotLoaded(self.pluginName)
amsoil.core.pluginmanager.PluginBootstrapModuleNotLoaded: mongodb (try looking at the log)

Remove rebase.sh and todo.md

Perform one last check if all todos are migrated to issues.

Enforce 'match' lookup authorization

After Authz discussion is resolved (#8), use permissions to determine dissemination of PRIVATE and IDENTIFYING values as per the PROTECT field.

Add parsing of the admin certificate

The admin certificate should be parsed in the SA and MA.

See comment below:

I don't think there is anything much to look for... the only thing that
is different is that the credential target is the authority itself.
(Normally, the CM generates credentials with a sliver as a target, and
of course then they are useless outside that sliver.)  I don't believe
that genadmincredential credentials issued by the SA do anything special
right now.

Also see discussion on Wednesday 26th of Aug.

Tests clean up after themselves

Currently, if you test CRUD on the SA/MA, the test typically removes the test data from the server. Adel said that sometimes you have to clear the database so that the tests can run twice in a row. He will add more details which tests does not clean up after itself.

Note: We are testing from the outside as a client to an external server and not from within the server.

Decouple Exceptions For Different API Versions

Currently, the exceptions for version 1 and version 2 are in the same file apiexceptions.py. Decoupling the exceptions into two separate files is suggested

Unit Tests For Slice Create and Update Methods

Adding more checks for Create and Update methods. For example, comparing the expiration time values to the ones at create time.

Add assertion to not create database entries on failures

Please add a lookup call with the slice name and assert that there is no such object after the failed call here and in the other new tests.

Remove redundancy in method parameters

Some methods have parameters that pass duplicated data. For example, the 'fields' value is passed in addition to the full 'options' parameters.

Test with jFed

iMinds has put out a testing framework to test testbed API specs. It is called jFed and contains a automated testing application (probe). This can be used to test Ohouse's implementation of the Federation API version 2.

It might make sense to test it with this framework so we have the same understanding as others. In the process of testing, we might encounter some issues with the actual spec. If so please write to the federation API mailinglist (see my mail from the 13th of May, 2014).

If you are using Mac OSX mind the security settings

Refactor delegate tools to use static methods

The apitools.py makes use static methods. Therefore, to make the implementation of the delegatetools.py consistent, some methods should be re-factored to be static.

travis-ci.org integration

Create configuration for http://travis-ci.org integrated testing.

We will need to install SWIG and xmlsec1 dependencies in the test environment. A MongoDB database is also necessary.