open-cloud/xos

Source code for XOS, the Cloud OS running on OpenCloud and CORD. This is a mirror of gerrit.opencord.org/xos.

xos's Issues

Base XOS image

Prepare a Docker image of "base" XOS and document installation.

Security Audit

Perform a complete security audit.

NameError: global name OpenStackDriver is not defined

INFO:xos.log:Step <class 'sync_controller_users.SyncControllerUsers'> succeeded
Step <class 'sync_controller_users.SyncControllerUsers'> succeeded
INFO:xos.log:Step <class 'sync_slivers.SyncSlivers'> succeeded
Step <class 'sync_slivers.SyncSlivers'> succeeded
INFO:xos.log:Step <class 'sync_controller_slices.SyncControllerSlices'> succeeded
Step <class 'sync_controller_slices.SyncControllerSlices'> succeeded
INFO:xos.log:Step <class 'sync_controller_sites.SyncControllerSites'> succeeded
Step <class 'sync_controller_sites.SyncControllerSites'> succeeded
ERROR:xos.log:sync step failed! BEG TRACEBACK
Traceback (most recent call last):
  File "/opt/xos/observer/syncstep.py", line 120, in call
    self.delete_record(o)
  File "/opt/xos/observer/steps/sync_controller_networks.py", line 81, in delete_record
    driver = OpenStackDriver().client_driver(caller=controller_network.network.owner.creator,
NameError: global name 'OpenStackDriver' is not defined
ERROR:xos.log:sync step failed! END TRACEBACK
ERROR:xos.log:sync step failed! BEG TRACEBACK
Traceback (most recent call last):
  File "/opt/xos/observer/syncstep.py", line 120, in call
    self.delete_record(o)
  File "/opt/xos/observer/steps/sync_controller_networks.py", line 81, in delete_record
    driver = OpenStackDriver().client_driver(caller=controller_network.network.owner.creator,
NameError: global name 'OpenStackDriver' is not defined
ERROR:xos.log:sync step failed! END TRACEBACK

Port forwarding does not work

I rebuilt opencloud-neutron-plugin to work on top of the ML2 neutron plugin. Instructions are in git.planet-lab.org: [opencloud-plugin.git] / ml2_plugin / README. Tested this on Washington.

Needs to be deployed to other Vicci headnodes and/or automated in install cloud.

Note the Juno supports "extension plugins" for ML2, and will yield a much cleaner approach.

Fix Ansible configuration in Docker image

In the image built using the project Dockerfile, Ansible is not configured with our patches.

Multiple user keys per slice on VM boot.

A current limitation is that only one user key is injected into the slice. That user can login and manually add the keys of other users, but an OpenCloud admin also needs to add the keys to the account used to support proxy login to the slice.

Controller Admin UI workflow / cleanup

1. Need a way to navigate to Controller objects that doesn't rely on selecting some other object, going to core, and then going to Controller.

2. ControllerAdmin should hide enacted/policied and make the backend_status/backend_register fields readonly.

3. Need to verify that nova.admin_user, nova.admin_password, nova.admin_tenant, etc., in xos_config are deprecated and if so remove them from the config file.

Instance flavor problems on non-OpenCloud backends

After connecting XOS to an OpenStack cluster running on CloudLab, I am able to create a VM but the flavor is wrong (e.g., I specify m1.small but get m1.tiny).

Public IP addresses not supported at most OpenCloud sites

Only Princeton and Arizona have a range of public IP addresses available to assign to VMs. For the other sites, we'll need to either secure a block of IPs, or else re-use some of the existing IPs assigned to VICCI nodes.

Mini-Dashboad

Plumb the monitoring mini-dashboard to the ceilometer data collection.

AttributeError: 'unicode' object has no attribute 'auth_url'

INFO:xos.log:Step <class 'sync_controller_images.SyncControllerImages'> is a leaf
INFO:xos.log:Step <class 'sync_controller_site_privileges.SyncControllerSitePrivileges'> is a leaf
INFO:xos.log:Step <class 'sync_controller_sites.SyncControllerSites'> succeeded
Step <class 'sync_controller_sites.SyncControllerSites'> succeeded
INFO:xos.log:Step <class 'sync_slivers.SyncSlivers'> succeeded
Step <class 'sync_slivers.SyncSlivers'> succeeded
INFO:xos.log:Step <class 'sync_controller_users.SyncControllerUsers'> succeeded
Step <class 'sync_controller_users.SyncControllerUsers'> succeeded
ERROR:xos.log:sync step failed! BEG TRACEBACK
Traceback (most recent call last):
  File "/opt/xos/observer/syncstep.py", line 133, in call
    self.delete_record(o)
  File "/opt/xos/observer/steps/sync_controller_slices.py", line 73, in delete_record
    driver = OpenStackDriver().admin_driver(controller=controller_slice.controller.name)
  File "/opt/xos/openstack/driver.py", line 48, in admin_driver
    client = OpenStackClient(tenant=tenant, controller=controller, cacert=self.config.nova_ca_ssl_cert)
  File "/opt/xos/openstack/client.py", line 177, in __init__
    self.keystone = KeystoneClient(*args, **kwds)
  File "/opt/xos/openstack/client.py", line 70, in __init__
    Client.__init__(self, *args, **kwds)
  File "/opt/xos/openstack/client.py", line 42, in __init__
    self.url = controller.auth_url
AttributeError: 'unicode' object has no attribute 'auth_url'
ERROR:xos.log:sync step failed! END TRACEBACK
INFO:xos.log:Step <class 'sync_controller_slices.SyncControllerSlices'> succeeded
Step <class 'sync_controller_slices.SyncControllerSlices'> succeeded
INFO:xos.log:Step <class 'sync_network_slivers.SyncNetworkSlivers'> succeeded
Step <class 'sync_network_slivers.SyncNetworkSlivers'> succeeded
INFO:xos.log:Step <class 'sync_network_slivers.SyncNetworkSlivers'> is a leaf
INFO:xos.log:Waiting for event

/stats endpoint is either returning Internal Server Error or showing traces for non-existed controllers (lie princeton-beta)

[12/Feb/2015 13:49:54] "GET /observer HTTP/1.1" 200 134
Internal Server Error: /stats/
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/xos/core/views/stats.py", line 12, in Stats
    controller = Controller.objects.filter(name=controller_name)[0]
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 177, in __getitem__
    return list(qs)[0]
IndexError: list index out of range
Internal Server Error: /stats/
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/xos/core/views/stats.py", line 12, in Stats
    controller = Controller.objects.filter(name=controller_name)[0]
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 177, in __getitem__
    return list(qs)[0]
IndexError: list index out of range
Internal Server Error: /stats/
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/xos/core/views/stats.py", line 12, in Stats
    controller = Controller.objects.filter(name=controller_name)[0]
  File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 177, in __getitem__
    return list(qs)[0]
IndexError: list index out of range
[12/Feb/2015 13:49:54] "GET /stats/?model_name=Sliver&pk=1&meter=network.incoming.bytes&controller_name=princeton-beta HTTP/1.1" 500 11860
[12/Feb/2015 13:49:54] "GET /stats/?model_name=Sliver&pk=1&meter=cpu&controller_name=princeton-beta HTTP/1.1" 500 11803
[12/Feb/2015 13:49:54] "GET /stats/?model_name=Sliver&pk=1&meter=network.outgoing.bytes&controller_name=princeton-beta HTTP/1.1" 500 11860
[12/Feb/2015 13:50:54] "GET /observer HTTP/1.1" 200 140
[12/Feb/2015 13:51:54] "GET /observer HTTP/1.1" 200 138

Spurious networks created

The Observer should only create a network in Neutron for private networks, not private-nat or public. Currently it creates a Neutron network for all types.

Sliver image ID is wrong

In the Ansible template for a sliver, the image_id is wrong. This should be the Glance image ID, but it is getting set to the ID of the image object in XOS.

Docker Hub image won't start

When the Docker Hub image (andybavier/xos) is run, it fails with the following:

could not access private key file "/etc/ssl/private/ssl-cert-snakeoil.key": Permission denied

I thought the Dockerfile accounted for this already, but clearly something is still amiss. Here's a link to a potential fix:

https://github.com/kartoza/docker-postgis/pull/10/files

When adding new controller, ControllerImage and ControllerNetwork are not created

Noticed this when the Singapore Controller was added -- existing images and networks did not acquire new ControllerImage and ControllerNetwork objects.

observer cannot handle image names with space in it

and raises following exception

Step <class 'sync_slivers.SyncSlivers'> succeeded
INFO:planetstack.log:Executing step SyncNetworkSlivers
Executing step SyncNetworkSlivers
INFO:planetstack.log:sync'ing network slivers
no previously-included directories found matching 'lib/ansible/modules/core/.git'
no previously-included directories found matching 'lib/ansible/modules/extras/.git'
ERROR: the playbook: /opt/opencloud/controller_images/Fedora could not be found

ERROR:planetstack.log:sync step failed! BEG TRACEBACK
Traceback (most recent call last):
  File "/root/xos/planetstack/observer/syncstep.py", line 123, in call
    self.sync_record(o)
  File "/opt/xos/observer/steps/sync_controller_images.py", line 37, in sync_record
    res = run_template('sync_controller_images.yaml', image_fields, path='controller_images', expected_num=1)
  File "/root/xos/planetstack/observer/ansible.py", line 89, in run_template
    raise Exception(error)
Exception
ERROR:planetstack.log:sync step failed! END TRACEBACK
INFO:planetstack.log:Step <class 'sync_controller_images.SyncControllerImages'> succeeded
Step <class 'sync_controller_images.SyncControllerImages'> succeeded
INFO:planetstack.log:Step <class 'sync_controller_images.SyncControllerImages'> is a leaf

element_name @ util/xml.py used-before-assignment

element_name is undefined...

110     def remove_elements(self, name):
111         """
112         Removes all occurences of an element from the tree. Start at
113         specified root_node if specified, otherwise start at tree's root.
114         """
115
116         if not element_name.startswith('//'):
117             element_name = '//' + element_name
118         elements = self.element.xpath('%s ' % name, namespaces=self.namespaces)
119         for element in elements:
120             parent = element.getparent()
121             parent.remove(element)

Clean up planetstack/plstackapi references

Clean up all planetstack and plstackapi references in source code. General housecleaning for release.

Exception: Error in creating instance: Duplicate networks are not allowed

observer throws following

INFO:xos.log:Step <class 'sync_controller_slice_privileges.SyncControllerSlicePrivileges'> succeeded
Step <class 'sync_controller_slice_privileges.SyncControllerSlicePrivileges'> succeeded
INFO:xos.log:Step <class 'sync_controller_slice_privileges.SyncControllerSlicePrivileges'> is a leaf
INFO:xos.log:sync'ing sliver:uninstantiated-1 slice:mysite_myslice controller:MyController OpenStack Juno

PLAY [127.0.0.1] **************************************************************

GATHERING FACTS ***************************************************************
ok: [127.0.0.1]

TASK: [nova_keypair ] *********************************************************
ok: [127.0.0.1] => {"changed": false, "result": "Key present"}

TASK: [nova_compute ] *********************************************************
failed: [127.0.0.1] => {"failed": true}
msg: Error in creating instance: Duplicate networks (8fe6b8b3-e04c-4ab6-9fea-5eaa28a7a21f) are not allowed

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
           to retry, use: --limit @/root/mysite_myslice-1.retry

127.0.0.1                  : ok=2    changed=0    unreachable=0    failed=1


ERROR:xos.log:sync step failed! BEG TRACEBACK
Traceback (most recent call last):
  File "/opt/xos/observer/syncstep.py", line 123, in call
    self.sync_record(o)
  File "/opt/xos/observer/steps/sync_slivers.py", line 125, in sync_record
    res = run_template('sync_slivers.yaml', tenant_fields,path='slivers', expected_num=2)
  File "/opt/xos/observer/ansible.py", line 89, in run_template
    raise Exception(error)
Exception: Error in creating instance: Duplicate net

and ansible playbook shows

root@cb70ab09e317:/opt/opencloud# cat /opt/opencloud/slivers/mysite_myslice-1

---
- hosts: 127.0.0.1
  connection: local
  tasks:

  - nova_keypair:
      state: present
      auth_url: http://172.25.148.240:5000/v2.0/
      login_username: [email protected]
      login_password: PASSWORD
      login_tenant_name: mysite_myslice
      name: padminATvicciorgmysite_myslice
      public_key: "ssh-rsa SSH_KEY"

  - nova_compute:
      auth_url: http://172.25.148.240:5000/v2.0/
      login_username: [email protected]
      login_password: PASSWORD
      login_tenant_name: mysite_myslice
      name: mysite_myslice-1

      state: present
      availability_zone: nova:a-rdo5.lcdn.corp.akalab.com
      image_id: 1c963e8d-aee7-4cad-b467-7d8c47bc9bcc
      key_name: padminATvicciorgmysite_myslice
      wait_for: 200
      flavor_id: 3
      user_data: "opencloud:\n   slicename: \"mysite_myslice\"\n   hostname: \"a-rdo5.lcdn.corp.akalab.com\"\n"
      nics:

          - net-id: 8fe6b8b3-e04c-4ab6-9fea-5eaa28a7a21f

          - net-id: 8fe6b8b3-e04c-4ab6-9fea-5eaa28a7a21f

Slivers fail to sync if any ControllerImage for the Sliver's Image fails to sync

For Example, Sliver onlab_hpc-195 on MPISWS failed to Sync because it was dependent on ControllerImage objects for Princeton and Arizona.

Deleting the ControllerImage objects for Princeton and Arizona temporarily resolved the issue.

Storm of cascading changes in model policies

When you dirty a sliver, all of the slivers in its slice seem to get dirtied and re-sync'd as a result. This is a bug, there's no reason for this to happen.

Propagate OpenCloud password changes to OpenStack

Ansible lacks the ability to change OpenStack passwords. TODO: Extend keystone_user ansible module with the ability to change passwords.

User mailing list

Harvest email addresses from OpenCloud and register them in the users mailing list.

Attribute Error

Attempting to use the REST API while not logged in returns "AttributeError: 'AnonymousUser' object has no attribute 'is_readonly'"

Dedicated Public IP Network

For slivers attached to a "Dedicated public IP" network, the IP address and SSH command shown on the Sliver Details page are both incorrect.

Interfaces/images binding

Automate the binding of interfaces to images.

Make instance-id and/or instance-name clickable in sliver list

When the IP address is empty, there's not much to click on in the sliver list, so make one or more of the other fields clickable.

Cleanup test cases

remove stuff in /tests that is obsolete, fix test cases that are broken

Services Button should bring up a list of Services

1. Remove HyperCache, RR, Syndicate from left-hand navigation

2. Add Services button to left-hand navigation

3. Service button brings up a list of all Services, regardless of whether they are subclassed or exist only as the base class.

4. Clicking on a service will bring up the service-specific view. For subclassed services like Hypercache, RR, or Syndicate, these are specialized views. For non-subclassed services they get a generic view.

Deployment Admin privilege

DeploymentPrivileges: Support Admin role (R/W access to all Deployment details).

Propagate write protect bit to dependencies

E.g. if slice onlab_onos is write protected, Site onlab should automatically be write protected.

Extend slice privileges

SlicePrivilege=Default should mean the user may ssh into the slice's slivers (not "no special privilege"). Rename "Default" to "Access" (or "User", although that's potentially confusing).

Images are synced to controllers that are part of deployments that the image is not enabled on

I think it would be best to only sync images to those controllers that relate to a deployment where the image has been enabled. Two reasons:

1. It would prevent syncing images to controllers where the image will never be used (i.e. there's no need to sync an image to Enterprise controllers if the image is only to be used on the Vicci deployment)

2. It would prevent the observer from syncing an image from /opt/xos/images before the administrator is ready. Right now the observer can and will sync an image while it is being uploaded.

Idea: tackle this the model_policy level instead of the observer level, prevent ControllerImage objects from being created unless "Controller.deployment in Image.deployments".

Observer Initscript Issues

1. The initscript doesn't work on Ubuntu.

2. How should the initscript support multiple Observers? (i.e.. Icehouse observer, Havana observer, and ec2 observer simultaneously)

One solution for #2 is to have a directory of config files, and launch an observer for each one of them.

Sliver flavor ID is wrong

In the Ansible template for a sliver, the flavor_id is wrong. This should be the flavor ID in Nova, but it is getting set to the ID of the flavor object in XOS.

On portal these two sets of IDs coincide so it happens to work. We shouldn't rely on this - it doesn't work if you bring up a new XOS (e.g., from the Docker image) or run XOS against an OpenStack installation that doesn't have the default flavors.

REST API throws validation error if enacted or policied is blank

Sliver created immediately after slice was missing private interface

When creating a slice and sliver via the Tenant UI, the sliver came up with its private interface missing. A sliver created a few minutes later came up fine with both interfaces.

I have not attempted to reproduce this yet.

'caller' field may not be being sent from REST API when instantiating new objects

This popped up when using the Tenant view to create a slice.

NameError: global name 'User' is not defined

INFO:xos.log:Step <class 'sync_controller_site_privileges.SyncControllerSitePrivileges'> is a leaf
INFO:xos.log:Step <class 'sync_controller_slice_privileges.SyncControllerSlicePrivileges'> is a leaf
ERROR:xos.log:sync step failed! BEG TRACEBACK
Traceback (most recent call last):
  File "/opt/xos/observer/syncstep.py", line 120, in call
    self.delete_record(o)
  File "/opt/xos/observer/steps/sync_controller_slices.py", line 71, in delete_record
    user = User.objects.get(id=controller_slice.slice.creator.id)
NameError: global name 'User' is not defined
INFO:xos.log:Step <class 'sync_controller_users.SyncControllerUsers'> succeeded
Step <class 'sync_controller_users.SyncControllerUsers'> succeeded
ERROR:xos.log:sync step failed! END TRACEBACK

global name 'Config' is not defined

2015-04-15 12:33:11,982 - ERROR - sync step failed! BEG TRACEBACK
Traceback (most recent call last):
File "/opt/xos/openstack_observer/syncstep.py", line 123, in call
self.sync_record(o)
File "/opt/xos/observer/steps/sync_controller_sites.py", line 30, in sync_record
res = run_template('sync_controller_sites.yaml', tenant_fields, path='controller_site
s', expected_num=1)
File "/opt/xos/observer/ansible.py", line 71, in run_template
if (Config().observer_steps):
NameError: global name 'Config' is not defined
2015-04-15 12:33:11,982 - ERROR - sync step failed! END TRACEBACK

Implement write protect bit

Ability to write protect certain objects

Implement second phase of deletions for non-sync'd objects

Objects that are sync'd get deleted when it's necessary. This is not true for non-sync'd objects such as Controllers.

Update user keys in running VMs

As new users are added to / removed from slices, it would be nice if the change is reflected in the authorized_keys file inside the VM as well as the user account used for proxy login. This functionality is outside of the scope of OpenStack's key injection feature, which is only triggered on VM creation.

Site Privileges

The full range of SitePrivileges is not implemented. Only Admin (currently named "PI") is supported. Should include Admin, Tech, and PI.

Object level dependencies not working for deletions

E.g. if deletion of Sliver princeton_sapan_5 fails, slice princeton_sapan may still get deleted, when it shouldn't because of a failed dependency.

Sync_network_slivers fails to add NetworkSliver for public-nat and public-dedicated

This is because the net_id for these networks is not stored anywhere in the data model, only the name is stored.

Duplicate ControllerNetwork objects created

I created a new slice using the Tenant View. Then I went to the developer view and created a sliver within that slice. The sliver failed to instantiate with an error message 'Error in creating instance: Network 3e59a10e-c872-414f-b9ea-c85925ee7269 is duplicated'.

Upon checking the ControllerNetwork objects for the -nat and -private networks associated with the slice, each network contained two objects per controller. Based on the numbering of the ControllerNetwork objects, it appeared that one full set of 8 objects (one per Controller) was created first, followed by a second full set of 8 objects.

CompositeKey support deployed to portal should prevent this data model corruption in the future, but the root cause of the duplicate objects was not determined.

Backend dependencies need reference links

Model dependencies are computed automatically. So finding links across those dependencies is easy, it is via the names of the fields on the basis of which the dependencies were computed.

With backend dependencies, we would have to tack on the names of the fields used to access them. E.g. "network.slices" for the ControllerNetwork->Slice dependency. Without this, the order of execution at the object level for such steps breaks.

REST API cleanup

• make xoslib and core use the same base classes [done]
• backend_icon stuff to one spot [done]
• use a consistent error-handling mechanism [to-do]