rhpit/manageiq-cli

Because we like managing the cloud via terminal!

A ManageIQ Command-Line Client

Documentation

The latest documentation is available on Read The Docs

manageiq-cli's Issues

Host the CLI documentation

Currently the project is using sphinx to document all of its documentation for the CLI. This issue is to determine and put in place how we plan to host the documentation. Some possible suggestions are:

• GitHub Pages
• Read the Docs

The nice take away with read the docs is it provides the ability to have versioned documentation. This way if users want to stick with a certain version of the CLI, they can follow the documentation that aligns with it. As far as I know, GitHub Pages does not provide this capability out of the box. I did find a sphinx versioning tool [1] that may be possible to provide this capability for GitHub Pages hosting the documentation.

[1] https://robpol86.github.io/sphinxcontrib-versioning/index.html#

Add an abort function to the log module

Currently the way we handle exiting the CLI is using the log.error(<message>, abort=True) function call.

Error logging should have one purpose and that is to provide the user with a detailed message when an error happens. The abort parameter was added in case the program wanted to exit right then. @seandst mentioned a comment in a previous PR that we should look into having a log.abort() function call to handle situations that need to exit right away when an error happens.

This issue is to add the new function, and revise the api module to use log.abort over log.error(abort=True) as needed. The function will be very simple to start but can be expanded over time.

ManageIQ/Cloudforms Authentication

From Tiago's comments on a previous issue, on how we should authenticate to the server:

We should have a token set in a temporary file to assist the command line to connect to the ManageIQ instance. But this is the responsibility of the CLI and not the user. The user has to provide only the username, password and the host wheren ManageIQ is installed.

The order for the token should be something like:

if a $HOME/.miqcli/auth file exist
validate token
if token doesn't work
create new token
update $HOME/.miqcli/auth
else
create $HOME/.miqcli/auth
create new token
update $HOME/.miqcli/auth
use token from $HOME/.miqcli/auth

Automate Method to obtain Cloud SubNetwork information for support of provisioning AWS

Currently through rest API no support for Cloud SubNetworks. Hence not supported in cli.
To provision an instance on AWS a cloud sub network is a required field. The field is supplied as an id through the api. Create and automate method that will return information for cloud sub networks.

Input: Cloud Network by Name or ID
Cloud SubNetwork by Name or ID
Returns: List of Cloud SubNetworks, Name, id, cidr and network_provider
All input is optional. No input given list of all Cloud SubNetworks returned.

Attempting to load an empty configuration file throws an exception

The first item the CLI tackles is handling reading configuration data that the user can set. User can set this data either in a file format, environment variable or as CLI global parameter options. I was testing the Config class which handles these actions and found when I have the directory /etc/miqcli created and an a empty file miqcli.yml created. It fails when trying to parse the file when it is empty.

Need to handle the use case where the file may be empty and we should ignore trying to parse it.

(miq-py36) [rywillia@laptop ~]$ tree /etc/miqcli/
/etc/miqcli/
└── miqcli.yml

0 directories, 1 file

(miq-py36) [rywillia@laptop ~]$ stat /etc/miqcli/miqcli.yml 
  File: /etc/miqcli/miqcli.yml
  Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
Device: fd01h/64769d	Inode: 1716768     Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/rywillia)   Gid: ( 1000/rywillia)
Context: unconfined_u:object_r:etc_t:s0
Access: 2018-01-11 10:56:06.944662412 -0500
Modify: 2018-01-11 10:55:58.056608016 -0500
Change: 2018-01-11 10:55:58.056608016 -0500
 Birth: -

(miq-py36) [rywillia@laptop ~]$ miqcli 
Traceback (most recent call last):
  File "/home/rywillia/.virtualenvs/miq-py36/bin/miqcli", line 6, in <module>
    from miqcli.cli.main import cli
  File "/home/rywillia/github/manageiq-cli/miqcli/cli/main.py", line 224, in <module>
    silent=True)
  File "/home/rywillia/github/manageiq-cli/miqcli/utils/__init__.py", line 74, in from_yaml
    for key in config_data:
TypeError: 'NoneType' object is not iterable

Define how we will connect to ManageIQ

In order for issue #23 to be started, we need to determine where we will establish the connection to the ManageIQ server defined by the user. Before the new year I started working on a prototype on how we could incorporate the ManageIQ API Client library but came across a problem. We can't establish the connection within the constructor for each collection class. If this is done, every time you run the CLI it would attempt to establish a connection to the server. This would potentially cause a DOS attack against your ManageIQ server. I.e. users install the client and run the client over and over to view the commands. It would be attempting to connect each time.

# connection would be attempted when running this command
$ miqcli <command>

I found that we can override the parent invoke method for the CLI sub-command class. This could allow us to perform the initial connection to ManageIQ (calling a method from a API class). Having this method available to be run at any point within the CLI is good because we may need to establish a connection again if a given token has expired.

I.e. When we are within the sub collections class we already know the command being run and have the command object. We can easily call a method to connect to the server.

class SubCollections(click.MultiCommand):

    def invoke(self, ctx):
        """Invoke the sub-command selected.

        Runs the sub-command given with all its arguments. Before it invokes
        the sub-command action to run, it will attempt to establish a
        connection to ManageIQ.

        :param ctx: Click context.
        :type ctx: Namespace
        """
        self.collection.connect()
        super(SubCollections, self).invoke(ctx)

Install manageiq-client from PyPI

Manageiq-client package has a new version on PyPI. This version contains the latest code changes we need for the client to connect to ManageIQ server by token authentication. Need to update the requirements.txt to remove installing from source.

Provider Support for OpenStack

Have CLI support for the OpenStack provider.
Support to create, teardown, add networking capabilities, query the resource (vm)

Improve how we return errors back to the user

Now that the CLI is progressing, we need to make sure we handle errors properly. These errors need to be handled properly and we should not just throw a stack trace back to the user. This is not user friendly and the user should not need to sift through the stack trace in order to determine what went wrong.

We currently have a couple areas in the source code that raise exceptions if user error happened or run time failure. This issue is to create a solution that will provide the user with a helpful explanation as to what went wrong. If users wish to receive a stack trace from the error received, we should provide a verbose or debug mode. When this is enabled it will allow users to gain more information at run time.

I think it will be good to define this standard and implement earlier on when the CLI is taking shape. This way it can be used by future commits.

Validate resources to be provisioned

Validate all the input for the resource that is to be provisioned:

• image, size, etc...

Define and create the documentation structure

The current documentation structure is the default one sphinx creates when setting up a new project. Before we start diving into adding documentation about each command and its sub-commands, etc. We need to come up with the structure on where content will be placed.

This issue is to create the structure for the documentation.

Possible sections:

• Introduction
• Installation
• Quickstart
• Commands

We need the Config class to be in the list

We need to add Config into the all variable if we later want to use import * from miqcli.utils

See line below the missing 'Config from the all_ variable.

manageiq-cli/miqcli/utils/__init__.py

Line 24 in 691775d

__all__ = ['get_class_methods']

Provide a provider class that can be usable for all collections

Collections interfacing directly with a cloud provider will require specific ID's for key:values in a given payload.

I.e: Provision request needs the ID for a template "image" within the payload over the string value.

This means collections will need to do look ups to get the ID for the given element a provider needs. Currently between automation_requests and provision_requests collection classes, they both require these look ups. This opens the door for potential code duplication between collections to get a specific IDs.

We should have a common provider class that is generic enough to work with any cloud provider to provide the ID back for the given key string value.

Query for overall status for a provisioning or teardown request

support querying the status of a provisioning or teardown request:

• Provisioning
• Provisioned
• Deleting
• Deleted

Automate methods created for project unable to handle multiples for fields given by name.

Some of the automate methods created for project (get_floating_ip, release_floating_ip allocate_floating_ip and retire_floating_ip) allow input by name. Name's are not unique. These methods need to be updated to handle situations where the input is given by name and multiple entries exist with the name. Some may require additional input fields. All require proper error handling if method can not narrow the action to just one target.

Id's are unique and the methods work fine for ids.

Provider Support for OpenShift

Have CLI support for the OpenShift provider.
Support to create, teardown, add routes & query the resource (pod)

Provider support for beaker

To provide support for provision requests in a beaker environment. The CLI should be able to perform actions that existing cloud providers can do within the beaker provider. These actions consist of the following:

1. Provision
2. Teardown
3. Query resource details (hostname, IP address)

Provider Support for AWS

Have CLI support for the AWS provider.
Support to create, teardown, add networking capabilities, query the resource (vm)

Teardown a Resource

support for an option to teardown a resource

CLI raises exception when an invalid command is given

Providing an invalid command/sub-command causes the CLI to raise an exception. Exception is raised due to the program is unable to locate the Python module for the given command.

i.e.

(miq-py36) [rywillia@laptop ~]$ miqcli  xyz
Traceback (most recent call last):
  File "/home/rywillia/.virtualenvs/miq-py36/bin/miqcli", line 10, in <module>
    sys.exit(cli())
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/rywillia/github/manageiq-cli/miqcli/cli/main.py", line 132, in invoke
    super(ManageIQ, self).invoke(ctx)
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/click/core.py", line 1061, in invoke
    cmd_name, cmd, args = self.resolve_command(ctx, args)
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/click/core.py", line 1100, in resolve_command
    cmd = self.get_command(ctx, cmd_name)
  File "/home/rywillia/github/manageiq-cli/miqcli/cli/main.py", line 86, in get_command
    miq_module = import_module(COLLECTIONS_PACKAGE + '.' + name)
  File "/home/rywillia/.virtualenvs/miq-py36/lib64/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'miqcli.collections.xyz'

(miq-py27) [rywillia@laptop ~]$ miqcli provision_requests xy
WARNING: Default configuration is used.
Traceback (most recent call last):
  File "/home/rywillia/.virtualenvs/miq-py27/bin/miqcli", line 10, in <module>
    sys.exit(cli())
  File "/home/rywillia/.virtualenvs/miq-py27/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/rywillia/.virtualenvs/miq-py27/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/rywillia/github/manageiq-cli/miqcli/cli/main.py", line 135, in invoke
    super(ManageIQ, self).invoke(ctx)
  File "/home/rywillia/.virtualenvs/miq-py27/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/rywillia/github/manageiq-cli/miqcli/cli/main.py", line 274, in invoke
    super(SubCollections, self).invoke(ctx)
  File "/home/rywillia/.virtualenvs/miq-py27/lib/python2.7/site-packages/click/core.py", line 1061, in invoke
    cmd_name, cmd, args = self.resolve_command(ctx, args)
  File "/home/rywillia/.virtualenvs/miq-py27/lib/python2.7/site-packages/click/core.py", line 1100, in resolve_command
    cmd = self.get_command(ctx, cmd_name)
  File "/home/rywillia/github/manageiq-cli/miqcli/cli/main.py", line 203, in get_command
    method = getattr(collection, name)
AttributeError: 'Collections' object has no attribute 'xy'

We should handle the exception and provide a message back stating the command is invalid.

Retrieve Client Configuration for ManageIQ Authentication

Provide a way for the client to retrieve client configuration defined by the user.
(Need ManageIQ host and credentials -> token or username/password, enabling ssl, etc...)

Do not attempt API connection when help parameter is given

Issue #34 brought the capability to have the CLI create an API connection to the declared ManageIQ server. This connection would only be created when the CLI goes to invoke "run" the commands sub-command.

$ miqcli providers create --option1 value1 --option2 value2 etc.

That is when we want the connection to be created when the sub-command is declared and its corresponding parameters needed to be invoked. That means all parameters including --help would allow the CLI to connect to the server. The help parameter is different than all other parameters. It is there to show all available parameters for the given sub-command and exit. It should not try to connect to the ManageIQ server.

Here is an example where the default MIQ configuration settings are defined but no ManageIQ server is running locally and the help parameter is given.

(miq-py36) [rywillia@laptop ~]$ miqcli providers create --help
Traceback (most recent call last):
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
    raise err
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f797992da20>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=8443): Max retries exceeded with url: /api (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f797992da20>: Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rywillia/github/manageiq-cli/miqcli/api.py", line 75, in connect
    verify_ssl=settings['enable_ssl_verify']
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/manageiq_client/api.py", line 44, in __init__
    self._load_data()
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/manageiq_client/api.py", line 47, in _load_data
    data = self.get(self._entry_point)
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/manageiq_client/api.py", line 86, in get
    partial(self._session.get, url, params=get_params))
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/manageiq_client/api.py", line 81, in _sending_request
    raise last_connection_exception
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/manageiq_client/api.py", line 77, in _sending_request
    return func()
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/requests/adapters.py", line 508, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='localhost', port=8443): Max retries exceeded with url: /api (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f797992da20>: Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rywillia/.virtualenvs/miq-py36/bin/miqcli", line 10, in <module>
    sys.exit(cli())
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/rywillia/github/manageiq-cli/miqcli/cli/main.py", line 123, in invoke
    super(ManageIQ, self).invoke(ctx)
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/rywillia/github/manageiq-cli/miqcli/cli/main.py", line 224, in invoke
    self.collection.connect()
  File "/home/rywillia/github/manageiq-cli/miqcli/collections/__init__.py", line 46, in connect
    super(Collection, self).connect(self.settings)
  File "/home/rywillia/github/manageiq-cli/miqcli/api.py", line 78, in connect
    raise RuntimeError(ex)
RuntimeError: HTTPSConnectionPool(host='localhost', port=8443): Max retries exceeded with url: /api (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f797992da20>: Failed to establish a new connection: [Errno 111] Connection refused',))

To fix this bug, the CLI code needs to be updated to not connect to the server when the help parameter is declared.

Networking Capabilities

• Create floating IP addresses
• Delete floating IP addresses
• Attach floating IP address to a cloud providers instance
• Detach floating IP address from a cloud providers instance

Investigation into Beaker Implementation

Investigation into how the client will support Beaker as a provider.

Unable to log debug level messages within any sub-command

Issue #44 brought the ability to log messages to the console. Debug level messages are only logged when the root option --verbose is enabled. Part of the debug function, I use click's global built-ins to get the current context. From here I use this to get the verbose value to determine if debug logging should happen. Well this works great for any root command. I.e:

$ miqcli

When you want to log messages at the sub-command layer it fails with the following exception. I.e.

# I mocked the providers create method to log a debug message

$ miqcli --verbose providers create
(miq-py36) [rywillia@laptop miqcli]$ miqcli --verbose providers create
Traceback (most recent call last):
  File "/home/rywillia/.virtualenvs/miq-py36/bin/miqcli", line 10, in <module>
    sys.exit(cli())
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/rywillia/github/manageiq-cli/miqcli/cli/main.py", line 119, in invoke
    super(ManageIQ, self).invoke(ctx)
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/rywillia/github/manageiq-cli/miqcli/cli/main.py", line 217, in invoke
    super(SubCollections, self).invoke(ctx)
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/rywillia/.virtualenvs/miq-py36/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/rywillia/github/manageiq-cli/miqcli/cli/main.py", line 169, in func
    return method(*args, **kwargs)
  File "/home/rywillia/github/manageiq-cli/miqcli/collections/providers.py", line 30, in create
    log.debug('Lets begin with creating a new provider.')
  File "/home/rywillia/github/manageiq-cli/miqcli/utils/log.py", line 55, in debug
    if click.get_current_context().parent.params['verbose']:
KeyError: 'verbose'

When the code gets run to log the debug message, it uses click.get_current_context to lookup the verbose value. The bug here is that the context it returns is for the sub-command context not the root command context. The root command is the one that allows you to set verbose parameter and saves its setting within the root context.

To fix this bug and allow the collection classes to log messages, we need to get the root context in order to determine if verbose mode was enabled or not. That will then allow the debug level messages to be logged.

Automate script to create & attach floating IP addresses to Openstack instances

Currently the MIQ REST API does not provide a collection with actions to create floating IP addresses and attach the IP to instances.

In order to provide networking capabilities #16 we need to leverage MIQ automate feature. We need to create an automate script (Ruby language) to create/attach IP to instances. Which can be called via the REST API automation_requests collection.

TBD

• Parameter input

Improve AUTHDIR to work on multiple platforms

The AUTHDIR is used in combination with the environment variable HOME, which is not set in all platforms. In order to be compatible with other platforms, we should use the function os.path.expanduser. Read more about expanduser here

Improvement for the ClientAPI constructor

I know we have discussed this before, but after exploring further some of the components of the client, I noticed that we are using settings object by reference, and if at any time we change the settings that will reflect in instability of the ClientAPI class.

Any line that uses settings within this class can be affected by a an external change to the given settings (if we don't generate a new dict, it will be float around as a reference).

Another point is the amount of times that the same statement is repeated in a single function:

manageiq-cli/miqcli/api.py

Line 87 in 691775d

succ_auth, exception = self.miq_auth(settings["token"])

manageiq-cli/miqcli/api.py

Line 79 in 691775d

succ_auth, exception = self.miq_auth(settings["token"])

manageiq-cli/miqcli/api.py

Line 95 in 691775d

succ_auth, exception = self.miq_auth(settings["token"])

manageiq-cli/miqcli/api.py

Line 109 in 691775d

succ_auth, exception = self.miq_auth(settings["token"])

With a better constructor we can create smaller functions that do one thing only and do it well instead of later we suffer with maintenance of the big connect() function.

With that in mind, I will propose a change to the ClientAPI class. Once the PR is sent, I would like to hear your opinion on that.

tox fails to build sphinx documentation

While working on issue #3 to enable a CI job for each pull request. I noticed the tox command would fail when it comes to building the sphinx documentation. The doc generator fails because the static folder does not exist. Console output below:

Running Sphinx v1.6.5
loading pickled environment... failed: unsupported pickle protocol: 4
building [mo]: all of 0 po files
building [html]: all source files
updating environment: 1 added, 0 changed, 0 removed
reading sources... [100%] index

looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index

generating indices... genindex
writing additional pages... search
copying static files... 
Warning, treated as error:
html_static_path entry u'/home/rywillia/github/manageiq-cli/doc/source/static' does not exist
ERROR: InvocationError: '/home/rywillia/github/manageiq-cli/.tox/docs/bin/python setup.py build_sphinx'
________________________________________________ summary ________________________________________________
  py36: commands succeeded
  py27: commands succeeded
  pep8: commands succeeded
ERROR:   docs: commands failed

There are two ways we can resolve this:

1. Commit the static folder with a empty place holder .gitkeep.

2. We can modify the docs conf.py module to remove the static folder from the html_static_path variable.

 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['static']
+html_static_path = []

I would suggest suggestion 1: to add an empty place holder. Reason being is when the documentation grows, we probably will have static files that will need to be included during the doc build process. This way then we do not need to edit the conf.py adding back the static folder to the path. @tvieira what are your thoughts on this?

Create the contributing guidelines

Create the CONTRIBUTING doc at the root of the project. This will define all the guidelines that existing and new contributors need to follow by when submitting pull requests.

https://help.github.com/articles/setting-guidelines-for-repository-contributors/

Query details for provisioned resource(s)

The CLI should provide the ability to query information related to provisioned resources. Resources will provide different information regarding the provider it was provisioned into.

I.e. query the host name for an instance or the public IP address to access the instance via SSH.

Consolidate README files for automate scripts into one location

PR #53 brought automate scripts with methods available to perform networking capabilities for Openstack provider. There is now a dedicated folder under the root of the manageiq-cli project (Automate-Datastore) which contains all the files and a README.md. It also added a AUTOMATE-DATASTORE.rst at the root of the project explaining how to use the automate scripts.

I think it would be best if we could combine these two files or move AUTOMATE-DATASTORE.rst to the Automate-Datastore folder. This way everything related to automate scripts is in one location. What does everyone think?

Incorporate the MIQ API Client Python library as the core to the CLI calls

We need to leverage the following project [1] to be used as the core underlying component to the CLI. This library provides access to MIQ REST API collections via Python. Our collections modules can use this library to perform the actions interfacing with MIQ.

This issue is to implement the solution on how we can incorporate this library into all collections. Maybe it is included as a mixin class to all collections classes, a utility module, etc.

[1] https://github.com/ManageIQ/manageiq-api-client-python

Unit tests to validate the command line structure

The command line structure design mirrors the available ManageIQ REST API collections. The command line has commands collections which have sub-commands collection actions. @tvieira brought up a nice point of how can we validate these commands. How can we know if our commands are compatible with the REST API collections. The goal here is to validate all our commands are valid commands on ManageIQ.

I.e. A future release of ManageIQ could depreciate certain collections, add collections, change collection name or even add new actions to collections. How would we easily know about this: users report bugs, we see it in release notes, etc. It would be nice to have an automated way that we could tell if our command line structure is valid or not.

The idea here is to create tests to verify all the command line commands are valid commands with ManageIQ. This could be an integration test requiring ManageIQ server to be running and fetch the collections/actions to compare.

For release 1.0 the command line structure is inline with the current collections [1] provided by ManageIQ.

[1] http://manageiq.org/docs/reference/latest/api/reference/collections

Automate Method to obtain information for VM's

Provision request returns very limited data. Need more Information to pass back to the user.
Need methods to return information for VM's and maybe floating IPs.
A list method where you can request all or narrow the request by input or even just ask for one.

Create List Methods for Floating ips and VM's.

Enable a bot to notify when new PRs are received

It would be nice to enable a bot listening as a web hook for new pull requests. This bot could notify us via IRC (similar to how Fedora Infra) has for their PRs. This would remove the need to check email for
reviewing PRs. Part of this made me think that it would also be nice addition to have a bot be able to auto recommend reviewers to a PR. I found the following bot that has an algorithm that will review modules modified and suggest reviewers based on authors for that given module.

https://github.com/facebook/mention-bot

This is not a critical issue. I just wanted to create an issue capturing the thought.

Provision a Resource

support for an option to provision a resource

Provide a class to use the client in python scripts

Currently you are able to use the client from command line.

$ miqcli <global-options> <command> <sub-command> <options>

This is great since it provides an easy way to interface with MIQ servers. To expand on this, we need to provide the ability to use the client as library import in Python modules. Right now the client is structured and built based on click library with each collection module accessing data from click's context.

Goal here is to have a unified class which you can import and instantiate to call MIQ collections/actions just like you would from the command line. If we did not provide a class like this, users would have to do something like the following:

from miqcli.api import ClientAPI
from miqcli.collections import provision_requests

# create client api
client_api = ClientAPI(<parameters>)

# build a click context for collections to use to access various attributes
ctx = Context(<paramerers>)

# create collection object
collection = provsion_requests.Collections()

and so on..

With the unified class, users would only have to do the following in their code:

from miqcli.api import ManageIQClient

miq = ManageIQClient(<manageiq_settings>)
collection = miq.get_collection('provision_requests')
collection.create(<parameters>)
collection.query(<paramerers>)

CLI raises an exception before it can generate the available commands

After pulling the latest changes, I got a problem when trying to run the CLI to view the available commands outside my local copy of the project. I now get an exception.

Steps to reproduce the problem:

# 1. create virtual environment 
$ virtualenv miqcli && source miqcli/bin/activate
(miqcli) [rywillia@laptop manageiq-cli]$

# 2. Install the CLI via pip
(miqcli) [rywillia@laptop manageiq-cli]$ pip install -e .

# 3. Change directory to a different one other than the source code
(miqcli) [rywillia@laptop manageiq-cli]$ cd ~

# 4 Run the CLI
(miqcli) [rywillia@laptop ~]$ miqcli
Traceback (most recent call last):
  File "/home/rywillia/miqcli/bin/miqcli", line 6, in <module>
    from miqcli.cli.main import cli
  File "/home/rywillia/github/manageiq-cli/miqcli/cli/main.py", line 26, in <module>
    from miqcli.api import ClientAPI
  File "/home/rywillia/github/manageiq-cli/miqcli/api.py", line 26, in <module>
    from miqcli.constants import AUTHDIR, DEFAULT_CONFIG
  File "/home/rywillia/github/manageiq-cli/miqcli/constants.py", line 29, in <module>
    VERSION = get_version(PACKAGE)
  File "/home/rywillia/miqcli/lib/python2.7/site-packages/pbr/packaging.py", line 755, in get_version
    name=package_name))
Exception: Versioning for this project requires either an sdist tarball, or access to an upstream git repository. It's also possible that there is a mismatch between the package name in setup.cfg and the argument given to pbr.version.VersionInfo. Project name miqcli was given, but was not able to be found.

If I switch back to the source code directory everything works fine.

(miqcli) [rywillia@laptop manageiq-cli]$ miqcli
Usage: miqcli [OPTIONS] COMMAND [ARGS]...

  ManageIQ command line interface.

Options:
  --version                       Show version and exit.
<Truncated>

Looks like pbr is unable to get the version based on the order they check for when outside a local copy of the git repository.

Automate script to delete & detach floating IP addresses from Openstack instances

Currently the MIQ REST API does not provide a collection with actions to delete floating IP addresses and detach the IP from instances.

In order to provide networking capabilities #16 we need to leverage MIQ automate feature. We need to create an automate script (Ruby language) to delete/detach IP to instances. Which can be called via the REST API automation_requests collection.

TBD

• Parameter input

Return data for get_floating_ip needs to be updated

When testing the Automate code for get_floating_ip, a floating ip was successfully generated.
However, I was unable to do anything with it as the return data did not include the id of that floating ip, it just looks like some guid:

"return":{"10.8.248.62":"8e8316c0-8d9e-456a-a579-a2d891c97e25"}

Remove raising exceptions in the client api class

Currently our ClientAPI class raises RuntimeError's on error. The CLI now has a dedicated logging module which uses click's built-in logging capabilities. This issue is to switch from raising exceptions to use the logging functions to log the message to the console. This will eliminate the stack trace logged and still will exit the program correctly.

Display a message when default configuration is used

Right now when running the CLI, you do not know if you are using the default configuration or not. To make this clear to the user, it would be a nice addition to log a message stating if you are using the default configuration.

Add Travis-CI badge in the readme file

It is good to have Travis-CI badge in the README.rst for master branch so we can check the status of the build right after getting to the Github page.

Document AWS subnet workaround for provisioning AWS instances via ManageIQ REST API

We have to create Ruby scripts to automate the creation of AWS subnet just like we did for creating Floating IPs at Openstack (#27). We want to have this documented to assist us when we deploy a stage instance of ManageIQ for testing purpose.

Create and Delete a Provider

support options to create and delete a provider

We need automation to run local tests

We have created a tox file to run tests in Python 3.6 + 2.7 + PEP8 standards check, but now we need an instance of ManageIQ running locally so we can run tests locally prior to create a new PR for the feature/fix.

There are few options that I have been testing:

1. Vagrant + ManageIQ image (http://manageiq.org/docs/get-started/vagrant)
2. Docker + ManageIQ image (http://manageiq.org/docs/get-started/docker)
3. Local machine installation

The 3rd option I would discard, I don't want to install and maintain a local installation of ManageIQ and I don't think anyone would want that.

The 1st and 2nd options are a lot easier to manage. I would adopt docker because when using Vagrant we still have to have Virtualbox. Some of us will have only libvirt, or VMWare and there's no official image available for these providers.

I would vote for Docker as, at least the core developers for this project, we all have the technology locally to run containers.

Is there anyone that would prefer another option other than using the docker image created by ManageIQ? Otherwise I already have some work done with the docker image.

Improvement to the Config class

While working on #50, I realized the existing logging messages will never be executed. Each Config class method accepts a silent parameter. This parameter will return out silencing any failures that may occur at run time. This is may be troubling to the user if they are trying to load configuration from a file but the file does not have the correct file extension. (This is just an example).

Now that we have a verbose option the user can define via the CLI. We should use this value to determine if messages in the Config methods should be silenced or not. This will require re-ordering when the Configuration is loaded by the CLI.

I will propose a change to when we load configuration when the CLI is run. Once the PR is submitted, please provide your feedback.

Cancel an Existing Provisioning Request

support for this option

Enable Travis CI

We should enable a Travis CI job to be run for any pull request to the project. This job would verify the code changes in the pull request meet our flake8 standards and verify all unit tests continue to pass. The job could easily run tox command to verify the latest changes for all supported Python versions.

https://docs.travis-ci.com/user/getting-started