bsdci/ioc

libioc command line tool for BSD jail management

ioc's Issues

ModuleNotFoundError: No module named 'ioc_cli.oops_typo'

When calling ioc with a CLI module that doesn't exist, we should catch ModuleNotFoundError and instead return a sensible error message, and our usage text.

ioc clone is broken

on the most current master (599fb6b) the ioc clone command fails with the following output:

# ioc clone jail1 jail2
No jail matching 'jail2' was found
Traceback (most recent call last):
  File "/usr/local/bin/ioc", line 10, in <module>
    sys.dd:exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/ioc_cli/clone.py", line 74, in cli
    host=ctx.parent.host
  File "/usr/local/lib/python3.6/site-packages/libioc/Jail.py", line 339, in __init__
    data["id"] = self._resolve_name(data["id"])
  File "/usr/local/lib/python3.6/site-packages/libioc/Jail.py", line 2065, in _resolve_name
    raise libioc.errors.JailNotFound(text, logger=self.logger)
libioc.errors.JailNotFound: jail2

ioc is in FreeBSD ports now, but misses jail module

Hi,

ioc is now in FreeBSD ports, so partly the ioc website is wrong stating that it is not there, but partly right, as it is not working.

Test with iocage jails running but iocage being deinstalled and right after ioc was installed:

ioc list
Traceback (most recent call last):
File "/usr/local/bin/ioc", line 13, in
from ioc_cli import cli
File "/usr/local/lib/ioc/ioc_cli/init.py", line 43, in
from libioc.Datasets import Datasets
File "/usr/local/lib/python3.6/site-packages/libioc/Datasets.py", line 31, in
import libioc.helpers
File "/usr/local/lib/python3.6/site-packages/libioc/helpers.py", line 37, in
import jail as libjail
ModuleNotFoundError: No module named 'jail'

So this is a bug report that there is a py-jail missing in the FreeBSD ports.

Bye,
Alexander.

ioc destroy error when destroying a release

When destroying a Release, the destroy CLI commands fails passing the skip_invalid_config argument to the Release Resource.

# ioc destroy -r 12.0-RELEASE
Traceback (most recent call last):
  File "/usr/local/bin/ioc", line 10, in <module>
    sys.dd:exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/ioc_cli/destroy.py", line 103, in cli
    skip_invalid_config=True

ioc clone fails due to non-existant jail

The ioc clone command is broken because it cannot find the target jail:

# ioc clone ros my-jail
No jail matching 'my-jail' was found
Traceback (most recent call last):
  File "/usr/local/bin/ioc", line 10, in <module>
    sys.dd:exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/ioc_cli/clone.py", line 74, in cli
    host=ctx.parent.host
  File "/usr/local/lib/python3.6/site-packages/libioc/Jail.py", line 355, in __init__
    data["id"] = self._resolve_name(data["id"])
  File "/usr/local/lib/python3.6/site-packages/libioc/Jail.py", line 2194, in _resolve_name
    raise libioc.errors.JailNotFound(text, logger=self.logger)
libioc.errors.JailNotFound: my-jail

Suggest to automatically configure Secure VNET firewall

The Secure VNET feature requires IPFW to be running and configured to filter ethernet bridges. Manual steps for a minimal configuration are:

sysrc firewall_type=open
sysrc firewall_enable=YES
service ipfw start
sysctl net.link.ether.ipfw=1
sysctl net.link.bridge.ipfw=1
# permanently enable sysctls
echo "net.link.ether.ipfw=1" >> /etc/sysctl.conf
echo "net.link.bridge.ipfw=1" >> /etc/sysctl.conf

see also https://bsdci.github.io/handbook/features/networking/

When the firewall is not properly configured, a dialog should ask the user to apply the above configuration.

ioc binary hard-codes python3.6

when installing ioc on FreeBSD-12.1-RELEASE, trying to run it will yield:

ioc: Command not found.
root@freebsd:~ # /usr/local/bin/ioc
/usr/local/bin/ioc: Command not found.
root@freebsd:~ # 

the reason is that the python version is hard-coded, and with 12.1 it's wrong:

#!/usr/local/bin/python3.6
import sys
import os.path

LIB_DIR = "/usr/local/lib/ioc"

if LIB_DIR.startswith("/") is False:
    __dirname = os.path.dirname(os.path.abspath(__file__))
    LIB_DIR = "/usr/local/lib/ioc"

# etc…

make (install) is missing python -m ensurepip

on a fresh system, python3.6 -m pip will fail because it doesn't have pip yet.
We should first run python3.6 -m ensurepip

ioc -d non-existent-log-level is taken, as-is.

When calling ioc -d with an inexistent log level, we should error out, printing a list of existing log levels.

ioc list -o unkown_property fails with a stacktrace in addition to its error message

since after the config refactor, bsdci/libioc#602 ioc list -o unknown_prop will now fail with a big long stack trace:

root@container-host1 ~# ioc list -o unknown_prop
The config property 'unknown_prop' is unknown
Traceback (most recent call last):
  File "/usr/local/bin/ioc", line 10, in <module>
    sys.dd:exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/src/ioc/ioc_cli/list.py", line 159, in cli
    _print_table(resources, columns, header, _sort)
  File "/usr/local/src/ioc/ioc_cli/list.py", line 178, in _print_table
    table_data.append(_lookup_resource_values(resource, columns))
  File "/usr/local/src/ioc/ioc_cli/list.py", line 245, in _lookup_resource_values
    columns
  File "/usr/local/src/ioc/ioc_cli/list.py", line 244, in <lambda>
    lambda column: str(_resource.getstring(column)),
  File "/usr/local/lib/python3.6/site-packages/ioc/Resource.py", line 356, in getstring
    value = self.get(key)
  File "/usr/local/lib/python3.6/site-packages/ioc/Jail.py", line 229, in get
    return self.jail.config[key]
  File "/usr/local/lib/python3.6/site-packages/ioc/Config/Jail/JailConfig.py", line 86, in __getitem__
    return super().__getitem__(key)
  File "/usr/local/lib/python3.6/site-packages/ioc/Config/Jail/BaseConfig.py", line 568, in __getitem__
    self._require_known_config_property(key)
  File "/usr/local/lib/python3.6/site-packages/ioc/Config/Jail/BaseConfig.py", line 773, in _require_known_config_property
    logger=self.logger
ioc.errors.UnknownConfigProperty

ioc cli parameters are very inconsistent

for reasons of backwards compatibility, we've made ioc's command line interface as inconsistent as its predecessors'.

currently we have three formats, which mostly differ where the <jail>, that is being operated on is placed, and how it's passed.

1. via the --name option:

   ioc create --name <jail>

2. as first (and only parameter)

   ioc start <jail>

3. as first parameter of infinitely many:

   ioc exec <jail> cat /etc/passwd

4. as last parameter of infinitely many:

   ioc set ip4_addr='vtnet0|172.16.0.9/12' template=no <jail>

i propose standardizing on one format.
if we want to provide backwards compatibility, we should do this shims that our users can install themselves in place of the iocace_legacy (/usr/local/sbin/iocage) or iocage (/usr/local/bin/iocage) binaries.

No jails started your input:

root@webserver ~# ioc provision webpirc
No jails started your input: 
root@webserver ~# ioc provision webirc
…

I think this should say: "No jails matched your input" or "No started jails matched your input" ?

Missing commands

I just pulled latest master and yesterday I spend half a day debugging what is going on. So the thing is that after make install I have ioc that is missing couple of commands

Usage: ioc [OPTIONS] COMMAND [ARGS]...

  A jail manager.

Options:
  --version             Show the version and exit.
  --source TEXT         Globally override the activated iocage dataset(s)
  -d, --log-level TEXT  Set the CLI log level ('critical', 'error', 'warn',
                        'info', 'notice', 'verbose', 'debug', 'spam',
                        'screen')
  --help                Show this message and exit.

Commands:
  activate    Set a zpool active for iocage usage.
  deactivate  Disable a ZFS pool for libioc.
  fstab       View and manipulate a jails fstab file.
  get         Gets the specified property.
  import      Import a jail from a backup archive
  migrate     Migrate jails to the latest format.
  pkg         Manage packages in a jail.
  promote     Clone and promote jails.
  provision   Trigger provisioning of jails.
  rename      Rename a stopped jail.
  restart     Restarts the specified jails.
  set         Sets the specified property.
  snapshot    Take and manage resource snapshots.
  start       Starts the specified jails or ALL.
  stop        Stops the specified jails or ALL.
  update      Update a jail to a new release or patchlevel.

I then checked out older versions and all of a sudden I got back all the commands, and even if I try to fetch latest master. Today I did pull latest master on new server and bam Again missing commands. Can you please help me understand what is going on here and what am I doing wrong.

change contributing guideline to adopt yapf (if possible)

Our contributing guidelines are still mostly copy-pasted from iocage.

I suggest we drop much of the words there dedicated to formatting, and use yapf — if that should be possible for us.

get of a nested config property should print a list of items

It would be nice to get a bunch of variables when querying a nested BaseConfig property with the ioc get CLI command. Printing multiple values is already supported (see --all), so this enhancement would allow to select an entire namespace in their output.

Nested sub-namespaces are expected to work in the same way.

Current result

$ ioc get provisioning. webirc
The config property 'provisioning.' is unknown
Traceback (most recent call last):
  File "/usr/local/bin/ioc", line 10, in <module>
    sys.dd:exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/src/ioc/ioc_cli/get.py", line 97, in cli
    value = lookup_method(source_resource, _prop)
  File "/usr/local/src/ioc/ioc_cli/get.py", line 131, in _lookup_jail_value
    value = resource.getstring(key)
  File "/usr/local/lib/python3.6/site-packages/libioc/Resource.py", line 356, in getstring
    value = self.get(key)
  File "/usr/local/lib/python3.6/site-packages/libioc/Jail.py", line 229, in get
    return self.jail.config[key]
  File "/usr/local/lib/python3.6/site-packages/libioc/Config/Jail/JailConfig.py", line 86, in __getitem__
    return super().__getitem__(key)
  File "/usr/local/lib/python3.6/site-packages/libioc/Config/Jail/BaseConfig.py", line 571, in __getitem__
    self._require_known_config_property(key)
  File "/usr/local/lib/python3.6/site-packages/libioc/Config/Jail/BaseConfig.py", line 806, in _require_known_config_property
    logger=self.logger
libioc.errors.UnknownConfigProperty
$ ioc get provisioning.source webirc
/usr/local/etc/puppet

ioc get non-existent-key defaults throws a traceback

ioc get non-existent-key defaults

ioc get running defaults
Traceback (most recent call last):
  File "/usr/local/bin/ioc", line 10, in <module>
    sys.dd:exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/src/libiocage/ioc/get.py", line 107, in cli
    value = lookup_method(source_resource, _prop)
  File "/usr/local/src/libiocage/ioc/get.py", line 130, in _lookup_config_value
    return str(iocage.helpers.to_string(resource.config[key]))
  File "/usr/local/src/libiocage/iocage/Config/Jail/BaseConfig.py", line 567, in __getitem__
    raise KeyError(f"Item not found: {key}")
KeyError: 'Item not found: running'

we know the error here, no need to throw a backtrace

Release missing for 0.7.0

We don't have a tag / release for 0.7.0

Non-root commands cannot be executed

$ ioc pkg myjail git-lite
Traceback (most recent call last):
  File "/usr/local/bin/ioc", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1061, in invoke
    cmd_name, cmd, args = self.resolve_command(ctx, args)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1100, in resolve_command
    cmd = self.get_command(ctx, cmd_name)
  File "/usr/local/lib/python3.6/site-packages/ioc_cli/__init__.py", line 156, in get_command
    if mod.__rootcmd__ and "--help" not in sys.argv[1:]:
AttributeError: module 'ioc_cli.pkg' has no attribute '__rootcmd__'

Port version info loading from libioc

in bsdci/libioc@d52d149 we load the version info from a file

we don't do that in this project yet

i also propose also creating a task for bumping the version and creating a tag

should `ioc get` support filters just like `ioc set`?

currently, ioc set supports filter to select jails, but ioc get does not

Is this a bug? or a specific omission?

how could we represent queries that return multiple jails?

ioc provision looking for an event that does not exist in libioc

root@webserver ~# ioc provision webproxy
[-] JailProvisioning@webproxy: ...
Traceback (most recent call last):
  File "/usr/local/bin/ioc", line 10, in <module>
    sys.dd:exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func                                                                                    
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/ioc_cli/provision.py", line 65, in cli
    **start_args
  File "/usr/local/lib/python3.6/site-packages/ioc_cli/provision.py", line 102, in _provision                                                                                
    print_function(_execute_provisioner(jail))
  File "/usr/local/lib/python3.6/site-packages/ioc_cli/__init__.py", line 91, in print_events                                                                                
    for event in generator:
  File "/usr/local/lib/python3.6/site-packages/ioc_cli/provision.py", line 125, in _execute_provisioner                                                                      
    if isinstance(event, libioc.events.JailCommandExecution):
AttributeError: module 'libioc.events' has no attribute 'JailCommandExecution'
root@webserver ~#

Where's the history?

There's tools in git to extract a subtree from a project into a new repo — while retaining the history.

Perhaps we should do that, rather than have a fake blank start.

Quit fetch command Release selection

It should be possible to leave the Release selection, that occurs when fetching releases, by entering q instead of any listed release identifier.

Current Result

Press [Enter] to fetch the default selection (12.0-RELEASE) [1]: q
Traceback (most recent call last):
  File "/usr/local/bin/ioc", line 10, in <module>
    sys.dd:exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/ioc_cli/fetch.py", line 102, in cli
    release = prompts.release()
  File "/usr/local/lib/python3.6/site-packages/libioc/Prompts.py", line 70, in release
    return available_releases[int(selection)]
ValueError: invalid literal for int() with base 10: 'q'

Output columns not working for lists of Releases

Output column names -o are ignored when combined with the -r (list releases) flag.

Current Result

$ ioc list -r -o full_name,dataset.name
+--------------+
|  FULL_NAME   |
+==============+
| 12.0-RELEASE |
+--------------+

Expected Result

$ ioc list -r -o full_name,dataset
+--------------+---------------------------------+
|  FULL_NAME   |  DATASET.NAME                   |
+==============+=================================+
| 12.0-RELEASE | zroot/ioc/releases/12.0-RELEASE |
+--------------+---------------------------------+