dmulyalin / salt-nornir Goto Github PK

View Code? Open in Web Editor NEW

26.0 4.0 3.0 2.9 MB

SALTSTACK Nornir based proxy minion, execution, state and runner modules

License: MIT License

Python 93.28% SaltStack 6.34% Jinja 0.16% Shell 0.03% RobotFramework 0.21%

network-automation network-engineers network-programming networking python

salt-nornir's People

Contributors

Stargazers

Watchers

Forkers

onemind-services-llc davama openssl-sg-insights

salt-nornir's Issues

import extended napalm drivers

Hello,

I am playing around with salt-nornir and it’s awesome! Ive been using plain nornir with my extended drivers for a few months now and wanted to try salt.

I understand that i can call napalm getters using the nornir_napalm…napalm_get task:

salt minion nr.task nornir_napalm.plugins.tasks.napalm_get getters='["get_facts"]' FG="procurve" FB="h-txa-*"

i have no issues with the above.

with plain nornir and having my extended drivers in my working directory under custom_napalm/driverbla.py i can easily call an extended function as so:

# where get_facts_custom is my extended function 
results = devices.run(task=napalm_get, getters=["get_facts", "get_facts_custom", "get_environment"])

Was trying to see if just by placing custon_napalm folder somewhere was enough for napalm to inherit it as shown here: https://github.com/napalm-automation/napalm/blob/d978c3db886f12d2b83938da63676adfcc16431b/napalm/base/__init__.py#L77

On the salt-master ive tried under/root/ (since salt is running as root), under /etc/salt (just a guess), and even tried /var/run/salt.

Is what im trying to do even possible?
Any suggestions?

I also have a custom nornir task we use, similar to nornir_napalm.plugins.tasks.napalm_get but one challenge at a time 😄

Any input is much appreciated!

Thank you for this amazing project!

Best,
Dave

Add support for InfluxDB2 for sending metrics

We use InfluxDB2 in our organization and would like this plugin to send metrics data to the database.

Docs: https://docs.influxdata.com/influxdb/v2.3/
API Guide: https://docs.influxdata.com/influxdb/v2.3/api-guide/
Python Library: https://docs.influxdata.com/influxdb/v2.3/api-guide/client-libraries/python/

Note: The module in the salt repo for InfluxDB is for v1 which has now been discontinued.

Unable to gracefully restart the VM

Today, I tried to reboot the VM using the command reboot and it is stuck waiting for salt-proxy for the last 5 hours.

I'm running two instances of proxy within this VM. One for managing the network devices and the other for HTTP interface for OOB servers access.

[FEATURE REQUEST] allow worker=x to select a particular nornir worker when using nr.test function

Worker=X isn't currently working with nr.test, it seems to use nr.cfg_gen and then uses a different worker from the selected which can mean that often the inventory is a different one that is expected and the previously updated keys are not available for the jinja2 to render the template.

Thanks,
Jon.

Jinja whitespace control ignored

Looks like whitespace control in Jinja templates is ignored.

[olielli@master ~]$ sudo salt nornirproxy nr.cfg_gen "{% for i in ['1', '2', '3'] -%}\n{{ i }}\n{% endfor -%}"
nornirproxy:
    ----------
    switch1:
        ----------
        salt_cfg_gen:

            1

            2

            3

These interim newlines shouldn’t be there, thats what the -%} statement should address.

Preferrably there should be a way to include Jinja config like lstrip_blocks=true and trim_blocks=true which is already configured on the master.

[FEATURE REQUEST] tighter structure with hcache data

When saving data to the Hcache from something like TTP parsing, the nesting includes "run_ttp" as a key.

Is there a way to have the dict it creates in the hcache without the run_ttp key to make it a little easier to navigate / more readable?

DOC - Add use case describing how to use hcache with nr.cfg function

Update documentation to include examples of sourcing data from hcache for nr.cfg and nr.cfg_gen and nr.nc functions.

Errors while running `nr.learn`

From the documentation's example, I ran salt-call nr.learn "show version" "show int brief" tf="cli_facts" fun="cli and was presented with the following error. This happens for all nr.learn commands.

Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    load_entry_point('salt==3004.1', 'console_scripts', 'salt-call')()
  File "/usr/lib/python3/dist-packages/salt/scripts.py", line 432, in salt_call
    client.run()
  File "/usr/lib/python3/dist-packages/salt/cli/call.py", line 55, in run
    caller.run()
  File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 111, in run
    ret = self.call()
  File "/usr/lib/python3/dist-packages/salt/cli/caller.py", line 218, in call
    ret["return"] = self.minion.executors[fname](
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/executors/direct_call.py", line 10, in execute
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/salt_nornir/modules/nornir_proxy_execution_module.py", line 2210, in learn
    return globals()[fun](*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/salt_nornir/modules/nornir_proxy_execution_module.py", line 1135, in cli
    default_kwargs = __proxy__["nornir.nr_data"]("nr_cli")
  File "/usr/lib/python3/dist-packages/salt/loader/context.py", line 78, in __getitem__
    return self.value()[item]
  File "/usr/lib/python3/dist-packages/salt/utils/context.py", line 221, in __getitem__
    return self._dict()[key]
KeyError: 'nornir.nr_data'

root@ubuntu:~# pip show nornir
Name: nornir
Version: 3.3.0
Summary: Pluggable multi-threaded framework with inventory management to help operate collections of devices
Home-page: https://github.com/nornir-automation/nornir
Author: David Barroso
Author-email: [email protected]
License: Apache 2.0
Location: /usr/local/lib/python3.8/dist-packages
Requires: typing-extensions, ruamel.yaml, mypy-extensions, importlib-metadata
Required-by: salt-nornir, nornir-utils, nornir-scrapli, nornir-salt, nornir-napalm
root@ubuntu:~#

PyATS Yang.connector and PyGNMI proto files issue

Atempted updating dependency - genie and pyats from 22.1 to 22.5 - getting gnmi proto files related errors, opened these github issues:

Unable to pass literal "\n" in config using nr.cfg or nr.cfg_gen

I am trying to render configuration to junos that contains literal newlines (used internally by Junos) but they are rendered as newlines no matter how I try to escape them.

Example of what I want to render to the switch:

system {
    login {
        message "test\nstring\n";
    }
}

I have been testing using nr.cfg_gen like this:

[olielli@saltmaster~]$ sudo salt nornirproxy nr.cfg_gen '{{ "test\nstring\n" }}' FB='ach1-fw-*' split_lines=False
phx1-nornirproxy-01:
    ----------
    ach1-fw-20:
        ----------
        salt_cfg_gen:
            test
            string

Escaping the newline renders the same way as above, ie '{{ "test\\nstring\\n" }}'

Escaping the escape renders an escaped backslash and a newline:

[olielli@saltmaster ~]$ sudo salt nornirproxy nr.cfg_gen '{{ "test\\\nstring\\\n" }}' FB='ach1-fw-*' split_lines=False
phx1-nornirproxy-01:
    ----------
    ach1-fw-20:
        ----------
        salt_cfg_gen:
            test\
            string\

I suspect some point in the salt -> nornir rendering pipeline the escapes are nullified.

DOC - fix reference to python requests library

In here - https://nornir-salt.readthedocs.io/en/latest/Connection%20Plugins.html#nornir_salt.plugins.connections.HTTPPlugin.HTTPPlugin - link toward python-requests library does not work- https://docs.python-requests.org/en/latest/api/ - need to fix it, presumably pointing to somewhere here - https://requests.readthedocs.io/en/latest/api/

Salt 3007.0 Compatability

With the release of Salt 3007, thought i'd open this up to show what is happening with SaltNornir when using the cfg_gen functions.

This is just an open placeholder to keep you in the loop.

For the interim i have hard-coded my version of Saltstack back to 3006.x

salt_cfg_gen:
          Traceback (most recent call last):
            File "/opt/saltstack/salt/extras-3.10/salt_nornir/proxy/nornir_proxy_module.py", line 1221, in _download_and_render_files
              rendered = __render(value)
            File "/opt/saltstack/salt/extras-3.10/salt_nornir/proxy/nornir_proxy_module.py", line 1199, in __render
              content = _file_download(ret, saltenv)
            File "/opt/saltstack/salt/extras-3.10/salt_nornir/proxy/nornir_proxy_module.py", line 767, in _file_download
              file_path = __salt__["cp.get_url"](url, dest="", saltenv=saltenv)
            File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
              ret = self.loader.run(run_func, *args, **kwargs)
            File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1233, in run
              return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
            File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1248, in _run_as
              return _func_or_method(*args, **kwargs)
            File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/cp.py", line 416, in get_url
              with _client() as client:
            File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/cp.py", line 171, in _client
              return salt.fileclient.get_file_client(__opts__)
            File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 54, in get_file_client
              return {"remote": RemoteClient, "local": FSClient, "pillar": PillarClient}.get(
            File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1128, in __init__
              self.channel = salt.channel.client.ReqChannel.factory(self.opts)
            File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 55, in factory
              return SyncWrapper(
            File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 77, in __init__
              self.obj = cls(*args, **kwargs)
            File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 133, in factory
              auth = salt.crypt.AsyncAuth(opts, io_loop=io_loop)
            File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 695, in __new__
              auth.__singleton_init__(opts, io_loop=io_loop)
            File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 742, in __singleton_init__
              self._authenticate_future = tornado.concurrent.Future()
            File "/opt/saltstack/salt/lib/python3.10/asyncio/events.py", line 656, in get_event_loop
              raise RuntimeError('There is no current event loop in thread %r.'
          RuntimeError: There is no current event loop in thread 'Thread-29 (_worker)'.

from the minion itself:

[INFO    ] User root Executing command nr.nc with jid 20240307202811956871
[ERROR   ] Exception while running callback
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 394, in consume
    await callback(msg)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 484, in wrap_callback
    await callback(decoded)
TypeError: object NoneType can't be used in 'await' expression
[INFO    ] Starting a new job with PID 9130
[INFO    ] Nornir-proxy MAIN PID 41 starting task 'nornir_salt.plugins.tasks.ncclient_call'
[INFO    ] Running task 'nornir_salt.plugins.tasks.ncclient_call' with args {'source': 'running', 'call': 'get_config', 'connection_name': 'ncclient'} on 1 hosts
[INFO    ] nornir_salt:RetryRunner Rtr1 - running task 'nornir_salt.plugins.tasks.ncclient_call'
[INFO    ] [host 10.48.235.183 session-id 75804] Requesting 'GetConfig'
[INFO    ] [host 10.48.235.183 session-id 75804] Sending:
<?xml version="1.0" encoding="UTF-8"?><nc:rpc xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="urn:uuid:048c9650-31df-48bd-bc94-190251a31437"><nc:get-config><nc:source><nc:running/></nc:source></nc:get-config></nc:rpc>]]>]]>
[INFO    ] [host 10.48.235.183 session-id 75804] Received message from host
[INFO    ] nornir_salt:RetryRunner Rtr1 - task 'get_config' completed
[INFO    ] Returning information for job: 20240307202811956871
[INFO    ] User root Executing command nr.cfg_gen with jid 20240307203139652979
[ERROR   ] Exception while running callback
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 394, in consume
    await callback(msg)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 484, in wrap_callback
    await callback(decoded)
TypeError: object NoneType can't be used in 'await' expression
[INFO    ] Starting a new job with PID 9536

Unable to execute custom tasks under /etc/salt/tasks but /etc/salt/bla works

Hello,

Playing around with custom tasks.

Copied over job_data_echo.py from source into /etc/salt/tasks.

When attempt to execute i get this result:

> salt minion nr.task plugin="salt://tasks/job_data_echo.py" job_data='{"foo": 123}'  FB="bla-ap-*"
jid: 20221108162024053981
minion:
    Nornir-proxy MAIN PID 13 job failed: {'task_fun': 'salt://tasks/job_data_echo.py', 'kwargs': {'job_data': {'foo': 123}, 'FB': 'bla-ap-*', '__pub_fun': 'nr.task', '__pub_arg': [{'plugin': 'salt://tasks/job_data_echo.py', 'job_data': {'foo': 123}, 'FB': 'bla-ap-*'}], '__pub_tgt': 'minion', '__pub_jid': '20221108162024053981', '__pub_ret': '', '__pub_master_id': 'master-amer', '__pub_tgt_type': 'glob', '__pub_user': 'root'}, 'identity': {'jid': '20221108162024053981', 'uuid4': '49ac5e51-6b39-4a28-8668-dae9abb3e996', 'user': 'root', 'function': 'exec.nr.task'}, 'name': 'salt://tasks/job_data_echo.py'}, error:
    'Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/salt_nornir/proxy/nornir_proxy_module.py", line 1048, in _worker
        task_fun = _get_or_import_task_fun(job["task_fun"], loader=loader)
      File "/usr/local/lib/python3.9/site-packages/salt_nornir/proxy/nornir_proxy_module.py", line 719, in wrapper
        return func(*args, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/salt_nornir/proxy/nornir_proxy_module.py", line 787, in _get_or_import_task_fun
        function_text = _file_download(plugin)
      File "/usr/local/lib/python3.9/site-packages/salt_nornir/proxy/nornir_proxy_module.py", line 765, in _file_download
        raise CommandExecutionError(
    salt.exceptions.CommandExecutionError: Salt-Nornir proxy pid 13, 'salt://tasks/job_data_echo.py' file download failed, saltenv 'base'

But if I place that same file under /etc/salt/bla it works fine:

> salt minion nr.task plugin="salt://bla/job_data_echo.py" job_data='{"foo": 123}'  FB="bla-ap-*"
jid: 20221108162151551395
minion:
    ----------
    bla-ap-1:
        ----------
        job_data_echo:
            ----------
            foo:
                123

Is there some configuration I may be missing?
Running:

nornir-salt                  0.16.0
salt                         3004.2
salt-nornir                  0.16.0

Any input is much appreciated!

Thank you,
Dave

Update nr.nc to have dedicated list of arguments

Continuation of our discussion on Slack;

When using the following nr.nc rpc command

salt 'nrp02' nr.nc rpc rpc="salt://templates/my_rpc.xml" FO='[{"hostname": "192.168.2.2"}]'

This would provide an invalid response, the needed command to be run was providing the full string of my_rpc.xml in the cli.

salt 'nrp02' nr.nc rpc rpc="FILTER_NAME" FO='[{"hostname": "192.168.2.2"}]'

Adding the render='["rpc"]' argument at the end of the first command was successful
Adding screenshot from slack for reference;

[POSSIBLE BUG] nr.file read and base_url parameter not accepted via cli

it seems that using:
salt nrp1 nr.file read filegroup=backup base_url="/var/salt-nornir/nrp1/files/kmx/backups/tf"

seem to get the error got multiple values for keyword argument 'base_url'

While the above base_url will work in nornir salt via Python directly.

nr_diagram

Unable to run nr.diagram with latest release 0.13.

[root@salt-master-3004 /]# salt nrp1 nr.diagram
nrp1:
'nr.diagram' is not available.
ERROR: Minions returned with non-zero exit code

[root@salt-master-3004 /]# salt-run nr.diagram L2 v3d
'nr.diagram' is not available.

Issues when using `run_ttp`

The proxy minion is throwing a lot of errors only when I have used run_ttp in my workflow

2022-05-03 07:14:53,327 [salt.loaded.ext.proxy.nornir_proxy_module                           :1001][ERROR   ][646543] Nornir-proxy MAIN PID 646543 job failed: {'task_fun': 'nornir_salt.plugins.tasks.netmiko_send_commands', 'kwargs': {'FH': 'x.x.x.x', 'add_details': True, 'enable': True, 'run_ttp': 'salt://ttp_interface_secondary_ip.txt', '__pub_fun': 'nr.cli', '__pub_arg': ['show run interface TenGigabitEthernet0/0/0.1830', {'FH': 'x.x.x.x', 'add_details': True, 'enable': True, 'run_ttp': 'salt://ttp_interface_secondary_ip.txt'}], '__pub_tgt': 'nornir_fremont', '__pub_jid': '20220503141450775369', '__pub_ret': '', '__pub_tgt_type': 'glob', '__pub_user': 'root', 'render': ['filename', 'commands'], 'commands': ('show run interface TenGigabitEthernet0/0/0.1830',), 'connection_name': 'netmiko'}, 'identity': {'jid': '20220503141450775369', 'uuid4': 'e738e0e2-a21c-42cb-8f37-31edc162683b', 'user': 'root', 'function': 'exec.nr.cli'}, 'name': 'nornir_salt.plugins.tasks.netmiko_send_commands'}, error:
'Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/salt_nornir/proxy/nornir_proxy_module.py", line 984, in _worker
    output = run(
  File "/usr/local/lib/python3.8/dist-packages/salt_nornir/proxy/nornir_proxy_module.py", line 1571, in run
    _download_files(download, kwargs, loader=loader)
  File "/usr/local/lib/python3.8/dist-packages/salt_nornir/proxy/nornir_proxy_module.py", line 662, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/salt_nornir/proxy/nornir_proxy_module.py", line 1153, in _download_files
    content = __salt__["cp.get_url"](kwargs[key], dest=None, saltenv=saltenv)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/modules/cp.py", line 424, in get_url
    result = _client().get_url(
  File "/usr/lib/python3/dist-packages/salt/fileclient.py", line 517, in get_url
    result = self.get_file(url, dest, makedirs, saltenv, cachedir=cachedir)
  File "/usr/lib/python3/dist-packages/salt/fileclient.py", line 1173, in get_file
    hash_server, stat_server = self.hash_and_stat_file(path, saltenv)
  File "/usr/lib/python3/dist-packages/salt/fileclient.py", line 1420, in hash_and_stat_file
    hash_result = self.hash_file(path, saltenv)
  File "/usr/lib/python3/dist-packages/salt/fileclient.py", line 1413, in hash_file
    return self.__hash_and_stat_file(path, saltenv)
  File "/usr/lib/python3/dist-packages/salt/fileclient.py", line 1405, in __hash_and_stat_file
    return self.channel.send(load)
  File "/usr/lib/python3/dist-packages/salt/utils/asynchronous.py", line 125, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/usr/lib/python3/dist-packages/salt/utils/asynchronous.py", line 131, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 454, in run_sync
    self.start()
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 757, in start
    raise RuntimeError("IOLoop is already running")
RuntimeError: IOLoop is already running
'

2022-05-03 07:14:53,325 [salt.loaded.ext.proxy.nornir_proxy_module                           :1001][ERROR   ][646543] Nornir-proxy MAIN PID 646543 job failed: {'task_fun': 'nornir_salt.plugins.tasks.netmiko_send_commands', 'kwargs': {'FH': 'x.x.x.x', 'add_details': True, 'enable': True, 'run_ttp': 'salt://ttp_interface_secondary_ip.txt', '__pub_fun': 'nr.cli', '__pub_arg': ['show run interface TenGigabitEthernet0/0/0.1829', {'FH': 'x.x.x.x', 'add_details': True, 'enable': True, 'run_ttp': 'salt://ttp_interface_secondary_ip.txt'}], '__pub_tgt': 'nornir_fremont', '__pub_jid': '20220503141450433928', '__pub_ret': '', '__pub_tgt_type': 'glob', '__pub_user': 'root', 'render': ['filename', 'commands'], 'commands': ('show run interface TenGigabitEthernet0/0/0.1829',), 'connection_name': 'netmiko'}, 'identity': {'jid': '20220503141450433928', 'uuid4': 'eb456242-50ff-4d54-8678-24d255cda8ca', 'user': 'root', 'function': 'exec.nr.cli'}, 'name': 'nornir_salt.plugins.tasks.netmiko_send_commands'}, error:
'Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/salt_nornir/proxy/nornir_proxy_module.py", line 984, in _worker
    output = run(
  File "/usr/local/lib/python3.8/dist-packages/salt_nornir/proxy/nornir_proxy_module.py", line 1571, in run
    _download_files(download, kwargs, loader=loader)
  File "/usr/local/lib/python3.8/dist-packages/salt_nornir/proxy/nornir_proxy_module.py", line 662, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/salt_nornir/proxy/nornir_proxy_module.py", line 1153, in _download_files
    content = __salt__["cp.get_url"](kwargs[key], dest=None, saltenv=saltenv)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/modules/cp.py", line 424, in get_url
    result = _client().get_url(
  File "/usr/lib/python3/dist-packages/salt/fileclient.py", line 517, in get_url
    result = self.get_file(url, dest, makedirs, saltenv, cachedir=cachedir)
  File "/usr/lib/python3/dist-packages/salt/fileclient.py", line 1173, in get_file
    hash_server, stat_server = self.hash_and_stat_file(path, saltenv)
  File "/usr/lib/python3/dist-packages/salt/fileclient.py", line 1420, in hash_and_stat_file
    hash_result = self.hash_file(path, saltenv)
  File "/usr/lib/python3/dist-packages/salt/fileclient.py", line 1413, in hash_file
    return self.__hash_and_stat_file(path, saltenv)
  File "/usr/lib/python3/dist-packages/salt/fileclient.py", line 1405, in __hash_and_stat_file
    return self.channel.send(load)
  File "/usr/lib/python3/dist-packages/salt/utils/asynchronous.py", line 125, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/usr/lib/python3/dist-packages/salt/utils/asynchronous.py", line 131, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "/usr/lib/python3/dist-packages/salt/ext/tornado/ioloop.py", line 458, in run_sync
    raise TimeoutError('Operation timed out after %s seconds' % timeout)
salt.ext.tornado.ioloop.TimeoutError: Operation timed out after None seconds
'

Note: This is an intermittent issue. It works 10% of the time without errors

nr.workflow not respecting filters.

When submitting a workflow using the following

main_workflow:
  nr.workflow:
    - options:
        fail_if_any_host_fail_any_step: []
        fail_if_any_host_fail_all_step: []
        fail_if_all_host_fail_any_step: []
        fail_if_all_host_fail_all_step: []
        report_all: False
        filters: {"FB": "ROUTER10"}
        hcache: True
        dcache: False
        sumtable: False
    # define pre-check steps
    - pre_check:
        - name: pre_check_pull_config
          function: nr.nc
          kwargs: {"source": "candidate", "filter": "<configuration><interfaces/></configuration>"}
          args: ["get_config"]

It appears the filter is not applied, my nrp that runs this specific host has about 400 managed nodes on it
It will not care about the filter and attempt the workflow across all 400 devices.

I've tried multiple iterations of this filter, tried using FL, FB, FO, and all reported the same issues.

Unable to run nr.cli via netbox external pillar

My master config for external pillar is

ext_pillar:
  - salt_nornir_netbox:
      url: 'http://192.68.100.1'
      token: 'Ggy2Fww1BEy@#dFGvwIv%DB4q!5K@N#Gu2YztTjI'
      use_minion_id_device: False
      use_minion_id_tag: True
      use_hosts_filters: False
      use_pillar: True
      host_add_netbox_data: True
      host_add_interfaces: True
      host_add_interfaces_ip: True
      host_add_interfaces_inventory_items: True
      host_add_connections: True
      secrets:
        resolve_secrets: True
        fetch_username: True
        fetch_password: True
        plugins:
          netbox_secretstore:
            private_key: /home/asaharan/.ssh/id_rsa

My pillar on master looks like

hosts:
  test:
    salt_nornir_netbox_pillar:
      use_minion_id_tag: true
    groups: [netbox_connection]
nornir:
  actions:
    facts:
      args:
      - show version
      description: Learn device facts
      function: nr.cli
proxy:
  memory_threshold_action: restart
  memory_threshold_mbyte: 2048
  multiprocessing: true
  proxytype: nornir

groups:
  netbox_connection:
    connection_options:
      netbox:
        extras:
          instances:
            production:
              token: Ggy2Fww1BEy@#dFGvwIv%DB4q!5K@N#Gu2YztTjI
              url: 'http://192.68.100.1'

The command that I'm trying

salt nb_minion nr.cli 'show clock'

Error:

nb_minion:
    ----------
    test:
        ----------
        nornir_salt.plugins.tasks.netmiko_send_commands:
            nornir_salt:RetryRunner test - connection netmiko, retry attempt 3, error: ''device_type''

salt-nornir version

Name: salt-nornir
Version: 0.16.1
Summary: Salt-Nornir Proxy Minion SaltStack Modules
Home-page: https://github.com/dmulyalin/salt-nornir
Author: Denis Mulyalin
Author-email: [email protected]
License: MIT
Location: /usr/local/lib/python3.8/dist-packages
Requires: nornir, nornir-salt, psutil, pydantic
Required-by:

Fresh install with 0.3.0 proxy does not start . missing nornir.grains ?

Hey @dmulyalin ,

I created a new venv with the new version , but the minion does not start.

╰─ pip list | egrep 'nornir|salt'
nornir                                            3.0.0
nornir-napalm                                     0.1.1
nornir-netmiko                                    0.1.1
nornir-salt                                       0.3.0
salt                                              3002.2
salt-nornir                                       0.3.0

Bad Trace from the minion startup
https://pastebin.com/dVs5qEa2

Created a fresh venv with 0.2.2 , minions starts fine

╰─  pip list | egrep 'nornir|salt'
nornir                                            3.0.0
nornir-napalm                                     0.1.1
nornir-netmiko                                    0.1.1
nornir-salt                                       0.3.0
salt                                              3002.2
salt-nornir                                       0.2.2

Good Trace https://pastebin.com/1vpmxy73 from the minion startup

best regards, sharky

Use external pillar for inventory

Is there a way that I can use my custom pillar which pulls data from netbox to feed it into nornir?

dependence on Pydantic

Сould you please up the version of pydantic to 1.10.1?

[REQUEST] Add parameter options for Cred Retry to accept different plugins

Hey!

As we discussed on Slack. This is a request to include the logic for looking at different plugin configurations when it comes to the retry runner (credential retry).

this use case is if someone would like to use Public Key Authentication as their primary method instead of Tacacs.
But if public key authentication is not available to still attempt to use Tacacs or local credentials.

With your help I was able to make this work with napalm only by using;

defaults: 
   data: 
      credentials:
         ssh_key_auth:
            username: "rsa-user"
         tacacs_account:
            username: "{{ salt['environ.get']('USERNAME') }}"
            password: "{{ salt['environ.get']('PASSWORD') }}"
            extras:
              optional_args:
                key_file: False
         local_account_1:
            username: admin
            password: "{{ salt['environ.get']('ADMIN_PASSWORD') }}"
            extras:
              optional_args:
                key_file: False

What I am suggesting if doable is to add the logic to allow other methods such as ncclient, scrapli_netconf, pyez, etc.

Snippet of the pillar configuration I was thinking of is like this (below is in the json format of the yaml file. apologies for that);

'defaults': {
    'data': {
        'credentials': {
            'ssh_key_auth': {
                'username': 'rsa-user'
            },
            'tacacs_account': {
                'username': "user",
                'password': "password",
                'extras': {
                    'napalm': {
                        'optional_args': {
                            'key_file': False
                        },
                    },
                    'scrapli_netconf': {
                        'auth_private_key': None
                    },
                    'ncclient': {
                        'key_filename': None
                    }
                }
            },

Confused with how to use ext_pillar

Hi,

I'm trying to generate the inventory in a module using ext_pillar, but every time I try this I get the following in the minion which doesn't start

No proxy key found in pillar or opts for id nrp1. Check your pillar/opts configuration and contents. Salt-proxy aborted.

I've tried returning all sorts but I must be missing something fundamental.

Just wondering what I should be looking at. As soon as I remover the ext_pillar from master file it starts again and works.

My nrp1.sls is as follows:

proxy:
  proxytype: nornir

hosts: {}
groups: {}
defaults: {}

and salt master file

interface: 0.0.0.0
pki_dir: /etc/salt/pki/master
timeout: 120

file_roots:
  base:
    - /etc/salt/
    - /etc/salt/states/
    - /etc/salt/templates/
    - /etc/salt/_modules

pillar_roots:
  base:
    - /etc/salt/pillar/
    - /etc/salt/states/

ext_pillar:
  - inventory_builder

and a snippet of the ext_pillar file (inventory_builder.py) which essentially just returns a dict

def ext_pillar(minion_id, pillar, *args, **kwargs):
    _copy_file_from_salt_master(source, destination)
    device_data = inventory_builder_download_data()
    inventory_builder_data_dict = convert_json_to_dict(device_data)
    devices_to_vendors, gen = generate_devices_to_vendors(inventory_builder_data_dict)


    yaml = generate_nornir_style_inventory(devices_to_vendors, gen)
    # create python dictionary from yaml
    inventory_dict = yaml_to_dict(yaml)
    write_inventory_to_file(inventory_dict)

    return inventory_dict

[FEATURE REQUEST] salt-nornir nr.inventory update_defaults

Something similar to the update_host functionality would also be useful to update the defaults section of the inventory.

e.g.

salt nrp1 nr.nornir inventory update_defaults data='{"username": "my_new_username", "password": "my_new_password123"}'

Which would be useful for updating things like global usernames and passwords in the inventory aswell as other things.

Thanks,

Jon.

[FEATURE REQUEST] execute custom salt python modules from workflow and return a pass/fail

Not sure if this one is actually possible but I was thinking would be nice to be able to integrate custom salt execution modules into the nr.workflow system in the following way.

Its already possible to call custom modules but they always return a pass.

It would be nice if say you were given a specific set of return data to return from the custom module, that it would be possible to return a true pass/fail from the custom module.

E.g. I have a few custom modules that might do something adhoc like upload some config backups to a nextcloud server and based on the results of that I might not want the next steps to run, so i I can return a data structure that it expects from the custom module, it would be nice to get a true pass/fail from it to integrate it into the workflow properly.

Thanks,
Jon.

[FEATURE REQUEST] - Add support for Nornir PYEZ plugins

I would like to use the Nornir-Pyez plugins found here: https://github.com/DataKnox/nornir_pyez

USECASE: Add support to validate network state using csv tables

Extend salt-nornir to validate network using data from csv table. Probably need to code support in TestsProcessor and/or create runner nr.test

For example, assuming we have this csv data:

host,interface,admin_state,line_state
R1,Eth1/1,up,up
R2,Gi0/0/0,down,down

Take above and compare with devices state and produce test results.

NAPALM has similar functionality by comparing getters output with expected output, but it accepts data in dictionary format, which might not scale well compared to csv - csv spreadsheet are easier to edit by humans as well as many systems provide export of data in csv format, allowing to extract desired state tables from them.

Another example, assuming this bgp peers table:

host,peer_ip,state
R123,10.1.2.3,established
R321,10.3.2.1,admindown

take output from devices and verify network state.

In essence, this feature will allow to define desired state of the network using tabulated data expressed in csv format. Optionally, can add support for markdown and its tables

[FEATURE REQUEST] ability to use job_data kwargs in nr.test

Would be great to be able to pass adhoc job_data kwargs dictionary into nr.test to allow the vars to be consumed inside the test suite jinja2

Thanks,
Jon.

pip install salt_nornir does not install reps

Hey ,

I created a fresh virtual environment . According to your docs if I run

pip install salt_nornir

Quote from Doc

nstalling``salt_nornir`` should automatically install these dependencies:

netmiko>=3.3.2
nornir>=3.0.0
nornir_netmiko>=0.1.1
nornir_napalm>=0.1.1
nornir_salt>=0.3.0
napalm>=3.0.0
psutil

This is my result .

╰─ pip install salt_nornir
Collecting salt_nornir
  Downloading salt_nornir-0.2.2.tar.gz (16 kB)
Using legacy 'setup.py install' for salt-nornir, since package 'wheel' is not installed.
Installing collected packages: salt-nornir
    Running setup.py install for salt-nornir ... done
Successfully installed salt-nornir-0.2.2
WARNING: You are using pip version 20.2.1; however, version 21.0.1 is available.
You should consider upgrading via the '/Users/sharky/.pyenv/versions/3.8.6/envs/salt_nornir/bin/python3.8 -m pip install --upgrade pip' command.
 OSX  sharkys-mbp  sharky  ~/python/salt/salt_nornir  salt_nornir                               16836
╰─ pip list
Package     Version
----------- -------
pip         20.2.1
salt-nornir 0.2.2
setuptools  49.2.1
WARNING: You are using pip version 20.2.1; however, version 21.0.1 is available.
You should consider upgrading via the '/Users/sharky/.pyenv/versions/3.8.6/envs/salt_nornir/bin/python3.8 -m pip install --upgrade pip' command.

I will go ahead and install it manually . No problem , but I'm not sure if the docs are outdated or there is something wrong with the package.

[Question] Integrate deltaproxy to avoid double targeting

Hey @dmulyalin ,

thanks for all your great work to combine the strength of nornir with salt and make salt more useable for network engineers.

I always find it non intuitiv to use the double targeting method.

salt nrp1 nr.cli "show clock" FB="R*"

Sure it is just syntax sugar , but I don't feel comfortable with it . If you use salt a lot salt-nornir is not the same syntax as salt.

Recently saltstack announced that they will open source there enterprise proxy aka delta proxy and control proxy which is something like you have built with the nornir proxy. I red through the documentation which is available now and it seems they don't need the double targeting . Not sure how , but the controlproxy seems to be transparent to the salt .

Do you think this is useful and you can adapt this concept for the nornir proxy ?

Not sure everything is open sourced yet , but this is what I red.

https://enterprise.saltstack.com/en/latest/_downloads/2b976b44110f45b5caeee4975c8c3139/delta-proxy-minions-3003.pdf

https://docs.vmware.com/en/vRealize-Automation/8.2/SaltStackConfig_Help_v64.pdf

https://github.com/saltstack/salt/pull/60090

From my point of view it would make the nornir proxy even more awesome :) , but I'm not that familiar with the implemention than you are. Let me know your thoughts.

best regards, sebastian

[FEATURE REQUEST] Ability to fail early on a host inside Task Workflow system

Fail quickly/early on a host if a certain step fails.

Say we have a step at the beginning that is very important, for backup.

If this step fails then halt all further steps for the specific hosts.

Something similar to what already exists as

    - options:
        fail_if_any_host_fail_any_step: []

maybe we could have something like
halt_if_any_host_fail_any_step: ["backups"]

Update ncclient to open connection with raise_mode option

I've opened a ticket with the ncclient community as well to see if this is the correct way of addressing this but nonetheless, the request would be add the option to ignore_warnings to the ncclient connection options.

This is similar to how Napalm has integrated it. The only difference I see is that Napalm is leveraging Junipers PyEz Package. I would prefer to have this integrated with ncclient instead.

This is my reference ticket with the ncclient team.

ncclient/ncclient#545

Snippet of the code i used is within the ticket, but essentially it would be to add in raise_mode optional argument to pass in when opening a connection to the device.

Code pasted below for reference:

#!/usr/bin/env python
 import os
 import sys

 import logging

 from ncclient import manager


 def connect(host, port, user, password):
     conn = manager.connect(host=host,
                            port=port,
                            username=user,
                            password=password,
                            timeout=60,
                            device_params={'name': 'junos'},
                            hostkey_verify=False)

     conn.lock()
     conn.raise_mode = 1

     # configuration as a string
     load_config_result = conn.load_configuration(action='set', config='delete system services telnet')
     logging.info(load_config_result)
     validate_result = conn.validate()
     logging.info(validate_result)

     compare_config_result = conn.compare_configuration()
     logging.info(compare_config_result)

     conn.commit()
     conn.unlock()
     conn.close_session()

 if __name__ == '__main__':
     LOG_FORMAT = '%(asctime)s %(levelname)s %(filename)s:%(lineno)d %(message)s'
     logging.basicConfig(stream=sys.stdout, level=logging.INFO, format=LOG_FORMAT)

     connect('192.168.2.2', '830', 'admin', 'password')

[FEATURE REQUEST] nr.gen_cfg to file then read from file with nr.cfg

Hi
Just creating one for the ability to read from file a previously saved generated config snippet when doing an nr.cfg config apply.
example workflow something like.

nr.cfg_gen tf=rollback_config
nr.cfg from_file=rollback_config or file_read... or rf=rollback_config   etc

Jon.

[FEATURE REQUEST] ability to exclude certain tasks from summary

Just a request to see if its possible to add another flag to the nr.workflow system that allows you to exclude certain steps/tasks from the end summary results.

Thanks,
Jon.

How to get failed status while running commands

I have been trying to figure out if there is a way to know whether the result I received is a success or false, but it's not evident from the response. Am I missing anything with the configuration? I have only done the configuration exactly as per the getting started guide

{
  "return":[
    {
      "home":{
        "home-csr1":{
          "nornir_salt.plugins.tasks.netmiko_send_commands":"nornir_salt:RetryRunner home-csr1 - connection netmiko, retry attempt 3, error: 'TCP connection to device failed.\n\nCommon causes of this problem are:\n1. Incorrect hostname or IP address.\n2. Wrong TCP port.\n3. Intermediate firewall blocking access.\n\nDevice settings: cisco_xe 172.16.14.54:22\n\n'"
        }
      }
    }
  ]
}

[bug?] Unable to get progress=bars working even with env vars exported

Hi,

Trying progress=bars but unable to get it working as it will always fail as per below:

PYTHONIOENCODING=utf-8 export PYTHONIOENCODING && salt-run nr.event progress=bars
/home/.venv/lib/python3.9/site-packages/salt/utils/event.py:124: DeprecationWarning: The 'transport' kwarg has been deprecated and it will be removed in the Chlorine release, as such, its usage is no longer required.
  salt.utils.versions.warn_until(
03-Sep-2023 14:00:45.147:root:415b90ce9aa26575035c:state.nr.workflow.deploy_syslog.b…  task:scrapli_send_commands                         P…
03-Sep-2023 14:00:48.636:root:415b90ce9aa26575035c:state.nr.workflow.deploy_syslog.b…  subtask:show configuration                         P…
03-Sep-2023 14:00:52.909:None:None:exec.nr.cli                                         task:scrapli_send_commands                         P…
03-Sep-2023 14:00:58.682:None:None:exec.nr.cli                                         subtask:show configuration system syslog | displa… P…

Passed invalid arguments: startswith first arg must be str or a tuple of str, not NoneType

Usage:

    Function to listen to events emitted by Nornir Proxy Minions. Matched
    event printed to terminal.

    :param tag: (str) tag regex string, default is ``nornir\-proxy/.*``
    :param jid: (int, str) Job ID to listen events for, default is ``all``
    :param progress: (str) progress display mode - ``log``, ``raw``, ``bars``, ``tree``
    :param stop_signal: (obj) thread Event object, stops listening to events if ``stop_signal.is_set()``,
        if ``stop_signal is None``, listens and print events until keyboard interrupt hit - ``ctrl+c``

    ``bars`` and ``tree`` progress display modes use Rich library, to properly display various
    symbols and characters need to make sure to use utf-8 encoding for your environment for example
    by running these commands::

        [root@salt-master ~]# PYTHONIOENCODING=utf-8
        [root@salt-master ~]# export PYTHONIOENCODING
    
root@salt-master-staging-3006

dmulyalin / salt-nornir Goto Github PK

salt-nornir's People

Contributors

Stargazers

Watchers

Forkers

salt-nornir's Issues

Recommend Projects

Recommend Topics

Recommend Org