Add Salt outputters

[whiteinge]

Add Salt outputters

any update on this issue ?

[bbinet]

For those interested, I've created the script https://gist.github.com/bbinet/d87bf2571652e4dec68df6b280b2af9d to format pepper output using salt outputters.
It can both format sync and async jobs, and can be used in the following ways:

$ pepper \* state.sls core | pepper_format.py -o highstate
$ pepper --client=local_async \* state.highstate | pepper_format.py
$ pepper_format.py 20170504160400750792

$ pepper_format.py -h
usage: pepper_format.py [-h] [-o OUTPUTTER] [-v] [-p] [-n] [jid]

Format pepper output using any salt outputter

positional arguments:
  jid                   Optional job id

optional arguments:
  -h, --help            show this help message and exit
  -o OUTPUTTER, --outputter OUTPUTTER
                        Outputter to use
  -v, --verbose         Verbose output
  -p, --profiling       Activate profiling
  -n, --no-color        No color in output

[hedrickbt]

On Ubuntu, if you
"apt install salt-common"

the output modules will be available. It will also default to the format you expect.

[whiteinge]

Option 2 is to have salt-api run the response through an outputter then send the rendered result back to Pepper.

[arthurzenika]

I think a build-dep would be ok.

Can salt-api already take an outputter option ?

[whiteinge]

If you specify the Accept header in your requests salt-api will happily oblige. It would be trivial to add support for Salt's nested outputter and return everything as a giant string. It's possible that color escape codes would even come through unscathed.

[arthurzenika]

Can you think of a way to pipe the json result to do something like :

pepper 'host' state.highstate | salt-call --output highstate --local something

[arthurzenika]

I've failed with

pepper 'host' state.highstate  | salt-call --local --output highstate test.outputter 

or

salt-call --local --output highstate test.outputter "`pepper 'host' state.highstate `" 

[whiteinge]

Outputter modules are written to run for each minion return, not on the data as a whole. To do this you'd need to write a small Python wrapper script.

[arthurzenika]

Here is a wrapper script that seems to do the job (so far) :

import __builtin__
import sys
import json

import salt.config
from salt.output.highstate import output

__opts__ = salt.config.minion_config('/etc/salt/minion')
__opts__['color']=True
__opts__['state_output_diff'] = True
__opts__['state_verbose'] = False
setattr(__builtin__, '__opts__', __opts__)
print output(json.load(sys.stdin))

[arthurzenika]

it works with highstate dictionnary data, but if you send it the result of a test.ping execution, it truncates the results.

I don't think there is a smart way of selecting the right outputter without further context in the results.

[whiteinge]

Thanks for posting your wrapper. I'm sure others will find that useful.

You're definitely right about not having enough information to decide which outputter to use. Although that information is in the job cache, most (all?) modules don't expose or display it.

[whiteinge]

The jobs.print_job runner function will output which outputter to use -- which may be helpful but would still require some indirection (run the job async, get the jid, poll for the jid, pipe the result through your wrapper).

[bbinet]

+1, being able to use Salt outputters directly from Pepper would be very useful indeed!

@arthurlogilab: I tried a slightly modified version of your python script (customized for running jobs.lookup_jid runner):

import __builtin__
import subprocess
import sys
import json

import salt.config
from salt.output.highstate import output

__opts__ = salt.config.minion_config('')
__opts__['color']=True
__opts__['state_output_diff'] = True
__opts__['state_verbose'] = False
setattr(__builtin__, '__opts__', __opts__)
#print output(json.load(sys.stdin))
print output(json.loads(subprocess.check_output(
    ['pepper', '--client=runner', 'jobs.lookup_jid', 'jid=%s' % sys.argv[1]])))

But for some reason, it does not work and returns:

return:
    Data failed to compile:
----------
[...]

Any idea why it does not work?

[cmclaughlin]

pepper is really nice... this feature would be icing on the cake!

[arthurzenika]

@bbinet nice one ! Thanks a bunch...

[bbinet]

Note that I've submitted PR saltstack/salt#41090 to make the profiling option work (it currently raises an exception)

[cmclaughlin]

@bbinet thanks!

[gtmanfred]

The next version of pepper will use the salt outputters if salt is installed.

[arthurzenika]

@gtmanfred glad to hear that... great news!

[lizhixin3016]

I solved the format problem by "pip/pip3 install salt"