atarola/pyjojo

Expose a directory of bash scripts as an API.

pyjojo's Issues

json only

go ahead and hard-code the json content type headers in the request server-side

scripts - version block

would be nice to have a "version" block so we could set versions for our scripts

Use OPTIONS instead of GET for script info requests

change GET /scripts/{script_name} to OPTIONS /scripts/{script_name}

Error when returning values in combined output mode

When you try to run a script that has output: combined and whose output has a return value match, Jojo throws an exception:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 1055, in _stack_context_handle_exception    
    raise_exc_info((type, value, traceback))
  File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 263, in _nested
    yield vars
  File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 236, in wrapped
    callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 551, in inner
    self.set_result(key, result)
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 480, in set_result
    self.run()
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 510, in run 
    yielded = self.gen.send(next)
  File "/usr/local/lib/python2.7/dist-packages/pyjojo/handlers.py", line 244, in post
    "return_values": self.find_return_values(stdout),
  File "/usr/local/lib/python2.7/dist-packages/pyjojo/handlers.py", line 277, in find_return_values
    key, value = [item.strip() for item in temp.split('=')]
ValueError: need more than 1 value to unpack

For example, here's a good script that runs as expected and which Jojo gives the correct return value for:

#!/bin/bash

# -- jojo --
# description: Solves simple math expressions
# param: expr - The expression to solve
# tags: math, solve
# http_method: post
# output: split
# -- jojo --

solution=`echo $EXPR | bc`
echo "jojo_return_value solution=$solution"

Here's something that gives correct combined output, but doesn't provide a return value:

#!/bin/bash

# -- jojo --
# description: Solves simple math expressions
# param: expr - The expression to solve
# tags: math, solve
# http_method: post
# output: combined
# -- jojo --

solution=`echo $EXPR | bc`
echo $solution

This throws the error reported above, though:

#!/bin/bash

# -- jojo --
# description: Solves simple math expressions
# param: expr - The expression to solve
# tags: math, solve
# http_method: post
# output: combined
# -- jojo --

solution=`echo $EXPR | bc`
echo "jojo_return_value solution=$solution"

Combined output line split gives unexpected output

Consider the following output of a script that does simple math:

2 + 7 = 9

When pyjojo encounters this output, it tries to prepare the stdout content by splitting it. Here's scripts.py:127:

callback((child.returncode, stdout.split()))

This means that what gets returned is this list:

"stdout": [
    "2",
    "+",
    "7",
    "=",
    "9"
]

I expect:

"stdout" : [
    "2 + 2 = 9"
]

By contrast, when the script runs in "split" mode, we use splitlines() instead, like here on scripts.py:143:

callback((child.returncode, stdout.splitlines(), stderr.splitlines()))

The former should use splitlines() as well.

ImportError: No module named config

Processing setuptools-0.6c11-py2.7.egg
setuptools 0.6c11 is already the active version in easy-install.pth
Installing easy_install script to /tmp/pyjojo/bin
Installing easy_install-2.7 script to /tmp/pyjojo/bin

Using /tmp/pyjojo/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
Finished processing dependencies for pyjojo==0.2

root@epoch:/tmp/pyjojo# source bin/activate

(pyjojo)root@epoch:/tmp/pyjojo# pyjojo -d
Traceback (most recent call last):
File "/tmp/pyjojo/bin/pyjojo", line 8, in
load_entry_point('pyjojo==0.2', 'console_scripts', 'pyjojo')()
File "/tmp/pyjojo/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 318, in load_entry_point
File "/tmp/pyjojo/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 2221, in load_entry_point
File "/tmp/pyjojo/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 1954, in load
File "/tmp/pyjojo/pyjojo/util.py", line 16, in
from pyjojo.config import config
ImportError: No module named config

Command Switches

I've tried your code and I think it's pretty powerful and useful. Great Job!

I'd like to propose a feature/enhancement: to be able to configure and pass command line switches in addition to env variables. That'd allow many people to use their scripts unmodified.

e.g. in addition to regular params:

param: text - text to echo back

Have something like:

switch: date:--date - Date of the month

switch: help:-h

In this cases "date" and "help" are the param names, and they're mapped into switches "--date" and "-h".

It'd also be powerful to be able to map arguments ($1, $2, etc) so any type of script could be accessed as a web service without any modifications.

SSL Medium Strength Cipher Suites Supported

When testing a server running pyjojo, we found it supports the use of SSL ciphers that offer weak strength encryption (56 bits and less than 112 bits).

Out of best practice, could it reconfigured to avoid use of any ciphers less than 128 bits?

sh testssl/test_ssl.sh ipaddress port

[*] Testing for weak ciphers (based on key length) ...
Accepted SSLv3 56 bits DES-CBC-SHA
Accepted TLSv1 56 bits DES-CBC-SHA

Thanks!

add 'tags' jojo block

tags would be nice to have to help with external system organization

GET action to execute scripts?

I'm attempting to use pyjojo as an option provider for rundeck, unfortunately Rundeck doesn't support POST url's for it's url option provider.

Is it possible to have scripts that don't need data passed in execute on a get request?

scripts - run time

would be nice to have a block that reports the time it took to run a script

Scripts do not execute when "lock" is "true"

There's a typo in scripts.py on line 103:

with (yield gen.Task(self.lock.aquire)):

should be

with (yield gen.Task(self.lock.acquire)):

Without the 'c' in there, it fails, and returns a 500. Put it back and try again, and it fails for a different reason:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/tornado/web.py", line 1055, in _stack_context_handle_exception
    raise_exc_info((type, value, traceback))
  File "/usr/local/lib/python2.7/site-packages/tornado/web.py", line 1178, in wrapper
    result = method(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/tornado/gen.py", line 154, in wrapper
    runner.run()
  File "/usr/local/lib/python2.7/site-packages/tornado/gen.py", line 508, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/local/lib/python2.7/site-packages/pyjojo/handlers.py", line 241, in post
    retcode, stdout = yield gen.Task(script.execute, self.params)
  File "/usr/local/lib/python2.7/site-packages/tornado/gen.py", line 534, in run
    self.yield_point.start(self)
  File "/usr/local/lib/python2.7/site-packages/tornado/gen.py", line 378, in start
    self.func(*self.args, **self.kwargs)
  File "/usr/local/lib/python2.7/site-packages/tornado/gen.py", line 154, in wrapper
    runner.run()
  File "/usr/local/lib/python2.7/site-packages/tornado/gen.py", line 508, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/local/lib/python2.7/site-packages/pyjojo/scripts.py", line 103, in execute
    with (yield gen.Task(self.lock.acquire)):
  File "/usr/local/lib/python2.7/site-packages/tornado/gen.py", line 534, in run
    self.yield_point.start(self)
  File "/usr/local/lib/python2.7/site-packages/tornado/gen.py", line 378, in start
    self.func(*self.args, **self.kwargs)
TypeError: acquire() got an unexpected keyword argument 'callback'

I don't know enough about Tornado to understand how (presumably) that @gen.engine decorator handles the function it's given, but something here is broken.

Centos openssl library doesnt support ciphers option.

2014-04-03 22:13:43,593 ERROR [ioloop.py:663] Exception in I/O handler for fd 6
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/tornado/ioloop.py", line 653, in start
self._handlers[fd](fd, events)
File "/usr/lib/python2.6/site-packages/tornado/stack_context.py", line 241, in wrapped
callback(_args, *_kwargs)
File "/usr/lib/python2.6/site-packages/tornado/netutil.py", line 141, in accept_handler
callback(connection, address)
File "/usr/lib/python2.6/site-packages/tornado/tcpserver.py", line 212, in _handle_connection
do_handshake_on_connect=False)
File "/usr/lib/python2.6/site-packages/tornado/netutil.py", line 322, in ssl_wrap_socket
return ssl.wrap_socket(socket, *_dict(context, *_kwargs))
TypeError: wrap_socket() got an unexpected keyword argument 'ciphers'

htpassword seems to only support crypt

https://github.com/atarola/pyjojo/blob/master/pyjojo/handlers.py#L59

the default behavior of htpasswd on Debian Wheezy is to use md5:

 -m  Force MD5 encryption of the password (default).
 -d  Force CRYPT encryption of the password.
 -p  Do not encrypt the password (plaintext).

using -s fails as well. this should be documented and/or more secure hashes should be allowed.

mod_wsgi support?

I would like to run pyjojo behind apache in order to support extended security and authentication options, notably LDAP, kerberos and/or SSL client certificates.

this in itself, isn't terribly difficult: bind pyjojo to localhost and then use ProxyPass. however, this leaves a gap: a user on localhost will still be able to send HTTP requests to localhost:3000 and bypass the apache layer, which contains all the authentication checks.

I've worked around this by using pyjojo's htpasswd support and using a directive like the following:

    <Location /api/>
        # this is where the magic happens
        RequestHeader set Authorization "Basic $BASE64(USER:PASSWORD)"
        ProxyPass http://localhost:3000/
        ProxyPassReverse http://localhost:3000/
    </Location>

now a user on localhost would need to steal the user+password to bypass authentication on port :3000.

running the whole application under mod_wsgi would mean that there is no port :3000 to use as a bypass. I did some research, but it wasn't clear how Tornado and mod_wsgi interact. it seems like Tornado has very little support for mod_wsgi and many caveats.

I'm not tied down to mod_wsgi, but I would like to explore/discuss some solutions towards securing pyjojo behind apache (or nginx) and contribute some example configs/documentation.

Add handler to report version

The upcoming changes (based on other issues filled here) look like they change the way requests and responses are formed in a few ways. In the interest of not having client/server miscommunication, there should be a way for the client to get the version of Jojo it's talking to so it can properly form requests and interpret responses from different server versions.

filtered_params option isn't recognized

If Pyjojo v0.6 finds this basic echo script:

#!/bin/bash

# -- jojo --
# description: echo
# param: text - Text to echo
# filtered_params: text
# -- jojo --

echo ${TEXT}
exit 0

...it outputs:

WARNING [scripts.py:212] unrecognized line in jojo block: filtered_params: text

Split output on newlines in stderr and stdout

stderr and stdout should each be an array of lines of text.