wooey/clinto

This converts an assortment of python command line interfaces into a language agnostic build spec for usage in GUI creation.

Wooey is a simple web interface to run command line Python scripts. Think of it as an easy way to get your scripts up on the web for routine data analysis, file processing, or anything else.

Wooey was envisioned as a system to allow data analysts to be able to easily:

1. Autodocument workflows for data analysis
    (simple model saving).
2. Enable fellow co-workers with no command line
    experience to utilize python scripts.
3. Enable the easy wrapping of any program in simple
   python instead of having to use language specific
   to existing tools such as Galaxy.

For installation and other instructions, check out our documentation or give Wooey a test drive at our demo-server!

clinto's Issues

Documentation Import

Provide the docstrings generated in python. Cross-ref from: wooey/Wooey#56

Support for CWL

Add in a parser for the CWL to be more agnostic towards the platform/kernel

Click support

Support for click.

Just opening a bug here in the right place.

Pass the default value for parameters

Currently, clinto does not report the default value for a field, only infers information about the default type. This will allow better interfaces to be constructed such as wooey/Wooey#119

Try / Except for Python 3 fails

Scripts using try...except blocks to conditionally import for Python 3 fail as the try...except block is collapsed during the parsing process.

Range Fields

In Python 3 a range(0,10,3) creates a range object rather than a list. We can make use of this to display a range-specific UI element such as slider.

This will require additions to clinto to detect and pass the object type and parameters somehow.

In flask-wooey this was handled and passed as follows:

jsons = []
for action, widget in filtered_actions:
    if PY3 and isinstance(action.choices, range):
        widget = 'RangeField'
        action.choices = (action.choices.start, action.choices.stop, action.choices.step)
    elif action.nargs == "+":
        # FIXME: We can be smarter here; other possibilities for multiple selections
        widget = 'SelectMany'
    else:
        widget = 'SelectOne'

    jsons.append(as_json(action, widget=widget))

return jsons

Since range supports integers only it might make sense to implement this as a range type in TYPE_FIELDS? The parameters for the range can then be passed to the field definition.

wooey/django-djangui#9

Import zip script error : UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 54

UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 54: character maps to

This is with 0.10 Wooey

Request Method:	POST
http://127.0.0.1:8000/admin/wooey/script/add/
1.11.13
ParserError
Traceback (most recent call last): File "C:\Users\cheta\Anaconda3\lib\site-packages\clinto\parsers\argparse_.py", line 252, in extract_parser ast_source = source_parser.parse_source_file(self.script_path) File "C:\Users\cheta\Anaconda3\lib\site-packages\clinto\ast\source_parser.py", line 39, in parse_source_file s = f.read() File "C:\Users\cheta\Anaconda3\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 54: character maps to
C:\Users\cheta\Anaconda3\lib\site-packages\wooey\signals.py in script_version_postsave, line 64
C:\Users\cheta\Anaconda3\python.exe
3.6.3
['D:\AUTOREPORT', 'C:\Users\cheta\Anaconda3\python36.zip', 'C:\Users\cheta\Anaconda3\DLLs', 'C:\Users\cheta\Anaconda3\lib', 'C:\Users\cheta\Anaconda3', 'C:\Users\cheta\Anaconda3\lib\site-packages', 'C:\Users\cheta\Anaconda3\lib\site-packages\Babel-2.5.0-py3.6.egg', 'C:\Users\cheta\Anaconda3\lib\site-packages\win32', 'C:\Users\cheta\Anaconda3\lib\site-packages\win32\lib', 'C:\Users\cheta\Anaconda3\lib\site-packages\Pythonwin', 'D:\AUTOREPORT']
Tue, 5 Jun 2018 23:05:37 +0000

Argparse fails if parse_args is never called

If a script never evaluates the parser, it will fail:

https://gist.github.com/Chris7/68dc60931ebe7d54dfa0

Subparser support

Support for subparsers so conditional logic can be represented. See: wooey/django-djangui#8

Docopt support

Support for docopt.

This is interesting as was can just grab the text and run it through docopt in clinto to generate the objects - no need for funky parsing. Not sure how flexible it is re parsing to our interface.

Source module names with library names

If we use imp.load_source with a library name, such as:

imp.load_source('click', '../clinto/clinto/tests/click_scripts/click_script.py')

it will fail to load because it will first go for the library. Make a check for importing script names that are library names.

Python 3 None

The AST source parser has issues when running under Python 3 code I think due to the fact that None is now a keyword in Python3. This means it doesn't "see" the None and returns a syntax error. e.g. the following:

parser.add_argument('-o', '--outfile', dest='outfile', default=None, type=str, help='Filename for output')

...gives a syntax error for missing value...

parser.add_argument('-o', '--outfile', dest='outfile', default=, type=str, help='Filename for output')

Use astor for AST->Python parsing

The current AST parsing fails on some python corner cases. We should use this library to do this conversion for us instead of maintaining our own copy of Armin's codegen code:
https://github.com/berkerpeksag/astor

Having both the argparser and the parser.parse_args() defined in global scope fails

As described by @CatherineH here wooey/Wooey#44

---

Okay, I think I've figured it out -

issue 1, the argparser problem that I started with, is caused by having both the argparser and the parser.parse_args() defined in global scope. For example, this script works:

import argparse
import sys

parser = argparse.ArgumentParser(description='testing wooey.')
parser.add_argument('--arg1', type=str, default="bloop", help='argument 1')
parser.add_argument('--arg2', help='argument 1', type=str, default="bloop")
parser.add_argument('--arg3', help='argument 1', type=str, default="bloop")

def main():
    args = parser.parse_args()
    print(vars(args))

if __name__ == "__main__":
    sys.exit(main())

So does this script:

import argparse

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description='testing wooey.')
    parser.add_argument('--arg1', type=str, default="bloop", help='argument 1')
    parser.add_argument('--arg2', help='argument 1', type=str, default="bloop") 
    parser.add_argument('--arg3', help='argument 1', type=str, default="bloop")

    args = parser.parse_args()
    print(vars(args))

but this one does not:

import argparse

parser = argparse.ArgumentParser(description='testing wooey.')
parser.add_argument('--arg1', type=str, default="bloop", help='argument 1')
parser.add_argument('--arg2', help='argument 1', type=str, default="bloop")
parser.add_argument('--arg3', help='argument 1', type=str, default="bloop")
args = parser.parse_args()
print(vars(args))

which used to work in a previous version of Wooey.

Some other (less widespread) command line systems

This is a list of other less well know command line systems we might want to support.

• argh
• blargs
• cliar
• hug
• optfunc

Don't think these have enough use in the wild to be a priority but worth bearing in mind for the generic parser support.

Support for entry_point scripts

A method to handle scripts that are called via entry_points.

Parser errors for argparse are not properly saved

If a script fails, the error will not be set since there is no valid parser.

Add zip support for clinto

OK So here is what I did :
I bundle up my project as below
lib/
__main__py

the lib folder has all libs specific to my project.
I renamed my main script to main.py

I then zip the contents up into reportgen.zip
I renamed this to reportgen.py

and now I can execute this directly as such : python reportgen.py locally.
However importing this into wooey gives the below error :

Request Method:	POST
http://URL/admin/wooey/script/add/
1.9.13
UnicodeDecodeError
'utf-8' codec can't decode byte 0x9a in position 12: invalid start byte
/opt/anaconda/anaconda3/lib/python3.6/codecs.py in decode, line 321
/opt/anaconda/anaconda3/bin/python
3.6.3

Support for action=append in argparse parser

We need to distinguish when an argument can be supplied with multiple arguments as well as be specified multiple times on the command line. This is related to issue wooey/Wooey#94

Return version numbers

Return version numbers from scripts. For argparse in Py2 this is in the constructor, for Py3 it's a parameter with an action='version' command.

zip support broken with v.12 wooey

I get the below error importing a zip file, works fine with 0.10

Request Method:	POST
http://testnnnnnn.com/admin/wooey/script/add/
3.0.5
ParserError
Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/clinto/parsers/argparse_.py", line 253, in extract_parser ast_source = source_parser.parse_source_file(self.script_path) File "/usr/local/lib/python3.7/site-packages/clinto/ast/source_parser.py", line 39, in parse_source_file s = f.read() File "/usr/local/lib/python3.7/codecs.py", line 322, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xda in position 14: invalid continuation byte
/wooey_build/wooey/signals.py in script_version_postsave, line 64
/usr/local/bin/python
3.7.7
['/docker_wooey', '/docker_wooey', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/site-packages', '/wooey_build']
Wed, 29 Apr 2020 18:02:21 +0000

method names

What should the core method be? I think import ArgParseNodeBuilder is a bit verbose (and assumes ArgParse).

how about:

from clinto import schema

initialize with: schema(script_name, script_path)

The user can naturally iterate over that, as well as call functions like getJSON and other intermediates.