ioistired/import-expression

Transpiles a superset of python to allow easy inline imports

Import Expression Parser (for lack of a better name)

Import Expression Parser converts code like this:

urllib.parse!.quote('hello there')

Into this equivalent code:

__import__('importlib').import_module('urllib.parse').quote('hello there')

Usage

>>> import import_expression
>>> import_expression.eval('collections!.Counter("bccdddeeee")')
Counter({'e': 4, 'd': 3, 'c': 2, 'b': 1})

The other public functions are exec, compile, parse, and find_imports. See their docstrings for details.

By default, the filename for SyntaxErrors is <string>. To change this, pass in a filename via the filename kwarg.

Reusing compiled code objects

import_expression.eval/exec/compile should not be passed strings in a tight loop.
Doing so will recompile the string every time. Instead, you should pre-compile the string to a code object and pass that to import_expression.eval / import_expression.exec. For example, instead of this:

for line in sys.stdin:
	print(import_expression.eval('foo!.bar(l)', dict(l=line))

Prefer this:

code = import_expression.compile('foo!.bar(l)', mode='eval')
for line in sys.stdin:
	print(import_expression.eval(code, dict(l=line)))

REPL usage

Run import-expression for an import expression enabled REPL.
Run import-expression -a for a REPL that supports both import expressions and top level await (3.8+).

See import-expression --help for more details.

Running a file

Run import-expression <filename.py>.

Limitations / Known Issues

• Due to the hell that is f-string parsing, and because ! is already an operator inside f-strings, import expressions inside f-strings will likely never be supported.
• Due to python limitations, results of import_expression.exec will have no effect on the caller's globals or locals without an explicit globals argument.
• Unlike real operators, spaces before and after the import expression operator (such as x ! .y) are not supported.

License

Copyright © io mintz <[email protected]>. All Rights Reserved.
Licensed under the MIT License. See the LICENSE file for details.

import-expression's Issues

ast.parse sometimes returns None

handle this and raise SyntaxError

tokenize no longer outputs error tokens for `!`

this completely breaks the entire library…

Broaden the scope of the language

I want to add features besides just import-expressions, such as null-aware chaining ops and a special AST node for statements that don't implicitly return a value. This requires a rebrand.

Make sure the API is what we want.

Look over the kwargs, args, and behavior of __init__.py. Make sure it's something we can commit to for 1.0.0. Consider making _parse_eval_exec_args public in some form, so that people can use vanilla eval and exec with less overhead.

encoding doesn't work

running a file with #coding:ie results in SyntaxError: encoding problem.

Tab completion no longer working for import expressions in the REPL

Previously, urllib.parse!.<TAB> would suggest quote but now, nothing happens.

lone import expression op parses successfully

See https://github.com/iomintz/import-expression-parser/blob/6906d00897baee1c757fcd623aef8a7bb0713580/tests.py#L92-L95.

This can probably easily be fixed by checking if the entire name is constants.MARKER, rather than just whether it endswith the marker.

Rewrite the parser

The current parser is hard to grok. Especially all those strange helper functions that return truthy values whose value is actually used. It's probably inefficient right now too.

Custom readline completer which supports import expressions

foo!.<TAB> should import foo and list its attributes.

REPL: ")" does not immediately raise SyntaxError like it should

➤ python
🐍 )
  File "<stdin>", line 1
SyntaxError: unmatched ')'
➤ import_expression
🐍 )
•• 
•• 
•• 

this is a bug with DONT_IMPLY_DEDENT or something.

**kwargs does not compile

problem step

import_expression.compile('f(**a)', mode='eval')

traceback

  File "import_expression/__init__.py", line 60, in compile
    source = parse(source, filename=filename, mode=mode)
  File "import_expression/__init__.py", line 55, in parse
    return parse_ast(tree, filename=filename)
  File "import_expression/_parser.py", line 27, in <lambda>
    parse_ast = lambda root_node, **kwargs: ast.fix_missing_locations(Transformer(**kwargs).visit(root_node))
  File "/usr/lib/python3.7/ast.py", line 262, in visit
    return visitor(node)
  File "/usr/lib/python3.7/ast.py", line 326, in generic_visit
    new_node = self.visit(old_value)
  File "/usr/lib/python3.7/ast.py", line 262, in visit
    return visitor(node)
  File "/usr/lib/python3.7/ast.py", line 317, in generic_visit
    value = self.visit(value)
  File "/usr/lib/python3.7/ast.py", line 262, in visit
    return visitor(node)
  File "import_expression/_parser.py", line 134, in visit_arg
    if has_any_import_op(node.arg):
  File "import_expression/_parser.py", line 37, in <lambda>
    has_any_import_op = lambda name: MARKER in name
TypeError: argument of type 'NoneType' is not iterable

comments fail lmao

https://travis-ci.org/bmintz/import-expression-parser/jobs/472693507#L510

line continuations broken in console on latest Python

➤ ie
🐍 def f():
  File "<console>", line 1
    def f():
            ^
IndentationError: expected an indented block after function definition on line 1
🐍
➤ python -V
Python 3.11.0

consider using parso

Parso is a python AST parser/unparser that could eliminate the need for our custom lexer and parser hack. It's an AST that recovers from errors. If this would work for our use case it would simplify the code base a lot.

tests: use pytest parameterization

All of these should use parameterization:

• test_valid_string_literals
• test_invalid_attribute_syntax
• test_del_store_import
• test_invalid_del_store_import
• test_invalid_argument_syntax
• test_invalid_def_syntax
• test_typehint_conversion
• test_import_statement
• test_normal_invalid_syntax

a!.b! parses successfully

It is interpreted as _IMPORT_MODULE('a.b'). It should raise SyntaxError.

urllib.parse!.<TAB> does not work

support tracebacks with line contents and line offsets

Currently tracebacks only have line and column number attached. Modify _parser and _syntax to store the original program text so that error messages will be more helpful