pfalcon/utemplate

Micro template engine in Python with low memory usage, designed for Pycopy, a minimalist Python dialect, but also compatible with other Pythons.

utemplate

utemplate is a lightweight and memory-efficient template engine for Python, primarily designed for use with Pycopy, a lightweight Python implementation (https://github.com/pfalcon/pycopy). It is also fully compatible with CPython and other compliant Python implementations.

utemplate syntax is roughly based on Django/Jinja2 syntax (e.g. {% if %}, {{var}}), but only the most needed features are offered (for example, "filters" ({{var|filter}}) are syntactic sugar for function calls, and so far are not planned to be implemented, function calls can be used directly instead: {{filter(var)}}).

utemplate compiles templates to Python source code, specifically to a generator function which, being iterated over, produces consecutive parts (substrings) of the rendered template. This allows for minimal memory usage during template substitution (with Pycopy, it starts from mere hundreds of bytes). Generated Python code can be imported as a module directly, or a simple loader class (utemplate.compiled.Loader) is provided for convenience.

There is also a loader class which will compile templates on the fly, if not already compiled - utemplate.source.Loader.

Finally, there's a loader which will automatically recompile a template module if source template is changed - utemplate.recompile.Loader. This loader class is the most convenient to use during development, but on the other hand, it performs extra processing not required for a finished/deployed application.

To test/manage templates, utemplate_util.py tool is provided. For example, to quickly try a template (assuming you are already in examples/ dir):

pycopy ../utemplate_util.py run squares.tpl

or

python3 ../utemplate_util.py run squares.tpl

Templates can take parameters (that's how dynamic content is generated). Template parameters are passed as arguments to a generator function produced from a template. They also can be passed on the utemplate_util.py command line (arguments will be treated as strings in this case, but can be of any types if called from your code):

pycopy ../utemplate_util.py run test1.tpl foo bar

Quick Syntax Reference

Evaluating Python expression, converting it to a string and outputting to rendered content:

• {{<expr>}}

Where expr is an arbitrary Python expression - from a bare variable name, to function calls, yield from/await expressions, etc.

Supported statements:

• {% args <var1>, <var2>, ... %} - specify arguments to a template (optional, should be at the beginning of a template if you want to pass any arguments). All argument types as supported by Python can be used: positional and keyword, with default values, *args and **kwargs forms, etc.
• {% if <expr> %}, {% elif <expr> %}, {% else %}, {% endif %} - similar to Python's if statement
• {% for <var> in <expr> %}, {% endfor %} - similar to Python's for statement
• {% while <expr> %}, {% endwhile %} - similar to Python's while statement
• {% set <var> = <expr> %} - assignment statement
• {% include "name.tpl" %} - statically include another template
• {% include {{name}} %} - dynamically include template whose name is stored in variable name.

File Naming Conventions

• The recommended extension for templates is .tpl, e.g. example.tpl.
• When template is compiled, dot (.) in its name is replaced with underscore (_) and .py appended, e.g. example_tpl.py. It thus can be imported with import example_tpl.
• The name passed to {% include %} statement should be full name of a template with extension, e.g. {% include "example.tpl" %}.
• For dynamic form of the include, a variable should similarly contain a full name of the template, e.g. {% set name = "example.tpl" %} / {% include {{name}} %}.

Examples

examples/squares.tpl as mentioned in the usage examples above has the following content:

{% args n=5 %}
{% for i in range(n) %}
| {{i}} | {{"%2d" % i ** 2}} |
{% endfor %}

More examples are available in the examples/ directory.

If you want to see a complete example web application which uses utemplate, refer to https://github.com/pfalcon/notes-pico .

License

utemplate is written and maintained by Paul Sokolovsky. It's available under the MIT license.

utemplate's Issues

Cannot run include.tpl

Was trying to test out some include logic. I can successfully run the more complex test...

$ python3 ./utemplate_util.py run examples/test1.tpl "something" "else"

...but if I try to run "include.tpl" it triggers this error...

$ python3 ./utemplate_util.py run examples/include.tpl 
Traceback (most recent call last):
  File "./utemplate_util.py", line 45, in <module>
    c.compile()
  File "/home/cefn/Documents/shrimping/git/utemplate/utemplate/source.py", line 128, in compile
    self.parse_line(l)
  File "/home/cefn/Documents/shrimping/git/utemplate/utemplate/source.py", line 105, in parse_line
    self.parse_statement(stmt)
  File "/home/cefn/Documents/shrimping/git/utemplate/utemplate/source.py", line 61, in parse_statement
    with open(self.loader.file_path(tokens[0][1:-1])) as inc:
AttributeError: 'NoneType' object has no attribute 'file_path'

What is the proper invocation of a template which might draw upon the loader argument?

P.S. Probably not related, but trying to use the compile or render commands create different errors...

$ python3 ./utemplate_util.py compile examples/include.tpl 
Traceback (most recent call last):
  File "./utemplate_util.py", line 33, in <module>
    render = loader.load(sys.argv[2])
  File "/home/cefn/Documents/shrimping/git/utemplate/utemplate/source.py", line 163, in load
    return super().load(name)
  File "/home/cefn/Documents/shrimping/git/utemplate/utemplate/compiled.py", line 13, in load
    return __import__(self.p + name, None, None, (name,)).render
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 954, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 896, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1148, in find_spec
  File "<frozen importlib._bootstrap_external>", line 947, in __init__
  File "<frozen importlib._bootstrap_external>", line 962, in _get_parent_path
KeyError: ''

$ python3 ./utemplate_util.py render examples/include.tpl 
Traceback (most recent call last):
  File "./utemplate_util.py", line 37, in <module>
    render = loader.load(sys.argv[2])
  File "/home/cefn/Documents/shrimping/git/utemplate/utemplate/compiled.py", line 13, in load
    return __import__(self.p + name, None, None, (name,)).render
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 954, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 896, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1148, in find_spec
  File "<frozen importlib._bootstrap_external>", line 947, in __init__
  File "<frozen importlib._bootstrap_external>", line 962, in _get_parent_path
KeyError: ''

Reuse of included templates

I was surprised to learn the {% include ... %} does not generate a separate function. If I include the same file into two other templates, the generated code does not get reused but is duplicated. There also does not seem to be a way to pass variables (or context) into an included file, which creates a problem for my usual nav-bar style menus that highlight the currently active page.

Installation fails under pipenv

Installation fais with pipenv over Python 3.8 and pip 20.3.1:

Installing utemplate...
Error:  An error occurred while installing utemplate!
Error text: Collecting utemplate
  Using cached utemplate-1.4.tar.gz (4.1 kB)

    ERROR: Command errored out with exit status 1:
     command: /Users/federico/.local/share/virtualenvs/notification-consumer-dispatcher-wU5PfN33/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/r3/2_zddds10zzdvx9sy_3p2stw0000gn/T/pip-install-2t38l5ez/utemplate_f50c993c62e84c119c9ab07e6441f709/setup.py'"'"'; __file__='"'"'/private/var/folders/r3/2_zddds10zzdvx9sy_3p2stw0000gn/T/pip-install-2t38l5ez/utemplate_f50c993c62e84c119c9ab07e6441f709/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/r3/2_zddds10zzdvx9sy_3p2stw0000gn/T/pip-pip-egg-info-dokffrvc
         cwd: /private/var/folders/r3/2_zddds10zzdvx9sy_3p2stw0000gn/T/pip-install-2t38l5ez/utemplate_f50c993c62e84c119c9ab07e6441f709/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/tokenize.py", line 392, in open
        buffer = _builtin_open(filename, 'rb')
    FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/r3/2_zddds10zzdvx9sy_3p2stw0000gn/T/pip-install-2t38l5ez/utemplate_f50c993c62e84c119c9ab07e6441f709/setup.py'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

This is likely caused by a bug in utemplate. Report this to its maintainers.
✘ Installation Failed

os and io imports do not match established Micropython naming conventions

Hi, and thanks for utemplate. Potentially really promising for a project I'm working on, unless you think there is a better candidate for python-oriented templating on ESP8266 Micropython

To get this version of utemplate to run the example template squares.tpl, I had to make the following changes so that _io was satisfied instead by uio and os satisfied by uos meaning that it could execute against micropython 1.8.7 on unix...

diff --git a/utemplate_util.py b/utemplate_util.py
index 348e8eb..1129264 100644
--- a/utemplate_util.py
+++ b/utemplate_util.py
@@ -1,6 +1,6 @@
 import sys
-import os
-import _io as io
+import uos as os
+import uio as io
 import utemplate.source
 import utemplate.compiled

After this, I successfully generated the example templated text....

| 0 |  0 |
| 1 |  1 |
| 2 |  4 |
| 3 |  9 |
| 4 | 16 |

Immediate inclusion breaks predictability of function names in compiled templates

This is demonstrated by https://github.com/cefn/utemplate/blob/demo_render_breakage/examples/immediateinclude.tpl which immediately hands off to another template, without sending even a character beforehand.

The compiled code from...
python3 ./utemplate_util.py run examples/immediateinclude.tpl
...looks as follows, with no render() function and two render1() functions...

# Autogenerated file
# Autogenerated file
def render1(*a, **d):
    for i in range(5):
        yield """| """
        yield str(i)
        yield """ | """
        yield str("%2d" % i ** 2)
        yield """ |
"""
def render1(*a, **d):
    yield from render1()

It is therefore impossible to run the template and utemplate_util.py predictably dies with...

Traceback (most recent call last):
  File "./utemplate_util.py", line 48, in <module>
    for x in ns["render"](*sys.argv[3:]):
KeyError: 'render'

Trouble running the PicoWeb examples when using utemplate

I have frozen PicoWeb (and uasyncio) into and ESP8266 firmware bin. Then I added utemplate, the templates directory with squares.tpl and example_webapp.py. On first run I get this...

`
33.000 <HTTPRequest object at 3fff36f0> <StreamWriter > "GET /squares"

33.000 <HTTPRequest object at 3fff36f0> <StreamWriter > AttributeError("'module' object has no attribute 'path'",)
Traceback (most recent call last):

File "picoweb/init.py", line 188, in _handle

File "example_webapp.py", line 22, in squares

File "picoweb/init.py", line 231, in render_template

File "picoweb/init.py", line 227, in _load_template

File "utemplate/source.py", line 153, in init

AttributeError: 'module' object has no attribute 'path'
`

If I add a path declaration to example_webapp.py I get this...

`
133.999 <HTTPRequest object at 3fff4c80> <StreamWriter > "GET /squares"

136.000 <HTTPRequest object at 3fff4c80> <StreamWriter > ImportError("no module named 'example_webapp.templates'",)
Traceback (most recent call last):

File "picoweb/init.py", line 188, in _handle

File "example_webapp.py", line 24, in squares

File "picoweb/init.py", line 231, in render_template

File "picoweb/init.py", line 228, in _load_template

File "utemplate/source.py", line 182, in load

File "utemplate/compiled.py", line 14, in load

ImportError: no module named 'example_webapp.templates'
`

Everything else works as expected so I think the problem is specifically with utemplate.

Template Inheritance

I tried to use template inheritance, but it does not work. Is template inheritance implemented in utemplate? If yes, how to use it?

Reuse of include functions

I was surprised to learn the {% include ... %} does not generate a separate function. If I include the same file into two other templates, the generated code does not get reused but is duplicated. There also does not seem to be a way to pass variables (or context) into an included file, which creates a problem for my usual nav-bar style menus that highlight the currently active page.

utemplate documentation

where can i find documentation of the utemplate?

Undocumented test1.tpl behaviour

Having successfully tested and run the example squares.tpl I tried a more advanced example test1.tpl like...

$ micropython ./utemplate_util.py run examples/test1.tpl

Although the compile stage seemed to properly complete (generating the code inlined below) attempting to run the compliled code through the utemplate_util.py test harness resulted in the following error.

$ micropython ./utemplate_util.py run examples/test1.tpl 
Traceback (most recent call last):
  File "./utemplate_util.py", line 38, in <module>
TypeError: render() takes 2 positional arguments but 0 were given

Of course it turned out that test1.tpl was expecting additional arguments (as implied by the positional arguments in the generated code) and passing these to the command line led to a correct run.

$ micropython ./utemplate_util.py run examples/test1.tpl "varValue" "anotherValue"

# Autogenerated file
def render(var, another):
    yield """hello
"""
    for i in range(5):
        if i % 2:
            yield """world
"""
        elif i % 3 == 0:
            yield """something else
"""
        else:
            yield """skies
"""
        yield """text"""
        yield str(i + 10)
        yield """
"""
    yield """trailer
"""

Packing of HTML and/or compression of longer HTML sections w/ zlib

I would like my templates to be "packed" as they are compiled. This would be for the offline compiling mode (never on-device)... so, spaces and blanks would be removed from the HTML template just before it becomes bytecode. Unfortunately, reliably HTML compression (packing) is hard to get right, and in this case, we have template logic in there as well.

It would be hard to do, but it's a big win because I only need the extra whitespace in the HTML template for my personal comprehension during development.

What might be easier to implement, and would achieve (IMHO) similar compression rates, would be to run longer HTML sections through zlib. There's no need to do this for short strings, but when you have a wall of HTML, it will get good compression rates.

compile() failure when run via mpremote mount on Pico

I ran into am issue when using utemplate on a Raspberry Pi Pico. I was using the mpremote command with the mount option. Something like

mpremote mount . run main.py

If you are nor familiar with that, it has the Pico mount a directory ('.' in this case) on the host and then run code from there. Using this you can run your code without having to upload to the Pico first.

It fails in source.py at the

for l in self.file_in:

line with an

TypeError: 'RemoteFile' object isn't iterable.

I presume that files opened on the host are of type RemoteFile and that you can't just iterate over each line like it was a local file. I quick fix is to make the following change

     def compile(self):
         self.header()
-        for l in self.file_in:
-            self.parse_line(l)
+        try:
+            for l in self.file_in:
+                self.parse_line(l)
+        except TypeError:
+            lines = self.file_in.readlines()
+            for l in lines:
+                self.parse_line(l)
         self.close_literal()
         return self.seq

It is a bit slow but it works -- maybe I should just always upload to the Pico :-)

Due to the above error, it created but did not finish writing the .py version of the template file (it only had the initial comment line). So later runs try to import it and they fail in compiled.py when trying to access the .render attribute AttributeError: 'module' object has no attribute 'render'. It might be helpful to others in the future, it that could be caught and a better error presented.

Feature Request - Parameterising Template Names in Include Directives

Hi again.

Been using your templating framework with great success, but have hit a brick wall on a specific feature which is the last I need to resolve to control memory overhead for our application.

Hopefully the feature I'm proposing makes some sense. I get very lost in the utemplate code so I'm not optimistic being able to translate the feature request into a patch without some help.

I think it cashes out as needing the names of included templates to be able to be computed themselves by runtime values. It's not the full power of 'eval' (which would be incompatible with the design intent of utemplate), but it would allow runtime state to parameterise the call to the template resolver, e.g. when iterating over lists or dicts.

This is not eval, every include would still end up backed by a named template from the resolver, with the potential that it hits a cache (e.g. a pre-compiled frozen module)). However, my workaround DOES effectively use eval, and is horrible.

Roughly speaking, instead of being limited to just...

{% include 'subpage' %}

...you could have extra expressive power by having computable template names...

{% for key in keys %}{% include 'page{{key}}' %}{% endfor %}

BACKGROUND

Our templating strategy is illustrated by this line from https://github.com/cefn/avatap/blob/master/python/milecastles.py which defines arguments available to all render functions
signature = "engine, story, box, node, card, sack"

This signature provides access to the text adventure engine, the whole story, the box the player has tapped on (these are RFID-sensing boxes in a museum), the node the player is at in the story, the card information of the player, and the player's sack (an inventory 'dict' loaded from their RFID)...

REQUIREMENT

The application requirement is illustrated by the (working) templates in the code-fragment under EVIL NESTED EVAL CODE below. To provide template authors with the needed features for our text adventure game, I have been forced to 'step outside' the template logic, and call the templating engine directly from within the render function. This implies a bit of an explosion of memory overhead, since it means eval is being called within eval! It also means that the embedded template is always evaled based on a runtime string and therefore can't be cached.

The motivation for the case is that authors write readable text used to render each choice you can make in the adventure (of one-to-many choices named by uid). However, these texts in turn use templating features, so they can't merely be static text, they should be loaded through the resolver/compiler with include.

Typically these parameterised template names will depend on a key from a Jinja2-style iteration. Our issue is that while the resolver can be augmented to handle special names like 'choices[choiceUid]' or 'choices[2]' or just 'choices_7' and can grab different backing templates in each parameterised case, there's no way to pass these values through include currently. Consequently I am using a reference to the templating engine directly within the template, and I am simply passing the template text (which can be accessed via Jinja2) to the engine from within the render function!

EVIL NESTED EVAL CODE

    choiceList = " {% for choiceUid in node.choiceNodeUids %}{% if not(node.isHidden(engine, choiceUid)) %}{% include 'choiceItem' choiceUid %}\n{% endif %}{% endfor %}"
    choiceItem = " {% args choiceUid %}{{ engine.concatenateGeneratedStrings(node, node.choices[choiceUid]) }} : {{story.lookupNode(choiceUid).getGoalBox(story).label}}"

This is so wrong, but it's ended up unavoidably unpacking this way. I wonder whether parameterised includes might offer important extra power without surrendering the efficiency/low overhead of cached generators. This would certainly fix our immediate problem.

DESIGN ISSUE

Of course, all of this implies that the resolver plays a part in the runtime execution of a render() function. I don't know if this is currently the case, or whether all includes are actually expanded by simply inlining at compile time. If all includes are always inlined, could it make sense to reverse this design decision, and keep the resolver in the loop even at runtime to get this extra power? It would also minimise duplication from embedding the same templates inline in multiple places.

Comments in templates

Sure would be nice to have comments in templates. Jinja-style {# ... #} style.

At first, single-line comments... that would easy, just a "regex and done". But I also use them for commenting-out sections, so really they need to support multiple lines.

no module named utemplate

im trying to run this code:
https://github.com/donskytech/micropython-wifi-robot-car

I am getting an error: ImporErrpr: no module named utemplate.
I have installed all the required libraries and tried it on multiple systems, yet this problem persists.
I think i am still missing something, please help me through it.

RFC: Dropping "compiled." subpackage prefix for compiled templates

The idea was to organize files more cleanly, but it only complicates matters, because it's unclear where that subpackage should reside: as the top-level subpackage, or in a particular template subdir, etc.

Dropping it, there should not be such ambiguities.

Another reason is to have cleaner mapping from template names to module names and to avoid fragmenting memory by constantly concatenating "compiled." in loader. So, with the new scheme, if there's a template

templates/group/my_temp_late

it will map to Python module name

templates.group.my_temp_late

To extend this a bit further, now "native" mode for naming templates are defined, where template name can't contain a dot (thus, no file extensions), ditto for directory components of a template. And component separator used then is ".". In other words, the native template name matches Python module name for the template. By following native naming, memory needed to look up template is minimized.

Then there's "compatibility" mode, where template names may contain dot/have extension, and they're addressed with directory separators. But looking up such templates may require extra memory.

Including dynamic template

Hello,
To include a dynamic template, the name of template do not have "underscore" character.
Ex. not working: nome_template.tpl
Ex working well: nome-template.tpl

But in any case, it's not working:
Work well include the static template {% include 'menu.html' %}
But not working the include dynamic template {% include {{body}} %} -> {{body}} result is menu.html

What is the exact syntax to use?
Thank you
Luca