pfalcon/picoweb

Really minimal web application framework for the Pycopy project (minimalist Python dialect) and its "uasyncio" async framework

picoweb

picoweb is a "micro" web micro-framework (thus, "pico-framework") for radically unbloated web applications using radically unbloated Python implementation, Pycopy, https://github.com/pfalcon/pycopy .

Features:

• Asynchronous from the start, using unbloated asyncio-like library for Pycopy (uasyncio). This means that picoweb can process multiple concurrent requests at the same time (using I/O and/or CPU multiplexing).
• Small memory usage. Initial version required about 64K of heap for a trivial web app, and since then, it was optimized to allow run more or less realistic web app in ~36K of heap. More optimizations on all the levels (Pycopy and up) are planned (but may lead to API changes).
• Has API affinity with some well-known Python web micro-framework(s), thus it should be an easy start if you have experience with them, and existing applications can be potentially ported, instead of requiring complete rewrite.

Requirements and optional modules

picoweb depends on uasyncio for asynchronous networking (https://github.com/pfalcon/pycopy-lib/tree/master/uasyncio). uasyncio itself requires Pycopy <https://github.com/pfalcon/pycopy>, a minimalist, lightweight, and resource-efficient Python language implementation.

It is also indended to be used with utemplate (https://github.com/pfalcon/utemplate) for templating, but this is a "soft" dependency - picoweb offers convenience functions to use utemplate templates, but if you don't use them or will handle templating in your app (e.g. with a different library), it won't be imported.

For database access, there are following options (picoweb does not depend on any of them, up to your application to choose):

• btree builtin Pycopy module. This is a recommended way to do a database storage for picoweb, as it allows portability across all Pycopy targets, starting with very memory- and storage-limited baremetal systems.
• btreedb wrapper on top of btree builtin module. This may add some overhead, but may allow to make an application portable between different database backends (filedb and uorm below). https://github.com/pfalcon/pycopy-btreedb
• filedb, for a simple database using files in a filesystem https://github.com/pfalcon/filedb
• uorm, for Sqlite3 database access (works only with Pycopy Unix port) https://github.com/pfalcon/uorm

Last but not least, picoweb uses a standard logging-compatible logger for diagnostic output (like a connection opened, errors and debug information). However this output is optional, and otherwise you can use a custom logging class instead of the standard logging/ulogging module. Due to this, and to not put additional dependencies burden on the small webapps for small systems, logging module is not included in picoweb's installation dependencies. Instead, a particular app using picoweb should depend on pycopy-ulogging or pycopy-logging package. Note that to disable use of logging, an application should start up using WebApp.run(debug=-1). The default value for debug parameter is 0 however, in which case picoweb will use ulogging module (on which your application needs to depend, again).

Details

picoweb API is roughly based on APIs of other well-known Python web frameworks. The strongest affinity is Flask, http://flask.pocoo.org, as arguably the most popular micro-framework. Some features are also based on Bottle and Django. Note that this does not mean particular "compatibility" with Flask, Bottle, or Django: most existing web frameworks are synchronous (and threaded), while picoweb is async framework, so its architecture is quite different. However, there is an aim to save porting efforts from repetitive search & replace trials: for example, when methods do similar things, they are likely named the same (but they may take slightly different parameters, return different values, and behave slightly differently).

The biggest difference is async, non-threaded nature of picoweb. That means that the same code may handle multiple requests at the same time, but unlike threaded environment, there's no external context (like thread and thread local storage) to associate with each request. Thus, there're no "global" (or thread-local "global") request and response objects, like Flask, Bottle, Django have. Instead, all picoweb functions explicitly pass the current request and response objects around.

Also, picoweb, being unbloated framework, tries to avoid avoidable abstractions. For example, HTTP at the lowest level has just read and write endpoints of a socket. To dispatch request, picoweb needs to pre-parse some request data from input stream, and it saves that partially (sic!) parsed data as a "request" object, and that's what passed to application handlers. However, there's no unavoidable need to have a "response" abstraction - the most efficient/lightweight application may want to just write raw HTTP status line, headers, and body to the socket. Thus, raw write stream is passed to application handlers as the "response" object. (But high-level convenience functions to construct an HTTP response are provided).

API reference

The best API reference currently are examples (see below) and the picoweb source code itself. It's under 10K, so enjoy: https://github.com/pfalcon/picoweb/blob/master/picoweb/__init__.py

Note that API is experimental and may undergo changes.

Examples

• example_webapp.py -A simple webapp showing you how to generate a complete HTTP response yourself, use picoweb convenience functions for HTTP headers generation, and use of templates. Mapping from URLs to webapp view functions ("web routes" or just "routes") is done Django-style, using a centralized route list.
• example_webapp2.py -Like above, but uses app.route() decorator for route specification, Flask-style.
• examples/ -Additional examples for various features of picoweb. See comments in each file for additional info. To run examples in this directory, you normally would need to have picoweb installed (i.e. available in your MICROPYPATH, which defaults to ~/.micropython/lib/).
• notes-pico - A more realistic example webapp, ported from the Flask original.

Running under CPython (regressed)

Initial versions of picoweb could run under CPython, but later it was further optimized for Pycopy, and ability to run under CPython regressed. It's still on TODO to fix it, instructions below tell how it used to work.

At least CPython 3.4.2 is required (for asyncio loop.create_task() support). To run under CPython, uasyncio compatibility module for CPython is required (pycopy-cpython-uasyncio). This and other dependencies can be installed using requirements-cpython.txt:

pip install -r requirements-cpython.txt

Reporting Issues

Here are a few guidelines to make feedback more productive:

1. Please be considerate of the overall best practices and common pitfalls in reporting issues, this document gives a good overview: How to Report Bugs Effectively.
2. The reference platform for picoweb is the Unix port of Pycopy. All issues reported must be validated against this version, to differentiate issues of picoweb/uasyncio from the issues of your underlying platform.
3. All reports must include version information of all components involved: Pycopy, picoweb, uasyncio, uasyncio.core, any additional modules. Generally, only the latest versions of the above are supported (this is what you get when you install/reinstall components using the upip package manager). The version information are thus first of all important for yourself, the issue reporter, it allows you to double-check if you're using an outdated or unsupported component.
4. Feature requests: picoweb is by definition a pico-framework, and bound to stay so. Feature requests are welcome, but please be considerate that they may be outside the scope of core project. There's an easy way out though: instead of putting more stuff into picoweb, build new things on top of it: via plugins, subclassing, additional modules etc. That's how it was intended to be from the beginning!
5. We would like to establish a dedicated QA team to support users of this project better. If you would like to sponsor this effort, please let us know.

picoweb's Issues

url parameters

Hi;

I am unable to successfully define a regex to handle url GET parameters such ads:

http://mydevice/?x=1&y=2&z=3

def json(req,resp):
  yield from picoweb.jsonify(resp,{"error":True})  

ROUTES = [
  (re.compile("^/\?.*"), json),
  ("/", lambda req, resp: (yield from app.sendfile(resp, path+"index.htm"))),
  (re.compile("^/(.+)"), lambda req, resp: (yield from app.sendfile(resp, path + req.url_match.group(1)))),
]   

I have attempted to look for the path starting with "/?" - what am I doing wrong ??

stream writer bigger than buffer

Hi, congratulations on the great work with picoweb.

I'm trying to use the render_template method and an AssertionError error is occurring, in the uasyncio library more precisely in the awrite method, there is a check of the size of the buffer size of the StreamWriter which occurs is that the template I'm trying to render is larger than the buffer , Is there any way around this?

Size Stream: 2456 bytes
Buffer: 2455 bytes

• Running on http://127.0.0.1:8081/
  Size Stream: 2457 and size buffer: 2455
  Traceback (most recent call last):
  File "server.py", line 16, in
  File "/home/murilobsd/.micropython/lib/picoweb/init.py", line 240, in run
  File "/home/murilobsd/.micropython/lib/uasyncio/core.py", line 121, in run_forever
  File "/home/murilobsd/.micropython/lib/uasyncio/core.py", line 85, in run_forever
  File "/home/murilobsd/.micropython/lib/picoweb/init.py", line 156, in _handle
  File "server.py", line 13, in home
  File "/home/murilobsd/.micropython/lib/picoweb/init.py", line 196, in render_template
  File "/home/murilobsd/.micropython/lib/uasyncio/init.py", line 155, in awrite
  AssertionError:

I'm using:

• ESP8266
• MicroPython v1.9-21-ga0dbbbeb-dirty on 2017-06-04; ESP module with ESP8266
• picoweb: 1.1.1
• uasyncio: 1.1.2
• utemplate: 1.1

Does picoweb support async handlers? If so, document this

I want to do some async operations in a handler eg. call an API, process and return the result. Would picoweb support something like :

@app.route("/temp")
async def temp(req, resp):       # an async handler
    temp = await httpClient.get('/status')
    yield from picoweb.start_response(resp)
    yield from resp.awrite("Temperature: "+temp['temperature'])

If so, please add an async example.

It seems that when a picoweb calls awrite, it is using StreamWriter from uasyncio, and the comment on the awrite method says it is a coroutine and may not return immediately, so awaiting it makes sense.
But picoweb handlers are called as generators ("yield from") but not awaited, so cannot be async ?
The word async hardly exists in the whole repository.
Apologies if I'm missing something, I'm new to Micropython.

App.run()

Calling app.run(...) gives an error regarding a logging dependency. Looking through the code, the import of the logging library is not of any use, so it would be nice to simply remove it, as it also does not get installed with the picoweb package and gives another error if you try to use it without installing logging manually.

I found a solution giving the parameter "debug" a value of '-1' when calling the run() method, so the condition breaks and the logging is never used.

picoweb memory leak

picoweb _handle() method will leak memory (rapidly) when yielded socket reads or writes return exceptions. Adding read.aclose() and write.aclose() to the exception exit code path in _handle() fixes this neatly.

Note: as-is, uasyncio does not return socket connection failures/errors. So, POLL_HUP and POLL_ERR events are not returned from the poller ioctl calls. As-is, bad sockets will hang picoweb co-routines and leak memory with yielded co-routines that never get data. Adding POLL_HUP and POLL_ERR to uasyncio poller read and write calls allows modwip.c to return these error events. However, modlwip.d ioctl really should be returning these unsolicited.

picoweb also does not handle socket errors thrown by uasyncio start_server() s.accept(). This will crash picoweb. Adding exception handling to re-start the server isn't ideal - its is preferred to add socket accept exception handling within the uasyncio start_server() loop so the server can ignore a bad connection attempt and return to waiting for new connections.

The modifications mentioned above were implemented and demonstrated to dramatically tolerate a high rate of bad socket connections while successfully serving requests to picoweb apps. Without these changes, a picoweb app and micropython can be crashed in seconds. Even by simply pressing F5 on a windows explorer browser GET request.

Error installing 'micropython-uasyncio': , packages may be partially installed

Was working a few days ago...

MicroPython v1.9.1-8-g7213e78d on 2017-06-12; ESP module with ESP8266
Type "help()" for more information.
>>> import upip
>>> upip.install("picoweb")
Installing to: /lib/
Warning: pypi.python.org SSL certificate is not validated
Installing picoweb 1.2 from https://pypi.python.org/packages/07/a1/ac79de53d38d6ae531c857f3d9291afaa27d90c58eca9a1d295d99f7546a/picoweb-1.2.tar.gz
Error installing 'micropython-uasyncio': , packages may be partially installed
>>> import uasyncio
>>> import picoweb
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/lib/picoweb/__init__.py", line 8, in <module>
ImportError: no module named 'pkg_resources'

Picoweb is eating memory: How to debug?

Hi,

I'm using the git version of picoweb running on an ESP32. Everything works smoothly. I do have a second ESP32 which posts data every 6 minutes to the "server". Everything looks good.

However, over time the "server" is showing less and less memory available. I've printed some debug messages with gc.mem_free().

This comes to a point where the server is not responding anymore. If you use telnet to connect to port 80 of the ESP "server" the connection is closed instantly again. No debug messages, no error messages.

There are still around 27000 bytes left of memory, so it's not that low.

I do have other functions running with uasyncio.create_task. The interesting fact is, that the server is not eating any memory if here is no web-interaction. If it just run the internal routines everything is fine and runs forever, but as soon as I start frequent server communication things are getting worse.

The function that is called frequently by the server is this:

def web_batteryslave(req, resp):
    gc.collect()
    print('%%% Web Start: Free Memory: ', gc.mem_free())
    method = req.method

    if req.method == "POST":
        size = int(req.headers[b"Content-Length"])
        data = yield from req.reader.read(size)
        #print('$$$ Data:    ', data)
        try:
                decoded = ujson.loads(str(data)[2:-1])
                yield from picoweb.start_response(resp)
                yield from resp.awrite("OK")
                yield from resp.awrite("\r\n")
                thisId = decoded['id'];
                # Save data from this sensor to measurements
                toolbox.measurements[thisId] = decoded
                print('%%% WEB End: Free Memory: ', gc.mem_free())
                return
        except:
                yield from picoweb.start_response(resp)
                yield from resp.awrite("NOK")
                yield from resp.awrite("\r\n")

The data in 'toolbox.measurements' is "just" updated every time, there is no new data added. The same functionality with storing the data works fine with local sensors without an memory leaks.

I've see other issues here and the discussion if the connection needs to be closed or not.

So how can this be debugged?

cannot read form data on ESP8266

I have been trying to read post data,

My code in MicroPython is

import picoweb
import uasyncio as asyncio

def info(req, resp):
----yield from picoweb.start_response(resp)
----yield from resp.awrite("Alrught")
----req.read_form_data()
----print(req.form)

ROUTES = [
("/info", info),
]
import logging
logging.basicConfig(level=logging.INFO)
app = picoweb.WebApp(__name__, ROUTES)
app.run(port=80,host="0.0.0.0",debug=1)

the server runs without hassles,
but problem is I am unable read post data, the error I keep getting is,

AttributeError: 'HTTPRequest' object has no attribute 'form'

I tried the parse_qs() and then access form it works, but same won't work for read_form_data()

I don't understand what is going wrong.

Does not work with logging from micropython-lib

The Logger class from micropython-lib/logging does not provide .exc(). Maybe switch to warning() or critical()?

cannot upip.install on nodemcu

Hi, I'm not sure if this is the correct place to raise this issue so feel free to let me know if now. Thanks for Picoweb, it looks to be exactly what I am after!

I have an ESP8266 flashed with the latest version of Micropython (1.8.5). On a fresh flash attempting to upip install picoweb generates an error:

`

import upip
upip.install('picoweb')
Installing to: /lib/
Traceback (most recent call last):
File "", line 1, in
File "upip.py", line 202, in install
File "upip.py", line 192, in install
File "upip.py", line 149, in install_pkg
File "upip.py", line 138, in get_pkg_metadata
File "upip.py", line 105, in url_open
OSError: -2
`

If you have any suggestions, or if I am doing something wrong please let me know.

Thanks,
Jon

Typo in picoweb/__init__.py

I think, dd68a0d introduced a error (typo) in line 50 of picoweb/__init__.py. h is supposed to be named k in order for the following lines to make sense afaics. (Or, of course, the other way around...)

picoweb/picoweb/__init__.py

Line 50 in dd68a0d

for h, v in headers.items():

Edit: Pull request for your convenience :-)

Warning: getaddrinfo constraints not supported on ESP8266

Using git and your fork of micropython, tried on two different esp8266

boot.py file

# This file is executed on every boot (including wake-boot from deepsleep)
import esp
esp.osdebug(None)
import gc
#import webrepl
#webrepl.start()
gc.collect()

main.py file

import network

def conectar():
    sta_if = network.WLAN(network.STA_IF)
    ap_if = network.WLAN(network.AP_IF)
    if not sta_if.isconnected():
        print('connecting to network...')
        sta_if.active(True)
        sta_if.connect('SSID', 'pass')
        while not sta_if.isconnected():
            pass
    print('network config:', sta_if.ifconfig())
    ap_if.active(False)

if __name__ == '__main__':
    conectar()
    import picoweb

    site = picoweb.WebApp(__name__)

    @site.route("/")
    def index(req, resp):
        yield from picoweb.start_response(resp)
        yield from resp.awrite("<h1>Hola</h1>")

    site.run(host="192.168.1.5", debug=True)

Nothing is showed when i've tried to access from web browser.
Console shows:

connecting to network...
network config: ('192.168.1.5', '255.255.255.0', '192.168.1.1', '208.67.222.123')
* Running on http://192.168.1.5:8081/
Warning: getaddrinfo constraints not supported

Modules directory on micropython:

ls modules/
apa102.py  dht.py      ffilib.py     inisetup.py  logging.py   ntptime.py  picoweb           port_diag.py  uasyncio  upip_utarfile.py  webrepl_setup.py
_boot.py   ds18x20.py  flashbdev.py  json         neopixel.py  onewire.py  pkg_resources.py  re.py         upip.py   webrepl.py        websocket_helper.py

Support for MAX-AGE in the response headers

I'm not sure if this is how it should be accomplished, but I would like specific responses to be cached by the client browser if possible. Javascript and style sheets could benefit by only having to be downloaded once.

Websockets support

Is there some hack how to start websockets endpoint on some path?

Add Cache-Control header

The HTML header "Cache-Control" might be useful for bigger static content.

A minimal version has already been implemented in https://github.com/joewez/picoweb/pull/1

I propose an even more minimal approach that does not include his changes in get_mime_type():

def sendfile(self, writer, fname, content_type=None, max-caching-age=0):

Merge request will follow.

Server crashes as the route is accessed

I have a very simple picoserver setup:

import picoweb
app = picoweb.WebApp(__name__)

@app.route("/")
def index(req, resp):
    yield from picoweb.start_response(resp)
    yield from resp.awrite("This is webapp #2")

app.run(debug=True, current_ip)

Then I see:

• Running on http://192.168.1.125:8081/

So the webserver did start. However, as soon as I click the link and access it in the browser. I get the crash:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/lib/picoweb/__init__.py", line 298, in run
  File "/lib/uasyncio/core.py", line 155, in run_forever
  File "/lib/uasyncio/core.py", line 130, in run_forever
  File "/lib/uasyncio/__init__.py", line 60, in remove_writer
TypeError: function takes 2 positional arguments but 3 were given

I am running this on a ESP32 using the latest esp32-20190113-v1.9.4-779-g5064df207.bin firmware.

add_reader/add_writer/remove_writer crash when using not using Pycopy, as otherwise required by README

Hi,

not sure if this belongs in Picoweb or in uasyncio - but when I try to run the example webserver I get the following crashlog:

Traceback (most recent call last):
  File "main.py", line 30, in <module>
  File "/lib/picoweb/__init__.py", line 298, in run
  File "/lib/uasyncio/core.py", line 155, in run_forever
  File "/lib/uasyncio/core.py", line 130, in run_forever
  File "/lib/uasyncio/__init__.py", line 60, in remove_writer
  TypeError: function takes 2 positional arguments but 3 were given

The webserver code isn't anything special, just wait for WiFi and host an app; both a route and a static folder.

I tried upip.install as well as copy & past the latest Picoweb, uasyncio & uasyncio.core from the Github repos.

TypeError: function expected at most 3 arguments, got 4

Hi guys,

I'm running the basic example of the picoweb in a ESP32 with MicroPython v1.12, and I'm getting this error:

Traceback (most recent call last):
File "main.py", line 20, in
File "/lib/picoweb/init.py", line 311, in run
File "/lib/picoweb/init.py", line 293, in serve
File "/lib/uasyncio/core.py", line 129, in run_forever
File "/lib/uasyncio/init.py", line 32, in add_reader
TypeError: function expected at most 3 arguments, got 4

======
File "main.py", line 20, in is "app.run(debug=True)"

Thanks

Consider making utemplate imported on demand

For very simple, light-weight web apps (eg just parse and/or return some json) there is no need for template machinery. There is (almost) one point where templating is used in picoweb, namely the render_template function. If it's never called then utemplate need never be imported (and hence take up lots of RAM). So one can just import utemplate in that funcion and save the template loader object for next time.

Exception on ESP8266

I'm not sure if this is a picoweb, uasyncio, or pycopy problem, leaning more towards pycopy (specifically micropython code), but unfortunately, I'm new to the project and not familiar at all with the sockets stack.

When starting a picoweb server, I'm getting the following exception in the uasyncio thread:

DEBUG:main:network config: ('172.16.1.127', '255.255.255.0', '172.16.1.1', '172.16.1.222')
* Running on http://0.0.0.0:8081/
Traceback (most recent call last):
  File "main.py", line 50, in <module>
  File "picoweb/__init__.py", line 311, in run
  File "picoweb/__init__.py", line 293, in serve
  File "uasyncio/core.py", line 116, in run_forever
  File "uasyncio/__init__.py", line 266, in start_server
OSError: [Errno 11] EAGAIN

This seems to work about 50% of the time if I put an arbitrary sleep (tested with 3s) after my network connection code, so perhaps it's some kind of race condition?

pycopy firmware was built with additional modules using:

wes@w-dock4:~/pycopy/ports/esp8266$ ../unix/pycopy -m upip install -p modules picoweb
Installing to: modules/
Warning: pypi.org SSL certificate is not validated
Installing picoweb from https://files.pythonhosted.org/packages/ba/e4/68983f6150f7fbd98b6b937bf7f76aeb4eb9ba4abb7c93dc05b2ccdbd25c/picoweb-1.8.1.tar.gz
Installing pycopy-uasyncio from https://files.pythonhosted.org/packages/3c/1e/e0953531408f1d237456bcb324b14c1b84218ad7ee5702a3022e7cde7a71/pycopy-uasyncio-3.5.tar.gz
Installing pycopy-pkg_resources from https://files.pythonhosted.org/packages/05/4a/5481a3225d43195361695645d78f4439527278088c0822fadaaf2e93378c/pycopy-pkg_resources-0.2.1.tar.gz
Installing pycopy-uasyncio.core from https://files.pythonhosted.org/packages/ca/b2/c5bba0bde7022b6d927a6144c026d7bf310d3f8a20e031571fbf1a08a433/pycopy-uasyncio.core-2.3.2.tar.gz

Any help would be greatly appreciated! Thanks!

how to handle OSError: [Errno 104] ECONNRESET

how can I handle those, gracefully ?

Traceback (most recent call last):
File "main.py", line 69, in
File "/lib/picoweb/init.py", line 240, in run
File "uasyncio/core.py", line 121, in run_forever
File "uasyncio/core.py", line 85, in run_forever
File "/lib/picoweb/init.py", line 156, in _handle
File "main.py", line 47, in
File "/lib/picoweb/init.py", line 214, in sendfile
File "/lib/picoweb/init.py", line 214, in sendfile
File "/lib/picoweb/init.py", line 209, in sendfile
File "/lib/picoweb/init.py", line 209, in sendfile
File "/lib/picoweb/init.py", line 28, in sendstream
File "uasyncio/init.py", line 144, in awrite
OSError: [Errno 104] ECONNRESET

Connection aborted with large body

I'm using picoweb on a ESP32 to share files between boards and I'm experimenting with the max body size it will handle. Of course I'm reading the request in chunks with the following code:

@app.route("/post")
def post(req, res):
  print("post")
  remaining = int(req.headers[b"Content-Length"])
  print("length", remaining)
  chunk = 128
  file = open("www/post.txt", "w")
  while remaining > 0:
    print("remaining...", remaining)
    to_read = min(remaining, chunk)
    data = yield from req.reader.read(to_read)
    file.write(data)
    remaining -= to_read
  file.close()
  del data
  print("OK")
  yield from picoweb.start_response(res)
  yield from res.awrite("OK")

With large body (15 to 50 kb), the REPL prints "OK" and file gets written well, but the connection gets aborted and the response from curl is

curl: (56) Recv failure: Connection aborted from sender (translated from italian " Connessione interrotta dal corrispondente")`

Of course I would like to get an "OK" response: does anyone have suggestions?

[RFC] Writing a new, even more minimal implementation vs optimizing picoweb (with not backward compatible changes)

There's growing need to make a choice for subj.

My idea: Writing another one only fragments the space. While a complete reimplementation may be needed eventually, doing incremental changes is almost always a better approach.

So, going to introduce a "breaking change" label to kinda pre-announce upcoming refactors: breaking change

Security

¿Is it possible to add basic security check to this webserver? I mean, like a popup when trying to access a certain URL.

Picoweb.run error

Using Picoweb on ESP32 devkit v1: one months ago on same board, all ok; tryed on other 2 identhical board, same error.

Here my simple code to test:

`import picoweb

app = picoweb.WebApp(name)

@app.route("/")
def index(req, resp):
yield from picoweb.start_response(resp)
yield from resp.awrite("Hello world from picoweb running on the ESP32")

app.run(debug=True, host="192.168.2.41")
`

Error after end of booting:

` ...
I (2373) network: CONNECTED
I (5213) event: sta ip: 192.168.2.41, mask: 255.255.255.0, gw: 192.168.2.1
I (5213) network: GOT_IP
:: connecting to network "<<<my_ssid> (try 2)"...
:: WLAN connected to ip 192.168.2.41 gateway 192.168.2.1
I (13773) modsocket: Initializing

• Running on http://192.168.2.41:8081/
  Traceback (most recent call last):
  File "main.py", line 12, in
  File "/lib/picoweb/init.py", line 302, in run
  File "/lib/uasyncio/core.py", line 129, in run_forever
  File "/lib/uasyncio/init.py", line 31, in add_reader
  TypeError: function expected at most 3 arguments, got 4
  MicroPython v1.11-269-gb29fae0c5 on 2019-09-03; ESP32 module with ESP32
  Type "help()" for more information.
  `

May be a bug?

Thank you,
Mauro

Importing picoweb fails on ESP8266

At import picoweb

MemoryError: memory allocation failed, allocating 554 bytes

MicroPython v1.8.7-7-gb5a1a20a3 on 2017-01-09; ESP module with ESP8266

Headers are stored unaltered, making them case sensitive. HTTP spec says they should be insensitive

The HTTP/1.1 & HTTP/2 standard, according to RFC 2616 says:

4.2:

Each header field consists of a name followed by a colon (":") and the field value. Field names are case-insensitive.

Picoweb parses headers and stores them unaltered, meaning if a client passes Content-Type:application/json in a header, req.headers.get('content-type') will return None.

Since different clients pass differently capitalized header keys, (and I guess that's OK, according to the standard) we can either check for all cases of headers ( if b'content-type in [k.lower() for k in req.header.keys()]:) or I would propose storing header keys in a single
case. The change would go in WebApp.parse_headers(): the original headers[k] = v.strip() would become, say, headers[k.lower()] = v.strip()

1.3: req.headers dictionary to contain bytes instead of str, and be optional at all

Currently, for each request, headers are being parsed, decoded to str's and stored in req.headers dict. The idea is to skip decoding to str's to save on memory pressure. But a lot of request handlers don't care about headers, so add a mode when headers aren't stored, but skipped instead. There also can be a need for request handler to parse headers itself, so allow that too. Allow to set the default header parsing mode for the WebApp (the default default will be "skip headers"). Finally, factor out header parsing to a method, to allow subclasses to override it.

Install picoweb in nodemcu

Hi, is it possible to install picoweb in nodemcu board? I run
upip.install('picoweb')

but i got:
Installing to: /lib/

Warning: pypi.org SSL certificate is not validated Error installing 'picoweb': memory allocation failed, allocating 7406 bytes, packages may be partially installed

Any tips? thanks in advance.

Try loading utemplate.compiled before utemplate.source (way less memory usage)

When working with render_templates(), the whole utemplate library is loaded:

picoweb/picoweb/__init__.py

Line 224 in 6e36850

import utemplate.source

This was a too high memory demand in my use case on an ESP8266. So I ended up comiling the template on my dev machine, putting this on the device and loading the compiled version manually (which works great):

import utemplate.compiled
template_loader = utemplate.compiled.Loader(None, "templates")
template = template_loader.load("config.html")

for s in template(config):
    yield from response.awrite(s)

But it would be nice to use the same API with source or compiled templates.
So please try loading the compiled files in _load_template() first, before importing utemplate.source.

AttributeError: 'module' object has no attribute 'get_event_loop'

Hello dear,
I am a beginner on picoweb with esp32 I followed your tutorial https://techtutorialsx.com/2017/09/01/esp32-micropython-http-webserver-with-picoweb/.
when I compile my test file I have it as an error
"
('192.168.1.50', '255.255.255.0', '192.168.1.1', '192.168.1.1')
Traceback (most recent call last):
File "", line 14, in
File "picoweb / __ init__.py", line 283, in run
AttributeError: 'module' object has no attribute 'get_event_loop'
"

I did import the lib uasyncio and pkg_resources using ftp because upip say having no module install.
Need help.
Best regards

Support encoded (compressed, etc.) content

I'd like to send static content with gzip encoding (see MDN dev docs) in order to increase the page load speed.

So ideally sendfile()

picoweb/picoweb/__init__.py

Line 238 in 6e36850

def sendfile(self, writer, fname, content_type=None):

provides an argument for the content encoding:

def sendfile(self, writer, fname, content_type=None, content_encoding=None):

as well as start_response():

picoweb/picoweb/__init__.py

Line 39 in 6e36850

def start_response(writer, content_type="text/html", status="200"):

def start_response(writer, content_type="text/html", status="200", content_encoding=None):

I don't want to gzip the files on the fly, but provide already gzipped content instead of raw content if available. An example for css files:

@app.route(re.compile('^\/(.+\.css)$'), methods=['GET'])
def styles(request, response):
    file_path = request.url_match.group(1)

    if b"gzip" in request.headers[b"Accept-Encoding"]:
        file_path_gzip = file_path + ".gz"
        import os
        if file_path_gzip in os.listdir("/"):
            yield from app.sendfile(response, file_path_gzip, 'text/css', 'gzip')
            return

    yield from app.sendfile(response, file_path, 'text/css')

failed to install on micropython-esp using upip

I'm really new to micropython and trying to install this package with upip:

import upip
upip.install('picoweb')

and it keep failing saying the package is partly installed

Running Picoweb on micropython

Hello,
I found a solution to run Picoweb on micropython esp32 wroom 32D
Here:
https://forum.micropython.org/viewtopic.php?f=18&t=8048&start=10
But I don't understand why it's runnig only on firmware v1.11?
Thanks to help me.
Regards.
Julien

Included examples may fail with "'module' object has no attribute '__path__'" on baremetal ports (e.g. ESP8266)

Environment: NodeMcu clone ESP8266 with picoweb built into custom firmware

1. Copy example_webapp2.py to esp8266 and squares.tpl into templates dir
2. Connect to network as station
3. Add IP address as host parameter in last line of example_webapp2.py
4. Reset
5. At REPL: import example_webapp2.py
6. From PC on network: http://ip-address-esp8266:8081/squares

Results in:

24.000 <HTTPRequest object at 3fff3e50> <StreamWriter <socket state=2 timeout=0 incoming=3fffac78 off=23>> "GET /squares"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "example_webapp2.py", line 24, in <module>
  File "picoweb/__init__.py", line 240, in run
  File "uasyncio/core.py", line 121, in run_forever
  File "uasyncio/core.py", line 85, in run_forever
  File "picoweb/__init__.py", line 156, in _handle
  File "example_webapp2.py", line 18, in squares
  File "picoweb/__init__.py", line 194, in render_template
  File "picoweb/__init__.py", line 190, in _load_template
  File "utemplate/source.py", line 153, in __init__
AttributeError: 'module' object has no attribute '__path__'

Work-around:
Update:
app = picoweb.WebApp(__name__)
To:
app = picoweb.WebApp(None)

Picoweb request doesn't return all form data

The function read_form_data() calls read() which can result in only part of the expected data being returned. Seems like it should be calling readexactly() instead. The problem is described in the micropython forum in detail:

https://forum.micropython.org/viewtopic.php?f=16&t=6647

Unicode characters in template problem

When I use unicode characters in template, a browser receive distorted chars. Using chars as bytes in template gives me an error. It also distorted when I send resp.write('абвгд') directly, without template.

I can send correctly using bytes - resp.write(b'абвгд'). So I rewrite render_template:

def render_template_b(writer, template_name, args=()):
  import utemplate.source
  template_loader = utemplate.source.Loader("__main__", "templates")
  template = template_loader.load(template_name)
  for s in template(*args):
    yield from writer.awrite(s.encode('utf-8'))

Is it a bug? Or how can I use unicode chars in template?

Optimize URL matching by dropping leading "/" from query string

Matching, at least for static files, might be possible to optimize, by having patterns like r"static/(.+)", where, if it matches, original request path is passed to resource_stream().

How to add image to the page?

Hi,

Please provide an example of hosting a picture on the website. I could not find any reference on google/github/blogpost how to do it.

Thank you!

app.run() fails with TypeError: function expected at most 3 arguments, got 4

i try to get picoweb running on the esp32. installed it with upip.install("picoweb") and upip.install("micropython-ulogging")`. i used the example code and cant get it running. no matter what i do, i run into this error:

>>> upip.install("picoweb")
Installing to: /lib/
Warning: micropython.org SSL certificate is not validated
Installing picoweb 1.7.1 from https://files.pythonhosted.org/packages/1b/4f/f7d35f90521e95d9d2307f69ff523133d7d4dd6da7ce1ce0c8382e7255fa/picoweb-1.7.1.tar.gz
Installing pycopy-uasyncio 3.1.1 from https://files.pythonhosted.org/packages/5f/24/fb08acdd7ebf1626dcdb3cdaf0d3f463c254a4a3aa2cab70b0ee6562a83d/pycopy-uasyncio-3.1.1.tar.gz
Installing pycopy-pkg_resources 0.2.1 from https://files.pythonhosted.org/packages/05/4a/5481a3225d43195361695645d78f4439527278088c0822fadaaf2e93378c/pycopy-pkg_resources-0.2.1.tar.gz
Installing pycopy-uasyncio.core 2.3 from https://files.pythonhosted.org/packages/6a/96/80a86b1ea4e2b8c7e130068a56f9e8b5bbb28369e48de56a753778e14faf/pycopy-uasyncio.core-2.3.tar.gz
>>> upip.install("micropython-ulogging")
Installing to: /lib/
Installing micropython-ulogging 0.3 from https://files.pythonhosted.org/packages/49/4f/fd39c77fb90dfd188e3e7716556bb3e211edf132d855d7860a8d835b9fbc/micropython-ulogging-0.3.tar.gz
>>> import picoweb
>>> app = picoweb.WebApp("foo")
>>> app
<WebApp object at 3ffd0080>
>>> app.run()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/lib/picoweb/__init__.py", line 299, in run
  File "/lib/uasyncio/core.py", line 163, in run_forever
  File "/lib/uasyncio/core.py", line 129, in run_forever
  File "/lib/uasyncio/__init__.py", line 31, in add_reader
TypeError: function expected at most 3 arguments, got 4

i know... no routes defined but i get the same error all over the place... even with routes and stuff. so i thought this might be the most simplest code to get something running. any ideas?

ESP32 doesn't support full getaddrinfo() params

Hey guys!

I'm trying to setup my ESP32 to run picoweb. I did a fresh installations of Micropython, did all the installs with upip. but when I try to run the example code i get the following response:

Traceback (most recent call last):
File "main.py", line 64, in
File "/lib/picoweb/init.py", line 287, in run
File "/lib/uasyncio/core.py", line 146, in run_forever
File "/lib/uasyncio/core.py", line 101, in run_forever
File "/lib/uasyncio/init.py", line 235, in start_server
TypeError: function takes 2 positional arguments but 4 were given

the code I'm currently running is:

import uasyncio
import picoweb

def index(req, resp):
    yield from picoweb.start_response(resp)
    yield from resp.awrite("""\
<!DOCTYPE html>
<html>
<head>
<script>
var source = new EventSource("events");
source.onmessage = function(event) {
    document.getElementById("result").innerHTML += event.data + "<br>";
}
source.onerror = function(error) {
    console.log(error);
    document.getElementById("result").innerHTML += "EventSource error:" + error + "<br>";
}
</script>
</head>
<body>
<div id="result"></div>
</body>
</html>
""")

def events(req, resp):
    print("Event source connected")
    yield from resp.awrite("HTTP/1.0 200 OK\r\n")
    yield from resp.awrite("Content-Type: text/event-stream\r\n")
    yield from resp.awrite("\r\n")
    i = 0
    try:
        while True:
            yield from resp.awrite("data: %d\n\n" % i)
            yield from uasyncio.sleep(1)
            i += 1
    except OSError:
        print("Event source connection closed")
        yield from resp.aclose()


ROUTES = [
    ("/", index),
    ("/events", events),
]

app = picoweb.WebApp(__name__, ROUTES)
app.run(debug=True)

my network configuration is done in a boot file before this code is being run but i don't think the problem is in there. Any of you have an idea where I might have made an error?

How to use with existing uasyncio loop?

I would like to incorporate picoweb into an existing project I have that uses an uasyncio loop for managing several blocking tasks (e.g. buttons), but I see that the WebApp.run implements a new uasyncio loop. Contrary to popular belief, running two of these loops concurrently doesn't seem to be working (I'm a newbie, please don't kill me).

Is the best way around this to have an optional argument to WebApp.run to specify an existing uasyncio loop to attach to and then return, and if an existing loop is not specified then create your own?

Here's a quick and dirty example of what I am talking about, I'm quite sure there's a more elegant way to implement this but I hope it expresses what I think the issue is and how it might be resolved.

    def run(self, host="127.0.0.1", port=8081, debug=False, lazy_init=False, loop=None):
        self.debug = int(debug)
        self.init()
        if not lazy_init:
            for app in self.mounts:
                app.init()
        if debug:
            print("* Running on http://%s:%s/" % (host, port))
        if not loop:
            loop = asyncio.get_event_loop()
            loop.run_forever()
            loop.close()
        else:
            loop.create_task(asyncio.start_server(self._handle, host, port))

Server often crashes with 'NameError: local variable referenced before assignment'

Rather often, my picoweb server crashes with the following traceback:

Traceback (most recent call last):
  File "main.py", line 39, in run
  File "/lib/picoweb/__init__.py", line 299, in run
  File "/lib/uasyncio/core.py", line 163, in run_forever
  File "/lib/uasyncio/core.py", line 116, in run_forever
  File "/lib/picoweb/__init__.py", line 210, in _handle
  File "/lib/picoweb/__init__.py", line 210, in _handle
NameError: local variable referenced before assignment

I added some logs to understand and the problem happens when calling uayncio's readline, at the beginning of the function. The original error is:

Traceback (most recent call last):
  File "/lib/picoweb/__init__.py", line 116, in _handle
  File "/lib/uasyncio/__init__.py", line 131, in readline
AssertionError: 

When readline is called, the req variable is not initialized. It explains the 'local variable referenced before assignment' error.

I moved the req = HTTPRequest() at the beginning of the try block to avoid the error. I now get the error log and my server does not crash. There is maybe a better solution.

Permission denied

git branch

• add_method
  master

git push -u origin add_method
remote: Permission to pfalcon/picoweb.git denied to Lisa999.
fatal: unable to access 'https://github.com/pfalcon/picoweb/': The requested URL returned error: 403

???

Trying to create a PR, for this in init.py:
methods = []
if len(e) > 2:
extra = e[2]
methods = extra['methods']

            if path == pattern and method in methods:

For using: @app.route("/", methods=['GET'])

Failing on ENOMEM on ESP8266

I am trying to use Picoweb on ESP8266. It is failing in memory, but I am not sure it that consumption is normal.

In REPL:

>>> gc.collect()
>>> gc.mem_free()
11888
>>> modules.webinterface.app.run(debug=True, port=80, run_loop=True)
* Running on http://127.0.0.1:80/
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "picoweb/__init__.py", line 299, in run
  File "uasyncio/core.py", line 155, in run_forever
  File "uasyncio/core.py", line 110, in run_forever
  File "uasyncio/__init__.py", line 240, in start_server
OSError: [Errno 12] ENOMEM

App definition looks like:

import picoweb

app = picoweb.WebApp(__name__, serve_static=True)

@app.route("/")
def index(req, resp):
    yield from picoweb.start_response(resp)
    yield from resp.awrite("test")

app.run(debug=True, port=80)

Libraries inc. Picoweb and uasyncio are frozen on ROM, so code should not eat RAM. There is also wifi_manager and some own code around.

Is it normal Picoweb can not run when there is 11888 free memory?

HTTPS support

Is there simple way how to make https webserver? Micropython has module ussl what can wrap socket to tls.

If possible I would like to generate elliptic curve self signed certificate on device, because it is much smaller than RSA certs and it is more secure.

Incorrect uasyncio function call awitestr() and memory leak issues as resolved in comments.

In init.py line 252 an incorrect call to a non existent function awritestr
yield from writer.awritestr(s)
Should be
yield from writer.awrite(s)

Some things I miss...

Hi!

I'm using Picoweb and it's great. However, I'm feeling that it lacks of some minor features and convenience methods. I'm working on these features as I need them in my project, and I think my work or ideas might be useful to improve Picoweb. These are my thoughts so far:

1. Request method filtering. Flask uses the route() function for it. I've created a decorator function to handle this.
2. Manage exceptions in views. Although Picoweb handles the exceptions, I like to send a custom (500) response. Right now I am using the same methods decorator that I created, although it is not an ideal solution.
3. A simple response class or method to avoid using the StreamWriter directly. I've created a Response class to send simple text/html and application/json responses. I've also created a subclass that format the response based on the JSend specification, although this is a very personal feature.
4. I miss a body attribute in the request object (or a read() method), although there is a read_form_data() method (only useful with web forms). A json() method would also be great :)
5. If a request has some body content but it is not read in the view (it may be not necessary), then the server doesn't send any response. That's why I think a body attribute could also help to empty and close the reader stream before the handler is called.

I've solved all of these needs, and I can adapt my code into Picoweb and make a few pull requests. However, I wanted to know what do you think about these features and if it makes sense to include them into the library.

Thank you, and sorry if my English doesn't make sense sometimes...

Picoweb and temperature read in the loop (uasyncio)

Hello,

As there is no forum or blog comments I must use this kind of public commutation chanel ;)

First of all - great job ! Thanks !

I would like to use ESP32 (with micropython) and picoweb to made some thermostat. I will get and set temperature via picoweb "web app". To communicate I will use simple json messages... but I need to check temperature every e.g. 2 minutes, compare it stored one and turn on or off my heater.

So my idea is:

• in file "init.py" I will add second scheduled job:

loop.create_task(asyncio.start_server(self._handle, host, port))
loop.create_task(check_temperature())

• then define this function (outside WebApp class) and put some code to read temperature and turn on/off heater

async def check_tempersture():
      while True:
        #check temperature
        # some IFs
        # turn on/off heater
        await asyncio.sleep(120)

And the question is... Is this the right way to do this? Any better solution?

Best regards,
MvincM

Tested on ESP32

Hi!

I don't know if this is the best place to share, but I've been able to install and use the Picoweb on an ESP32.

I'm leaving the instructions that worked for me bellow, in case someone else wants to give it a try.
https://techtutorialsx.com/2017/09/01/esp32-micropython-http-webserver-with-picoweb/

Nonetheless, I've not been able to use the temple features, due to unavailability of memory to allocate. But running the server and creating some routes works pretty well.

Thanks for your great work on this module.

Best regards,
Nuno Santos