marcoskirsch/nodemcu-httpserver

A (very) simple web server written in Lua for the ESP8266 firmware NodeMCU.

nodemcu-httpserver's Issues

Not working project at SDK 1.4.0

Simple text files are not UTF-8 encoded

I think that in the year 2017, if we can support only one string encoding, it should be UTF-8.
This should apply to all text MIME types (text/css, text/html, etc).

Added example hello_world.txt for testing. By default it looks incorrect, see top window in screenshot. It should look like the bottom window.

Running "background" process

I would like to try to combine nodemcu-httpserver with a ledstrip controller. For that i need to constantly send codes to the ledstrip. How can i share cpu time with the http server process (i'm not sure if process is the right name here, maybe it's a coroutine or something like that). It's ok if the ledstrip goes down under any traffic (request/response of webserver) because i just need to configure what colors i want once and then it can run that pattern until i would like to configure something else.

When i make GET/POST requests to the webserver how do i get my background process to pick up on the new values i'm setting? Should i use some global variable for that?

Lua Panic on uri.file length > 27

In spite of what the nodemcu doco says, it appears that filenames > 27 characters (not 32) cause file.open to throw a panic.

Utterly knucklehead patch attached;
httpserver.lua.patch.txt

Hangs after first connection

I formatted my MCU, flashed all these files to it, and changed the configuration so it could connect to my WiFi. It connects successfully, but the first time I try to hit it with any type of request, the console registers the call (below), but the browser just hangs. Any subsequent calls from any other clients are not registered on the MCU or responded to.

set (mode=3)
AP MAC:     1a:fe:34:a3:38:47
Client MAC:     18:fe:34:a3:38:47
chip:   10696775
heap:   25144
nodemcu-httpserver running at http://192.168.1.1:80
IP:     192.168.1.242
Requested URI: /
Method: GET

I disabled the basicauth and it functions exactly the same - no response of any kind (even status codes or headers) ever makes it back to the browser.

I've been digging through recent pulls a little bit to see if there was something that could cause this, but I haven't seen anything obvious yet. I also checked with a different power supply to make sure it wasn't hanging due to a power issue.

Can anyone confirm this on the latest master branch, or am I just crazy?

Change Makefile to use nodemcu-uploader

Right now we use esptool which doesn't support binary files.

Base64 decoding fails on nodemcu float builds

Base64 decoding gives unexpected results on nodemcu builds with floating point support.

nodemcu_integer_0.9.6-dev_20150704.bin

NodeMCU 0.9.6 build 20150704  powered by Lua 5.1.4
> print(math.floor(1.5))
stdin:1: malformed number near '1.5'
> print(dofile("httpserver-b64decode.lua")("dGVzdGluZw=="))
testing

nodemcu_float_0.9.6-dev_20150704.bin

NodeMCU 0.9.6 build 20150704  powered by Lua 5.1.4
> print(math.floor(1.5))
1
> print(dofile("httpserver-b64decode.lua")("dGVzdGluZw=="))
tUstYnf

Problems with nodemcu dev built with 1.4.0 SDK

It appears that the semantics of connection:send (as used in httpserver-error) has changed in recent builds. In particular, it appears that you can only do one :send before you have to wait for the "sent" event. Subsequent :send calls appear to fail. Not clear what the right solution is.....

Warning on boot (deprecated API)

Warning, deprecated API! Argument style station configuration is replaced by table style station configuration. It will be removed in the next version. See documentation for details.

Needs to be updated.

bufferedconnection:send() should have fragmentation logic for large payloads, and pre-concats should be avoided

Hi,

I've commented on this in the relevant commits, but I thought I'd better open an issue.
The ESP currently has an issue with sending payloads over a socket that are larger than one MTU, minus some overhead bytes. If this is done, there is a chance that the ESP will become unstable and could reboot.
There is a guideline (I forget where I read it) to keep payloads below about 1400 in size.
The buffered connection implementation refactored in this commit currently has 2 thresholds: 800 and 1000. I think these were meant as conservative margins, and they do work as long as you keep buffered sends small. The current implementation accumulates payload in a table, then once a threshold is reached, it concatenates and sends. The use of a table and a late concat is an efficient way to handle this.
The implicit assumption is that the total payload size will still be less than one MTU. However, there is no provision for the case where a single payload is very large, or where the payload plus what is already accumulated in the table is greater than one MTU (i.e.: 1400).
There needs to be some fragmentation logic implemented to make sure that the socket:send() calls are always done with payloads smaller than 1400. With this, buffered:send() can be called in any way the user wants as long as memory hold out.
In addition, this commit made changes to prematurely concatenate payloads, and then send the large results to buffered:send(). The premature concatenations are done in the "bad" way, by using the ".." operator, which is known to have performance issues because it trashes the GC, especially when used in a tight loop. These concats are not needed, because the buffered connection already handles it, and in a more efficient way.
I suspect that there was confusion between the socket:send() and the buffered:send() (at one point I made the same mistake). It is true that the new firmware can only handle one socket:send in each callback, and so if the socket were used as connection, then these concats would be needed. However, the -static, -error, -header, etc files receive as argument a buffered connection, and so the concats should not be done, because they make the previous point worse.

args.lua saved to disk instead of opening in browser

Hi, in the example, when I enter to a index.html from a web browser, and then click on link to a .lua (ie arguments link), the browser want to download the arguments.lua instead of open a new window with the html content, Why?

Thank you, awesome job!

init.lua error

Hello,

I have this problem after make upload_server:

init.lua:55: cannot open/write to file

I'm using nodemcu_float_0.9.6-dev_20150704.bin.

Any suggestions?
Thanks in advance

Replace httpserver-b64decode.lua with encoder module

We use Base64 decoding in the implementation of Basic HTTP Authentication.

NodeMCU provides Base64 decoding through the encoder module

We should get rid of httpserver-b64decode.lua (too bad, at the time it was a pain in the ass to get it to work on NodeMCU's extreme memory constraints) and instead make the encoder module a requirement for HTTP Authentication. It should be much faster and use less memory.

Can't handle page with embedded images

The server quickly runs out of memory and crashes if a page with a few embedded images is requested. See example cars.html

  Requested URI: /cars.html
  Requested URI: /cars-mas.jpg
  Requested URI: /cars-ferrari.jpg
  Requested URI: /cars-lambo.jpg
  PANIC: unprotected error in call to Lua API (not enough memory)
  ������1�)m�1�)�!������B

http requests HEAD and POST causing reboot

Requested URI: /
PANIC: unprotected error in call to Lua API (?:0: attempt to index a nil value)
PANIC: unprotected error in call to Lua API (attempt to index a nil value)
PANIC: unprotected error in call to Lua API (attempt to index a nil value)
.
.
.
PANIC: unprotected error in call to Lua API (attempt to index�?�)���19@H
)��4!����

No support for simple HTTP Authentication

http GET request from our software

After our IOT software request a HTTP GET for your HTTP SERVER, we got the following error
https://dl.dropboxusercontent.com/u/83554849/bug_http.txt
Please, see if you find out the bug.
Miguel

Cancelled requests fills up memory and causing reset

Output from serial console attached below. Heap size printed every 100ms using:

tmr.alarm(2, 100, 1, function()
    print(node.heap())
end)

I'm refresing file_list.lua page in my browser every 400-500ms. Important thing is page loading time slower than my refreshing speed - which effectively means cancelling previous request and making new request every time. These cancelled requests do not free its resources, will add up and causing reset.

If catching these half-completed and cancelled requests possible, this issue can be fixed.

26768
Requested URI: /file_list.lua
18280
20040
20040
18216
Requested URI: /file_list.lua
14400
15624
15624
14696
Requested URI: /file_list.lua
10840
12072
12048
11112
10800
Requested URI: /file_list.lua
7216
8512
7640
7800
Requested URI: /file_list.lua
3616
4888
4888
4000
4176
PANIC: unprotected error in call to Lua API (not enough memory)
~~ reboot and bootloader gibberish ~~ 

Sorry for my poor English and I hope the issue above understandable.

Improve usability/factorization of bufferedConnection

The bufferedConnection code has been pulled out into its own file. This improves usability, but the result needs more work.
Specifically, usage depends on a socket onSent() callback, which is outside the bufferedConnection file, and on coroutine creation/handling/destruction code, which are also outside the bufferedConnection file.
The ideal buffered connection would have a usage model as close as possible to the current socket usage model, have all the coroutine code hidden away, have a small mem footprint, and trivial to integrate into an application for general use.
I would suggest creating something along the lines of another class which wraps the bufferedConnection, maybe called a "threaded" connection, which contains the missing pieces of dependent code. This is just an idea of course, and open for discussion.

not enough memory

I am getting:
NodeMCU 0.9.5 build 20150213 powered by Lua 5.1.4
set mode=STATION (mode=1)
MAC: 18-FE-34-9F-E0-A5
chip: 10477733
heap: 20032
lua: init.lua:15: not enough memory

What is the minimum heap for server to run?

Compiling: httpserver-conf.lua

Flashed:
NodeMCU custom build by frightanic.com
branch: master
commit: 2e67ff5a639a13260fd4fb3c3b627ccdc2845616
SSL: true
modules: adc,ads1115,bit,ds18b20,enduser_setup,file,gpio,i2c,mdns,mqtt,net,node,ow,pwm,sntp,spi,struct,tmr,uart,websocket,wifi,tls
build built on: 2017-08-30 18:00
powered by Lua 5.1.4 on SDK 2.1.0(116b762)

upload with make and get this error:

Compiling: httpserver-conf.lua
lua: httpserver-conf.lua:25: unexpected symbol near '}'
stack traceback:
[C]: in function 'compile'
?: in function <?:5>
?: in main chunk
[C]: in function 'dofile'
init.lua:3: in main chunk
[C]: ?

it compiles if I remove the brace

Add ability to serve gzip compressed content

Many js and css libraries are greater then available space but fit in case of gzipped.

Cannot get web page to load

I am using Esplorer to load NodeMCU 8266 12E. I have tried every thing I can think of to get the server to give me a response.

I get an IP address 192.168.1.140
I have tried several urls like
http://192.168.1.140/index.html)[http://192.168.1.140/index.html]
http://192.168.1.140/index.html

Have index.html and http/index.html loaded on the ESP.

Any ideas would be great, this looks just like what I need!

Thanks

init.lua failing to start server

Apologies for what I expect is a noob issue rather than a genuine issue but perhaps the answer will be educational to other people.

I'm using init.lua as supplied except for the obvious additions. For some reason it appears the server is attempting to start before even trying to join the network, therefore wifi.sta.getip() is NIL and dofile("httpserver.lc")(80) does not execute.

So adding this print statement

-- Uncomment to automatically start the server in port 80
print(not not wifi.sta.getip(),  not not wifi.ap.getip())
if (not not wifi.sta.getip()) or (not not wifi.ap.getip()) then
    dofile("httpserver.lc")(80)
end

results in the following output upon reset

NodeMCU custom build by frightanic.com
    branch: master
    commit: c8037568571edb5c568c2f8231e4f8ce0683b883
    SSL: false
    modules: file,gpio,net,node,ow,tmr,uart,wifi
 build  built on: 2016-02-25 15:48
 powered by Lua 5.1.4 on SDK 1.4.0
set (mode=1)
Client MAC:     18:fe:34:a4:b7:a0
chip:   10794912
heap:   37168
false false
> 
Communication with MCU...
Got answer! AutoDetect firmware...

NodeMCU firmware detected.
=node.heap()
41856
> Connecting to WiFi Access Point ...
IP:     192.168.1.198

At that point I can manually execute init.lua again and it works fine. I can see various ways of rectifying this but before I do I'd like to understand why it's happening in the first place. Thanks!

Long file names crash server

Allow static or dynamic error pages

The idea is to alter the filename of the request and remember, that it is an error url. and then reprocess it.
I am afraid this will require some refactoring
I would suggest to use a new name scheme for error pages to allow displaying them in unauthorized mode and prevent calling them directly
like
error/404.* for a 404 error and the like.

What do you think?

PANIC: unprotected error in call to Lua API

Hello @marcoskirsch,

Your code is awesome! it connected to my wifi and it shows it has access as a client, however i cant seem to load the index.html. im using the ff:
-esplorer
-d1miniPro
Can you help me with this, I was reading your read me and you said that the filename should have a prefix "http/" but how do i do this cuz i know filenames can't have slashes. hope to hear from you sir, you have a good day!

Thanks!

Root directory is cluttered. Move files to a folder.

When this project started, all the server code was in a single file. Now it's spread across lots of little files, all in the root directory:

httpserver-b64decode.lua
httpserver-basicauth.lua
httpserver-compile.lua
httpserver-conf.lua
httpserver-connection.lua
httpserver-error.lua
httpserver-header.lua
httpserver-init.lua
httpserver-request.lua
httpserver-static.lua
httpserver-wifi.lua
httpserver.lua

I'm toying with the idea of reorganizing and moving all the server files into a folder. I'd remove the prefix to each file. The folder may be named httpserver/ or simply server/

I'm not about to go and do this, as it can be pretty disruptive to other contributors. So I'm floating the idea here in order to get some feedback.

Thanks!

Jquery zipped file is too large

There's an example in the http/ folder for testing out serving compressed file. But the file we are demoing with is too large.

We should replace with something smaller so that all the files fit into flash well.

WiFi configuration is not ready to operate as AP only

init.lua is not ready to set wifi mode to AP only. IP setting is missing at all and init is forced to get IP as client, otherwise http server will not start

Crash after POST request from iOS & MacOs Safari

Using the post.lua example from Safari web browser (on both iOS and MacOS) leads to a crash of the server. No problem using other web browser or cURL tool with POST method.

Here's the log :

Requested URI: /post.lua
Method: GET
Requested URI: /post.lua
Method: POST
Getting Request Data
Parsing Form Data
PANIC: unprotected error in call to Lua API (?:0: attempt to index a nil value)
PANIC: unprotected error in call to Lua API (attempt to index a nil value)
PANIC: unprotected error in call to Lua API (attempt to index a nil value)
PANIC: unprotected error in call to Lua API (attempt to index a nil value)
PANIC: unprotected error in call to Lua API (attempt to index a nil value)
...
...
...
PANIC: unprotected error in call to Lua API (attempt to index a nil value)
PANIC: unprotected error in call to Lua API (attempt to index a nil value)
PANIC: unprotected error in call to Lua API (attempt to index a nil value)
PANIC: unprotected error in call to Lua API (attempt to index a nil value)
PANIC: unprotected error in call to Lua API (loop in gettable)
PANIC: unprotected error in call to Lua API (attempt to concatenate a nil value)
PANIC: unprotected error in call to Lua API (attempt to call a string value)

PS: anyway, let me thank you for your great work, this server is just awesome ! :)

Websocket implementation

Seems like it's close
https://gist.github.com/creationix/079e11345249525c7e2e

node_info in standalone mode

Hi
Thank for all your work, it's usefull for me and my IOT project(s)

I want to use my nodeMCU in a nomad project, so it's configured in Station and Access Point, to be connected to the network of my house if available, or by the access point of the node itself.

I detect a problem with the node_info page, because when I'm without my home network, it's try to concat the nil IP in the string to send.

my quick patch is to check the wifi.sta.status in an IF statement before ask the IP

Thanks for all,
Luc

Cannot serve multiple large files at once

Need to keep track of coroutine per connection, right now there's just a single global coroutine.

Specific files are not delivered correctly

Some files (f.e. the gif and png images in this package) are not delivered fully to the client.

Strip down server

Hello Marcos,

I am totally in love with your nodemcu-server. I was trying to write you a mail but obviously you didnt want be reached like that :).

I have a project where I was trying to use it but with the other Lua code I need everything seems to heavy to run on my ESP. So I thought about stripping down the server to the bare minimum. I only want to serve a single file and parse some arguments from the browser. Extracting the parser function worked quite well. The problem I am facing now, is to do multiple conn:send()'s.

I already looked it up and everything results in nasty cascading of function. Is soo convinient to do multiple conn:send()'s in the Lua files I can serve with your server. I really would like to have that for myself as well. I checked your code and you seem to use a coroutine but I couldnt figure out how exactly. I would really appreciate it if you would give me a minimal example for that.

What I would like to do is following:

Page request

• parse arguments
• execute some code depending on arguments
• send file (remote.html)...
• while sending manipulate one or two lines of remote.html (preferablyon the fly). like so:

if file.open("remote.html", "r") then
    while true do
        local line=file.readline()
        if line == nil then break end
        line = line:gsub("STR-2-RPLC",str(depending on arguments from above))
        conn:send(line)
    end
end
file.close()

As you might know it is not that easy to understand a hugh bunch of code someone else wrote, plus I'm not a Lua guru. So I would be really glad if you could help me out with that.

Best regards

Support for POST method

Is it possible to post like form, and fetch some text entered in textbox on webpage in lua variable?

Only works once

Hello, I am having an issue with the web server on my ESP8266 boards. It works fine for the first request, but then the server is not accessible anymore & I need to restart the chip. Do you know if this is a known issue? Thanks

Serving bigger static files causes watchdog restart

I have patch to prevent this. How can i commit it ?

---------------------------- httpserver-static.lua ----------------------------
@@ -23,6 +23,7 @@ return function (connection, req, args)
          connection:send(chunk)
          bytesSent = bytesSent + #chunk
          chunk = nil
+         tmr.wdclr() -- loop can take a while for long files. tmr.wdclr() prevent watchdog to restart module
          --print("Sent" .. args.file, bytesSent)
       end
    end

Basic HTTP Authentication is broken

When I enable it in httpserver-conf.lua and try to go to any page, I get sent straight to the "401 - Unauthorized" error page.

This used to work, need to narrow it down.

Why not using file.exists instead of file.open()+file.close()?

In plenty of location there are construction like:

nodemcu-httpserver/httpserver.lua

Line 64 in 28fbb6e

local fileExists = file.open(uri.file, "r")

nodemcu-httpserver/httpserver.lua

Line 65 in 28fbb6e

file.close()

where a file is open; then close to check if the file exist.

Why not just using file.exists()?

Maybe I miss something but as far as I can see that construction is going use more CPU in the end, and probably create some object in memory that we will need to trash on the next GC

Make httpserver "git submodule" friendly

I've made a fork of the Lua WebIDE from @moononournation (cf: https://github.com/Godzil/nodemcu-webide )

To make thing easier to evolve and always use the latest version of httpserver (and not use an hardcoded version) I moved things around to use both httpserver and uploader as git submodules. That works great but httpserver makefile was not thought to not be used as the main makefile as set some parameter internally and we have to edit the file by hand.

To get around that, the makefile of my project after getting the data from the httpserver project patch the makefile to be more compatible. I also do other patches, but these one are really specific for my needs and I don't think they would be really usefull and welcome on the normal version.

The main patch is basically some small changes on the makefile:

See: https://github.com/Godzil/nodemcu-webide/blob/master/patchs/httpserver_makefile.patch

For the rest I basically rename init.lua to httpserver-start.lua and add it to the compilation list.

I still have to figure a good way to be able to let the webide user change the configuration without changing too much of httpserver, if you have any idea, it is welcome!

To make thing more clear, I would like to try to do as least changes as possible on httpserver through patches and keep it as clean as possible so updating the submodule would break as little as possible.

Does default mimetype should not be "text/plain"?

Wouldn't be better to use a mime type like "application/octet-stream" than "text/plain" for unknown types?

nodemcu-httpserver/httpserver-header.lua

Line 18 in 28fbb6e

if mt[ext] then return mt[ext] else return "text/plain" end

I know that we are not going to serve really complicated thing with this server, but if the file is not text we will probably get corrupted data as the browser will expect 7bit ASCII

Not enough memory after ~120 refresh of the same page

Hello,

I've just tested the latest version with a esp8266-01 (1MB flash)
I added only html/index.html with a little content <H1>OK</H1>

I can load the content with my browser, but I made a test : refresh ~120x the page...then the esp crashed (not enough memory)...

I have the same problem with my home-made version of a http server...I think there is a memory-leaks with the socket management, but where?

Files get mixed up when being served simultaneously.

If you request a large file that also pulls another large file, the coroutines get mixed up and start sending chunks incorrectly.

The reason is that file API is global and you cannot have two files open at the same time. We will have to open and seek on each chunk being sent from the coroutine.

Cannot upload files using make upload_*

Preparing esp for transfer.
Transfering init.lua as init.lua
Verifying...
Transfering httpserver.lua as httpserver.lua
Verifying...
Traceback (most recent call last):
File "/usr/local/bin/nodemcu-uploader.py", line 509, in
uploader.write_file(f, d, args.verify)
File "/usr/local/bin/nodemcu-uploader.py", line 233, in write_file
data = self.download_file(destination)
File "/usr/local/bin/nodemcu-uploader.py", line 179, in download_file
if bytes_read > int(size):
ValueError: invalid literal for int() with base 10: 'connection:send(table.concat(self.data, ""))\r'
make: *** [upload_all] Error 1

How is working the Basic HTTP authentication

How is working the basic authentication.
Sent the sample code, please!
Thank You.

Not working at SDK 1.4.0

NodeMCU custom build by frightanic.com
branch: master
commit: c8037568571edb5c568c2f8231e4f8ce0683b883
SSL: false
modules: adc,bit,cjson,file,gpio,i2c,net,node,ow,pwm,rtcfifo,rtcmem,rtctime,sntp,spi,tmr,uart,wifi
build built on: 2016-05-31 14:01
powered by Lua 5.1.4 on SDK 1.4.0

Some questions.

hello,marcos.I try to complete your project, but found that uploads the firmware, I can not upload files, use make upload_server and make upload_http command errors. There are other solution? ?Looking forward to your reply

Support protecting only certain pages using Basic HTTP Authentication

Right now it's all-or-nothing.
I would like to have the ability to only use HTTP Authentication on some pages but not others. I imagine this by having a regular expression that matches the pages to protect in httpserver-conf.lua.