dbohdan/jimhttp

A library collection and web microframework

jimhttp

A collection of standalone libraries and a web microframework prototype for Jim Tcl. Most of the libraries also work in Tcl 8.x. The libraries implement command line and proc argument parsing, an HTML DSL, parsing and generating JSON, templates, and persistent storage powered by SQLite3. The web microframework provides a rough implementation of the HTTP/1.1 protocol and a routing DSL.

Components

The components listed below work in Tcl 8.5, Tcl 8.6, Tcl 8.7a3, and Jim Tcl 0.76 or later unless indicated otherwise. Each component is versioned separately. Component version numbers follow semantic versioning. A major version number of zero indicates an unstable API.

Filename	Function	Version
arguments.tcl	Command line argument parsing.	1.0.0
example.tcl 1	A sample web server that demonstrates the use of the other components.	—
entities.tcl	A dictionary mapping characters to HTML entities.	1.0.0
html.tcl	A DSL for HTML generation. Requires entities.tcl.	0.2.1
http.tcl 1	The titular web microframework. Requires mime.tcl.	0.15.2
json.tcl	JSON generation with schema support. 3 JSON parsing. 4	2.1.3
mime.tcl	Rudimentary MIME type detection based on the file extension.	1.2.0
rejim.tcl 2	A basic RESP2 Redis/Valkey/KeyDB/etc. client.	0.2.0
storage.tcl 1	SQLite persistence of static variables.	0.2.0
template.tcl	tmpl_parser templating.	1.0.0
testing.tcl	A test framework with support for tcltest-style constraints.	0.5.0
tests.tcl	Tests for the other components. 5	—

1. Jim Tcl-only.

2. Does not support Tcl 8.5.

3. Schemas define data types. See the example below.

4. Warning: parsing is fairly slow in general and extremely slow in UTF-8 builds of Jim Tcl. (Obsolete benchmark.) This may matter to you if you need to decode more than a few dozen KiB of JSON at a time. Since version 0.79 Jim Tcl can be built with a fast binary extension for parsing and encoding JSON. The jq module is an option for faster JSON parsing in earlier versions. It requires an external binary.

5. Only compatible components are tested in Tcl 8.

Use examples

http.tcl

source http.tcl

::http::add-handler GET /hello/:name/:town {
    ::http::respond [::http::make-response \
            "Hello, $routeVars(name) from $routeVars(town)!"]
}

::http::start-server 127.0.0.1 8080

http.tcl and storage.tcl

source http.tcl
source storage.tcl

::http::add-handler GET /counter-persistent {{counter 0}} {
    ::storage::restore-statics

    incr counter

    ::storage::persist-statics
    ::http::respond [::http::make-response $counter]
}

::storage::init
::http::start-server 127.0.0.1 8080

json.tcl

# This produces the output
# {"a": "123", "b": 123, "c": [123, 456], "d": "true", "e": true}
source json.tcl

puts [::json::stringify {
    a 123
    b 123
    c {123 456}
    d true
    e true
} 0 {
    a string
    c {N* number}
    d string
}]

Requirements

Compile Jim Tcl 0.76 or later from its Git repository. Stable releases prior to that (0.75 and earlier) will not work. You will need an SQLite3 development package (libsqlite3-dev on Debian and Ubuntu, libsqlite3x-devel on Fedora, sqlite3-devel on openSUSE Tumbleweed) to do this and optionally AsciiDoc (asciidoc on Debian and Ubuntu, Fedora, and openSUSE) to generate the documentation (don't use the option --disable-docs in that case).

git clone https://github.com/msteveb/jimtcl.git
cd jimtcl
./configure --with-ext="oo tree binary sqlite3" --enable-utf8 --ipv6 --disable-docs
make
sudo make install

Once you have installed Jim Tcl you can clone this repository and try out the example by running

git clone https://github.com/dbohdan/jimhttp.git
cd jimhttp
jimsh example.tcl

and then pointing your web browser at http://localhost:8080/.

License

MIT.

static.jpg photo by Steven Lewis. License: CC0.

jimhttp's Issues

Compression support

See msteveb/jimtcl#36 - adding support for optional compression should be fairly easy:

for encoding in $headers(acceptEncoding):
    if (command exists $encoding) {
      body = $encoding body
      headers(contentEncoding) = $encoding
    }

However, ::http::make-response cannot access the request headers to determine which encodings are supported by the client, so this requires some refactoring.

Cannot parse empty object.

Hi there!

The JSON parser has exactly the same problem with empty objects as it had with empty arrays. A similar fix will do it, i.e. testing for the presence of }at the beginning of the string as the first thing you do in the while loop.

Fix static file handlers

Support preprocessing channels in ::http

Cannot parse empty arrays.

The current version of the parser is not able to parse empty arrays, e.g.

::json::parse {{"test":[]}}

will fail. The following patch fixes it:

proc ::json::decode-array {str {numberDictArrays 0}} {
    set strInitial $str
    set result {}
    set value {}
    set i 0
    if {[string index $str 0] ne "\["} {
        error "can't parse JSON array: $strInitial"
    } else {
        set str [string range $str 1 end]
    }
    while 1 {
        # Empty array, get out from here.
        if { [string index [string trimleft $str] 0] eq "\]" } {
        set str [string range [string trimleft $str] 1 end]
        break
        }
        # Value.
        lassign [::cirrus::json::decode-value $str $numberDictArrays] value str
        set str [string trimleft $str]
        if {$numberDictArrays} {
            lappend result $i
        }
        lappend result $value

        # ","
        set sep [string index $str 0]
        set str [string range $str 1 end]
        if {$sep eq "\]"} {
            break
        } elseif {$sep ne ","} {
            error "can't parse JSON array: $strInitial"
        }
        incr i
    }
    return [list $result $str]
}

Sorry if the formatting got a little bit messed up. The idea is basically to peek forward for the end of the array and jump out of the loop at once. I haven't had time to test this thoroughly, so might need to do some extra thinking for corner cases.

Force string output

May I make the following suggestion? To be frank, I am not entirely satisfied with the solution, but it actually suits my current needs. You might want to take the idea further for refinement though...

I've had to find a way not to automatically convert things that integer or doubles to their JSON counterparts, i.e. a way to ensure that value 123 does not become the integer 123, but rather the string "123". To do this, I have slightly modified (see below) stringify to:

1. Take a new argument that requests string-only output
2. Recognises key that look like xxx!string as the key xxx which value should only be output as a string.

What do you think?

proc ::json::stringify {dictionaryOrValue {numberDictArrays 1} {allstring 0}} {
    set result {}
    if {[llength $dictionaryOrValue] <= 1} {
        if {[string is integer $dictionaryOrValue] || \
                [string is double $dictionaryOrValue]} {
        # Number.
        if { $allstring } {
        set result "\"$dictionaryOrValue\""
        } else {
        set result $dictionaryOrValue
        }
        } else {
            # String.
            set result "\"$dictionaryOrValue\""
        }
    } else {
        # Dict.
        set allNumeric 1

        if {$numberDictArrays} {
            set values {}
            set i 0
            foreach {key value} $dictionaryOrValue {
                set allNumeric [expr {$allNumeric && ($key == $i)}]
                if {!$allNumeric} {
                    break
                }
                lappend values $value
                incr i
            }
        }

        if {$numberDictArrays && $allNumeric} {
            # Produce array.
            set arrayElements {}
            foreach x $values {
                lappend arrayElements [::json::stringify $x 1 $allstring]
            }
            set result "\[[join $arrayElements {, }]\]"
        } else {
            # Produce object.
            set objectDict {}
            foreach {key value} $dictionaryOrValue {
        foreach {k type} [split $key "!"] break
        if { $type eq "string" } {
            lappend objectDict "\"$k\": [::json::stringify $value 0 1]"
        } else {
            lappend objectDict "\"$k\": [::json::stringify $value 0 $allstring]"
        }
            }
            set result "{[join $objectDict {, }]}"
        }
    }
    return $result
}

json.tcl: "array" and "object" semantics in schemas are broken

Erroneous boolean parsing.

I have failed parsing JSON that looks like this:

{"action":"set","node":{"key":"/dir4","dir":true,"modifiedIndex":17,"createdIndex":17}}

This all fails on the parsing of true. To fix this, I propose the following implementation:

proc ::json::decode-value {str {numberDictArrays 0}} {
    set str [string trimleft $str]
    switch -regexp -- $str {
        {^\".*} {
            return [::json::decode-string $str]
        }
        {^[0-9-].*} {
            return [::json::decode-number $str]
        }
        {^\{.*} {
            return [::json::decode-object $str $numberDictArrays]
        }
        {^\[.*} {
            return [::json::decode-array $str $numberDictArrays]
        }
        {^(true|false|null)} {
            regexp {^(true|false|null)} $str val
            return [list $val [regsub {^(true|false|null)} $str ""]]
        }
        default {
            error "cannot decode value as JSON: \"$str\""
        }
    }
}

The only difference is in the implementation of the {^(true|false|null)} branch of the switch.

I haven't made extensive testing of this, but my solution seems to solve the problem. You might want to introduce a json::decode-boolean for completion?