swindon-rs/tk-http

Full featured HTTP and Websockets library for rust/tokio

Tk-HTTP

Status: Beta

Documentation | Github | Crate

A full-features asynchronous HTTP implementation for tokio-rs stack, including websockets.

Features:

• HTTP 1.1 and 1.0 support (plans to support for HTTP/2 with same API)
• Flexible configuration of pipelining both for client and server
• Comprehensive configuration of timeouts both for client and server
• Strict parsing of few selected headers which influence security
• Other headers go unparsed to keep CPU usage low
• Minimum copies of data: i.e. you can decode JSON directly from network buffer
• Generic over transport (so can be used over TLS or unix sockets)

License

Licensed under either of

• Apache License, Version 2.0, (./LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (./LICENSE-MIT or http://opensource.org/licenses/MIT) at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

tk-http's Issues

Use a smaller but growable buffers for headers (httparse)

I mean this one: https://github.com/servo/rust-smallvec

Also, we shouldn't create Vec::with_capacity(MAX_HEADERS) for headers table. We already know number of headers at this point (at it's much smaller than MAX_HEADERS in 99.9% use cases)

Encoder.done_headers should return false when content-length=0

encoder.add_length(0).unwrap();
if encoder.done_headers().unwrap() {
  // this should be unreachable
}

handle "100 continue" response

TLS server example

There is now a TLS client example, but not an example available for TLS server side. It would be awesome to have a TLS server example.

Implement request body

currently parse_body is a stub; must be implemented.
Todo:

• fixed sized
• chunked

Integrating with tokio-tls

Hi,

I've seen in the example for tls_client tokio-rustls is used, it would be great to have an example of using tokio-tls, which will use the native tls libs.

I'm still new to the tokio/futures libs, so not sure will be able to pull it off without reading the source for tokio/futures/tk-http/tokio-tls.

So, if you could please provide an example for it working or confirm that it should work, that would be great.

Thanks

Check header validity

Both in add_header and in format_header we should check whether \r\n are present in header name or value.

A way to handle it:

1. Store start of header value
2. Check data just written into buffer
3. Truncate buffer back and return error

Alternative would be just panicking, but I think error makes more sense here.

Another arguable thing is if we want to do this only in debug build. Probably we can do this in debug build only if we are going to panic. Otherwise, users expect error to be returned regardless of build settings.

Sending websocket `Packet`s of shared data

I'm trying to use BufferedDispatcher::new_with_websockets to handle HTTP and websocket requests on the server side. It works great, however I have a lot of cases where I need to send the same thing to multiple websocket sinks (e.g., broadcast the same message to all users).

I couldn't find a good way to do this without having to clone the data for every user, since everything accepts a Packet, which owns its data in a Vec<u8> or String (although encoding a Packet only requires a reference to the bytes).

Is there a recommended way to do this without cloning?

---

On a related note, I'm guessing the reason Packet is using String/Vec is so that it's compatible with futures/tokio, which requires 'static in most places. However if Packet were defined as:

enum Packet<T> where T: AsRef<[u8]> {
  // ...
  Binary(T),
  // ...
}

it would allow Packet to still "own" the data, and allow end-users to use something like the bytes crate to share the data internally (or define their own Arc<Vec<u8>> wrapper that can dereference to an array of bytes, etc). This would probably make the API more annoying to use though, for what might be an uncommon use case.

I could try implementing a Codec in my own code, but since most of the logic is the same, it looks like I would have to copy/paste a lot of the private stuff from BufferedCodec and websocket::ServerCodec / websocket::zero_copy.

Websocket example is missing updated Tokio support

Websocket example is missing updated Tokio support where Core + handle is not used.

Windows 10: Connection error: I/O error: An established connection was aborted by the software in your host machine.

I am testing out using tk-http instead of Hyper, and I noticed I can quite easily get my code to crash with some low-level looking network error by holding down the F5 key in Firefox. I haven't been able to trigger an error like this with Hyper.

I'm running this code on Windows 10 with the MSVC toolchain.

To reproduce, checkout this revision of my pandt project. Then do the following in the checkout:

$ cd ptrpi
$ cargo run --bin tkrpi 1337 dd.yaml

Then load up http://localhost:1337/ in your browser and spam it until it crashes, which only takes a few seconds for me.

Here's a link to quickly see my mainloop: arpeggiorpg/arpeggiorpg@254dbed#diff-23e1c8400fe6268f1a67705b9c60cde2R84

Connection error: I/O error: An established connection was aborted by the software in your host machine. (os error 10053)
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ()', src\libcore\result.rs:860
stack backtrace:
   0:     0x7ff75d7cd847 - std::panicking::default_hook::{{closure}}
                        at C:\projects\rust\src\libstd\panicking.rs:356
   1:     0x7ff75d7ccd74 - std::panicking::default_hook
                        at C:\projects\rust\src\libstd\panicking.rs:367
   2:     0x7ff75d7d0691 - std::panicking::rust_panic_with_hook
                        at C:\projects\rust\src\libstd\panicking.rs:545
   3:     0x7ff75d7d0548 - std::panicking::begin_panic<collections::string::String>
                        at C:\projects\rust\src\libstd\panicking.rs:507
   4:     0x7ff75d7d0464 - std::panicking::begin_panic_fmt
                        at C:\projects\rust\src\libstd\panicking.rs:491
   5:     0x7ff75d7d03f9 - std::panicking::rust_begin_panic
                        at C:\projects\rust\src\libstd\panicking.rs:467
   6:     0x7ff75d7d93a7 - core::panicking::panic_fmt
                        at C:\projects\rust\src\libcore\panicking.rs:69
   7:     0x7ff75d5f0879 - core::result::unwrap_failed<()>
                        at C:\projects\rust\src\libcore\macros.rs:29
   8:     0x7ff75d5c9a99 - core::result::Result<(), ()>::unwrap<(),()>
                        at C:\projects\rust\src\libcore\result.rs:737
   9:     0x7ff75d6a9e93 - tkrpi::main
                        at C:\Users\radix\Projects\pandt\ptrpi\src\bin\tkrpi.rs:93
  10:     0x7ff75d7d1481 - panic_unwind::__rust_maybe_catch_panic
                        at C:\projects\rust\src\libpanic_unwind\lib.rs:98
  11:     0x7ff75d7d096a - std::rt::lang_start
                        at C:\projects\rust\src\libstd\rt.rs:51
  12:     0x7ff75d6abc6b - main
  13:     0x7ff75d7dfea8 - __scrt_common_main_seh
                        at f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:253
  14:     0x7ff90d36451c - BaseThreadInitThunk
error: process didn't exit successfully: `C:\Users\radix\Projects\pandt\target\debug\tkrpi.exe 1337 .\dd.yaml` (exit code: 101)

Fix pushback

This issue tracks problem described here:
f849057#commitcomment-19486411

Handling of `Connection: close`

1. Should not receive more requests if current one contains Connection: close in input headers
2. Should close connection automatically after sending response if RequestConfig::do_close is true

Consider convenience method to stamp Date header in response

Date can be cached thread-locally and formatted only when formatted value is expected to change (seconds-level granularity).
e.g. Encoder.add_date() or Encoder.add_auto_date()

Implement 100-Expect support

Limit number of `InFlight` requests

Should be easy to do. Just measure size of the Deque and stop/restart reading