boostorg/beast

HTTP and WebSocket built on Boost.Asio in C++11

beast's Issues

Update Parser concept documentation

The Parser concept doc omits write(bs, ec) where bs is a BufferSequence. After deciding what should be in the concept and doc, it might be necessary to update the trait.

Commit limit should be adaptive

The commit limit blocks insertions when the sustained throughput of inserts exceeds the throughput limit of the commit process. Currently this limit is set to an arbitrary high value determined empirically. It should be set using an adaptive calculation.

Question: WebSocket server implementation segfaults

Hey there,

I saw in the README I can ask questions here, so I'll give it a try. I just started to implement an async WebSocket server. I've tried to keep it as close to my implementation of an async boost::asio::ip::tcp socket server, but still followed Beast's documentation as much as possible. This is so I can keep it simple for myself, as I'm still a fanatic hobbyist C++ at this point and the project is heavily in development. :-)

It works as following:

• WebSocketAcceptor
  1. it sets up an end point, opens a socket, binds to it and starts listening - lines 36-63
  2. it starts an async accept at the io_service - line 73
  3. when one is accepted, it accepts it on the WebSocket layer, creates/starts a WebSocketSession and starts another async accept at the io_service - lines 88-101
• WebSocketSession
  1. Once created, it registers itself at the MessageCenter (currently an echo server) and starts an async read - lines 51-53 and line 73
  2. Once a message is received, it is stored in a string and processed at this MessageCenter (which just echos it back to the session using the Write method) - lines 94-99

So far so good. Once I connect from the console of a browser, it accepts the connection and I can write a message, which shows up in the console as a received message. However, as soon as I connect again (whether it's from another computer or another browser) the program segfaults inside Beast library. Below you can find some GDB output.

If it's not too much, could one of you maybe point me in the right direction? I just don't see it, but I really would like to get it to work with Beast. Also, other tips are welcome as well, still learning here. :-)

---

GDB output:

Thread 3 "HarmonIQ" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff6345700 (LWP 4591)]
0x00000000005d5f84 in beast::websocket::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> > >::build_response<beast::http::string_body, beast::http::basic_headers<std::allocator<char> > > (this=0x97c210, req=...) at /home/tiesjan/Documents/HarmonIQ/lib/beast/websocket/impl/stream.ipp:1000
1000        (*d_)(res);
Missing separate debuginfos, use: dnf debuginfo-install boost-system-1.60.0-7.fc24.x86_64 libgcc-6.1.1-3.fc24.x86_64 libstdc++-6.1.1-3.fc24.x86_64

Head of the GDB stack trace:

#0  0x00000000005d5f84 in beast::websocket::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> > >::build_response<beast::http::string_body, beast::http::basic_headers<std::allocator<char> > > (this=0x97c210, req=...) at /home/tiesjan/Documents/HarmonIQ/lib/beast/websocket/impl/stream.ipp:1000
#1  0x00000000005d020e in beast::websocket::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> > >::accept<beast::http::string_body, beast::http::basic_headers<std::allocator<char> > > (this=0x97c210, req=..., ec=...) at /home/tiesjan/Documents/HarmonIQ/lib/beast/websocket/impl/stream.ipp:306
#2  0x00000000005cbe60 in beast::websocket::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> > >::accept<boost::asio::null_buffers> (
    this=0x97c210, buffers=..., ec=...) at /home/tiesjan/Documents/HarmonIQ/lib/beast/websocket/impl/stream.ipp:259
#3  0x00000000005c9771 in beast::websocket::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> > >::accept (this=0x97c210, ec=...)
    at /home/tiesjan/Documents/HarmonIQ/lib/beast/websocket/impl/stream.ipp:205
#4  0x00000000005c140f in network::WebSocketAcceptor::handle_async_accept (this=0x97c1e8, error_code=...) at /home/tiesjan/Documents/HarmonIQ/src/network/WebSocketAcceptor.cpp:90

the rest of the stack trace is boost::asio invoking WebSocketAcceptor.handle_async_accept

BuildRequest not using resource parameter

I'm just getting started with websocket programming and thought that the Beast library would be a good place to started. I've been trying to put together a simple app that communicates with a websockets server and I noticed that the handshake is always issued as GET / rather than, for example, GET /myserver/serverpoint.

I notice that although you are expected to provide a resource during the handshake call this parameter, which is passed into the build_request function, is ignored and the url is explicitly set to "/". This looks like an oversight - shouldn't the resource be used in the construction of the req.url?

PortableNuke.

Problems with message constructor interface

From @ja11sop

First off, I really, really like Beast from what I've seen so far. Its integration with asio and the careful thought put into the design is evident unlike most other libraries in this space. As a result I've been tracking master in some local work and I've had to make a few changes based on construction of http requests - which is fine.

However the change in this commit as referenced by the comment, "Previous constructors are removed as they were a notational convenience for assembling HTTP/1 requests and responses. They are not necessary as this library aims at library writers and not end users." leaves me a little perplexed.

This is not about convenience - this is about providing a way to assemble an object so that construction, as far as possible, equals useful state. You have an object and it is in a useful state, or you don't. In general piecemeal construction is a bad idea and error-prone. If anything there should be a way to create a fully formed message at construction time alone. Modifiers should really only be there to handle the case where there is a time disparity with availability of information, in other words the information cannot be known at construction time or it is prohibitively expensive to make it so.

I'd be interested in why piecemeal, property-based construction is seen as attractive? I don't see an advantage here, just opportunity for error and more code to write. I may of course be missing something.

Inccorect use of rfc2616 in basic_headers

Currently basic_headers applies rfc2616 to every header but the rfc states

"Multiple message-header fields with the same field-name MAY be
present in a message if and only if the entire field-value for that
header field is defined as a comma-separated list [i.e., #(values)].
It MUST be possible to combine the multiple header fields into one
"field-name: field-value" pair, without changing the semantics of the
message, by appending each subsequent field-value to the first, each
separated by a comma."

which implies that this is not valid for all headers. As such basic_headers should retain the structure as is.

Rename websocket server sources

To "server" instead of "peer", so they are more easily recognized

Refactor http example server to abstract the handler

The HTTP example servers (sync and async) should be refactored so that the handler for requests is supplied as a template argument. This would make it easier for people to re-use the server without having to modify the boilerplate.

If the server is generic enough, we can consider moving it to extras/ as a partly-official API.

Beast message interface and middleware

I am searching C++11 library for embedding http server inside my application, and I found Beast as a perfect solution. I have found that current message interface is very good and offers full control over every parth of message

What I dont like:

1. I can forget to fill some fields (status, reason...) and message will be sent with no errors on server side
2. Too many steps to fill data. Current message is good for low-level operations, but after I tried Nodejs, I want to write less code... This can be done with constructing messages from pre-defined templates or setting some missed fields with default values.
   I would like to send a file/json answer/custom html or any other typical message with using only one line of code. Yes, I can make such templates myself and use them, but I think that Beast should show users how to make it the right way
   
   And do we really need an explicit call of prepare() ?

One more question is not clear for me about message construction and its structure. How to make middleware? For example, something like connect-multiparty
This middleware should be able to add new field "files" into message structure, check request path and intercept messages. Should I derive from request_v1 and write overloaded functions for reading/wiriting? Do you have any plans about middleware affecting message structure? I think, Beast should offer a simple way how to write and use middleware

Boost library conformance

http://www.boost.org/development/requirements.html

URL is wrong in copyright header of doc/reference.xsl

An unfortunate find/replace error

SSL short read not handled correctly

I am playing with http client and SSL. And I am having problem with getting the response from the server after calling beast::http::async_read.

When I examined the code in read.ipp: parse_op::operator() I found the there is nothing to handle "ssl short read errors". I am not SSL expert but from my previous attempts with ASIO I remember ASIO often finished read operations on SSL socket by receiving this kind of error. I think it could be interpreted as error::eof.

When I made following changes to read.ipp I can get correct http response in my handler. Without this change SSL short read error is propagated to my handler but response doesn't contain parsed response although it was received successfully. cpp-netlib library also contains special code to handle it.

Wikipedia article seems to consider this to have websocket server functionality

https://en.wikipedia.org/wiki/Comparison_of_WebSocket_implementations
As I only see this is a client library, probably the wikipedia article needs to be updated

Rename WebSocket APIs to DynamicBuffer, dynabuf

Several websocket interfaces still use Streambuf instead of DynamicBuffer, they should be renamed for clarity.

Wrong error code starting values

All error code enums should start with

success = 0

Move websocket echo servers to examples

NT

Build error on raspberry pi

I get missing boost/endian/buffers.hpp, not sure if there is boost module for this
I installed boost with this command *sudo apt-get install libboost-all-dev *

In file included from Beast/include/beast/websocket/detail/stream_base.hpp:14:0,
                Beast/include/beast/websocket/option.hpp:12,
                Beast/include/beast/websocket.hpp:12,
                Beast/examples/websocket_example.cpp:9:
Beast/include/beast/websocket/detail/frame.hpp:18:36: fatal error: boost/endian/buffers.hpp: No such file or directory
 #include <boost/endian/buffers.hpp>

http_sync_server unhandled exception if file not found

Hello, I have found Beast as an amazing implementation of C++ embeddable web-server.
http_sync_server sample crashes if file not found with throwing unhandled exception boost::system_error
With debugger I have found two problems:

1. We need to put the code of sending file into else branch of if(! boost::filesystem::exists(path)) here
   Because if file was not found - there will be sent 404 error and program will continue trying to send file with 200 result code
2. Inside catch(std::exception const& e) { res = {};
   handler we need to costruct resposne with string_body, because it tries to put std::string{"An internal error occurred"} + e.what(); into file body and crashes with preparing this response

suite::fail needs overloads

We need a fail that takes the reason, file, and line.

Any Plan for Socks Proxy?

do you have a plan to support socks proxy(at least socks v5)?

Better control over request and response objects with websockets

It would be nice to have better control over request and response objects with websockets. Having access to the entire HTTP response object could be very useful. For instance, a unit test may want to inspect the response object after a failed handshake. As a client, you should be able to fully customize a request. For example by providing a user name and password for authentication if needed.

Use BOOST_ASSERT instead of assert

Per the Boost library guidelines

http_server example based on boost::asio::spawn

boost::asio offers asynchronous implementation based on Coroutines that can use single thread for multiple parallel requests

It would be nice to have such example code for http server

basic_parser_v1 and parser_v1 should have interfaces that throw

e.g. write

Doc update for websocket decorator

The documentation for stream::set_option lists the decorate overload as having a parameter type from the detail namespace, and it could use more exposition to make its use easier to understand. An example, especially one that shows how to set websocket subprotocols, would be great.

docs should use modern C++

Documentation example code needs to be checked for modern C++. Specifically that constructed objects use braces instead of parenthesis.

This snippet

boost::asio::ip::tcp::socket&& sock;
...

should read

boost::asio::ip::tcp::socket sock;

split admin command

Need a command to split a database into two. This could work conceptually by iterating the data file and writing a new data/key file until reaching the halfway point in terms of file size, then continue the iteration and write the second data/key file. In practice to make this work in reasonable time frames (days instead of weeks) it would render large portions of the key file in memory the way that rekey works.

Changes in b4?

From @Randian
I just checked in and saw many things have changed. For example, methods in request objects are manually specified strings and not enums now? What is the _v1 stuff? What does prepare do? How about a before/after code example to explain these changes?

Response:

Thank you for your interest in Beast!

The method in the request is now a string instead of an enum. Previously, it was an enumeration because the parser being used (https://github.com/nodejs/http-parser) only supported a fixed set of methods. I have since written a new HTTP parser which is header-only (Beast is now completely header-only) and supports any string for the method.

In the previous version, http::write and http::async_write made a copy of the message and inserted these fields for you. This design caused a number of problems. Now, the prepare function will set certain fields in the message for you (especially Content-Length):
http://vinniefalco.github.io/beast/beast/ref/http__prepare.html
Use of prepare is optional, if you want to set Content-Length yourself you may do so. For example, when sending the response to a HEAD request.

I've been collaborating with @vinipsmaker (author of the Boost.Http proposal) and in the interest of making Beast "HTTP/2 ready", the message model has been adjusted. The base class message represents any HTTP message including HTTP/2, while the derived class message_v1 contains the parts of the message that are specific to HTTP/1.0 and HTTP/1.1. For example, message_v1 has a version, while message does not. The _v1 suffix indicates that the interface is for HTTP/1.X only.

The only explanation of the changes is in the commit messages. This library is currently in the Boost incubator awaiting feedback for subsequent iteration. And Beast is not yet at version 1.0.0. From what I can tell there are not that many users. All interfaces should be considered subject to change, although from a practical perspective I do not anticipate any more significant changes unless Boost reviewers present a strong case for it (with the exception of http::read and http::async_read, which have some problems in their current interface). However in the future I will try to document API breaking changes.

Once again thank you for your valuable feeback!

explore find_package for MSVC CMake

It would be nice to use find_package for Boost when building Visual Studio projects but I couldn't get that to work and support both 32 and 64 bit compiled boost and projects on the same machine.

Error handling questions

In my code I catch boost::system::system_error. That works great for boost::asio::error::operation_aborted or boost::asio::error::eof, but since it looks like you map beast errors onto boost::system::system_error how do I distinguish the regular boost asio errors from beast http errors in my catch blocks? I don't know what boost::asio::error::whatever has the same value as what beast is throwing.

Tidying

• Remove boost/system includes (e.g. <boost/system/error_code.hpp>) from places that don't need it.
• Remove unused headers_type from write_preparation (https://github.com/vinniefalco/Beast/blob/master/include/beast/http/impl/write.ipp)
• Use braced-init style construction everywhere (e.g. https://github.com/vinniefalco/Beast/blob/master/include/beast/core/buffer_cat.hpp)
• Doc for handler_alloc should indicate that copies of the handler will be made as needed (https://github.com/vinniefalco/Beast/blob/master/include/beast/core/handler_alloc.hpp)

CI scripts should test CMake

To make sure CMake based builds succeed, one of the CI targets should attempt to use it.

Appveyor support

Need Appveyor CI integration

Documentation

Replace the main documentation page with a custom written index, like Asio. This will hide the rather ugly auto-generated list of refs below the Quick Reference page.

WebSocket doc changes:

• Each stream member function needs to have as a precondition, that the NextLayer represents an active, valid connection
• Typo, "approached" in http://vinniefalco.github.io/beast/beast/types/DynamicBuffer.html
• Some code snippets need brace style constructions
• Exposition should link to the stream::handshake member on http://vinniefalco.github.io/beast/beast/websocket/handshaking.html
• This page is confusing http://vinniefalco.github.io/beast/beast/websocket/messages.html
• Some Javadocs incorrectly state that "pongs are noted" when they are actually routed to the pong callback if set.
• Incomplete sentence on this page http://vinniefalco.github.io/beast/beast/websocket/pongs.html
• stream remarks should read "A stream object must not be moved or destroyed while there are pending asynchronous operations associated with it"
• In the last code snippet for WebSocket/Handshaking, the call to beast::http::read is missing the sb parameter.

HTTP doc changes:

• read and write descriptions should present the functions as generic serialization and deserialization algorithms.
• Parser concept should use ConstBufferSequence not ConvertibleToConstBuffer in write (http://vinniefalco.github.io/beast/beast/types/Parser.html)

General doc changes:

• Refer to TLS instead of SSL
• Consider renaming beast/websocket/ssl.hpp to tls.hpp
• Return type for DynamicBuffer::prepare missing a colon (http://vinniefalco.github.io/beast/beast/types/DynamicBuffer.html)

HTTP docs are slightly out of date

Docs refer to beast::http::request where they should be saying beast::http::request_v1 for example

Documentation error

In the last code snippet for WebSocket/Handshaking, the call to beast::http::read is missing the sb parameter.

arena block size should be adaptive

The arena should calculate an adaptive block size instead of requiring callers to provide a size hint upon construction.

http servers should respect Keep-Alive and Close

Currently they don't

Websocket handshake will not use the given resource path

It will use a regular path '/'.
Here is the code in stream.ipp .

template<class NextLayer>
http::request_v1<http::empty_body>
stream<NextLayer>::
build_request(boost::string_ref const& host,
    boost::string_ref const& resource, std::string& key)
{
    http::request_v1<http::empty_body> req;
    req.url = "/";
    req.version = 11;
    req.method = "GET";
    req.headers.insert("Host", host);
    req.headers.insert("Upgrade", "websocket");
    key = detail::make_sec_ws_key(maskgen_);
    req.headers.insert("Sec-WebSocket-Key", key);
    req.headers.insert("Sec-WebSocket-Version", "13");
    (*d_)(req);
    http::prepare(req, http::connection::upgrade);
    return req;
}

The README structure should be to revised to include installation / inclusion examples

Following my own confusion adopting Beast into a recent project, I believe the README could use an update. This should help to make the Beast library more accessible.

Note: this is a follow-up to a recent comment discussion on issue 53.

Make the websocket handshake request customizable

Currently the websocket handshake request is built up by the lib under the hood, but i need to decorate it with additional header values, please extend the interface with a handshake method which allows the request customization.

Build instructions

The README.md needs a section with complete instructions for building and running the tests and examples. Especially how to install Boost with bjam in the path.

Boost.Asio like async_read

Hi Vinnie,

I asked a question about async_read in person at CppCon 2016. Thank you for writing a great library. And you lightning talks was fantastic!

I couldn't describe my question at that time due to my poor English. Let me describe my question.

See the following Boost.Asio's example:
http://www.boost.org/doc/libs/1_61_0/doc/html/boost_asio/example/cpp03/chat/chat_server.cpp

  void start()
  {
    room_.join(shared_from_this());
    boost::asio::async_read(socket_,
        boost::asio::buffer(read_msg_.data(), chat_message::header_length),
        boost::bind(
          &chat_session::handle_read_header, shared_from_this(),
          boost::asio::placeholders::error));
  }

boost::asio::async_read calls the handler chat_session::handle_read_header when chat_message::header_length bytes of data are received. The caller can assume that the data size is exactly chat_session::handle_read_header bytes if error code is success. Even if a sender write the concatenated header and body as one call. That means the receiver can control the length of receiving data.

In chat_session::handle_read_header, calls async_read as the same mannar. The length is read_msg_.body_length().

  void handle_read_header(const boost::system::error_code& error)
  {
    if (!error && read_msg_.decode_header())
    {
      boost::asio::async_read(socket_,
          boost::asio::buffer(read_msg_.body(), read_msg_.body_length()),
          boost::bind(&chat_session::handle_read_body, shared_from_this(),
            boost::asio::placeholders::error));
    }
    else
    {
      room_.leave(shared_from_this());
    }
  }

I'd like to do the same thing using Beast C++ websocket library's async_read. Because I have many those kinds of codes using Boost.Asio, and if Beast C++ websocket library supports the similar functionality, it is easy to write abstraction layer that supports both TCP/IP (Boost.Asio) and Websocket (Beast).

It is a feature request. But if there is better solution, any advises are welcome.

Check 32/64 bit builds on Windows

There are some integer warnings building 64 bit on Windows (bjam and CMake default to 32). The Windows 64-bit CMake build in particular needs to be checked, with

cmake -G"Visual Studio 14 2015 Win64" ..

http-server did not handle the connections which are need to be closed.

When I use Apache Bench to test the performance of beast, however i found that
the connection did not closed, this leading the test failed.
how to repair the problem?

Use C++ tags in markdown

README.md has a code block or two that aren't marked as C++

Order of output using suite::log

if you use suite::log before calling pass or fail (directly or indirectly) the output is out of order. e.g.:

nudb.test.callgrind
0 inserts
nudb.test.recover
10 inserts

should read

nudb.test.callgrind
nudb.test.recover
0 inserts
10 inserts

The easy fix is to output the suite preamble when writing to the log for the first time, if the preamble has not already been output.

A workaround is to call

        testcase("");

from overrides of suite::run as needed.

1.0.0-b8 cmake broken

CMake with no DVARIANT specified fails to build, the error is:

CMake Error at CMakeLists.txt:26 (if):
  if given arguments:

    "STREQUAL" "coverage"

  Unknown arguments specified


-- Configuring incomplete, errors occurred!
See also "D:/src/Beast/bin/CMakeFiles/CMakeOutput.log".

Abstract WebSocket servers

The echo WebSocket servers in the test directory should be refactored and made abstract, and moved to extras, so they may serve as a foundation for users to write their own application specific servers.

The server will follow Asio best practices especially for managing concurrency properly, and shutting down without race conditions.

How to get HTTP response content from streambuf ?

Can it auto parse http response and separate headers and content