cpp-netlib/uri

cpp-netlib URI

C++ Network Library

Modern C++ network programming libraries.

Join us on Slack: http://slack.cpp-netlib.org/

Subscribe to the mailing list: https://groups.google.com/forum/#!forum/cpp-netlib

Downloading cpp-netlib

You can find official release packages of the library at:

http://github.com/cpp-netlib/cpp-netlib/downloads

If you want the latest code from the master branch of the project, you can follow these instructions for cloning the project repository:

$ git clone https://github.com/cpp-netlib/cpp-netlib
$ cd cpp-netlib
$ git submodule init
$ git submodule update

Introduction

cpp-netlib is a collection of network-related routines/implementations geared towards providing a robust cross-platform networking library. cpp-netlib offers the following implementations:

• Common Message Type -- A generic message type which can be used to encapsulate and store message-related information, used by all network implementations as the primary means of data exchange.
• Network protocol message parsers -- A collection of parsers which generate message objects from strings.
• Adapters and Wrappers -- A collection of Adapters and wrappers aimed towards making the message type STL friendly.
• Network protocol client and server implementations -- A collection of network protocol implementations that include embeddable client and server types.

This library is released under the Boost Software License (please see http://boost.org/LICENSE_1_0.txt or the accompanying LICENSE_1_0.txt file for the full text.

Building and Installing

To build the libraries you will need to have CMake version 2.8 or higher installed appropriately in your system.

$ cmake --version
cmake version 2.8.1

It is recommended that you build cpp-netlib outside of the source directory, to avoid having issues with CMake generated files polluting the source directory:

$ mkdir ~/cpp-netlib-build
$ cd ~/cpp-netlib-build
$ cmake -DCMAKE_BUILD_TYPE=Debug     \
>       -DCMAKE_C_COMPILER=clang     \
>       -DCMAKE_CXX_COMPILER=clang++ \
>       $HOME/cpp-netlib    # we're assuming this is where cpp-netlib is.

Once CMake is done with generating the Makefiles and configuring the project, you can now build the tests and run them:

$ cd ~/cpp-netlib-build
$ make
$ make test

You can also download and install cpp-netlib using the ` vcpkg`_ dependency manager:

$ git clone https://github.com/Microsoft/vcpkg.git $ cd vcpkg $ ./bootstrap-vcpkg.sh $ ./vcpkg integrate install $ vcpkg install cpp-netlib

The cpp-netlib port in vcpkg is kept up to date by Microsoft team members and community contributors. If the version is out of date, please create an issue or pull request on the ` vcpkg`_ repository.

If for some reason some of the tests fail, you can send the files in Testing/Temporary/ as attachments to the cpp-netlib developers mailing list.

Running Tests

If you want to run the tests that come with cpp-netlib, there are a few things you will need. These are:

• A compiler (GCC 4.x, Clang 3.6, MSVC 2008)
• A build tool (CMake is required)
• OpenSSL headers (optional)

Note

This assumes that you have cpp-netlib at the top-level of your home directory.

Hacking on cpp-netlib

cpp-netlib uses git for tracking work and is hosted on GitHub. cpp-netlib is hosted on GitHub following the GitHub recommended practice of forking the repository and submitting pull requests to the source repository. You can read more about the forking process and submitting pull requests if you're not familiar with either process yet. cpp-netib follows the GitHub pull request model for accepting patches. You can read more about the process at http://cpp-netlib.org/process.html#pull-requests.

Because cpp-netlib is released under the Boost Software License it is recommended that any file you make changes to bear your copyright notice alongside the original authors' copyright notices on the file. Typically the copyright notices are at the top of each file in the project.

You can read about the cpp-netlib style guide at http://cpp-netlib.org/style-guide.html.

The main "upstream" repository is at http://github.com/cpp-netlib/cpp-netlib.

Contact and Support

In case you have any questions or would like to make feature requests, you can contact the development team through the developers mailing list or by filing issues at http://github.com/cpp-netlib/cpp-netlib/issues.

Join us on Slack: http://slack.cpp-netlib.org/

You can reach the maintainers of the project through:

Dean Michael Berris ([email protected])

Glyn Matthews ([email protected])

uri's Issues

Add extension points for scheme and protocol based normalization and comparison

MSVC debug suffix

When I use network-uri in my application with -DUri_USE_STATIC_CRT=OFF, compiler shows errors when linking in debug mode.
To solve, I have to create different network-uri.lib for release and debug mode.
I have to modify src/CMakeLists.txt to specify the debug_postfix when MSVC is detected.

if (MSVC)
#When compiling with MSVC we use different names for release and debug library.
#We solve problems when linking in debug mode
	set(CMAKE_DEBUG_POSTFIX d)
endif()

add_library(network-uri ${Uri_SRCS})

uri_builder.hpp: 'set_authority' was not declared in this scope

{
    network::uri_builder builder;
    builder.authority("host.com:80");
}


..cpp-netlib/uri/src/network/uri/uri_builder.hpp:134: error: 
'set_authority' was not declared in this scope

There is an authority() interface but no set_authority in uri_builder.
Solution: write an authority parser or remove the authority interface.

Copy assignment operator invalidates URI path

Hello,

I have noticed a difference in behavior between the copy constructor and the copy assignment operator in the uri class. This is the code I used:

#include <iostream>
#include <string>

#include "network/uri.hpp"

using network::uri;

void PrintParts(const uri &u) {
  std::cout << "scheme    : " << u.scheme() << std::endl;
  std::cout << "user_info : " << u.user_info() << std::endl;
  std::cout << "host      : " << u.host() << std::endl;
  std::cout << "port      : " << u.port() << std::endl;
  std::cout << "path      : " << u.path() << std::endl;
  std::cout << "query     : " << u.query() << std::endl;
  std::cout << "fragment  : " << u.fragment() << std::endl;
}

int main(int argc, char *argv[])
try {
  // Using copy constructor
  uri u1 = uri(argv[1]);
  PrintParts(u1);

  std::cout << "==============================" << std::endl;
  
  // Using copy assignment
  uri u2;
  u2 = uri(argv[1]);
  PrintParts(u2);

  return 0;
} catch (std::exception &e) {
  std::cerr << "error: " << e.what() << std::endl;
  return 1;
}

Then, when I run it with a URI beginning with file://, I get the following result:

$ ./test file:///path/to/file.txt
scheme    : file
user_info : 
host      : 
port      : 
path      : /path/to/file.txt
query     : 
fragment  : 
==============================
scheme    : file
user_info : 
host      : 
port      : 
path      : path/to/file.txt
query     : 
fragment  : 

Notice that the copy assignment operator suppresses the leading forward slash in the path. Pull request #93 doesn't seem to solve this problem.

Failed test: uri_normalization_test

Test uri_normalization_test.normalize_with_lower_case failed with message:

uri/test/uri_normalization_test.cpp:40: Failure
Value of: instance.normalize(network::uri_comparison_level::syntax_based)
  Actual: { 'h' (104, 0x68), 't' (116, 0x74), 't' (116, 0x74), 'p' (112, 0x70), ':' (58, 0x3A), '/' (47, 0x2F), '/' (47, 0x2F), 'w' (119, 0x77), 'w' (119, 0x77), 'w' (119, 0x77), '.' (46, 0x2E), 'e' (101, 0x65), 'x' (120, 0x78), 'a' (97, 0x61), 'm' (109, 0x6D), 'p' (112, 0x70), 'l' (108, 0x6C), 'e' (101, 0x65), '.' (46, 0x2E), 'c' (99, 0x63), 'o' (111, 0x6F), 'm' (109, 0x6D), '/' (47, 0x2F), '*' (42, 0x2A) }
Expected: "http://www.example.com/%2A"
Which is: 0x1015c8496

Environment: mac osx, clang 5.0

Calling `authority` on URI with empty-string port crashes

Setting the authority with an empty port and then getting that authority causes a crash.

auto uri = network::uri_builder()
    .authority("localhost:")
    .uri();
std::cout << *uri.authority() << '\n';

Root cause

The root cause is in the definition of the uri::authority() function.

boost::optional<boost::string_ref> uri::authority() const {
   auto host = this->host();
   if (!host) {
     return boost::optional<boost::string_ref>();
   }

   auto first = std::begin(*host), last = std::end(*host);
   auto user_info = this->user_info();
   if (user_info) {
     first = std::begin(*user_info);
   }

   auto port = this->port();
   if (port) {
     last = std::end(*port);
   }

   return to_string_ref(first, last);
 }

The local variable port will be initialized as the empty string referencing a default-initialized string_type, subsequently causing the last variable to be assigned a null value. That causes the range (first, last) to be invalid.

Add functionality to split the query into (key,value) pairs

It would be great if it was possible to split the query component into a series of (key,value) pairs.

I would suggest having a query_key_values() method which either returns an input_iterator which has a value type of pair<string_view, string_view> or receives an output_iterator which has a value type of pair<string_view, string_view>.

Make `uri_builder::append_query()` more robust against misuse

The current interface

template <typename Source>
uri_builder &append_query(const Source &query)

expects query to be an unencoded key/value pair like key=value, which is then percent-encoded internally. However, characters in /.@&%;= are excluded from encoding which leads to broken URI's.

Example:

uri_builder ub(...);
ub.append_query("q=" + get_unsafe_data());

If get_unsafe_data() returns a string that contains characters like % or =, these characters are never percent-encoded. In the former case, a URI decoder will usually expect a percent-encoded octet when reading a % character. In the latter case, a URI decoder may try to split the string again in a key and value part.

To fix the % issue, this character could simply be removed from the exclude set. On the other hand, = cannot be removed from the exclude set, because it is needed literally as a separator between the key and value part.

Proposed solution:

I think the basic design problem here is that the key and value parts are not treated separately. An interface which does that already exists in the form of uri_builder::append_query_key_value_pair (this function has been recently updated because it suffered from similar encoding issues).

uri_builder::append_query(input) should reuse uri_builder::append_query_key_value_pair(key, value) internally by splitting input at the first = character -- the left part goes as key, the right part goes as value. If no = character is found, input is passed as key leaving value empty. This way everything gets percent-encoded properly.

I can prepare a PR if you like.

Consider how to extend the URI according to the scheme-specific/protocol-specific parts of the comparison ladder

1. string comparison
2. case normalization
3. percent encoding
4. path segment normalization
5. *scheme-specific
6. *protocol specific

This can be done rather cleanly by passing functions to the transformer functions instead of enum values, and by using the decorator pattern. I'll experiment without and see how it works out...

Is this project still active?

network::uri_builder::uri() method crashes with special user_info string

I tried setting the following as basically blank userid and password fields for the FTP credentials. The mlfBldr.uri() crashes with the expression string iterator + offset out of range.

    static const std::string gUserInfo(":");
    network::uri_builder mlfBldr;
    mlfBldr.scheme("ftp").host(nicHost).user_info(gUserInfo);
    auto uri = mlfBldr.uri();

The code breaks in the 3rd last line of the snuppet below from uri.cpp (from the uri subproject). I was expecting to either have an error indicating that the ":" paramter was illegal or am pretty sure that the problem is due to the while ((':' == *it) || ('/' == *it)) as the code there does not expect a : to follow (as is the case when the userid password are set to null. as per calling the method uri_builder::user_info(":") - this leaves the iterator pointing to the 3rd occurrence of a colon in this URI. Also if I change the userid/password to " :" leaving a single space in for the userid then things work fine.

auto it = std::begin(uri_);
if (scheme) {
  uri_parts_.scheme.reset(
      copy_range(std::begin(*scheme), std::end(*scheme), it));
  // ignore ://
  while ((':' == *it) || ('/' == *it)) {
    ++it;
  }
}

if (user_info) {
  uri_parts_.hier_part.user_info.reset(
      copy_range(std::begin(*user_info), std::end(*user_info), it));
  ++it;  // ignore @
}

if (host) {
  uri_parts_.hier_part.host.reset(
      copy_range(std::begin(*host), std::end(*host), it));
}

Versioning

@glynos any chance you could apply some kind of tagged versioning on this? Even if it not a stable release. I am writing a conanfile.py for it to allow easier deployment and it would be nice to tie each conanfile update with some versioned release rather than just master.

Building a URI with query is awkward and dangerous

The uri_builder class has two query methods, one that appends but incorrectly encodes and another that correctly encodes but replaces instead of appending.

First method

auto uri = network::uri_builder()
    .scheme("http")
    .host("example.com")
    .query("q1=foo bar")
    .query("q2=biz baz")
    .uri();
std::cout << uri << '\n';

What I expect: http://example.com?q1=foo%20bar&q2=biz%20baz
What I get: http://example.com?q2=biz%20baz

The actual output is correctly encoded, but the second query call replaced result of the first query call.

Second method

auto uri = network::uri_builder()
    .scheme("http")
    .host("example.com")
    .query("q1", "foo bar")
    .query("q2", "biz baz")
    .uri();
std::cout << uri << '\n';

What I expect: http://example.com?q1=foo%20bar&q2=biz%20baz
What I get: http://example.com?q1=foo bar&q2=biz baz

The actual output is incorrectly encoded, though at least the second query call appended to the result of the first query call.

Commentary

The inconsistency in the API is asking to be misused by Murphy. At the least, the two query methods should behave consistently with regards to each other.

Even better, both query methods should always percent-encode and always append. I struggle to think of a use case where I would want an incorrectly encoded URI, and I struggle to think of a use case where, when using uri_builder, I would want to undo a previous call to query by replacing its result with a subsequent call to query.

'::' in url result in broken uri object

cppnet-lib 0.11.0-rc1 downloaded from website.

This code

uri::uri uri("http:://google.com"); //note double : in the url
assert(uri.port_range().empty());

result in a uri object with broken hierarchical part. some of internal uri_parts iterator ranges pointing to random memory areas.

This problem could be critical for security sensitive applications because probably opens remote code execution vulnerabilities if application read url from outside.

Swap the order of arguments to urri::resolve

The network::uri::resolve function takes an argument of a uri object that is a URI reference. The URI proposal to the standard library states the argument is a base URI and this is a reference.

So swap the arguments around. This also makes it consistent with make_relative

move constructor and destructor can be declared with noexcept

uri(uri &&other) NETWORK_URI_NOEXCEPT:
~uri() NETWORK_URI_NOEXCEPT:

throw network::uri_syntax_error when contains square brackets

case: http://www.example.com/path/?key1=value1&key2=val[ue2

Add nothrow versions of the normalize, make_reference and resolve transformers

Parsing http://user@host gives bad host iterator and will crash

network::uri address("http://user@host"); // bad last_colon for host in parse here
auto tmp = address.host();
std::string host = tmp.to_string(); // crashes here due to bad iterator

Crash when clearing query parameters using network::uri::uri_builder if no parameters exist in original URI

When clearing query parameters using the URI builder, a debug assertion (from SCL) is tripped if the original URI doesn't have query strings.

clear_query
network::uri original("http://example.com/path");
network::uri_builder builder(original);
builder.clear_query(); // BOOM

Note that this doesn't crash if original had a query string. Details:

Debug Assertion Failed!

Program: ...path\to\program.exe
File: C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\xmemory0
Line: 100

Expression: "(_Ptr_user & (_BIG_ALLOCATION_ALIGNMENT - 1)) == 0" && 0

Stack trace:

program.exe!std::_Deallocate(void * _Ptr, unsigned __int64 _Count, unsigned __int64 _Sz) Line 99	C++
program.exe!std::allocator<char>::deallocate(char * _Ptr, unsigned __int64 _Count) Line 721	C++
program.exe!std::_Wrap_alloc<std::allocator<char> >::deallocate(char * _Ptr, unsigned __int64 _Count) Line 988	C++
program.exe!std::basic_string<char,std::char_traits<char>,std::allocator<char> >::_Tidy(bool _Built, unsigned __int64 _Newsize) Line 2260	C++
program.exe!std::basic_string<char,std::char_traits<char>,std::allocator<char> >::~basic_string<char,std::char_traits<char>,std::allocator<char> >() Line 1017	C++
program.exe!network::optional<std::basic_string<char,std::char_traits<char>,std::allocator<char> > >::operator=(network::nullopt_t __formal) Line 241	C++
program.exe!network::uri_builder::clear_query() Line 127	C++

Compiler is VC++ 2015.

uri_builder unable to reset fields.

The following code still prints the userid:password field when I try to obfuscate it by making a clone of an URI and explicitly resetting the user_info field. The reason this happens is that the fields are boost::optional. There should be some way to invoke reset on these optional fields to make them reset once more. The alternative is to either explicitly parse out the 'userid:password@' field from my ftp URI or start a uri_builder from scratch and reinitialize the fields.

John

if (CURLE_OK != res) {
    // we failed
    network::uri_builder bldr(rFTPUri);
    // reset the boost optional field
    bldr.user_info(network::uri::string_type());
    std::stringstream ss;
    ss << "error [" << curl_easy_strerror(res)
        << "] uri [" << bldr.uri() << "]" << std::endl;
    if (res == CURLE_REMOTE_FILE_NOT_FOUND) {
        throw UtlFileNotFoundException(ss.str());
    } else {
        throw UtlIOException(ss.str());
    }
} else {

The unit tests for encoding/decoding do not look right

They look quite wrong in fact.

See also cpp-netlib/cpp-netlib#314

Incorrect port information after applying uri's assignment operator

After overriding an URI that has port info with an URI that does not have port info, the URI object still tells that it has port info. I guess there is a bug in the implementation of network::uri's assignment operator.

Furthermore, when I pass this re-assigned URI object into the network::uri_builder constructor, the constructed new URI not only still tells it has port info, but also creates a broken URI string when calling network::uri::string(). Checking with Google's AddressSanitizer it reports a heap-use-after-free error. It seems that the string_view that points to the port info is dangling.

Some unit tests for reproduction:

#include <gtest/gtest.h>
#include "network/uri/uri_builder.hpp"

TEST(UriTest, AssignmentOperator) {
  network::uri a("http://a.com:1234");
  ASSERT_TRUE(a.has_port());

  const network::uri b("http://b.com");
  ASSERT_FALSE(b.has_port());

  a = b;
  ASSERT_FALSE(a.has_port()) << a.string();
}

TEST(UriTest, CopyConstructor) {
  network::uri a("http://a.com:1234");
  const network::uri b("http://b.com");
  a = b;

  const network::uri c(a);
  ASSERT_FALSE(c.has_port()) << c.string();
}

TEST(UriBuilderTest, ConstructFromUri) {
  network::uri a("http://a.com:1234");
  const network::uri b("http://b.com");
  a = b;

  network::uri_builder ub(a);  // ASAN reports heap-use-after-free here
  const network::uri c(ub.uri());
  ASSERT_FALSE(c.has_port()) << c.string();
}

int main(int argc, char** argv) {
  ::testing::InitGoogleTest(&argc, argv);
  return RUN_ALL_TESTS();
}

Output

[==========] Running 3 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 2 tests from UriTest
[ RUN      ] UriTest.AssignmentOperator
src/uri_test.cc:12: Failure
Value of: a.has_port()
  Actual: true
Expected: false
http://b.com
[  FAILED  ] UriTest.AssignmentOperator (0 ms)
[ RUN      ] UriTest.CopyConstructor
src/uri_test.cc:21: Failure
Value of: c.has_port()
  Actual: true
Expected: false
http://b.com
[  FAILED  ] UriTest.CopyConstructor (0 ms)
[----------] 2 tests from UriTest (0 ms total)

[----------] 1 test from UriBuilderTest
[ RUN      ] UriBuilderTest.ConstructFromUri
src/uri_test.cc:31: Failure
Value of: c.has_port()
  Actual: true
Expected: false
http://b.com:1234
[  FAILED  ] UriBuilderTest.ConstructFromUri (0 ms)
[----------] 1 test from UriBuilderTest (0 ms total)

[----------] Global test environment tear-down
[==========] 3 tests from 2 test cases ran. (0 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 3 tests, listed below:
[  FAILED  ] UriTest.AssignmentOperator
[  FAILED  ] UriTest.CopyConstructor
[  FAILED  ] UriBuilderTest.ConstructFromUri

 3 FAILED TESTS

I can also provide ASAN output if needed. My compiler is gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10) and I use the latest code from the master branch. Thank you for developing cppnetlib, btw -- it's awesome!

Encoding errors must be handled using an exception

At the moment, encoding errors are not handled gracefully. As stated here:

http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3625.html#error-reporting

We can define an exception.

huge list of errors

hello, i have a huge problem with compiling:
i use vs 2015
errors.txt

The uri class does not support allocators

Pass allocators as arguments to all constructors, not the move constructor.

Add support for scheme-specific normalization, resolution and comparison.

There is a simple way to do this by overloading the accessors that use the comparison ladder with function objects, then passing specific functions that do the work. e.g.

namespace network { namespace uri_normalize { namespace scheme {
  uri http(const uri &other);
  uri https(const uri &other);
}}

namespace network {
  class uri {
public:
    template <class NormalizeScheme>
    uri normalize(NormalizeScheme scheme) {
      return scheme(normalize(uri_comparison_level::path_segment_normalization));
    }
  };
}
} 

// Test code
network::uri instance("http://www.example.com:80/");
assert(instance.normalize(network::uri_normalize::scheme::http) == "http://www.example.com/");

Add convenience overloads to the uri_builder

For example:

class uri_builder {
public:
  uri_builder &port(std::uint16_t);

  // Key and Value must be converted to strings, but can be any type
  template <typename Key, typename Value>
  uri_builder &query(const Key &key, const Value &value);
};

Probably others.

Move constructor allocates memory and therefore potentially throws std::bad_alloc

It's considered bad form the throw from a move constructor. There are two options that I can think of:

1. Use new with nothrow when constructing pimpl_
2. Assign null to pimpl_
3. is error prone

Missing install rules

There's no install rule for cpp-netlib/uri.

When installing cpp-netlib, conforming to the README.rst from cpp-netlib/cpp-netlib, files such as network/uri/uri.hpp aren't installed.

Trying to compile an application using cpp-netlib's server results in a message such as this:

boost/network/protocol/http/impl/request.hpp:20:31: fatal error: network/uri/uri.hpp: No such file or directory

I'm going to work around it, but perhaps there should be an install rule for cpp-netlib/uri, or perhaps the CMakeLists from cpp-netlib/cpp-netlib should take care of installing these files.

optional<uri> returned from function is broken

Maybe this is more an issue with boost::optional. When I pass an optional<uri> returned from a function to uri::resolve the result has trailing '\0'.

namespace {
    boost::optional<network::uri> TryParseUri(const std::string& uri) {
        try {
            return network::uri{uri};
        }
        catch(network::uri_syntax_error) {
            return boost::optional<network::uri>();
        }
    }
}

TEST_F(uri_resolve_test, resolve_issue_xxx) {
    network::uri base{"http://a.com/"};

    boost::optional<network::uri> invalid1 = TryParseUri("http:/invalid//");
    boost::optional<network::uri> invalid2{"http:/invalid//"};

    auto result1 = base.resolve(*invalid1, network::uri_comparison_level::string_comparison);
    auto result2 = base.resolve(*invalid2, network::uri_comparison_level::string_comparison);

    ASSERT_EQ(result1.string(), result2.string());
}

Output:

 Value of: result2.string()
   Actual: "http://a.com/invalid//"
 Expected: result1.string()
   Which is: "http://a.com/invalid//\0"

The library cannot be build with Visual Studio 2013 U5

The library cannot be build because of constexpr and noexcept direct usage instead of using the boost macros which handle this in a backward compatible way.

Is it possible to reintroduce these macros or equivalent so the library can be consumed by Visual Studio 2013 U5?

uri::has_host() returns true for "http://"

If the URI is "http://" then uri::has_host() returns true even though there is no host. I expected it to return false.

Calling `authority` on URI with empty-string host crashes

This issue is similar to Issue #62, including the root cause. See that issue for more details.

For this issue, the following program exposes the crash.

auto uri = network::uri_builder()
    .authority(":1234")
    .uri();
std::cout << *uri.authority() << '\n';

The problem is the same as with #62: calling uri.authority causes a read access violation.

Backport to compile on VS2012

Grumble...

Percent-decoding does not accept non-ASCII octets

When building a network::uri object using network::uri_builder, query parameters that contain non-ASCII multibyte characters (e.g. UTF-8) are percent-encoded as expected. For example, http://example.com/q=법정동 becomes http://example.com/q=%EB%B2%95%EC%A0%95%EB%8F%99.

However, the other way around, when applying network::uri::decode to the encoded query parameter, a percent_decoding_error exception is thrown. I think this behavior is incorrect. According to RFC 3986 section 2.5 percent-encoding and decoding work at octet-level and should be otherwise agnostic about character encodings.

Suggested fix in network/uri/detail/decode.hpp:

-  if (h0 >= '8') {
-    // unable to decode characters outside the ASCII character set.
-    throw percent_decoding_error(uri_error::conversion_failed);
-  }

Unit tests for reproduction:

• Percent-encoding a UTF-8 query parameter works
• Percent-decoding a UTF-8 query parameter does not work

TEST(UriBuilderTest, PercentEncodingAcceptsNonAsciiOctets) {
  const std::string decoded = u8"법정동";
  const std::string encoded = "%EB%B2%95%EC%A0%95%EB%8F%99";

  network::uri_builder ub(network::uri("http://example.com"));
  ASSERT_NO_THROW(ub.append_query_key_value_pair("q", decoded));

  const network::uri uri = ub.uri();
  ASSERT_EQ(network::string_view(encoded), uri.query_begin()->second);
}

TEST(UriDecodeTest, PercentDecodingAcceptsNonAsciiOctets) {
  const std::string decoded = u8"법정동";
  const std::string encoded = "%EB%B2%95%EC%A0%95%EB%8F%99";

  std::string output;
  ASSERT_NO_THROW(network::uri::decode(encoded.begin(), encoded.end(),
                                       std::back_inserter(output)));
  ASSERT_EQ(decoded, output);
}

Output:

[==========] Running 2 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 1 test from UriBuilderTest
[ RUN      ] UriBuilderTest.PercentEncodingAcceptsNonAsciiOctets
[       OK ] UriBuilderTest.PercentEncodingAcceptsNonAsciiOctets (0 ms)
[----------] 1 test from UriBuilderTest (1 ms total)

[----------] 1 test from UriDecodeTest
[ RUN      ] UriDecodeTest.PercentDecodingAcceptsNonAsciiOctets
src/uri_test.cc:53: Failure
Expected: network::uri::decode(encoded.begin(), encoded.end(), std::back_inserter(output)) doesn't throw an exception.
  Actual: it throws.
[  FAILED  ] UriDecodeTest.PercentDecodingAcceptsNonAsciiOctets (0 ms)
[----------] 1 test from UriDecodeTest (0 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 2 test cases ran. (1 ms total)
[  PASSED  ] 1 test.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] UriDecodeTest.PercentDecodingAcceptsNonAsciiOctets

 1 FAILED TEST

Copy constructor fails!!!

network::uri u("http://google.com");
cerr << u.host() << '\n';
u = network::uri("http://microsoft.com");
cerr << u.host() << '\n';

yields:

google.com
microsoft.

URI builder doesn't accept non-array string literals

The uri_builder class rejects pointer-type string literals with a compiler error. Instead, string literals must be of array type.

const char* const p = "http";
const char* const q = "foo";
const auto u =
  network::uri_builder()
  .scheme(p)           // ERROR: 'impl' uses undefined struct 'network::detail::translate_impl<Source>'
  .host("example.com") // OK: string literal is array type
  .path(*reinterpret_cast<const char(*)[1]>(q)) // OK: cast to array type
  .uri();

This problem affects all functions that pass through the translate_impl trait, which is implemented for std::string and char arrays—but not char pointers.

URI templates

Test parsing of URI templates

http://tools.ietf.org/html/rfc6570

Adding a path to a url without the leading / separator produces an invalid url

I'm not sure if this is really a bug - more of a nice to have - but implementing it would seem trivial. The problem is that in order to fetch named files on the root directory of an FTP server I typically have to prepend a root separator before the name of the file as typically files listed from the FTP server do not individually have the leading separator character. I typically do the following:

memBufferPath = fs::path("/") / iter->getPath();

Given that the optional path() component is not present (it is optional) could you add logic to prepend the / separator if the specified path does not already contain one?

network::uri_builder bldr;
bldr.scheme("ftp")
    .host("192.168.1.1")
    .path("default_path.ext");
network::uri uri = bldr.uri();
std::cout << uri << std::endl;

prints the following

ftp://192.168.1.1default_path.ext

Error on removing adjacent slashes

Hello,

Adjacent slashes are not removed in some cases.
Please find the test case for the problem below:

TEST(uri_normalization_test, path_double_dash2) {
  network::uri instance("http://www.example.com/abc//elementary");
  ASSERT_EQ("http://www.example.com/abc/elementary",
            instance.normalize(network::uri_comparison_level::syntax_based).string());
}

When I run make test, CTest is failing with following output:

/home/user/cpp-netlib-uri_test/test/uri_normalization_test.cpp:143: Failure
Value of: instance.normalize(network::uri_comparison_level::syntax_based).string()
Actual: "http://www.example.com/abc//elementary"
Expected: "http://www.example.com/abc/elementary"
[ FAILED ] uri_normalization_test.path_double_dash2 (0 ms)

If we try to normalize http://www.example.com/abc//elementary, the normalized value should be http://www.example.com/abc/elementary.

Thanks.

Copying an uri without host component crashes.

The following will crash inside copy_range() when u is assigned to u2.

uri u("file:///foo");
uri u2 = u; // crashes

The problem is in the advance_parts() function that wrongly skips all three '/' after "file:" when it only should skip two because the third one belongs to the path component.

The root cause seems to be the same as in #64. The current implementation does not expect the host component to be empty.

Will not build with android NDK

The android NDK does not have std::stoi or std::to_string defined (see http://stackoverflow.com/questions/17950814/how-to-use-stdstoul-and-stdstoull-in-android/18124627#18124627 for some discussion about it).

These functions are used by the uri library to handle port conversion.

I am willing to look into fixing the issue and submitting a pull request, but am wondering what the best approach would be? Just defining these function within an #ifdef __ANDROID__? Using something like boost::lexical_cast? Creating new functions? Other suggestion that I may have missed?

Comparing a URI containing a percent-encoded query triggers assert

Comparing two nonempty URI instances triggers an assert if at least one of the URI instances contains at least one percent-encoded character in its query.

The following program demonstrates the problem.

#include <iostream>
#include <network/uri.hpp>

int main()
{
  const auto u = network::uri_builder()
    .scheme("http")
    .host("example.com")
    .query("foo", "bar\\baz")
    .uri();

  const auto v = network::uri_builder()
    .scheme("http")
    .host("example.com")
    .uri();

  std::cout << u.string() << '\n';

  const auto p = (u == v);
  std::cout << p << '\n';
}

Program output:

http://example.com?foo=bar\baz
Assertion failed: is_valid, file C:\Source\uri\src\uri.cpp, line 401

Whereas, the program behaves correctly by changing the bar\\baz query value to bar/baz. Likewise, the program fails for any query string that contains at least one percent-encoded character.

Add tests for internationalization of the URI

This includes:

• Passing internationalized URI strings to the URI constructor and checking that they parse correctly
• Passing internationalized / encoded strings as arguments to the URI builder
• Testing the uri accessors

placement new issues with gcc6

This project does not compile with gcc6 (at least on FreeBSD 10), apparently because of a new issue detection mechanism implemented in gcc6.

There seems to be a misunderstanding in the following parts.
src/boost/function/function_template.hpp at line 572
src/boost/function/function_base.hpp at line 308

Here are the errors as reported by gcc:

src/boost/function/function_template.hpp:572:11: error: placement new constructing an object of type 'network_boost::algorithm::detail::token_finderF<network::detail::normalize_path_segments(network::string_view)::<lambda(char)> >' and size '8' in a region of type 'char' and size '1' [-Werror=placement-new=]

src/boost/function/function_base.hpp:308:13: error: placement new constructing an object of type 'network_boost::detail::function::functor_manager_common<network_boost::algorithm::detail::token_finderF<network::detail::normalize_path_segments(network::string_view)::<lambda(char)> > >::functor_type {aka network_boost::algorithm::detail::token_finderF<network::detail::normalize_path_segments(network::string_view)::<lambda(char)> >}' and size '8' in a region of type 'char' and size '1' [-Werror=placement-new=]

Sadly, I'm not familiar enough with this 'new placement' thing to understand what the heck is going on here.
It's just a warning, so it should be possible to compile it anyway, but not without disabling the "warning as error" option.

Assigning a default-constructed URI crashes

Assigning a default-constructed URI to a non-default-constructed URI crashes.

TEST(uri_test, default_constructed_assignment_test) {
  network::uri instance("http://www.example.com/");
  instance = network::uri(); // <-- CRASHES HERE
  ASSERT_EQ(instance, network::uri());
}

This seems to be the case no matter what the destination URI is, so long as that URI is not empty.

Whereas, assigning a default-constructed URI to another default-constructed URI succeeds.

Add config macros to autodetect C++11 features that are not implemented on all platforms

This will partly solve the problem in #2.

network::detail::decode function not compiling

Hello!

I have been trying to use your library for a project of mine in order to parse the query string of a request. The problem I had was that I receive the request URL encoded and I have to decode it. I saw that your library offers such capabilities and I have been trying to use them, but unfortunately, it doesn't compile and fails with the following error:

/my-project/third-party/cpp-netlib-uri/include/network/uri/detail/decode.hpp:64:11: error: no viable overloaded '='
      out = c;
      ~~~ ^ ~
/my-project/third-party/cpp-netlib-uri/include/network/uri/uri.hpp:590:20: note: in instantiation of function template specialization 'network::detail::decode<std::__1::__wrap_iter<char *>, std::__1::__wrap_iter<char *> >' requested
      here
    return detail::decode(first, last, out);
                   ^
src/my-file.cpp:60:12: note: in instantiation of function template specialization 'network::uri::decode<std::__1::__wrap_iter<char *>, std::__1::__wrap_iter<char *> >' requested here
      uri::decode(requestUri.begin(), requestUri.end(), decodedRequestUri.begin());
           ^
/usr/include/c++/v1/iterator:1258:7: note: candidate function (the implicit copy assignment operator) not viable: no known conversion from 'char' to 'const std::__1::__wrap_iter<char *>' for 1st argument
class __wrap_iter
      ^
/usr/include/c++/v1/iterator:1258:7: note: candidate function (the implicit move assignment operator) not viable: no known conversion from 'char' to 'std::__1::__wrap_iter<char *>' for 1st argument

Looking through the code of decode function:

template <class InputIterator, class OutputIterator>
OutputIterator decode(InputIterator in_begin, InputIterator in_end,
                      OutputIterator out_begin) {
  auto it = in_begin;
  auto out = out_begin;
  while (it != in_end) {
    if (*it == '%') {
      if (std::distance(it, in_end) < 3) {
        throw percent_decoding_error(uri_error::not_enough_input);
      }
      char c = '\0';
      it = decode_char(it, &c);
      out = c;
      ++out;
    } else {
      *out++ = *it++;
    }
  }
  return out;
}

it looks like the out variable is an iterator when the c variable is of type char. A possible fix would be to replace

out = c;

with:

*out = c;

and everything would be fine.

I am looking forward to your answer! Thank you!

`make install` results in nested include directory

If I build with cmake, make, make install, I get include/include/network in my install prefix.