w3c/wptserve

Web server designed for use with web-platform-tests

Repository Moved

This repository has moved; its new home is under web-platform-tests. Please use that repository for future pull requests and issues.

wptserve's Issues

Endorse as a feature the possibility to change the document root while the server is running

There's an interesting behaviour that allows us to change the document root while the server is running.

Once the server is running if you change httpd.router.doc_root the next request will use the new value as document root.

I would like to propose that this becomes a supported feature.
This is a web server created for running tests. Changing the document root without requiring a restart is a really nice feature.

Log spew reporting "An existing connection was forcibly closed..."

I see this on Windows 8 using the 1.1 release. It was not present on 1.0.1:
ERROR:wptserve:Traceback (most recent call last):
File "C:\mozbench\venv\lib\site-packages\wptserve\server.py", line 182, in han
dle_one_request
request_line_is_valid = self.get_request_line()
File "C:\mozbench\venv\lib\site-packages\wptserve\server.py", line 247, in get
_request_line
self.raw_requestline = self.rfile.readline(65537)
File "C:\Python27\Lib\socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
error: [Errno 10054] An existing connection was forcibly closed by the remote ho
st
This is logged continuously and prevents the application from closing cleanly.

Add support for full-duplex streaming

@annevk
@domenic

As discussed over on whatwg/fetch#229 in order to validate the now formalized support for full-duplex streaming in the WHATWG Fetch API we need to be able to test it.

One suggestion is to use hyper-h2 and implement an HTTP2 capable server that can be used for testing. @Lukasa & @tbetbetbe might be able to help out with that.

Support HTTP/2

It would be good to also have tests that go over HTTP/2. There are a couple of subtle differences with HTTP/1 and if HTTP/2 is on the rise it would be good to start covering it.

Differences I'm aware of:

• Header casing can matter in HTTP/1
• WebSocket only works in HTTP/1
• Chunked encoding is only in HTTP/1 (this will matter when Fetch gets "upload streams")

Handling of multiple responses

If we want to test 1xx responses, we need to be able to respond with one, and then follow up with another or a final non-1xx response.

Add feature to unregister existing handlers

It would be great to have a method in the Router class which would allow you to easily unregister a formerly registered handler.

.sub templating for optional keys

Is there a graceful way to use .sub processing when a value may not exist?

The simplest possible example is:

foo.sub.txt contains the following:

bar on the GET string is "{{GET[bar]}}"

If I make a request for "foo.sub.txt?bar=baz", I get what I expect:

bar on the GET string is "baz"

But If I make a request for just "foo.sub.txt" (leaving bar undefined on the GET string), I get a KeyError and status code 500. This seems rather fragile for a templating system.

is there a syntax to make it gracefully default to an empty string if a key doesn't existt?

Make request._raw_headers an official API

To be able to test whatwg/fetch#304 and XMLHttpRequest properly, we need to be able to tell what casing ends up being used for a header name.

JavaScript part:

promise_test(() => {
  return fetch("resources/echo-headers.py", {headers: [["THIS-is-A-test", 1], ["THIS-IS-A-TEST", 2]] }).then(res => res.text()).then(body => {
    assert_true(body.indexOf("THIS-is-A-test: 1, 2") !== -1)
  })
})

echo-headers.py (can probably be simplified):

def main(request, response):
    response.writer.write_status(200)
    response.writer.write_header("Content-Type", "text/plain")
    response.writer.end_headers()
    response.writer.write(str(request._raw_headers))
    response.close_connection = True

Class members for the WebTestHttpd class should be properties which access values via inner httpd instance

The WebTestHttpd class has not a private marked members like router. All those are getting forwarded to the inner httpd instance within the __init__() method. Customer code would be able to change the router value but without any affect. Instead cls.httpd.router would have to be updated.

As result this gives an inconsistency between class members and actually used values. Best here would be to make all those members properties which read/write the values from/to the inner httpd class instance.

WebKit bots having too many TIME_WAIT sockets

As shown in https://bugs.webkit.org/show_bug.cgi?id=159166#c21, some WebKit bots WPT servers (localhost-running) may have too many sockets in the TIME_WAIT state. This seems to prevent running the tests correctly for some runs.

Has anybody encountered the issue?
Maybe keep-alive should be implemented in wptserve to reduce the risk?

Accessing URLs with spaces included causes a 404 response

As I noticed today while working on mozilla/mozdownload#368, wptserve fails in handling of URLs with spaces included. An example URL like this:

http://127.0.0.1:8081/firefox/releases/23.0.1/win32/en-US/Firefox Setup 23.0.1.exe

will cause the following failure:

{"error": {"message": "", "code": 404}}

missing referrer causes exception

ERROR:web-platform-tests:referer
Traceback (most recent call last):
  File "/home/ubuntu/work/web-platform-tests/tools/wptserve/wptserve/server.py", line 253, in handle_one_request
    handler(request, response)
  File "/home/ubuntu/work/web-platform-tests/tools/wptserve/wptserve/handlers.py", line 142, in __call__
    response = pipeline(request, response)
  File "/home/ubuntu/work/web-platform-tests/tools/wptserve/wptserve/pipes.py", line 36, in __call__
    response = func(request, response, *args)
  File "/home/ubuntu/work/web-platform-tests/tools/wptserve/wptserve/pipes.py", line 129, in inner
    return f(request, response, *arg_values)
  File "/home/ubuntu/work/web-platform-tests/tools/wptserve/wptserve/pipes.py", line 362, in sub
    new_content = template(request, content)
  File "/home/ubuntu/work/web-platform-tests/tools/wptserve/wptserve/pipes.py", line 425, in template
    new_content, count = template_regexp.subn(config_replacement, content)
  File "/home/ubuntu/work/web-platform-tests/tools/wptserve/wptserve/pipes.py", line 413, in config_replacement
    value = value[item[1]]
  File "/home/ubuntu/work/web-platform-tests/tools/wptserve/wptserve/request.py", line 356, in __getitem__
    values = dict.__getitem__(self, key.lower())
KeyError: u'referer'

Add support to include files as data URLs

Tests containing data URLs are horrible because data URLs are opaque strings. We should improve on this somehow, IMO by providing some feature to substitute in a file as a data URL with a given Content-Type and with a base64 flag.

Disable printing of routes

On every server init I get the following printed to the console:

^/(.*)$
^/(.*)\.asis$
^/(.*)\.py$

This is done here:

wptserve/wptserve/router.py

Line 139 in 3b401bf

print self.routes[-1][1].pattern

There should be a way to turn this off.

0x0B and 0x0C stripped from header values

These are not HTTP whitespace. They should be preserved I think. Where is the header value parsing code?

Creation of a handler which doesn't set a string as response content hangs the server

The following handler will cause the server to hang while delivering the page content:

  def handler(request, response):
        response.content = 1
        return ()

This happens because a non-string value gets set as the content.

Multiple request.headers handling

There seems to be code to handle multiple request headers, but when I attempt sending them into a wptserve.handlers.handler I only get the last one sent in...despite request.headers being a list.

I'm still digging through the code, but if someone has a hunch what might be up, I'm eager for pointers. 😄

Multiple response headers

This is kinda the counter-part to #84.

Given a headers file containing:

Link: <support/cascade-import-002d.css>; rel="stylesheet"
Link: <support/cascade-import-002f.css>; rel="stylesheet"

This ends in the response headers being:

HTTP/1.1 200 OK
Content-Type: application/xhtml+xml
Link: <support/cascade-import-002f.css>; rel="stylesheet"
Server: BaseHTTP/0.3 Python/2.7.12
Date: Wed, 17 Aug 2016 16:56:16 GMT

i.e., only the last of the Link headers gets output.

While in theory Link: <support/cascade-import-002d.css>; rel="stylesheet", <support/cascade-import-002f.css>; rel="stylesheet" is equivalent, not all UAs have done this for all headers historically. As such, we should probably have some way of actually sending them in the response.

File names with spaces are 404

In web-platform-tests/wpt#2361 (comment) I wanted to test navigating to a file named "url foo", with a space, but wptserve returns a 404 response. Similarly for getting a directory "/foo bar/" gets 404. But a wptserve directory listing includes such files.

I tried looking at handlers.py to fix this myself and search around a bit, but without luck so far. @jgraham can you take a look?

Trickled responses are cacheable

With http://web-platform.test:8000/images/blue.png?pipe=trickle(d5) included as an image, Firefox will take 7s (as expected) to load initially; it will thereafter load instantly from cache. This is problematic. Trickled responses must not be cacheable.

Add support for `application/x-xpinstall` Content Type for `xpi` files

Lack of output encoding creates unavoidable security issues for .sub files

Template substitution is a nice way to avoid "active" content like .py or .php files where security errors can give server-side code execution, but without an encoding context it introduces unavoidable XSS and other injection issues into the .sub files. (as compared to PHP where you can apply a sanitization function to data before including it into a page)

Consider example.sub.js:

var userData = "{{GET[userData]}}";

I can GET /example.sub.js?%22%3B%20var%20evil%20%3D%20%22somethingBad

Which yields:

var userData = ""; var evil = "somethingBad";

The same issue exists for .html, .sub.headers, etc...

It is probably necessary to add syntax and functionality to the template engine to describe the context into which the substitution will occur so that it can be properly escaped.

Support different escaping for substitution pipe

Currently .sub.html changes & to & for {{GET[foo]}} which is nice if you use it in e.g. an attribute value, but less nice if you use it inside a <script>.

It would be nice to be able to ask for specific escaping rules for different contexts. At least for {{GET[]}} but possibly also for {{headers[]}}.

html: entity-escape & " ' < >
script-string: backslash-escape \ " ' <!-- </script

Support IPv6

This is needed for the future, and needed for IPv6-only networks and devices (which are increasingly prevalent).

Broken pipe error

I can get wptserve to produce the following:

ERROR:wptserve:Traceback (most recent call last):
  File "/Projects/html-ts/web-platform-tests/tools/wptserve/wptserve/server.py", line 220, in handle_one_request
    response.write()
  File "/Projects/html-ts/web-platform-tests/tools/wptserve/wptserve/response.py", line 200, in write
    self.write_status_headers()
  File "/Projects/html-ts/web-platform-tests/tools/wptserve/wptserve/response.py", line 190, in write_status_headers
    self.writer.end_headers()
  File "/Projects/html-ts/web-platform-tests/tools/wptserve/wptserve/response.py", line 401, in end_headers
    self.write_default_headers()
  File "/Projects/html-ts/web-platform-tests/tools/wptserve/wptserve/response.py", line 386, in write_default_headers
    self.write_header(name, f())
  File "/Projects/html-ts/web-platform-tests/tools/wptserve/wptserve/response.py", line 378, in write_header
    self.write("%s: %s\r\n" % (name, value))
  File "/Projects/html-ts/web-platform-tests/tools/wptserve/wptserve/response.py", line 418, in write
    self._wfile.write(self.encode(data))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe

It is difficult to reproduce reliably, but I believe it happens if wptserve tries to write a file to the socket when the browser has closed it already.

I can reproduce it only by going to a page and reloading it like a madman. It can take quite a few tries to obtain the result. (Hardware: super recent top of the line MBP with SSD — so it might reproduce differently on something slower — works in several browsers.)

Once it has happened once I get the impression that it becomes easier to produce it again if the server isn't killed (but I can't prove it, it may be an impression). That might mean that the worker in which the exception occurs survives but is in a corrupt state.

Feature request: handler for invalid HTTP responses

In web-platform-tests/wpt#5037, a test concerning an invalid HTTP response (specifically: a 204 response with a body) was found to be failing intermittently. I created a reduced demonstration here:

web-platform-tests/wpt@master...bocoup:serve-bug-demo

The underlying problem stems from the fact that each response handler shares a single stream for all requests. This behavior makes writing invalid HTTP responses generally unsafe. This is by design--wptserve's handlers are derived from Python's built-in BaseHTTPRequestHandler, whose documentation reads:

BaseHTTPRequestHandler has the following instance variables:

[...]

wfile

Contains the output stream for writing a response back to the client.
Proper adherence to the HTTP protocol must be used when writing to this
stream.

Maintainers expressed interest in explicitly testing such conditions, but in light of the above constraint, I believe wptserve will need to be extended with a new handler type designed explicitly for this purpose.