cherrypy/cheroot

Cheroot is the high-performance, pure-Python HTTP server used by CherryPy. Docs -->

Cheroot is the high-performance, pure-Python HTTP server used by CherryPy.

Status

The test suite currently relies on pytest. It's being run via GitHub Actions CI/CD workflows.

For Enterprise

Professional support for Cheroot is available as part of the Tidelift Subscription. The CherryPy maintainers and the maintainers of thousands of other packages are working with Tidelift to deliver one enterprise subscription that covers all of the open source you use. Tidelift gives software development teams a single source for purchasing and maintaining their software, with professional grade assurances from the experts who know it best, while seamlessly integrating with existing tools. Learn more.

Contribute Cheroot

Want to add something to upstream? Feel free to submit a PR or file an issue if unsure. Please follow CherryPy's common contribution guidelines. Note that PR is more likely to be accepted if it includes tests and detailed description helping maintainers to understand it better 🎉

Oh, and be pythonic, please 🐍

Don't know how? Check out How to Contribute to Open Source article by GitHub 🚀

License

cheroot's Issues

Outdated docstring in cheroot.server

The docstring says:

Simplest example on how to use this server::
    import cheroot.server
    def my_crazy_app(environ, start_response):
        status = '200 OK'
        response_headers = [('Content-type','text/plain')]
        start_response(status, response_headers)
        return ['Hello world!']
    server = cheroot.server.WSGIServer(
                ('0.0.0.0', 8070), my_crazy_app,
                server_name='www.cherrypy.example')
    server.start()

However it looks like that class got moved to cheroot.wsgi.

Add tests against CherryPy (master or latest stable)

Currently, CherryPy has tests against cheroot master.

And while that's interesting and good to know, it often too late to effectively capture regressions in cheroot (as they would have to already have been merged into master and the tests in cherrypy run). It's additionally difficult to correlate changes in cheroot with failures in CherryPy.

I propose instead that cheroot should run CherryPy's tests against itself, such that a PR could be validated against CherryPy before being accepted (and other commits as well).

I'm not yet fully sure how this proposal could work. There are some challenges:

• Which CherryPy version to use?
• Which Python(s) to use?
• Which tox env to use?
• What about chicken/egg problem (CherryPy changes depend on Cheroot changes)?
• How to integrate with CI? Should it be allowed to fail?

For now, I'm tempted to start with a tox env that can be invoked manually to validate the current code against CherryPy master... and we can build from there.

socket.timeout: The write operation timed out

Copy of cherrypy #1575
cherrypy/cherrypy#1575

• I'm submitting a ...
  [x] bug report
  [ ] feature request
  [ ] question about the decisions made in the repository

• Do you want to request a feature or report a bug?
  bug

• What is the current behavior?
  Several socket.timeout stack traces on error logs. No documentation to address this issue.

• If the current behavior is a bug, please provide the steps to reproduce and if possible a screenshots and logs of the problem.

Lots of following trace in error logs.

[10/Mar/2017:11:13:44] ENGINE socket.error 'The write operation timed out'
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/cheroot/server.py", line 960, in communicate
    req.respond()
  File "/usr/local/lib/python3.5/dist-packages/cheroot/server.py", line 786, in respond
    self.server.gateway(self).respond()
  File "/usr/local/lib/python3.5/dist-packages/cheroot/wsgi.py", line 101, in respond
    self.write(chunk)
  File "/usr/local/lib/python3.5/dist-packages/cheroot/wsgi.py", line 183, in write
    self.req.write(chunk)
  File "/usr/local/lib/python3.5/dist-packages/cheroot/server.py", line 839, in write
    self.conn.wfile.write(chunk)
  File "/usr/local/lib/python3.5/dist-packages/cheroot/makefile.py", line 26, in write
    self._flush_unlocked()
  File "/usr/local/lib/python3.5/dist-packages/cheroot/makefile.py", line 35, in _flush_unlocked
    n = self.raw.write(bytes(self._write_buf))
  File "/usr/lib/python3.5/socket.py", line 593, in write
    return self._sock.send(b)
  File "/usr/lib/python3.5/ssl.py", line 861, in send
    return self._sslobj.write(data)
  File "/usr/lib/python3.5/ssl.py", line 586, in write
    return self._sslobj.write(data)
socket.timeout: The write operation timed out

• What is the expected behavior?

If fatal or of concern, guide to address if not, ability to silence.

• What is the motivation / use case for changing the behavior?

Need a guide that google can pick up for any one else having similar issues.

• Please tell us about your environment:

• Python version: Python 3.5.2
• OS: Linux n2 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
• Browser: all

[TODO] Change Travis CI config to use newly presented build stages feature

Refs:

• https://blog.travis-ci.com/2017-05-11-introducing-build-stages
• https://docs.travis-ci.com/user/build-stages#Examples

Note: I'm going to move PYPI deployment to separate stage, so it would wait for tests in all envs to complete.

Server docstring example incompatible with Python 3

• I'm submitting a ...
  [x] bug report
  [ ] feature request
  [ ] question about the decisions made in the repository

• Do you want to request a feature or report a bug?
  Report a bug

• What is the current behavior?
  The wsgi docstring "hello world" example is incompatible with Python 3

• If the current behavior is a bug, please provide the steps to reproduce and if possible a screenshots and logs of the problem. If you can, show us your code.
  Requests to the server return HTTP ERROR 408
  The console shows ValueError: WSGI Applications must yield bytes

• What is the expected behavior?
  200 OK, "Hello world!"

• What is the motivation / use case for changing the behavior?
  This was frustrating as someone evaluating cheroot

• Please tell us about your environment:

• Python version: 3.6.4
• Cheroot version: 6.0.0
• OS: Mac- Browser: [all ]

cheroot.server.Server() does not correctly parse the request_uri and set the query string.

• I'm submitting a bug report

• What is the current behavior?

Unfortunately the code is broken and doesn't correctly set self.qs.

cheroot/cheroot/server.py

Lines 709 to 724 in 60ec5b6

	scheme, authority, path = self.parse_request_uri(uri)
	if path is None:
	self.simple_response('400 Bad Request',
	'Invalid path in Request-URI.')
	return False
	if NUMBER_SIGN in path:
	self.simple_response('400 Bad Request',
	'Illegal #fragment in Request-URI.')
	return False
	if scheme:
	self.scheme = scheme
	qs = EMPTY
	if QUESTION_MARK in path:
	path, qs = path.split(QUESTION_MARK, 1)

Does not currently correct parse the reques_uri and set self.qs.

Anyone using cheroot with CherryPy will not have environ['QUERY_STRING'] set correctly, which was fun to debug.

• What is the expected behavior?

To parse the request_uri correctly and set self.qs so that it may be set in the WSGI environ.

• What is the motivation / use case for changing the behavior?

urlparse will not return the query string in the path, so it needs to be appended again, or returned from parse_request_uri, then the check on line

cheroot/cheroot/server.py

Line 723 in 60ec5b6

if QUESTION_MARK in path:

will no longer be necessary.

• Please tell us about your environment:

• Python version: 2.7.x
• Cheroot version: 5.7.0 (issue has existed since 5.0.0: https://github.com/cherrypy/cheroot/blob/v5.0.0/cheroot/server.py#L785)

test_core.HTTPTests.test_parse_uri_invalid_uri assertion invalid

Currently, the test_parse_uri_invalid_uri asserts that a UTF-8 encoded resource identifier is invalid. However, the HTTP spec allows for any byte sequence for the URI, and in fact, when using curl to request a unicode string, I see that it transmits unquoted UTF-8 bytes:

curl http://localhost:8080/йопта!

It's difficult for me to illustrate here how that transmits UTF-8, but trust me, it does.

It seems we're not alone, but I still consider curl to be a widely-accepted HTTP client (perhaps the most widely accepted command-line client), so I consider them a de facto authority in correctness.

Moreover, older versions of CherryPy supported these requests (at least on Python 2), as I reported in #55. Therefore, I believe this test should be removed.

Cheroot uses _pyio incorrectly

Attempting to use Cheroot (via CherryPy) on Cygwin has revealed that Cheroot imports a private module, _pyio. "Outside" use of _pyio this way is not supported.

CherryPy should use a different module or interface for this functionality (

cheroot/cheroot/makefile.py

Line 5 in 86c6b24

import _pyio as io

).

A side-effect of fixing this issue is that this may help CherryPy run on Cygwin once again.

See https://bugs.python.org/issue32287 for more discussion or context.

(originally posted here: cherrypy/cherrypy#1704 (comment))

Windows regression: tests get stuck in CI after first one

It looks like smth in #80 breaks the build in CI.

• I'm submitting a ...
  [x] bug report
  [ ] feature request
  [ ] question about the decisions made in the repository

• Do you want to request a feature or report a bug?
  bug

• What is the current behavior?
  win build gets stuck

• If the current behavior is a bug, please provide the steps to reproduce and if possible a screenshots and logs of the problem. If you can, show us your code.
  see: https://ci.appveyor.com/project/CherryPy/cheroot/build/291

• What is the expected behavior?
  win build succeeds

• What is the motivation / use case for changing the behavior?
  green CI

• Please tell us about your environment:

• Python version: 3.6/2.7
• Cheroot version: 6.1.1
• CherryPy version: ‒
• OS: Windows
• Browser: ‒

• Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, e.g. stackoverflow, gitter, etc.)
  build gets stuck at this point

cheroot/server.py::cheroot.server PASSED                                 [  1%]
cheroot/test/test_conn.py::test_HTTP11_persistent_connections 

Port tests from CherryPy

Following the refresh of cheroot from CherryPy and the new release of Cheroot 5.0, the codebase is stable and suitable for release, but the test suite is basically disabled.

We'll need to invest some time to adapt the test suite to work with the cheroot code and get the tests passing again, at which point most if not all of those tests can be removed from CherryPy, which can rely on the tests executed here.

Tests hang on macOS

While running the tests on my macOS 10.13.1, Python 3.6.3, the tests got to this point and stalled:

...
 cheroot/test/test_conn.py::ConnectionCloseTests.test_Streaming_with_len_10 ✓       43% ████▍     
 cheroot/test/test_conn.py::ConnectionCloseTests.test_Streaming_with_len_11 ✓       45% ████▌     
 cheroot/test/test_conn.py::ConnectionCloseTests.test_readall_or_close ✓            47% ████▊     ^CERROR: KEYBOARDINTERRUPT

Python 2.7 test failures since 5.9 release

Tests are failing thus in CherryPy. Offending commit is e0c117a.

HTTP Keep-Alive Connections not handled correctly

• I'm submitting a ...

• bug report
• feature request
• question about the decisions made in the repository

• Do you want to request a feature or report a bug?
  Bug

• What is the current behavior?
  When the number of open Keep-Alive sessions exceeds CherryPy's thread count specified by server.thread_pool, CherryPy queues all new Keep-Alive connections up to the timeout specified by server.socket_timeout.

• If the current behavior is a bug, please provide the steps to reproduce and if possible a screenshots and logs of the problem. If you can, show us your code.

# mini_cherrypy_server.py
import cherrypy


class HelloWorld(object):
    @cherrypy.expose
    def index(self):
        print(cherrypy.request.headers)
        return "Hello world!"


if __name__ == '__main__':
    cherrypy.config.update({'server.thread_pool': 5})
    cherrypy.quickstart(HelloWorld())

# mini_client.py
import requests
from multiprocessing.dummy import Pool as ThreadPool

urls = ['http://localhost:8080/index'] * 10

def do_request(url):
    sess = requests.Session() # forces a keep-alive session
    resp = sess.get(url)
    print(resp.text)

def main():
    thread_pool = ThreadPool()
    results = thread_pool.map(do_request, urls)
    thread_pool.close()
    thread_pool.join()
    
if __name__ == '__main__':
    main()

• What is the expected behavior?
  Allow threads in the thread pool to answer and process new connections by rotating "sleeping" sessions back into the queue.

• What is the motivation / use case for changing the behavior?
  Correctly support the HTTP/1.1 spec.

• Please tell us about your environment:

• Cheroot version: 6.0.0
• CherryPy version: 13.1.0 (if applicable)
• Python version: 3.5.2 32-bit
• OS: Windows 10
• Browser: [all]

Introduce GitHub templates for issues and PRs

Cheroot 5.9.1 has invalid files

Yegor Y. sent me an e-mail reporting an issue with the cheroot 5.9.1 package. It contains invalid files like "test_core_BACKUP_1440.py". These files cause issues in compilation.

These files were not intended to be in the distribution. They happened to be on my machine when I cut the release manually. I cut the release manually because the CI process takes half an eternity to complete.

I'll cut a new 5.9.2 release to correct the issue.

Unix socket mode not set to 0o777

Currently, cheroot deletes any pre-existing AF_UNIX socket file, then attempts to chmod the now non-existent file to 0o777, and only then creates it.

The correct behaviour would be to chmod the socket after creating it.

[TODO] Add valid .gitignore and .gitattributes

Need support for certificate FILETYPE_ASN1

• I'm submitting a ...
  [ ] bug report
  [x ] feature request
  [ ] question about the decisions made in the repository

• Do you want to request a feature or report a bug?

I would call this a new feature.

• What is the current behavior?

Currently, cheroot hardcodes the certificate as FILETYPE_PEM in two places. The first in the call to SSL.Context.use_certificate_file() by not passing the FILETYPE parameter. The second is in the call to crypto.load_certificate() where FILETYPE_PEM is hard-coded.

• If the current behavior is a bug, please provide the steps to reproduce and if possible a screenshots and logs of the problem. If you can, show us your code.

• What is the expected behavior?

• What is the motivation / use case for changing the behavior?

Since both FILETYPEs exist, it would be nice for cheroot to support them.

• Please tell us about your environment:

• Python version: 2.7
• Cheroot version: 5.7.0
• CherryPy version: X.X.X (if applicable)
• OS: Windows 7 (I know...)
• Browser: Chrome 57.0.2987.133 (64-bit)

• Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, e.g. stackoverflow, gitter, etc.)

is there any right way to interrupt a started server?

• I'm submitting a ...
  [x] bug report
  [ ] feature request
  [ ] question about the decisions made in the repository

• Do you want to request a feature or report a bug?

bug? or may be it is a feature request?... no.. I think it is a bug (I don't know)

• What is the current behavior?

the method server.interrupt [of class cherrot.Server] seems like useless.

I can not invoke the method server._set_interrupt() [the property server.interrupt] from the same thread where I just invoked server.start(), because the method server.start() is blocking the thread.

and I can not invoke server.interrupt() in another thread, because server._set_interrupt() invokes server.stop() inside himself. but server.stop() isn't threadsafe!

the method server.stop() isn't threadsafe -- because it has a race condition with the method server.start(). and there isn't a way how to wait for the moment when the method server.start() will pass his initialization (dangerous) point.

how can I use right way to interrupt a server?

I've found the way (the workaround?) like this:

### thread A ###
try:
    server.start()
except MyStopError:
    pass
finally:
    server.stop() # ---- will be invoked always after ``server.start()`` and always in the same thread

### thread B ###
server._interrupt = MyStopError() # ---- doesn't do anything except setting a flag

i.e I've tried to avoid using server._set_interrupt(), so I've got ability to invoke the method server.stop() inside the thread A after invoking the method server.start() in the same thread (the race condition wouldn't happen).

• What is the expected behavior?

if the method server.start() an server.stop() would be use synchronization primitives -- it would NOT fix the problem(!) . because in this case the race contition with server.stop() will still happen at moment before starting the method server.start() (in detail: in thread B we can at random invoke stop() before invoking start(), so the stop() will not stop anything).

and of course -- I can not set server.ready = False in the thread B, because I still have race condition for this variable with the method server.start() invoked in the thread A. it is the real problem..

I think one of the way -- it is to split the method server.start() to two separated methods:

• the first method for a start-initialization phase (not blocking method)
• the second method for a serving phase (blocking method).

this separation will allow to know that you can correctly invoke the method server.stop() -- after the start-initialization phase is done.

thank you in advance!

TypeError('WSGI response header value 469 is not of type str.',)

I'm submitting a ...

• bug report
• feature request
• question about the decisions made in the repository

Describe the bug. What is the current behavior?

I'm using web.py and Vue.js, when it gives a response header the bug occurs. I found that was because of in the header, Content-Length field was provided in number not string. I have to look up for the standard, and the standard didn't specify the Content-Length field has to be a string.

I tried my compiled Vue page with default python http.server, it works fine.

What is the motivation / use case for changing the behavior?

To Reproduce

Steps to reproduce the behavior:

1. Run '...'
2. Make request '....'
3. See error

To be honest, I'm pretty new to python web dev, I'm not sure if this is a bug or it's a mistake made by myself. Here's the log:

127.0.0.1:49575 - - [06/May/2018 20:56:23] "HTTP/1.1 GET /" - 200 OK
-------------------[('Connection', 'close'), ('Content-Type', 'text/html;charset=utf-8'), ('Content-Length', 469)]
TypeError('WSGI response header value 469 is not of type str.',)
-------------------[('Connection', 'close'), ('Content-Type', 'text/html;charset=utf-8'), ('Content-Length', 469)]
-------------------[('Connection', 'close'), ('Content-Type', 'text/html;charset=utf-8'), ('Content-Length', 469)]
TypeError('WSGI response header value 469 is not of type str.',)
-------------------[('Connection', 'close'), ('Content-Type', 'text/html;charset=utf-8'), ('Content-Length', 469)]
TypeError('WSGI response header value 469 is not of type str.',)
Traceback (most recent call last):
  File "D:\Env\Python\lib\site-packages\cheroot\server.py", line 1193, in communicate
    req.respond()
  File "D:\Env\Python\lib\site-packages\cheroot\server.py", line 997, in respond
    self.server.gateway(self).respond()
  File "D:\Env\Python\lib\site-packages\cheroot\wsgi.py", line 140, in respond
    for chunk in filter(None, response):
  File "D:\Env\Python\lib\site-packages\web\httpserver.py", line 234, in __iter__
    self.start_response(self.status, self.headers)
  File "D:\Env\Python\lib\site-packages\web\httpserver.py", line 287, in xstart_response
    out = start_response(status, response_headers, *args)
  File "D:\Env\Python\lib\site-packages\cheroot\wsgi.py", line 176, in start_response
    'WSGI response header value %r is not of type str.' % v)
TypeError: WSGI response header value 469 is not of type str.
Traceback (most recent call last):
  File "D:\Env\Python\lib\site-packages\cheroot\server.py", line 1193, in communicate
    req.respond()
  File "D:\Env\Python\lib\site-packages\cheroot\server.py", line 997, in respond
    self.server.gateway(self).respond()
  File "D:\Env\Python\lib\site-packages\cheroot\wsgi.py", line 140, in respond
    for chunk in filter(None, response):
  File "D:\Env\Python\lib\site-packages\web\httpserver.py", line 234, in __iter__
    self.start_response(self.status, self.headers)
  File "D:\Env\Python\lib\site-packages\web\httpserver.py", line 287, in xstart_response
    out = start_response(status, response_headers, *args)
  File "D:\Env\Python\lib\site-packages\cheroot\wsgi.py", line 176, in start_response
    'WSGI response header value %r is not of type str.' % v)
TypeError: WSGI response header value 469 is not of type str.
TypeError('WSGI response header value 469 is not of type str.',)
Traceback (most recent call last):
  File "D:\Env\Python\lib\site-packages\cheroot\server.py", line 1193, in communicate
    req.respond()
  File "D:\Env\Python\lib\site-packages\cheroot\server.py", line 997, in respond
    self.server.gateway(self).respond()
  File "D:\Env\Python\lib\site-packages\cheroot\wsgi.py", line 140, in respond
    for chunk in filter(None, response):
  File "D:\Env\Python\lib\site-packages\web\httpserver.py", line 234, in __iter__
    self.start_response(self.status, self.headers)
  File "D:\Env\Python\lib\site-packages\web\httpserver.py", line 287, in xstart_response
    out = start_response(status, response_headers, *args)
  File "D:\Env\Python\lib\site-packages\cheroot\wsgi.py", line 176, in start_response
    'WSGI response header value %r is not of type str.' % v)
TypeError: WSGI response header value 469 is not of type str.
Traceback (most recent call last):
  File "D:\Env\Python\lib\site-packages\cheroot\server.py", line 1193, in communicate
    req.respond()
  File "D:\Env\Python\lib\site-packages\cheroot\server.py", line 997, in respond
    self.server.gateway(self).respond()
  File "D:\Env\Python\lib\site-packages\cheroot\wsgi.py", line 140, in respond
    for chunk in filter(None, response):
  File "D:\Env\Python\lib\site-packages\web\httpserver.py", line 234, in __iter__
    self.start_response(self.status, self.headers)
  File "D:\Env\Python\lib\site-packages\web\httpserver.py", line 287, in xstart_response
    out = start_response(status, response_headers, *args)
  File "D:\Env\Python\lib\site-packages\cheroot\wsgi.py", line 176, in start_response
    'WSGI response header value %r is not of type str.' % v)
TypeError: WSGI response header value 469 is not of type str.
-------------------[('Connection', 'close'), ('Content-Type', 'text/html;charset=utf-8'), ('Content-Length', 469)]
TypeError('WSGI response header value 469 is not of type str.',)
Traceback (most recent call last):
  File "D:\Env\Python\lib\site-packages\cheroot\server.py", line 1193, in communicate
    req.respond()
  File "D:\Env\Python\lib\site-packages\cheroot\server.py", line 997, in respond
    self.server.gateway(self).respond()
  File "D:\Env\Python\lib\site-packages\cheroot\wsgi.py", line 140, in respond
    for chunk in filter(None, response):
  File "D:\Env\Python\lib\site-packages\web\httpserver.py", line 234, in __iter__
    self.start_response(self.status, self.headers)
  File "D:\Env\Python\lib\site-packages\web\httpserver.py", line 287, in xstart_response
    out = start_response(status, response_headers, *args)
  File "D:\Env\Python\lib\site-packages\cheroot\wsgi.py", line 176, in start_response
    'WSGI response header value %r is not of type str.' % v)
TypeError: WSGI response header value 469 is not of type str.
-------------------[('Connection', 'close'), ('Content-Type', 'text/html;charset=utf-8'), ('Content-Length', 469)]
TypeError('WSGI response header value 469 is not of type str.',)
Traceback (most recent call last):
  File "D:\Env\Python\lib\site-packages\cheroot\server.py", line 1193, in communicate
    req.respond()
  File "D:\Env\Python\lib\site-packages\cheroot\server.py", line 997, in respond
    self.server.gateway(self).respond()
  File "D:\Env\Python\lib\site-packages\cheroot\wsgi.py", line 140, in respond
    for chunk in filter(None, response):
  File "D:\Env\Python\lib\site-packages\web\httpserver.py", line 234, in __iter__
    self.start_response(self.status, self.headers)
  File "D:\Env\Python\lib\site-packages\web\httpserver.py", line 287, in xstart_response
    out = start_response(status, response_headers, *args)
  File "D:\Env\Python\lib\site-packages\cheroot\wsgi.py", line 176, in start_response
    'WSGI response header value %r is not of type str.' % v)
TypeError: WSGI response header value 469 is not of type str.

Lines starting with "-----------" are my debug output.

Expected behavior

As I posted above.
Details

Environment

• Cheroot version: 6.2.4
• CherryPy version: X.X.X (if applicable)
• Python version: 3.6.5
• OS: Windows 10
• Browser: Chrome 66 | Edge

Additional context

[TODO] Enable PyPy v5.9.0 in Travis CI once it's available there

Ref: https://morepypy.blogspot.com/2017/10/pypy-v59-released-now-supports-pandas.html

CherryPy tests crashing on macOS

While testing on a mac, I find the CherryPy tests fail reliably if I include the change from #53 and proceed normally if I use 3e3ef23 instead.

Annoyingly, the tests crash silently and apparently between tests, so it's difficult to tell why.

$ tox -- -k test_conn.py
python develop-inst-noop: /Users/jaraco/Dropbox/code/public/cherrypy
python installed: backports.unittest-mock==1.3,certifi==2017.11.5,chardet==3.0.4,-e git+gh://cherrypy/cheroot@93d77a279728bdf22de2fac4e7ecab1205988d46#egg=cheroot,-e git+gh://cherrypy/cherrypy@32471496cf39901b6cae587dd36f9759dde27969#egg=CherryPy,codecov==2.0.9,coverage==4.4.2,graphviz==0.8.1,idna==2.6,jaraco.classes==1.4.3,objgraph==3.1.1,path.py==10.5,portend==2.2,py==1.5.2,pytest==3.2.5,pytest-cov==2.5.1,pytest-sugar==0.9.0,pytz==2017.3,repoze.lru==0.7,requests==2.18.4,Routes==2.4.1,six==1.11.0,tempora==1.9,termcolor==1.1.0,urllib3==1.22
python runtests: PYTHONHASHSEED='3343766152'
python runtests: commands[0] | pytest -k test_conn.py
Test session starts (platform: darwin, Python 3.6.3, pytest 3.2.5, pytest-sugar 0.9.0)
cachedir: .cache
rootdir: /Users/jaraco/Dropbox/code/public/cherrypy, inifile: pytest.ini
plugins: sugar-0.9.0, cov-2.5.1, backports.unittest-mock-1.3

 cherrypy/test/test_conn.py::ConnectionCloseTests.test_HTTP10_KeepAlive ✓            5% ▌         
 cherrypy/test/test_conn.py::ConnectionCloseTests.test_HTTP11 ✓                      9% ▉         
 cherrypy/test/test_conn.py::ConnectionCloseTests.test_Streaming_no_len ✓           14% █▍        
 cherrypy/test/test_conn.py::ConnectionCloseTests.test_Streaming_with_len ✓         18% █▊        
 cherrypy/test/helper.py::ConnectionCloseTests.test_gc ✓                            23% ██▍       ERROR: InvocationError: '/Users/jaraco/Dropbox/code/public/cherrypy/.tox/python/bin/pytest -k test_conn.py'
____________________________________________ summary _____________________________________________
ERROR:   python: commands failed

Pytest 3.3.0 (released Nov 23) dropped support for 2.6 and 3.3

Ref: https://docs.pytest.org/en/latest/changelog.html

TODO: act accordingly

macOS Python nightly tests failing

The nightly tests are failing and blocking releases.

[TODO] Re-enable `--testmon` once the coverage issue is fixed

Because of tarpas/pytest-testmon#86 --testmon corrupts coverage data, so I'm dropping it from pytest.ini (8a161e8). This should be reverted once upstream issue fixed.

TODO: git revert 8a161e87

Make HTTP server correctly read trailers (Trailer HTTP headers, which come after body)

• I'm submitting a ...

• bug report
• feature request
• question about the decisions made in the repository

• Do you want to request a feature or report a bug?
  It's a bug I noticed during tests rewrite for pytest runner #67

• What is the current behavior?
  Unread ending of the request with headers gets read by the next request parser, which recognises it as garbage (because it's not a valid Request-Line).

Extracted from the xfail message (removed in f55e41d):

Server does not correctly read trailers/ending of the previous HTTP request, thus the second request fails as the server tries to parse b'Content-Type: application/json\r\n' as a Request-Line. This results in HTTP status code 400, instead of 413.

• If the current behavior is a bug, please provide the steps to reproduce and if possible a screenshots and logs of the problem. If you can, show us your code.
  See conn_test.py::test_Chunked_Encoding.

• What is the expected behavior?
  Payload of the request with trailers is completely read out from the socket, so that it won't affect the next request.

• What is the motivation / use case for changing the behavior?
  Emm.. My perfectionism, I guess.

• Please tell us about your environment:

• Python version: 3.6.2 (but shouldn't version-agnostic)
• Cheroot version: v6.0.0 (and probably all prev versions)
• CherryPy version: N/A
• OS: Gentoo Linux
• Browser: N/A

• Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, e.g. stackoverflow, gitter, etc.)

TypeError in catch exception ssl.SSLerror

• I'm submitting a bug report

I got traceback error.

cheroot/server.py line 1515, in start
cheroot/server.py line 1590, in tick
cheroot/ssl/builtin.py line 100, in wrap
TypeError: argument of type 'int' is not iterable

builtin.py line 100 code:

elif 'handshake operation timed out' in ex.args[0]:

• Python version: 2.7.11
• Cheroot version: 5.8.3
• CherryPy version: 11.0.0 (if applicable)
• OS: debian7.7
• Apache-HttpClient/4.2.1 (java 1.5)

Worker thread is exiting abnormally

Hello,

We are getting following error in cherrypy worker thread and leading it to exit.

[2017-06-22 04:15:43,892] [ERROR] 'module' object has no attribute 'esocket_errors_to_ignore'. Traceback -   File "/usr/local/lib/python2.7/dist-packages/cheroot/workers/threadpool.py", line 100, in run
    conn.communicate()
  File "/usr/local/lib/python2.7/dist-packages/cheroot/server.py", line 962, in communicate
    req.simple_response('408 Request Timeout')
  File "/usr/local/lib/python2.7/dist-packages/cheroot/server.py", line 813, in simple_response
    if x.args[0] not in errors.esocket_errors_to_ignore:

There is a typo in 'errors.(e)socket_errors_to_ignore'. Can you please fix it?

PyPy test hangs on test_conn

Beginning with 3e1efa3, in test_conn, the test is consistently hanging in Travis. As you can see, the only changes were to the project layout, not to any of the code. Additionally, these tests pass without error on my local version of PyPy (5.6).

Ship stubs with detached annotations

Now that there's a PEP and mypy support for shipping stubs, it would be useful to have them.

• I'm submitting a ...

• bug report
• feature request
• question about the decisions made in the repository

• Do you want to request a feature or report a bug?
  idea 💡

• What is the current behavior?
  No annotations

• If the current behavior is a bug, please provide the steps to reproduce and if possible a screenshots and logs of the problem. If you can, show us your code.

• What is the expected behavior?

• What is the motivation / use case for changing the behavior?
  Having autocomplete for at least public interfaces. Consider this as an essential part of the public API.

• Please tell us about your environment:

• Cheroot version: master
• CherryPy version: -
• Python version: -
• OS: -
• Browser: -

• Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, e.g. stackoverflow, gitter, etc.)

User story based docs

I'm submitting a ...

• bug report
• feature request
• question about the decisions made in the repository

Do you want to request a feature or report a bug?
Docs

What is the current behavior?
Lack of docs.

What is the motivation / use case for changing the behavior?
Having well-documented ways of how to use Cheroot.

Details
I think, we should try out user story based documentation approach. What users are looking for is the usage patterns of Cheroot along with other tooling. Also, we need to specify the boundaries of the project, so that ppl won't expect unrealistic things from it.

get the access log

• I'm submitting a ...

• bug report
• feature request
• question about the decisions made in the repository

Is there any way to display (and optionally write to a file) the access log?
If yes, how?
If not, would it possible in the near future?

• Please tell us about your environment:

• Python version: 3.6.4
• Cheroot version: 6.2.4
• OS: Windows

[Task] Add encrypted password to Appveyor config

@jaraco I'm implementing support for PEP 513 and want to have win32 tagged wheel built as well. Since I don't have maintainer access to PYPI, I'm asking you to do the following:

1. Please use this tool to encrypt your PYPI password.
2. After that put it into Appveyor config or post in here and I'll do the rest.

Ref: 8593dc2

Make Python 2 behave closer to Python 3 standard behavior

Add the following snippet to the top of all modules:

from __future__ import absolute_import, division, print_function, unicode_literals
__metaclass__ = type

• I'm submitting a ...

• bug report
• feature request
• question about the decisions made in the repository

• Do you want to request a feature or report a bug?
  code cross-compatibility improvement for better Python 3/2 support.

• What is the current behavior?
  Has old-style classes. Python 2-centric syntax present.

• If the current behavior is a bug, please provide the steps to reproduce and if possible a screenshots and logs of the problem. If you can, show us your code.

• What is the expected behavior?
  New-style classes by default. New syntax all over the place.

• What is the motivation / use case for changing the behavior?
  Make project code in Python 2 env behave more like Python 3.

• Please tell us about your environment:

• Cheroot version: master branch
• CherryPy version: N/A
• Python version: N/A
• OS: N/A
• Browser: N/A

• Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, e.g. stackoverflow, gitter, etc.)
  Ageed via Gitter: https://gitter.im/cherrypy/cherrypy?at=5ae1d28f60ddf96c97e3eed3
  cc: @jaraco

Non-ascii no longer allowed in URIs on Python 2 only

• I'm submitting a ...
  [X] bug report
  [ ] feature request
  [ ] question about the decisions made in the repository

• What is the current behavior?

Since the commit in 51c004a, committed in #39 and released as 5.8.0, cheroot no longer accepts non-ascii in the URI. Prior versions of CherryPy allowed this (under some conditions) and one of our key applications relies on that behavior.

• If the current behavior is a bug, please provide the steps to reproduce and if possible a screenshots and logs of the problem. If you can, show us your code.

I'll put together a test in the suite.

• What is the expected behavior?

cheroot should provide some mechanism to accept non-ascii in URIs.

• What is the motivation / use case for changing the behavior?

Existing applications still stuck on Python 2 require the ability to support these URIs.

• Please tell us about your environment:

• Python version: 2.7
• Cheroot version: 5.8.3
• CherryPy version: 11.x (if applicable)
• OS: macOS 10.13.1
• Browser: n/a

Refactor cheroot.server.HTTPServer.bind_addr: don't mutate object state in bind()

The real bound host/port pair should be decoupled into smth like bound_addr property, like:

@property
def bound_addr(self):
    return self.socket.getsockname()[:2]

It's a follow-up for @jaraco's review comment https://github.com/cherrypy/cheroot/pull/67/files#r160028806

P.S. This requires locating all bind_addr reads in the codebase.

Cannot import cheroot.ssl.pyopenssl

On Windows 10, using Python 3.4, cheroot 5.1 import cheroot.ssl.pyopenssl fails:

(wsgidav34) C:\Prj\git\wsgidav>pip list
cheroot (5.1.0)
defusedxml (0.5.0)
pip (9.0.1)
portend (1.8)
pypiwin32 (219)
pytz (2016.10)
setuptools (18.2)
six (1.10.0)
tempora (1.6.1)
WsgiDAV (2.1.1.dev20170218, c:\prj\git\wsgidav)

(wsgidav34) C:\Prj\git\wsgidav>python
Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 19:28:18) [MSC v.1600 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import cheroot
>>> import cheroot.ssl.pyopenssl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\prj\env\wsgidav34\lib\site-packages\cheroot\ssl\pyopenssl.py", line 49, in <module>
    class SSL_fileobject(MakeFile):
TypeError: function() argument 1 must be code, not str
>>>

It looks like the problem is in makefile.py, where a function object is assigned to MakeFile

def MakeFile_PY3(sock, mode='r', bufsize=io.DEFAULT_BUFFER_SIZE):
    if 'r' in mode:
        return io.BufferedReader(socket.SocketIO(sock, mode), bufsize)
    else:
        return BufferedWriter(socket.SocketIO(sock, mode), bufsize)

[...]
MakeFile = MakeFile_PY2 if six.PY2 else MakeFile_PY3

and then in pyopenssl.py this function is used as a base class?

from ..makefile import MakeFile

class SSL_fileobject(MakeFile):
    [...]

Drop in integration testing for SSL/TLS stuff

Consider:

• using python-trio/trustme, while looks like a really simple interface to what we need (spawns a fake CA, issues certificates and lets us set up ssl context to trust it) / https://trustme.readthedocs.io
• checking pyca/cryptography test suite for their ways of generating SSL certificates for testing
• studying https://github.com/wolfSSL/wolfssl/blob/master/certs/renewcerts.sh and https://github.com/wolfSSL/wolfssl/blob/master/certs/gen_revoked.sh for their approaches
• using jsha/minica
• https://letsencrypt.org/docs/certificates-for-localhost/
• https://www.decalage.info/security/cert4tests
• https://gist.github.com/scottvrosenthal/5691305
• https://devcenter.heroku.com/articles/ssl-certificate-self
• https://github.com/aio-libs/aioredis/blob/master/tests/ssl/Makefile#L2
• https://github.com/hynek/pem
• https://github.com/tiran/multissl
• https://github.com/secure-systems-lab/securesystemslib / https://github.com/theupdateframework/tuf
• https://github.com/FiloSottile/mkcert
• https://pypi.org/project/certauth/
• https://github.com/globalsign/certlint
• https://github.com/sgallagher/sscg

Processing OPTIONS request with query parameters in URL

• I'm submitting a ...
  [*] bug report
  [ ] feature request
  [ ] question about the decisions made in the repository

• Do you want to request a feature or report a bug?
  I want to report a bug.

• What is the current behavior?
  I use CherryPy as WSGI server to serve my Flask application. When I send OPTIONS request with query parameters, for example http://0.0.0.0/profile?param=1, server returns 404 NOT FOUND. It occurs due to wrong URL processing, query string doesn't remove from URL in request processing.

That's behaviour appears after upgrading CherryPy from 11.0.0 to 12.0.0

• If the current behavior is a bug, please provide the steps to reproduce and if possible a screenshots and logs of the problem.

• CherryPy 12.0.0:

andrey@asus-K55N:~$ curl -i -X OPTIONS http://localhost:5000/api/profile/tariffs?monthly=1
HTTP/1.1 404 NOT FOUND
Content-Type: text/html
Content-Length: 233
Access-Control-Allow-Origin: *
Date: Mon, 20 Nov 2017 13:15:34 GMT
Server: 0.0.0.0

andrey@asus-K55N:~$ curl -i -X OPTIONS http://localhost:5000/api/profile/tariffs
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Allow: GET, OPTIONS, HEAD
Access-Control-Allow-Origin: *
Content-Length: 0
Date: Mon, 20 Nov 2017 13:15:37 GMT
Server: 0.0.0.0

• CherryPy 11.0.0:

andrey@asus-K55N:~$ curl -i -X OPTIONS http://localhost:5000/api/profile/tariffs?monthly=1
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Allow: HEAD, OPTIONS, GET
Access-Control-Allow-Origin: *
Content-Length: 0
Date: Mon, 20 Nov 2017 13:18:55 GMT
Server: 0.0.0.0

• What is the expected behavior?
  Query string parameters should be cut off from URL, as it was in CherryPy 11.0.0.

• What is the motivation / use case for changing the behavior?
  Properly processing of OPTIONS requests

• Please tell us about your environment:

• Cheroot version: 5.9.1
• CherryPy version: 12.0.0
• Python version: 3.5.4
• OS: Kubuntu 16.04 LTS
• Browser: [all ]

Tests take too long to run

It shouldn't take two hours to cut a release.

A stand-alone command for running a Cheroot server

• I'm submitting a ...
  [ ] bug report
  [x] feature request
  [ ] question about the decisions made in the repository

• What is the current behavior?
  Applications wishing to support Cheroot have to provide a loader stub to interface with the generic WSGI application.

• What is the expected behavior?
  Applications can start a Cheroot instance for a WSGI application from the command line without any additional Python code.

• What is the motivation / use case for changing the behavior?
  My organisation is in the process of migrating several Python WSGI apps to an automated deployment environment (Red Hat OpenShift + S2I). We would be interested in testing Cheroot as a server backend now it has been decoupled from CherryPy, but we do miss being able to spin up an instance from the command line in the manner of Gunicorn and uWSGI.
  In short, having a stand-alone application is useful from a testing and deployment standpoint, as it removes the need to ship a dedicated Cheroot loader for a web app. All that is required is a Python WSGI application exposed as a module.

• Other information
  The attached pull request contains a proof of concept script (bin/cheroot) which starts a cheroot.wsgi.Server() based on command line arguments, similar in functionality to the standalone gunicorn and uwsgi programs. Given that this is an early stage I would love feedback on the viability of this proposal, the specific command line arguments, etc. (I am holding off on changing the documentation until the design is approved)

[TODO] Get rid of unittest.TestCase

This is needed to fully take advantage of py.test testing framework.

Insufficient arguments are passed while calling handle_no_ssl method.

Please refer the the image attached. have taken this screenshot from cheroot/server.py file.

Is it possible to log an error to server logger when worker exits for unexpected reasons? All threads in threadpool has exited in our cherrypy web server and no requests were served(have checked it with the help of all python threads stackdump). I am expecting that its happening due to arguments mismatch.

Integrate support for setup.cfg declarative config into distribution

I've been playing with this during some time in other projects and I think it would make distribution declaration cleaner.

• I'm submitting a ...
  [ ] bug report
  [X] feature request
  [X] question about the decisions made in the repository

• Do you want to request a feature or report a bug?

It's a suggestion to use declarative config for static stuff/metadata for setuptools

• What is the current behavior?

There's just purely pythonic module setup.py.

• What is the motivation / use case for changing the behavior?

I want config data to be in a separate file setup.cfg, which restricts doing dynamic non-deterministic stuff, which is currently possible within setup.py.

Advantages

• setup.cfg is clean and readable ini-config
• in prevents us from making stupid mistakes
• the config leaves in a structured file, not in code

Drawbacks

• setup.py is still needed
• some of stuff will still go to setup.py
• adds prerequisite dependency of higher setuptools version, but this affects only for dev env and ones who install source distribution instead of binary one, which we also provide

@jaraco any objections against implementing this?

[Idea] Create an example with Sentry integration

New option to drop headers that contain underscores

I'd like to propose a new option to drop headers that contain underscores ('_').

Cheroot / CherryPy convert HTTP headers for WSGI to upper case and replace hyphen ('-') with underscore ('_'):

env.update(
    ('HTTP_' + bton(k).upper().replace('-', '_'), bton(v))
    for k, v in req.inheaders.items()
)

So this headers

headers = {"Sm-User": "val1",
           "Sm_User": "val2",}
res = requests.get("http://127.0.0.1:8080", headers=headers)

will result in a single WSGI environment entry "HTTP_SM_USER": "val1" or "HTTP_SM_USER": "val2".
(The concrete behavior is undefined, since the order of dict key iteration is hard to control.)

Proposed solution

This could be fixed by modifying the wsgi server code above, but I would suggest to drop those headers in the core code.

While underscores in header fields are allowed (rfc7230), they are uncommon, AFAIK.
For example Apache and nginx seem to drop them by default:
http://stackoverflow.com/a/18205049/19166
http://stackoverflow.com/a/22856867/19166

So a solution could look like this:

def read_headers(rfile, hdict=None, drop_underscores=False):
    ...
    while True:
        line = rfile.readline()
            ...
            if drop_underscores and '_' in k:
                continue
            k = k.strip().title()
            v = v.strip()
            hname = k
            ...

I tried to prepare a PR, but I could need some advice how and where to make this configurable:

        try:
            read_headers(self.rfile, self.inheaders)
            # Configurable using "`server.underscores_in_headers": False`
            drop_underscores = not self.server.server_adapter.underscores_in_headers
            read_headers(self.rfile, self.inheaders, drop_underscores)
        except ValueError:
            ex = sys.exc_info()[1]
            self.simple_response('400 Bad Request', ex.args[0])

For example I tried adding this to cheroot.HTTPServer:

class HTTPServer(object):
    ...
    underscores_in_headers = True
    """If False, silently discards headers containing underscores ('_')."""

and then make it configurable from cherrypy like so:

cherrypy.config.update({"server.underscores_in_headers": False})
cherrypy.quickstart(HelloWorld())

=> this seems to set the attribute on the ServerAdapter instead of the HttpServer.

or

cherrypy.quickstart(HelloWorld(), config={
    "/": {
        "server.underscores_in_headers": False,
        }
    })

doesn't work, because the servernamespace is not registered when quickstart is running?

cheroot 6.1.x breaks cherrypy.test.webtest 14.0.1 - TerseTestResult

cherrypy.test.webtest import fails with the latest 6.1.x cheroot.

Python 2.7.12 (default, Feb 9 2018, 15:46:55)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.

from cherrypy.test import webtest
Traceback (most recent call last):
File "", line 1, in
File "/home/lsmithso/yg/panoptic_1/.tox/python/lib/python2.7/site-packages/cherrypy/test/webtest.py", line 4, in
from cheroot.test.webtest import ( # noqa
ImportError: cannot import name TerseTestResult

This works if cheroot is pinned to 6.0.0. Is this a breaking change in 6.1.x?

Thanks.

Get rid of default Server header being non-ascii/unicode

Problem

• socket.gethostname() returns current hostname and is used by cheroot.server.HTTPServer as a default fallback;
• it is a user-defined input (one may set it to any value);
• it may use unicode charset (confirmed at Linux and Win);
• we forcefully .encode('ISO-8859-1') it when constructing headers, which leads to UnicodeEncodeError being raised therefore.

Proposal

• urlencode header contents;
• this looks allowed (and recommended by RFC);
• think about other headers.

Links

Ref: https://tools.ietf.org/html/rfc2231#section-4
Ref: werwolfby/monitorrent#214

test_core.py cannot pass on Python3.5

While running pytest, I ran into a problem.

The test_core.py cannot pass on codes line.522 to line.555 .

# Get a partial file.
        if cherrypy.server.protocol_version == 'HTTP/1.1':
            self.getPage('/ranges/slice_file', [('Range', 'bytes=2-5')])
            self.assertStatus(206) # ERROR: Status ('404 Not Found') != 206
            self.assertHeader('Content-Type', 'text/html;charset=utf-8')
            self.assertHeader('Content-Range', 'bytes 2-5/14') # ERROR: 'Content-Range':'bytes 2-5/14' not in headers
            self.assertBody('llo,') # ERROR: much more than that

            # What happens with overlapping ranges (and out of order, too)?
            self.getPage('/ranges/slice_file', [('Range', 'bytes=4-6,2-5')])
            self.assertStatus(206) # ERROR: Status ('404 Not Found') != 206
            ct = self.assertHeader('Content-Type')
            expected_type = 'multipart/byteranges; boundary='
            self.assert_(ct.startswith(expected_type)) # AssertionError: False is not true
            boundary = ct[len(expected_type):]
            expected_body = ('\r\n--%s\r\n'
                             'Content-type: text/html\r\n'
                             'Content-range: bytes 4-6/14\r\n'
                             '\r\n'
                             'o, \r\n'
                             '--%s\r\n'
                             'Content-type: text/html\r\n'
                             'Content-range: bytes 2-5/14\r\n'
                             '\r\n'
                             'llo,\r\n'
                             '--%s--\r\n' % (boundary, boundary, boundary))
            self.assertBody(expected_body) # ERROR: much more than that
            self.assertHeader('Content-Length')

            self.getPage('/ranges/slice_file', [('Range', 'bytes=2300-2900')])
            self.assertStatus(416) # ERROR: Status ('404 Not Found') != 416
            self.assertHeader('Content-Range', 'bytes */14') # ERROR: 'Content-Range':'bytes */14' not in headers

I removed the original comments, and put test error messages as comments behind actually unpassed codes.

---

It looks like need a static file named static/index.html.

I try to create the file manually, which content is "Hello". But the response code is 416 other than 206.

---

I set the content to be Hello.

The first ERROR is fixed, now it is the 2nd ERROR: 'Content-Range':'bytes 2-5/14' not in headers

---

Right, it's "Hello, world."

I solved this, how can I remove this issue?

wsgi.Server reports CherryPy version

As I understand it, Cheroot brings a standalone server with seperate versioning (e.g. 5.1), but
cheroot.wsgi.Server.version returns either "CherryPy/undefined" or "CherryPy/10.1.1", depending on the additional cherrypy package beeing installed or not.