servirtium/servirtium-python

A Python version of Servirtium

Main Servirtium site: http://servirtium.dev

A Servirtium library for Python

Demo project that uses it: https://github.com/servirtium/demo-python-climate-data-tck

Alpha quality software

Working so far:

• Record and playback of GET/POST/PUT

Not Working yet:

• Support for "Transfer-Encoding" header for "chunked" response

Help needed from Pythonistas!

Building Servirtium lib/package for Python3

Execute the following commands to install the package locally:

pip3 install requests
cd servirtium-python
pip3 install -e .

OR install it from https://pypi.org/ using following:

pip3 install servirtium

The 'demo-python-climate-data-tck' demo needs you to have built the Python version of Servirtium first as it is presently unpublished in pip-land.

Running unit tests

pip3 install pytest
cd servirtium-python
pytest

Running the compatibility suite

This should record a bunch of interactions, using a Mocha test suite that we launch via Selenium-WebDriver

python3 -m test.compatibility-suite record

servirtium-python's Issues

Needs to cater for POST/PUT etc

imports need to be corrected

Getting error in IDE: Unresolved reference 'markdown_parser'

Fix:
Change imports from

from markdown_parser import SimpleMarkdownParser, parse_markdown_string, get_markdown_file_strings

to

from servirtium.markdown_parser import SimpleMarkdownParser, parse_markdown_string, get_markdown_file_strings

Change all imports to use from servirtium.xxx

because of incorrect imports pytest fails with error:

(servirtium-python)  ~/Yogesh/workspace/servirtium-python-workspace/yogeshrnaik/servirtium-python   master  pytest
============================================================================================= test session starts ==============================================================================================
platform darwin -- Python 3.7.5, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /Users/yogeshnaik/Yogesh/workspace/servirtium-python-workspace/yogeshrnaik/servirtium-python
collected 9 items / 1 error / 8 selected

==================================================================================================== ERRORS ====================================================================================================
____________________________________________________________________________________ ERROR collecting test/test_recorder.py ____________________________________________________________________________________
ImportError while importing test module '/Users/yogeshnaik/Yogesh/workspace/servirtium-python-workspace/yogeshrnaik/servirtium-python/test/test_recorder.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../../../.pyenv/versions/3.7.5/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test/test_recorder.py:6: in <module>
    from definitions import MOCKS_DIR
E   ImportError: cannot import name 'MOCKS_DIR' from 'definitions' (/Users/yogeshnaik/Yogesh/workspace/servirtium-python-workspace/yogeshrnaik/servirtium-python/definitions.py)
=========================================================================================== short test summary info ============================================================================================
ERROR test/test_recorder.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=============================================================================================== 1 error in 0.31s ===============================================================================================

Fix missing functions/methods

The https://github.com/servirtium/demo-python-climate-tck contains following errors due to missing functions/methods in https://github.com/servirtium/servirtium-python library

Cannot find reference 'set_markdown_files' in 'recorder.py' 
Unresolved attribute reference 'set_invoking_method' for class 'RecorderHttpHandler' 

Cannot find reference 'set_markdown_files' in 'playback.py' 
Unresolved attribute reference 'set_invoking_method' for class 'MockServiceHttpHandler' 

Transfer-Encoding header needs to be removed when responding back from recorder HTTP server

At this line:

servirtium-python/servirtium/recorder.py

Line 158 in be73f05

self.wfile.write(response.content)

we get below error due to Transfer-Encoding header sent with value = chunked. This is header is used when communicating with real service but should not be sent when responding back to caller (test). Instead we need to send Content-Length header to fix the below issue.

FAILED                                  [100%]127.0.0.1 - - [27/Dec/2020 14:57:23] "GET /climateweb/rest/v1/country/annualavg/pr/1980/1999/fra.xml HTTP/1.1" 200 -

test/test_recorder.py:8 (test_something)
self = <urllib3.response.HTTPResponse object at 0x10c663410>

    def _update_chunk_length(self):
        # First, we'll figure out length of a chunk and then
        # we'll try to read it from socket.
        if self.chunk_left is not None:
            return
        line = self._fp.fp.readline()
        line = line.split(b";", 1)[0]
        try:
>           self.chunk_left = int(line, 16)
E           ValueError: invalid literal for int() with base 16: b'<list>\n'

/Users/yogeshnaik/.pyenv/versions/3.7.5/envs/servirtium-python/lib/python3.7/site-packages/urllib3/response.py:696: ValueError

During handling of the above exception, another exception occurred:

self = <urllib3.response.HTTPResponse object at 0x10c663410>

    @contextmanager
    def _error_catcher(self):
        """
        Catch low-level python exceptions, instead re-raising urllib3
        variants, so that low-level exceptions are not leaked in the
        high-level api.
    
        On exit, release the connection back to the pool.
        """
        clean_exit = False
    
        try:
            try:
>               yield

/Users/yogeshnaik/.pyenv/versions/3.7.5/envs/servirtium-python/lib/python3.7/site-packages/urllib3/response.py:436: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

            except (HTTPException, SocketError) as e:
                # This includes IncompleteRead.
>               raise ProtocolError("Connection broken: %r" % e, e)
E               urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(7 bytes read)', IncompleteRead(7 bytes read))

/Users/yogeshnaik/.pyenv/versions/3.7.5/envs/servirtium-python/lib/python3.7/site-packages/urllib3/response.py:454: ProtocolError

During handling of the above exception, another exception occurred:

    def test_something():
        # servirtium_site = "http://localhost:61417"
    
        recorder.set_real_service('http://climatedataapi.worldbank.org')
        recorder.set_request_header_replacements({'User-Agent': 'Servirtium-Testing'})
        recorder.set_response_header_removals({'Set-Cookie: AWSALB', 'X-', "Date:"})
    
        servirtium_daemon = threading.Thread(target=recorder.start, daemon=True)
        servirtium_daemon.start()
    
        recorder.RecorderHttpHandler.set_markdown_filename("just_testing_ignore_me")
    
>       rsp = requests.get('http://localhost:61417/climateweb/rest/v1/country/annualavg/pr/1980/1999/fra.xml')

test_recorder.py:21: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/Users/yogeshnaik/.pyenv/versions/3.7.5/envs/servirtium-python/lib/python3.7/site-packages/requests/api.py:76: in get
    return request('get', url, params=params, **kwargs)
/Users/yogeshnaik/.pyenv/versions/3.7.5/envs/servirtium-python/lib/python3.7/site-packages/requests/api.py:61: in request
    return session.request(method=method, url=url, **kwargs)
/Users/yogeshnaik/.pyenv/versions/3.7.5/envs/servirtium-python/lib/python3.7/site-packages/requests/sessions.py:542: in request
    resp = self.send(prep, **send_kwargs)
/Users/yogeshnaik/.pyenv/versions/3.7.5/envs/servirtium-python/lib/python3.7/site-packages/requests/sessions.py:697: in send
    r.content
/Users/yogeshnaik/.pyenv/versions/3.7.5/envs/servirtium-python/lib/python3.7/site-packages/requests/models.py:831: in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def generate():
        # Special case for urllib3.
        if hasattr(self.raw, 'stream'):
            try:
                for chunk in self.raw.stream(chunk_size, decode_content=True):
                    yield chunk
            except ProtocolError as e:
>               raise ChunkedEncodingError(e)
E               requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(7 bytes read)', IncompleteRead(7 bytes read))

/Users/yogeshnaik/.pyenv/versions/3.7.5/envs/servirtium-python/lib/python3.7/site-packages/requests/models.py:756: ChunkedEncodingError