threema-ch/threema-msgapi-sdk-python

Threema Gateway Message SDK for Python

threema-msgapi-sdk-python's Issues

Get proxies information from HTTP_PROXY / HTTPS_PROXY environment variables

By default "trust_env" variable is false, so aiohttp.ClientSession does not get information from HTTP_PROXY / HTTPS_PROXY environment variables. For this reason, the following code was received:

aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host msgapi.threema.ch:443 ssl:default [Name or service not known]

In order to fix this issue, it is necessary to modify "_gateway.py" module, including in init the following line:
self._session = session if session is not None else aiohttp.ClientSession(trust_env=True)

_gateway.zip

Long standing py_lru_cache bug.

Hello and first of all thanks for making this library.

I have a question/suggestion and I was wondering if noone ever ran into this issue.
There is a longstanding bug in py_lru_cache that has not been fixed by now:

stucchio/Python-LRU-cache#18

We went around by pinning to: 'py_lru_cache @ git+https://github.com/kirill578/Python-LRU-cache.git'

Add asyncio support

Add compatibility for usage in asyncio events loops. Only the requests module needs to be changed over to something that supports asyncio. The CLI would have to start most commands in an event loop instance. This should be fairly easy and would allow a much easier integration into asynchronous applications.

Missing description for prerequisites

Obviously you use requests but in the Readme it is not listed under "Prerequisites", so you should add it there as a required library:

pip install requests

And you also - obviously - use libnacl so you should also mention it.
pip install libnacl

Update click package to 7.x version

Hi there

Thank you very much for the python-sdk for the threema.gateway. I am facing some dependency issues with click.

You are depending on 'click>=6.7,<7'. Other projects have adapted to click >= 7.

Would it be possible to update this on your side?

Regards tbi88

https_proxy env is ignored

It is currently not possible to use threema-gateway with a web proxy, as aiohttp does not check the $https_proxy environment variable unless it is explicitly told to do so.

See aiohttp documentation: https://docs.aiohttp.org/en/stable/client_advanced.html?highlight=proxy

threema-gateway could either allow to submit the web proxy setting as command line option and pass it to aiohttp, or it could use trust_env to let aiohttp use $https_proxy.

It works for me by changing gateway/_gateway.py and pass trust_env=True to aiohttp:

        self._session = aiohttp.ClientSession(connector=connector, trust_env=True)

That way, aiohttp is instructed to use $https_proxy.

Send ImageMessage with caption

From the Android client it is possible to send an Image with caption. Is that also supported by the API/ this SDK?

RuntimeError with aiohttp >= 3.8

I just updated aiohttp to 3.8.1, and my Bot was not able to receive any messages anymore:

2021-12-10 08:05:22,779 - aiohttp.server - ERROR - Error handling request
Traceback (most recent call last):
  File "/home/bot/covidbot/venv/lib/python3.8/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
    reset = await self.finish_response(request, resp, start_time)
  File "/home/bot/covidbot/venv/lib/python3.8/site-packages/aiohttp/web_app.py", line 504, in _handle
    """gunicorn compatibility"""
  File "/home/bot/covidbot/venv/lib/python3.8/site-packages/threema/gateway/e2e.py", line 212, in handle_callback
    message = await context.receive_handler(context.connection, {
  File "/home/bot/covidbot/venv/lib/python3.8/site-packages/threema/gateway/e2e.py", line 481, in receive
    key_pair = await cls.get_decrypt_key_pair(connection, parameters)
  File "/home/bot/covidbot/venv/lib/python3.8/site-packages/threema/gateway/e2e.py", line 585, in get_decrypt_key_pair
    public = await connection.get_public_key(parameters['from_id'])
  File "/home/bot/covidbot/venv/lib/python3.8/site-packages/threema/gateway/util.py", line 352, in _wrapper
    value = await func(*args, **kwargs)
  File "/home/bot/covidbot/venv/lib/python3.8/site-packages/threema/gateway/_gateway.py", line 176, in get_public_key
    response = await self._get(self.urls['get_public_key'].format(id_))
  File "/home/bot/covidbot/venv/lib/python3.8/site-packages/threema/gateway/_gateway.py", line 317, in _get
    return await self._session.get(*args, **kwargs)
  File "/home/bot/covidbot/venv/lib/python3.8/site-packages/aiohttp/client.py", line 466, in _request
    and hdrs.AUTHORIZATION in headers
  File "/home/bot/covidbot/venv/lib/python3.8/site-packages/aiohttp/helpers.py", line 701, in __enter__
    @property
RuntimeError: Timeout context manager should be used inside a task

Unfortunately, I currently do not have the time to look further into this, the simple solution is to sepcify aiohttp<3.8, this was my quick fix. Can anyone reproduce this with aiohttp >= 3.8, or is it related to my whole setup?

Improve decryption of E2E messages

The decryption functionality of the Python SDK is insufficient. We should at least be able to decrypt messages we are able to send. Missing types:

• ImageMessage
• FileMessage

Also required for #8

Official fork? & What's your plan?

Looking at this repo confuses me. From the commit history it seems it was once hosted officially under threema-ch\msgapi-sdk-python.
It seems you were a contributor, because afterwards the whole repo was moved here and is deleted at threema-ch. As Threema did not do this with the other repos it is quite confusing.

But anyway it is nice to see that you maintain it here - I'll maintain my PHP fork too. There I choose the approach to have a branch with the official files (just as a mirror) while developing ad Pull Requests on the main branch are possible.
What's you plan?

Edit: Related: Threema hat GitHub Repositories eingestellt

Add to PyPI

Upload the project to the Python Package Index for easy installation via pip.

Connection reuse

When sending

Example:

from threema.gateway import Connection              
from threema.gateway.e2e import TextMessage

connection = Connection(
    identity='*YOUR_GATEWAY_THREEMA_ID',
    secret='YOUR_GATEWAY_THREEMA_ID_SECRET',
    key='private:YOUR_PRIVATE_KEY',
    blocking=True,
)

with connection:
    TextMessage(
        connection=connection,
        to_id='ECHOECHO',
        text='test1'
    ).send()
    TextMessage(
        connection=connection,
        to_id='ECHOECHO',
        text='test2'
    ).send()

with connection:
    TextMessage(
        connection=connection,
        to_id='ECHOECHO',
        text='test3'
    ).send()

I can send test1 and test2, but test3 fails:

  ...
  File "/mnt/data/Data/.virtualenvs/test/lib/python3.6/site-packages/aiohttp/client.py", line 585, in __iter__
    resp = yield from self._coro
  File "/mnt/data/Data/.virtualenvs/test/lib/python3.6/site-packages/aiohttp/client.py", line 163, in _request
    raise RuntimeError('Session is closed')
RuntimeError: Session is closed

Padding with PRNG

For encrypting (text) messages a padding it added. However this uses a pseudo-random-number generator, which is cryptographically not secure.
In the Threema Whitepaper (page 9) the purpose of this is described:

Padding

In order to thwart attempts to guess the content of short messages by looking at the amount of data, Threema adds a random amount of PKCS#7 padding to each message before end-to-end encryption.

So it is not that serious if this is weak and does not affect the security of NaCl/Libsodium as it is just an additional feature, but in the Threema Whitepaper it is claimed (page 8) that the clients (aka the Threema app) use real RNG for "Padding amount determination", so obviously this is a different implementation there.

I've asked on Stackexchange whether this is a problem, but also when it is not there are no arguments against using a real cryptographically-secure RNG there AFAIK...

This issue also exists in these other implementations:

• Java: implementation, PRNG
• PHP: rugk/threema-msgapi-sdk-php#26

Threema was contacted about this issue.

Invalid environment marker: python_version<="3.4"

:/tmp/threema-msgapi-sdk-python-3.0.4# pip3 install . Unpacking /tmp/threema-msgapi-sdk-python-3.0.4 Running setup.py (path:/tmp/pip-42xc6mwq-build/setup.py) egg_info for package from file:///tmp/threema-msgapi-sdk-python-3.0.4 error in threema.gateway setup command: Invalid environment marker: python_version<="3.4" Complete output from command python setup.py egg_info: error in threema.gateway setup command: Invalid environment marker: python_version<="3.4"

Tested clean installation with Release 3.0.4 on an Cubietruck with Python 3.4.2

Configurable route for callback-server CLI

• Configurable route
• Maybe print the resulting URL
• Add to readme

Add Script for libsodium installation

Hi,

here is a little script to automate libsodium installation on linux systems.
Need to be executed as root:
install-sodium.txt

Allow usage of asyncio.run

Hey 👋🏻 Maybe you can help me.

Would it be possible to use asyncio.run for execution? Since python3.6 is the last version not to support that, that would be a big improvement for me, since I have to handle the eventloop myself.

https://github.com/python/cpython/blob/d9d63f13cf5b6b896c245099c42b08f6da083d05/Lib/asyncio/runners.py#L8

I haven't got it working with the current code, maybe you can provide some pointers.

Receive E2E messages

Receiving E2E messages is already supported (at least the processing part). However, an example is missing, yet. A minimal server implementation which can receive and process incoming E2E messages might be interesting, too.

Public Key Pinning

If the public key for the HTTPS connected could be pinned, this would certainly improve the security of the TLS connection.

However it seems this is not implemented into Python yet, but it seems there was at least interest in implementing it.

Dependency pinning

Currently this library is broken when installing it via pip:

$ python threema/gateway/bin/gateway_client.py
...
AttributeError: module 'aiohttp' has no attribute 'FingerprintMismatch'

The dependency is specified as aiohttp>=0.19.0. Current version is at 2.0.7.

Pinning to aiohttp>=1,<2 already seems to help in this case, although an update would probably be good.

Can we add semver compatible version pinnings to all dependencies? I can provide a PR against develop if desired.

Codecov doesn't recognise CLI tests

I think that was the problem I had with find_packages @dbrgn

OSError: Could not locate nacl lib

Hi there,
just installed via pip and trying to generate the certs.

threema-gateway generate privateKey.txt publicKey.txt
Traceback (most recent call last):
  File "/home/pi/.local/share/virtualenvs/sms-YuepyvnI/bin/threema-gateway", line 5, in <module>
    from threema.gateway.bin.gateway_client import main
  File "/home/pi/.local/share/virtualenvs/sms-YuepyvnI/lib/python3.7/site-packages/threema/gateway/__init__.py", line 26, in <module>
    from . import _gateway
  File "/home/pi/.local/share/virtualenvs/sms-YuepyvnI/lib/python3.7/site-packages/threema/gateway/_gateway.py", line 4, in <module>
    import libnacl.encode
  File "/home/pi/.local/share/virtualenvs/sms-YuepyvnI/lib/python3.7/site-packages/libnacl/__init__.py", line 80, in <module>
    nacl = _get_nacl()
  File "/home/pi/.local/share/virtualenvs/sms-YuepyvnI/lib/python3.7/site-packages/libnacl/__init__.py", line 75, in _get_nacl
    raise OSError(msg)
OSError: Could not locate nacl lib, searched for libsodium.so, libsodium.so.23, libsodium.so.18, libsodium.so.17, libsodium.so.13, libsodium.so.10, libsodium.so.5, libsodium.so.4, 

I am on debian armv7l with python3.7.3.

Inconsistent format for message IDs

DeliveryReceipt.message_ids contains a list of IDs in bytes whereas the various send methods of the Connection instance usually return a hex-encoded ID. This makes it really annoying to handle.

We should unify the format. My personal tendency is to use bytes. This would definitely require a major version increase because it breaks API. Any thoughts, @dbrgn?

Ability to "pin" TLS version and/or ciphers

It would be nice if you could not only pin the cert hash (#9), but also the ciphers and the TLS version to use. This would effectively make any downgrade attacks impossible and ensure that always the best encryption is used.

E.g. you can also do this in the PHP-SDK.

Add file message based variants for images and videos

Threema is moving away from the old image message and video message format towards one that is based on the file message with additional metadata. The lib should still be able to decode the old format but only encode in the new format. Note that those properties are not yet documented in the Gateway API documentation.

receive messages

Do I understand correctly, that I have to set up a callback server to receive messages? It is not possible just to send a request to the threema api and get new messages like I do with sending messages?

Fetching of remaining credits

Threema Gateway SDKs version 1.1.2 allows fetching the remaining credits of your account. This should be implemented here too.

Sign commits & releases (tags)

You should consider signing git commits & releases.

• onionshare#221 (comment)
• It is also displayed on GitHub: https://github.com/blog/2144-gpg-signature-verification
• Here is GitHub's help for this: https://help.github.com/articles/generating-a-gpg-key/
• And here is how you can sign releases: https://wiki.debian.org/Creating%20signed%20GitHub%20releases

Update fingerprint

Dear Threema Gateway User

For more “Swissness”, and to increase the level of trust (Organization Validation), we are switching our server certificates to SwissSign. On 01.12.2016 at 10:00 CET, the certificate for the web host https://msgapi.threema.ch (which you use for communicating with Threema Gateway) will be changed.

If the HTTPS client that you employ for communication with Threema Gateway uses one of the common CA lists (e.g. Mozilla CA store/NSS) or does not verify server certificates, then you don't need to do anything. The root certificate of SwissSign is already contained in common CA lists. If you have included our old certificate (GeoTrust RapidSSL) manually, you need to make the root certificate of SwissSign Gold G2 available to your HTTPS client.

The root certificate of SwissSign Gold G2 can be found here: https://swisssign.net/cgi-bin/authority/download?ca=Gold%20G2 (other formats see: https://swisssign.net/cgi-bin/trust/import).

If you have any questions concerning this certificate change, contact us at [email protected].

Best regards,
Threema Gateway

Add Travis CI

Continuous Integration!

Sending Files, File Upload and Download

The Threema Gateway version 1.1.2 allows sending files and uploading/downloading them to/from the Threema blob server. An update of the Python SDK is required.

Please publish/tag a release

Please use the git tags to publish releases of the versions of the Phyton-SDK.

Improve public key pinning

Currently the fingerprint of the certificate is pinned.
There are things to improve this pinning:

1. When cert expires and/or is renewed,. the fingerprint will change, so this breaks clients regularly!
   To prevent this, pin the public key of the key pair instead of the fingerprint.
2. When the key pair has to be changed (e.g. in case of a server breach) there should be a backup key pinned, which can be used in this case. As Threema uses HPKP, they specify one. More information here.

As this may be an upstream issue with aiohttp, I also opened an issue there: aio-libs/aiohttp#1187