deepstream.io client for Python

[deepstreamio]

deepstream.io client for Python

[YavorPaunov]

@viridia I'm working on a pure Python based client at the moment on YavorPaunov/deepstreampy. Much of the functionality is there but it is not complete yet. Will probably have some time around the holidays when I can speed up development a bit.

[AlexBHarley]

Hi guys, I have been working on this here alexbharley/deepstream.py. It's slow but steady going as I keep running into problems (my first time doing something like this)

[jlowin]

@YavorPaunov has a nice Tornado implementation. My read of the latest blog post suggests that 1) the HTTP API may be open sourced in Deepstream 3.0 (which would solve @viridia's use case) and 2) that unfortunately there may be breaking changes to @YavorPaunov's version.

As for different Python async environments, while that's true, Python 3 has introduced a canonical framework (asycnio) as well as async/await primitives. Most async libraries have introduced compatibility with these native coroutines. The path of least resistance, absent some external factor, would be to build a library on asyncio that could be bridged to other compatible frameworks.

Unless of course we get a green light that @YavorPaunov's implementation is the right direction. It would be nice to hear from the devs about their plans here. @WolframHempel @yasserf Deepstream is such a neat technology but I've had to pass on it for a number of projects because there is no clarity on whether working to integrate it with Python is a waste of time. In particular, adding the current community-supported Tornado client and finding out that 3.0 obsoletes it or introduces a non-Tornado official client would be bad.

[YavorPaunov]

@albrnick
Basically, it should work. Feel free to try it and please let me know if you find any issues.
I wouldn't say the client as a whole is in a production ready state though.

I am also planning to add documentation (complete README, examples and docstrings) in the near future. Life is pretty busy at the moment, unfortunately :(

[charleswhchan]

Just tested @YavorPaunov's deepstreampy (Python 2.7), and was able to emit() an event to DS. I created a PR as a really basic example to help others get started.

Look forward to an official Python client from DS team in future. 😃

[alv000h]

+1 python client w/ asyncio

[thakkardharmik]

Is this still open?

[rajanshah]

@YavorPaunov I am interested in utilizing your deepstreampy client. I can install your package in my virtual environment. Is there a way, you can add quick-start guide on how to use it in standalone python program and publish some messages, etc...?

[Serkan-devel]

Does it support Python 3?

[AlexBHarley]

@viridia @jlowin the deepstream v3.0 release won't be breaking compatibility with any clients. The changes are mainly internal and we've marked this as a major release due to other reasons (stay tuned for the blog post soon).

@YavorPaunov's library looks great and is definitely what we'd like out of community SDKs. As with most open source projects I'm sure he would appreciate more contributors.

[Allan-Nava]

Does it support Python 3?

Any news?

[yasserf]

Unfortunately not no, writing clients is very time consuming and it would be required to be a community effort

[jwanglof]

I would love to help out on this! Love Python (thought about creating an issue for this actually). Do you guys have any deadlines at all?

[WolframHempel]

That'd be amazing. No deadlines at all. There are only so many languages we know, so we're grateful for any help we can get.

[YavorPaunov]

@jwanglof
Have you started working on this? If so, let me know if I can help out. Otherwise, I'd be happy to get it started.

[jwanglof]

Sadly no, real busy this spring. I would love to help if you get it started though =)

[WolframHempel]

Looking really promising. Happy to tweet it to get some initial test users (although might be good to have a basic readme first)

[AlexBHarley]

@WolframHempel I'll get the events working to 100% and let you know on slack? At the moment the functionality is too limited

[abagshaw]

Is there any quick-start guide for any of the currently implemented features? Or is this not ready for testing yet? Would love to give it a shot!

[smorin]

If you do this please implement it as a non-blocking python client with asyncio or twisted support

[SneakBug8]

As i see, Python client by Alex is dead.

[srossross]

Is this still valid in v2? There no longer seems to be a TCP port.

(and all of the links above are dead.)

[WolframHempel]

Afraid so, the current Python client is unofficial. We'll release a high performance version based on the C++ client in the near future

[t3db0t]

@WolframHempel which c++ client? This one? https://github.com/CenacleResearch/deepstream-cpp-client

[viridia]

Any update on this? We're considering looking into deepstream but our backend is Python based.

Also, all of the links in the initial message are 404.

[juicycleff]

I will also love to contribute to the .NET cient. I am relatively new to deepstream to be exact 3 day old.

[yasserf]

@YavorPaunov I just looked through your python client, I don't have much experience in python tbh but the builds + steps + structure looks great!

If there is anything we can help with just let us know!

[albrnick]

@YavorPaunov Your client looks very interesting! I was wondering if you the basic record getting/setting/subscripting, etc.. is working?

Thanks! :)

[hipkiss91]

What would be a rough date for official release?
Just started using python. I'm eager to get deepstream involved :)

[viridia]

Turns out that there's a difficulty involved in writing a deepstream client for Python, which is that there are many different environments for handling asynchronous events - twisted, asyncio, etc - and the design of the deepstream client is probably going to be quite different depending on which environment it is built for. You won't be able to take a deepstream client written for asyncio and use it within a twisted environment, and vice versa.

In my own use case, for example, I need to be able to publish deepstream events from within a Django-REST server. Django-REST is based on WSGI, which is a simple run loop that processes incoming HTTP requests and invokes callbacks. Any asynchronous processing - such as handling incoming messages - has to be done in the context of servicing some request. Otherwise, there's no place for the incoming data to go. This doesn't fit with the deepstream model at all.

My solution in this case will be to implement just a tiny portion of the deepstream protocol - basically the parts that deal with authentication and sending events - and treat deepstream like a conventional RESTful service. My Django server won't be able to receive events or rpc calls, but that's OK, there are other microservices in the system (written in Node.js) that can perform that job.

Note that in languages like Node.js, this is not a problem since the async primitives (like Promises) are pretty much built into the language, so all of the various packages for dealing with async operations are somewhat interoperable.

[jlowin]

@AlexBHarley thanks!

[luis-kaufmann-silva]

"Writing a deepstream client" page not found :(

[viridia]

Right now the Tornado-based python client is not going to satisfy my use case.

What I am trying to do is replace WAMP/Crossbar in my environment with deepstream.io, since deepstream supports multiple server instances and Crossbar does not. We've already managed to do this for the main server and for the frontend, but there are a couple of other pieces which are stuck because of lack of driver support.

One of these pieces is a small daemon/app launcher that runs windows machines that our users connect to via VNC. One of the launcher's jobs is to mount various filesystems in response to various kinds of cloud events. It currently does this using the Autobahn/WAMP driver which is written on top of Twisted. What I want to do is replace the Autobahn/WAMP driver with an equivalent deepstream.io driver.

The daemon does not understand HTTP or any other kinds of network protocols other than RPCs over real-time websockets. Using deepstream would be ideal, since the daemon needs to communicate with the browser frontend, but the browser shouldn't need to know the exact IP address of the machine the daemon is running on. With something like deepstream the daemon can simply register rpc methods whose name includes the instance id.

The problem I have is lack of driver support for deepstream. Since the daemon doesn't talk HTTP, dropping in a webserver like Tornado makes no sense. (Also, Tornado is not recommended on Windows machines.) Right now a twisted-based driver would probably be the easiest, although if an asyncio driver showed up I would probably consider migrating to Python3. The other thing we've been considering is re-writing the daemon entirely in node.js, but that's a bigger job and would take more engineering time than we have to spare right now.

[yasserf]

I'm not very versed in python, but I'm guessing you can't easily our websocket libraries right? In java we allow websocket libraries to be injected if needed.

[mentiflectax]

@WolframHempel @AlexBHarley I'm trying to understand the scope of Python client project. I looked through the spec project and wrote out all features in this gist. Is it correct? If a Python client implementation has all the features listed there, is it ready then?

[WolframHempel]

@dpisarenko Your list looks very conclusive and is correctly based on the unterlying specs. From experience though I know that the devil is in the detail. While e.g. the protocol to record behaviour spec is fairly short, the actual client behaviour for merging paths, providing received notifications etc. can be quite challenging to get right. In general though I think the answer is "yes, a Python client implementing all features listed in this gist would be considered complete and would cover the full range of deepstream features"

[mentiflectax]

@WolframHempel Thanks!

[yasserf]

I'm closing this issue since the messages within it are outdated to V3 and Text Protocol, and would probably just confuse anyone starting to look into it.

The goal is to write a client in C or C++ around the protobuf protocol and then get other clients to just call into that.