Comments (4)
I think a larger point to possibly be made here is the difficulty of using pulsar with other libraries without these features. For example, consider a user trying to write a simple FastAPI server that simply exposes a REST endpoint for a client that gives that latest value in a topic.
If they were using Kafka, they could use broadcaster
(like suggested here), or they could use aiokafka
(an example of this would be here and here). These third party libraries that allow background processing generally require coroutines to be passed in and don't support synchronous functions because that's obviously a nightmare with python and GIL.
I'm not even sure on how to nicely implement the FastAPI example with Pulsar client (apart from implementing blocking threads), or if the increased complexity means that for many Python developers they might simply see the provided kafka solutions and be swayed.
from pulsar-client-python.
@zbentley Integrating this client with asyncio is frustrating, because it's not entirely clear which thread things are running in and much of asyncio is not threadsafe. In order to get these callbacks to work with asyncio you have to resort to low level threadsafe facilities of asyncio, making it far less trivial to safely integrate this client with asyncio than you portray.
For example if you want to use send_async
. Intuition might lead you to just pass a future into that callback, and set the result of that future when you're done. You cannot do that, because Future is not threadsafe. You instead need to call future.set_result
using loop.call_soon_threadsafe(future.set_result, res)
to ensure that set_future
is called from the main thread. This is actually a simpler example, whereas wrapping message_listener callbacks to support async callbacks is even more frustrating. Basically you're always forced to use low-level threadsafe facilities to schedule into the main threads event loop.
I think the argument that this is strictly syntactical is simply not true. These concurrency models are fundamentally different. Pulsar is using a threaded blocking network model, where threads handle blocking operation as to not block the main thread, these threads are put to sleep by the OS when they make a blocking syscall. Asyncio is driven by an event loop and non-blocking operations running on a single thread. I understand if an asyncio wrapper is outside the scope of this project, but these concurrency models diverge on more than just syntax, and integrating with asyncio is not trivial or straightforward especially for most python developers who may not understand the low level details of how asyncio works.
from pulsar-client-python.
This package already provides asynchronicity (Producer.send_async
and the message_listener
argument to the client).
While those don't provide async
or await
syntactic constructs, they are asynchronous (nonblocking) and can be integrated with Python's asyncio (or trio etc.) futures to provide awaitable tokens.
How would async/await syntax save on IO waiting time if this client is already asynchronous? Something would still have to wait on I/O as is already being done.
from pulsar-client-python.
Assign this issue to myself. I wrote a demo locally and I'm going to start this work.
![image](https://private-user-images.githubusercontent.com/18204803/293215889-84633fc9-d2b0-406a-b4d4-4c19a1845c30.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjIxODA1MTEsIm5iZiI6MTcyMjE4MDIxMSwicGF0aCI6Ii8xODIwNDgwMy8yOTMyMTU4ODktODQ2MzNmYzktZDJiMC00MDZhLWI0ZDQtNGMxOWExODQ1YzMwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MjglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzI4VDE1MjMzMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTg4NTgzYjRkMDhjYTY4NGU3ZDA1MDIyMThjZDZlNWY4ZWUxMmM0ODc3MDViZTRmZmNiMGU0N2M3NzYyZTdlNWQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0._-MH826VdPrq4OVQl7DMF3AIhCi6RoRgo7a4UW_B71w)
from pulsar-client-python.
Related Issues (20)
- [Bug] Memory issue using Record with Array HOT 2
- What is the latest python version currently supported by this? I did not find it HOT 5
- Negative acknowledge doesn't work with message_id HOT 2
- Producer callback is unable to utilise python raise to propagate errors HOT 14
- Document for the pulsar.Result enum
- Document for the batching arguments when creating producer
- Add proper type hints for public APIs HOT 1
- Reader set to MessageId.latest and inclusive start message does not work HOT 6
- Custom logger doesn't work with created resources (like producers) HOT 5
- Avro EOF Error HOT 1
- publish sdist for pypi HOT 5
- Reader.has_message_available() returns True even if no messages to consume, when start_message_id is earliest HOT 2
- Segmentation fault sometimes happens when running unit tests HOT 2
- 3.5.0 release? HOT 2
- There is a memory leak,I need help HOT 3
- Support on PPC64LE
- Cannot seek on partitioned topic HOT 3
- Singleton Pulsar Producer Using Python Client Not Working HOT 7
- Multi Topic consumer not using client provided logger - MultiTopicsConsumerImpl:526 | Received Message from one of the topic HOT 2
- Support for tableview in python client.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pulsar-client-python.