Apache Beam Dataframe and SQL support

[alxmrs]

Apache Beam Dataframe and SQL support

[goodwanghan]

Thanks for the suggestion. We may consider Flink in the future, however, I am not sure about Beam.

I am curious have you seen beam performing well compared to native Flink and Spark? I personally didn't have much positive experience with Beam. And the streaming first philosophy may be the fundamental problem in my opinion.

I'd love to learn different opinion from you.

Thanks!

[alxmrs]

Interesting! What are your negative experiences with Beam? I personally have had great experiences with Beam. I haven’t used native Spark in a while, I’ve been primarily using Beam for the last few years. I have searched around a bit and haven’t found benchmarks comparing Beam’s Dataframes on Spark vs Native Spark Dataframes. There are other comparisons on the JVM that show that native Spark and Flink are faster than using Beam, but I’m not sure how useful they would be for the Fugue/Python context. I have two main motivations for an integration with Beam, even if there are separate integrations with Flink: 1. I need to be able to deploy on GCP Dataflow. 2. I need to integrate a dataframes library of some kind with Xarray-Beam ( https://github.com/google/xarray-beam). Using Fugue for this would be quite streamlined, but it’s not a dealbreaker.

And the streaming first philosophy may be the fundamental problem in my

opinion. Would you elaborate here? It’s a problem with respect to what? I’m happy to share my experience and this context would help.

…

On Tue, Feb 20, 2024 at 4:28 AM Han Wang ***@***.***> wrote: Thanks for the suggestion. We may consider Flink in the future, however, I am not sure about Beam. I am curious have you seen beam performing well compared to native Flink and Spark? I personally didn't have much positive experience with Beam. And the streaming first philosophy may be the fundamental problem in my opinion. I'd love to learn different opinion from you. Thanks! — Reply to this email directly, view it on GitHub <#535 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AARXAB6OSF25XPS3F2SYWALYUO7XHAVCNFSM6AAAAABDNHAIPGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJTGE3TOOBXGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>