Comments (11)
I guess I'll have to think about it a bit. If you have any good opinions on whether it is better to put a dialect in the pipeline, please proceed. I will continue to try. I think making another adapter should be postponed to a later date. I think the most important thing is to handle the flink job well.
from hoptimator.
Thanks @HoganEdwardChu for #31
from hoptimator.
Hi, I didn't mean to disturb you, I was watching with interest. The line that you mean is here?
from hoptimator.
The line that you mean is here?
Yep. It looks like the hard-coded dialect is only used in a couple methods that return SQL strings. It might make sense to change those to return the underlying ScriptImplementer, which is dialect agnostic. Then callers could do rel.query(...).sql(dialect)
and get whatever dialect they want.
from hoptimator.
Oh you mean https://github.com/HoganEdwardChu/Hoptimator/pull/1/files this way? I haven't seen all the code, so I don't know the specific details, so it might seem lacking(sorry for that) But than i think we have to get input dialect.
from hoptimator.
this way?
Exactly, tho I'm not sure where to go from there. One way would be to add a dialect parameter to pipeline()
, s.t. the operator can generate pipelines in whatever dialect it wants. For now, the operator can just use the MySQL dialect.
Looking ahead, we'd like to be able to implement alternative runtimes, e.g. using Spark SQL instead of Flink SQL. Ideally, those changes would be in the operator, not the planner. So this would be a step in that direction.
from hoptimator.
Thanks for the detailed and contextual explanation. I'll think about what you said too. (if there's a good direction, I'll try it and share it again too!) Also would try implementing alternative runtimes Spark SQL instead of Flink SQL.(operator part)
from hoptimator.
I haven't given much thought to how we'd support multiple runtimes in the operator. There is a flink adapter, which is really the only place in the control plane that we assume the SQL will run on flink: https://github.com/linkedin/Hoptimator/blob/main/hoptimator-flink-adapter/src/main/java/com/linkedin/hoptimator/operator/flink/FlinkSqlJobReconciler.java#L57
As you can see, the flink adapter just passes the SQL from SqlJob
to a FlinkDeployment
(which is handled by the external flink-kubernetes-operator). Theoretically, we could do something similar for Spark or any other SQL runtime. I'm not sure how valuable that would be, but it seems possible.
from hoptimator.
I think adding sqlDialect to pipeline is best ... maybe at this point
https://github.com/HoganEdwardChu/Hoptimator/blob/90f4c459e1bb71f79c2625f19d79753eb0fbfbd5/hoptimator-planner/src/main/java/com/linkedin/hoptimator/planner/PipelineRel.java#L95
from hoptimator.
adding sqlDialect to pipeline
Makes sense to me!
from hoptimator.
Thanks!!!!!
I am trying to add another sql time like spark. If there is any valuable progress I will pr! @ryannedolan
from hoptimator.
Related Issues (12)
- hoptimator-cli hangs/times-out for unbounded queries
- !test command
- Command-line arguments for hoptimator-operator HOT 2
- Expose whether a table is being used as a source or a sink HOT 1
- Support multiple input dialects
- Support generating multiple YAML documents from one template
- CLI logs are too verbose HOT 3
- Expose "hints" to the planner
- Support no-op Resources
- Integration tests can only handle one Flink job
- Sub operator ignores changes to hints
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hoptimator.