substrait-io / substrait-java Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
Hi, I noticed that the time type parsed from calcite will have a precision value check before generating substrait plan.
I added some timestamp related sql tests(sql ddl like "CREATE TABLE test(col_date DATE, col_time TIME(6), col_timestamp TIMESTAMP(6));"οΌand found time type precision value parsed from calcite will not be greater than three, will always make this check fail. Is there a bug in the use of time types?
unsupported timestamp precision 3
java.lang.UnsupportedOperationException: unsupported timestamp precision 3
at io.substrait.isthmus.TypeConverter.convert(TypeConverter.java:77)
at io.substrait.isthmus.TypeConverter.convert(TypeConverter.java:109)
at io.substrait.isthmus.TypeConverter.toNamedStruct(TypeConverter.java:37)
at io.substrait.isthmus.SubstraitRelVisitor.visit(SubstraitRelVisitor.java:76)
at io.substrait.isthmus.SubstraitRelVisitor.visit(SubstraitRelVisitor.java:32)
at io.substrait.isthmus.RelNodeVisitor.reverseAccept(RelNodeVisitor.java:85)
at io.substrait.isthmus.SubstraitRelVisitor.apply(SubstraitRelVisitor.java:319)
at io.substrait.isthmus.SubstraitRelVisitor.visit(SubstraitRelVisitor.java:122)
at io.substrait.isthmus.SubstraitRelVisitor.visit(SubstraitRelVisitor.java:32)
at io.substrait.isthmus.RelNodeVisitor.reverseAccept(RelNodeVisitor.java:95)
at io.substrait.isthmus.SubstraitRelVisitor.apply(SubstraitRelVisitor.java:319)
at io.substrait.isthmus.SubstraitRelVisitor.convert(SubstraitRelVisitor.java:340)
at io.substrait.isthmus.SubstraitRelVisitor.convert(SubstraitRelVisitor.java:332)
at io.substrait.isthmus.SqlToSubstrait.lambda$executeInner$0(SqlToSubstrait.java:120)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at io.substrait.isthmus.SqlToSubstrait.executeInner(SqlToSubstrait.java:113)
at io.substrait.isthmus.SqlToSubstrait.execute(SqlToSubstrait.java:81)
Contribution guidelines does state that the project uses JDK 17 and Gradle, however detailed developer environment setup, build and test instructions will be helpful for beginners.
query 15 failed due to comments in .sql file. CommandLine cannot recognize the comments which start from --
https://github.com/substrait-io/substrait-java/blob/main/isthmus/src/test/resources/tpch/queries/15.sql
hi, folks
When i open substrait-java
project in IntlliJ IDEA, output below error message.
How should I solve this problem? Please help, thanks!
A problem occurred configuring root project 'substrait'.
> Could not resolve all files for configuration ':classpath'.
> Could not find spotless-lib-extra-2.30.0.jar (com.diffplug.spotless:spotless-lib-extra:2.30.0).
Searched in the following locations:
https://plugins.gradle.org/m2/com/diffplug/spotless/spotless-lib-extra/2.30.0/spotless-lib-extra-2.30.0.jar
> Could not find spotless-lib-2.30.0.jar (com.diffplug.spotless:spotless-lib:2.30.0).
Searched in the following locations:
https://plugins.gradle.org/m2/com/diffplug/spotless/spotless-lib/2.30.0/spotless-lib-2.30.0.jar
Possible solution:
- Declare repository providing the artifact, see the documentation at https://docs.gradle.org/current/userguide/declaring_repositories.html
For example in q44
has the expression: ss_customer_sk is null
Queries in tpcds having this issue: [40, 44, 49, 75, 76, 77, 78, 80, 93, 97]
How to reproduce:
Add a test to TpcdsQueryNoValidation
like this:
@ParameterizedTest
@ValueSource(ints = {44})
public void tpcdsQ12(int query) throws Exception {
testQuery(query);
}
On fixing, please move the fixed queries from the tpcdsFailure
list to the tpcdsSuccess
list
In q67
has the expression: rank() over (partition by i_category order by sumsales desc)
This is after applying #56
Error Statcktrace:
java.lang.UnsupportedOperationException: RexOver not supported
at io.substrait.isthmus.expression.RexExpressionConverter.visitOver(RexExpressionConverter.java:76)
at io.substrait.isthmus.expression.RexExpressionConverter.visitOver(RexExpressionConverter.java:17)
at org.apache.calcite.rex.RexOver.accept(RexOver.java:122)
at io.substrait.isthmus.SubstraitRelVisitor.toExpression(SubstraitRelVisitor.java:68)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
...
How to reproduce:
Add a test to TpcdsQueryNoValidation
like this:
@ParameterizedTest
@ValueSource(ints = {67})
public void tpcdsQ12(int query) throws Exception {
testQuery(query);
}
On fixing, please move the fixed queries from the tpcdsFailure
list to the tpcdsSuccess
list
Based on our experiments with Substrait plans and conversations with the Substrait folks highlighted on the Google Group, it looks like the Substrait plans currently generated by Isthmus are unoptimized.
We're actively looking to use Substrait plans infused with Calcite optimizations for our internal projects. Please keep us updated on when support for the same could be added, so that we can incorporate and experiment with the optimized plans.
Additionally, if there're any other (ideally pythonic) tools/APIs that successfully generate optimized Substrait plans (from any other query optimizers) now or in the future, please let us know and we'd love to utilize them for our projects.
Thank you very much for the continued support!
(disclaimer this is me poking at the project to learn about it)
I wrote a simple reflection based test, that lists all methods in ExpressionCreator
and for those using simple types (primitives and things like Dates
or BigDecimal
etc). I generate a synthetic set of params and invoke the method doing a roundtrip conversion (lifted from LiteralRoundtripTest
which was only covering decimal so far). Two questions:
Do you agree it makes sense to have something like this in the codebase, to automatically cover (at least a few) cases as they get added to the system? Substrait strikes me as a project that will grow pretty broad and automating high-coverage tests seems worthy. (happy to post code if you agree it makes sense).
I saw several of the methods failing on the roundtrip because the "nullability" seems not retained correctly. Is this by design and we should ignore it in the comparison or is it a bug? (example below with timestamp
method).
Invoking timestamp ([boolean, long]) with values ([true, -5750464908882337892])
expected: <TimestampLiteral{nullable=true, value=-5750464908882337892}> but was: <TimestampLiteral{nullable=false, value=-5750464908882337892}>
Expected :TimestampLiteral{nullable=true, value=-5750464908882337892}
Actual :TimestampLiteral{nullable=false, value=-5750464908882337892}
I merged the release patch it looks like the release build is failing.
Additionally, it looks like it is tagging version on each merge (maybe) as opposed to once a week...
Failure here: https://github.com/substrait-io/substrait-java/actions/runs/3544635664/jobs/5952198598
@davisusanibar, can you take a look?
Currently, the converter from substrait rel plan to calcite relnode plan does not support SortRel / FetchRel, and would not produce sql with order by clause, limit offset clause.
This ticket is to track this missing functionality.
The converter from substrait plan to sql currently does not support a subquery, either in the form of correlated or non-correlated subquery.
This ticket is to track this missing functionality.
Add support for set operations UNION and MINUS in SQL -> Substrait Proto -> Substrait POJO. This would involve adding implementation of methods in SubstraitRelVisitor, RelCopyOnWriteVisitor, ProtoRelConverter, and RelProtoConverter. Queries such as these should roundtrip between SQL, Substrait Proto, and Substrait POJO:
select p_partkey as partkey from part where p_partkey > cast(100 as bigint)
union select l_partkey as partkey from lineitem where l_orderkey > cast(100 as bigint);
select p_partkey as partkey from part where p_partkey > cast(100 as bigint)
minus select l_partkey as partkey from lineitem where l_orderkey > cast(100 as bigint);
Below is error log of q8, q12 is almost same.
Processing tpc-h query, 8
select o_year, sum(case when nation = 'EGYPT' then volume else 0 end) / sum(volume) as mkt_share from ( select extract(year from o.o_orderdate) as o_year, l.l_extendedprice 1.txt 1y.txt count.txt date.txt decimal.txt filter.txt null.txt smoke.sh str.txt tpch_smoke.sh where.txt (1 - l.l_discount) as volume, n2.n_name as nation from "part" p, "supplier" s, "lineitem" l, "orders" o, "customer" c, "nation" n1, "nation" n2, "region" r where p.p_partkey = l.l_partkey and s.s_suppkey = l.l_suppkey and l.l_orderkey = o.o_orderkey and o.o_custkey = c.c_custkey and c.c_nationkey = n1.n_nationkey and n1.n_regionkey = r.r_regionkey and r.r_name = 'MIDDLE EAST' and s.s_nationkey = n2.n_nationkey and o.o_orderdate between date '1995-01-01' and date '1996-12-31' and p.p_type = 'PROMO BRUSHED COPPER' ) as all_nations group by o_year order by o_year
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.calcite.linq4j.tree.BlockBuilder.<clinit>(BlockBuilder.java:52)
at org.apache.calcite.rex.RexExecutorImpl.compile(RexExecutorImpl.java:83)
at org.apache.calcite.rex.RexExecutorImpl.compile(RexExecutorImpl.java:68)
at org.apache.calcite.rex.RexExecutorImpl.reduce(RexExecutorImpl.java:132)
at org.apache.calcite.rex.RexSimplify.simplifyCast(RexSimplify.java:2216)
at org.apache.calcite.rex.RexSimplify.simplify(RexSimplify.java:289)
at org.apache.calcite.rex.RexSimplify.simplifyList(RexSimplify.java:662)
at org.apache.calcite.rex.RexSimplify.simplifyComparison(RexSimplify.java:506)
at org.apache.calcite.rex.RexSimplify.simplifyComparison(RexSimplify.java:498)
at org.apache.calcite.rex.RexSimplify.simplify(RexSimplify.java:307)
at org.apache.calcite.rex.RexSimplify.simplifyCase(RexSimplify.java:1138)
at org.apache.calcite.rex.RexSimplify.simplify(RexSimplify.java:285)
at org.apache.calcite.rex.RexSimplify.simplifyUnknownAs(RexSimplify.java:248)
at org.apache.calcite.rex.RexSimplify.simplifyPreservingType(RexSimplify.java:187)
at org.apache.calcite.rex.RexSimplify.simplifyPreservingType(RexSimplify.java:182)
at org.apache.calcite.tools.RelBuilder.project_(RelBuilder.java:1947)
at org.apache.calcite.tools.RelBuilder.project_(RelBuilder.java:1938)
at org.apache.calcite.tools.RelBuilder.project(RelBuilder.java:1797)
at org.apache.calcite.tools.RelBuilder.projectNamed(RelBuilder.java:2088)
at org.apache.calcite.sql2rel.SqlToRelConverter.createAggImpl(SqlToRelConverter.java:3289)
at org.apache.calcite.sql2rel.SqlToRelConverter.convertAgg(SqlToRelConverter.java:3192)
at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:738)
at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:664)
at org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3589)
at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:589)
at io.substrait.isthmus.SqlToSubstrait.lambda$sqlToRelNode$1(SqlToSubstrait.java:153)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
at io.substrait.isthmus.SqlToSubstrait.sqlToRelNode(SqlToSubstrait.java:162)
at io.substrait.isthmus.SqlToSubstrait.executeInner(SqlToSubstrait.java:112)
at io.substrait.isthmus.SqlToSubstrait.execute(SqlToSubstrait.java:81)
at io.substrait.isthmus.PlanEntryPoint.call(PlanEntryPoint.java:41)
at io.substrait.isthmus.PlanEntryPoint.call(PlanEntryPoint.java:15)
at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
at picocli.CommandLine.access$1300(CommandLine.java:145)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
at picocli.CommandLine.execute(CommandLine.java:2078)
at io.substrait.isthmus.PlanEntryPoint.main(PlanEntryPoint.java:49)
Caused by: java.lang.RuntimeException: Unknown field 'FALSE' in class class java.lang.Boolean
at org.apache.calcite.linq4j.tree.Types.getField(Types.java:110)
at org.apache.calcite.linq4j.tree.Types.getField(Types.java:121)
at org.apache.calcite.linq4j.tree.Expressions.field(Expressions.java:849)
at org.apache.calcite.linq4j.tree.OptimizeShuttle.<clinit>(OptimizeShuttle.java:47)
... 48 more
Caused by: java.lang.NoSuchFieldException: FALSE
at java.lang.Class.getField(DynamicHub.java:1103)
at org.apache.calcite.linq4j.tree.Types.getField(Types.java:108)
... 51 more
For example in q72
has the expression: d3.d_date > d1.d_date + 5
Queries in tpcds having this issue: [72]
How to reproduce:
Add a test to TpcdsQueryNoValidation
like this:
@ParameterizedTest
@ValueSource(ints = {72})
public void tpcdsQ72(int query) throws Exception {
testQuery(query);
}
On fixing, please move the fixed queries from the tpcdsFailure
list to the tpcdsSuccess
list
As discussed in the PR when it was on the other project, whether we go with Java 17 minimum or something older, either way we should probably not make use of preview features.
For example:
q27
has the expression: group by rollup (i_item_id, s_state)
rank() over (partition by grouping(i_category)+grouping(i_class)..)
Queries in tpcds having this issue: [27, 36, 86]
How to reproduce:
Add a test to TpcdsQueryNoValidation
like this:
@ParameterizedTest
@ValueSource(ints = {27})
public void tpcdsQ27(int query) throws Exception {
testQuery(query);
}
On fixing, please move the fixed queries from the tpcdsFailure
list to the tpcdsSuccess
list
Calcite produces RexLiteral with Sarg to handle ORed together column value comparison. This Sarg encapsulates multiple literal values, but our code assumes there will be a single value. For example, tpc-h query 12 fails:
select
l.l_shipmode,
sum(case
when o.o_orderpriority = '1-URGENT'
or o.o_orderpriority = '2-HIGH'
then 1
else 0
end) as high_line_count,
sum(case
when o.o_orderpriority <> '1-URGENT'
and o.o_orderpriority <> '2-HIGH'
then 1
else 0
end) as low_line_count
from
"orders" o,
"lineitem" l
where
o.o_orderkey = l.l_orderkey
and l.l_shipmode in ('TRUCK', 'REG AIR')
and l.l_commitdate < l.l_receiptdate
and l.l_shipdate < l.l_commitdate
and l.l_receiptdate >= date '1994-01-01'
and l.l_receiptdate < date '1994-01-01' + interval '1' year
group by
l.l_shipmode
order by
l.l_shipmode
This query ends up with a RexLiteral with these values: Sarg[_ISO-8859-1'1-URGENT ', _ISO-8859-1'2-HIGH ']
ending up producing the following exception:
java.lang.UnsupportedOperationException: Unable to handle char type: Sarg[_ISO-8859-1'1-URGENT ', _ISO-8859-1'2-HIGH ']
at io.substrait.isthmus.expression.LiteralConverter.convert(LiteralConverter.java:96)
at io.substrait.isthmus.expression.RexExpressionConverter.visitLiteral(RexExpressionConverter.java:92)
at io.substrait.isthmus.expression.RexExpressionConverter.visitLiteral(RexExpressionConverter.java:21)
at org.apache.calcite.rex.RexLiteral.accept(RexLiteral.java:1217)
at io.substrait.isthmus.expression.RexExpressionConverter.lambda$visitCall$1(RexExpressionConverter.java:81)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at io.substrait.isthmus.expression.RexExpressionConverter.visitCall(RexExpressionConverter.java:82)
at io.substrait.isthmus.expression.RexExpressionConverter.visitCall(RexExpressionConverter.java:21)
at org.apache.calcite.rex.RexCall.accept(RexCall.java:189)
at io.substrait.isthmus.expression.RexExpressionConverter.lambda$visitCall$0(RexExpressionConverter.java:57)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
at io.substrait.isthmus.expression.CallConverters.lambda$static$1(CallConverters.java:51)
at io.substrait.isthmus.expression.CallConverters$SimpleCallConverter.convert(CallConverters.java:83)
at io.substrait.isthmus.expression.RexExpressionConverter.visitCall(RexExpressionConverter.java:57)
at io.substrait.isthmus.expression.RexExpressionConverter.visitCall(RexExpressionConverter.java:21)
at org.apache.calcite.rex.RexCall.accept(RexCall.java:189)
at io.substrait.isthmus.SubstraitRelVisitor.toExpression(SubstraitRelVisitor.java:76)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
at io.substrait.isthmus.SubstraitRelVisitor.visit(SubstraitRelVisitor.java:131)
at io.substrait.isthmus.SubstraitRelVisitor.visit(SubstraitRelVisitor.java:36)
at io.substrait.isthmus.RelNodeVisitor.reverseAccept(RelNodeVisitor.java:95)
at io.substrait.isthmus.SubstraitRelVisitor.apply(SubstraitRelVisitor.java:317)
at io.substrait.isthmus.SubstraitRelVisitor.visit(SubstraitRelVisitor.java:192)
at io.substrait.isthmus.SubstraitRelVisitor.visit(SubstraitRelVisitor.java:36)
at io.substrait.isthmus.RelNodeVisitor.reverseAccept(RelNodeVisitor.java:113)
at io.substrait.isthmus.SubstraitRelVisitor.apply(SubstraitRelVisitor.java:317)
at io.substrait.isthmus.SubstraitRelVisitor.visit(SubstraitRelVisitor.java:243)
at io.substrait.isthmus.SubstraitRelVisitor.visit(SubstraitRelVisitor.java:36)
at io.substrait.isthmus.RelNodeVisitor.reverseAccept(RelNodeVisitor.java:109)
at io.substrait.isthmus.SubstraitRelVisitor.apply(SubstraitRelVisitor.java:317)
at io.substrait.isthmus.SubstraitRelVisitor.convert(SubstraitRelVisitor.java:334)
at io.substrait.isthmus.SubstraitRelVisitor.convert(SubstraitRelVisitor.java:326)
at io.substrait.isthmus.SqlToSubstrait.lambda$executeInner$0(SqlToSubstrait.java:114)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at io.substrait.isthmus.SqlToSubstrait.executeInner(SqlToSubstrait.java:107)
at io.substrait.isthmus.SqlToSubstrait.execute(SqlToSubstrait.java:80)
at io.substrait.isthmus.TpchQueryNoValidation.tpch(TpchQueryNoValidation.java:18)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:92)
at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:210)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:206)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:131)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:65)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
at org.junit.platform.engine.support.hierarchical.NodeTestTask$DefaultDynamicTestExecutor.execute(NodeTestTask.java:212)
at org.junit.platform.engine.support.hierarchical.NodeTestTask$DefaultDynamicTestExecutor.execute(NodeTestTask.java:192)
at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:139)
at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.lambda$execute$2(TestTemplateTestDescriptor.java:107)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$15$1.accept(ReferencePipeline.java:541)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:107)
at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:42)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:75)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at jdk.proxy1/jdk.proxy1.$Proxy2.stop(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:133)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:71)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
For example:
q5
has the expression: 'web_site' || web_site_id as id
q66
has the expression: 'DHL' || ',' || 'BARIAN' as ship_carriers
q80
has expression using ||Implicit casting is not
Queries in tpcds having this issue: [5, 66, 80, 84]
How to reproduce:
Add a test to TpcdsQueryNoValidation
like this:
@ParameterizedTest
@ValueSource(ints = {5})
public void tpcdsQ5(int query) throws Exception {
testQuery(query);
}
On fixing, please move the fixed queries from the tpcdsFailure
list to the tpcdsSuccess
list
Current code does not support aggregation MIN/MAX
in sql -> substrait plan conversion. As such, tpc-h q2 and q15 would fail with error like "Unable to find binding for call MIN($0)".
Two parts of places probably need fix:
Open this ticket to track the missing MIN/MAX support.
substrait-io/substrait#405 added an ExtendedExpression
message to represent standalone expressions (expressions that are not part of plans). It would be nice to have this implemented in the Substrait Java library.
Currently Isthmus only supports outputting the substrait plan in json format. It would be great to have the options of getting the binary output as well.
Looks like there was some breaking change in more recent substrait release. When attempting to upgrade substrait-java to latest substrait (v 0.9.0), I get the following exception when running the tpch tests:
java.lang.ExceptionInInitializerError
at io.substrait.isthmus.TpchQueryNoValidation.tpch(TpchQueryNoValidation.java:15)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:92)
at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:210)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:206)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:131)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:65)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
at org.junit.platform.engine.support.hierarchical.NodeTestTask$DefaultDynamicTestExecutor.execute(NodeTestTask.java:212)
at org.junit.platform.engine.support.hierarchical.NodeTestTask$DefaultDynamicTestExecutor.execute(NodeTestTask.java:192)
at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:139)
at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.lambda$execute$2(TestTemplateTestDescriptor.java:107)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$15$1.accept(ReferencePipeline.java:541)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:107)
at org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:42)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:143)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:32)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:108)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:96)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:75)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at jdk.proxy1/jdk.proxy1.$Proxy2.stop(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:133)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:71)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: java.lang.RuntimeException: Failure while parsing /functions_arithmetic.yaml
at io.substrait.function.SimpleExtension.load(SimpleExtension.java:605)
at io.substrait.function.SimpleExtension.lambda$load$1(SimpleExtension.java:571)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
at io.substrait.function.SimpleExtension.load(SimpleExtension.java:576)
at io.substrait.function.SimpleExtension.loadDefaults(SimpleExtension.java:558)
at io.substrait.isthmus.SqlConverterBase.<clinit>(SqlConverterBase.java:76)
... 136 more
Caused by: com.fasterxml.jackson.databind.exc.ValueInstantiationException: Cannot construct instance of `io.substrait.function.ImmutableSimpleExtension$AggregateFunctionVariant`, problem: Cannot build AggregateFunctionVariant, some of required attributes are not set [intermediate]
at [Source: (sun.net.www.protocol.jar.JarURLConnection$JarURLInputStream); line: 721, column: 7] (through reference chain: io.substrait.function.ImmutableSimpleExtension$FunctionSignatures$Json["aggregate_functions"]->java.util.ArrayList[5]->io.substrait.function.ImmutableSimpleExtension$AggregateFunction$Json["impls"]->java.util.ArrayList[0])
at app//com.fasterxml.jackson.databind.exc.ValueInstantiationException.from(ValueInstantiationException.java:47)
at app//com.fasterxml.jackson.databind.DeserializationContext.instantiationException(DeserializationContext.java:1907)
at app//com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapAsJsonMappingException(StdValueInstantiator.java:587)
at app//com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.rewrapCtorProblem(StdValueInstantiator.java:610)
at app//com.fasterxml.jackson.databind.deser.std.StdValueInstantiator._createUsingDelegate(StdValueInstantiator.java:647)
at app//com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createUsingDelegate(StdValueInstantiator.java:306)
at app//com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1397)
at app//com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:362)
at app//com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:195)
at app//com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:355)
at app//com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
at app//com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
at app//com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
at app//com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:324)
at app//com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:225)
at app//com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:197)
at app//com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1398)
at app//com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:362)
at app//com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:195)
at app//com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:355)
at app//com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
at app//com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
at app//com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
at app//com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:324)
at app//com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:225)
at app//com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:197)
at app//com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1398)
at app//com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:362)
at app//com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:195)
at app//com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
at app//com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4593)
at app//com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3585)
at app//io.substrait.function.SimpleExtension.load(SimpleExtension.java:586)
... 149 more
Caused by: java.lang.IllegalStateException: Cannot build AggregateFunctionVariant, some of required attributes are not set [intermediate]
at io.substrait.function.ImmutableSimpleExtension$AggregateFunctionVariant$Builder.build(ImmutableSimpleExtension.java:3224)
at io.substrait.function.ImmutableSimpleExtension$AggregateFunctionVariant.fromJson(ImmutableSimpleExtension.java:2950)
at jdk.internal.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at com.fasterxml.jackson.databind.introspect.AnnotatedMethod.call1(AnnotatedMethod.java:107)
at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator._createUsingDelegate(StdValueInstantiator.java:631)
... 177 more
For example in q12
has the expression: sum(sum(ws_ext_sales_price)) over (partition by i_class)
Queries in tpcds having this issue: [12, 20, 47, 51, 53, 57, 63, 89, 98]
How to reproduce:
Add a test to TpcdsQueryNoValidation
like this:
@ParameterizedTest
@ValueSource(ints = {12})
public void tpcdsQ12(int query) throws Exception {
testQuery(query);
}
On fixing, please move the fixed queries from the tpcdsFailure
list to the tpcdsSuccess
list
See tpcds q9
How to reproduce:
Add a test to TpcdsQueryNoValidation
like this:
@ParameterizedTest
@ValueSource(ints = {9})
public void tpcdsQ9(int query) throws Exception {
testQuery(query);
}
On fixing, please move the fixed queries from the tpcdsFailure
list to the tpcdsSuccess
list
Hi, it seems we don't support null data in filter currently, e.g
table :
CREATE TABLE table_test(col_a BIGINT)
query :
SELECT col_a FROM table_test WHERE col_a IS NOT NULL
or
SELECT col_a FROM table_test WHERE col_a != NULL
Any help will be appreciated!
Isthmus may run into problems when converting a string concatenation call to substrait plan.
The input to Isthmus is
"SELECT 'foo' || 'bar' FROM test;" -c "CREATE TABLE test(col_1 INTEGER NOT NULL, col_2 VARCHAR(10));"
and it throws an error. I suspect that this is because substrait only have two signatures for concat
: concat:vchar_vchar
and concat:str_str
. In this case, both inputs are fixed char fchar
so the signature match fails. Perhaps adding a cast somewhere can fix this?
Full stack trace below:
java.lang.IllegalArgumentException: Unable to convert call ||(char<3>, char<3>).
at io.substrait.isthmus.expression.RexExpressionConverter.visitCall(RexExpressionConverter.java:78)
at io.substrait.isthmus.expression.RexExpressionConverter.visitCall(RexExpressionConverter.java:19)
at org.apache.calcite.rex.RexCall.accept(RexCall.java:189)
at io.substrait.isthmus.SubstraitRelVisitor.toExpression(SubstraitRelVisitor.java:99)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.util.Collections$2.tryAdvance(Collections.java:4853)
at java.util.Collections$2.forEachRemaining(Collections.java:4861)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at io.substrait.isthmus.SubstraitRelVisitor.visit(SubstraitRelVisitor.java:156)
at io.substrait.isthmus.SubstraitRelVisitor.visit(SubstraitRelVisitor.java:61)
at io.substrait.isthmus.RelNodeVisitor.reverseAccept(RelNodeVisitor.java:95)
at io.substrait.isthmus.SubstraitRelVisitor.apply(SubstraitRelVisitor.java:351)
at io.substrait.isthmus.SubstraitRelVisitor.convert(SubstraitRelVisitor.java:374)
at io.substrait.isthmus.SubstraitRelVisitor.convert(SubstraitRelVisitor.java:366)
at io.substrait.isthmus.SqlToSubstrait.lambda$executeInner$0(SqlToSubstrait.java:80)
at java.util.ArrayList.forEach(ArrayList.java:1511)
at io.substrait.isthmus.SqlToSubstrait.executeInner(SqlToSubstrait.java:73)
at io.substrait.isthmus.SqlToSubstrait.execute(SqlToSubstrait.java:41)
at io.substrait.isthmus.PlanEntryPoint.call(PlanEntryPoint.java:59)
at io.substrait.isthmus.PlanEntryPoint.call(PlanEntryPoint.java:16)
at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
at picocli.CommandLine.access$1300(CommandLine.java:145)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
at picocli.CommandLine.execute(CommandLine.java:2078)
at io.substrait.isthmus.PlanEntryPoint.main(PlanEntryPoint.java:51)
a || b
as the concatenation operator, and does not recognize CONCAT(a, b)
The input query is
"SELECT CONCAT('foo', col_2) FROM test;"
This seems to be limited by Apache Calcite. "SELECT 'foo' || col_2 FROM test;"
would work just fine.
Full stack trace below:
org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to line 1, column 27: No match found for function signature CONCAT(<CHARACTER>, <CHARACTER>)
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.lang.reflect.Constructor.newInstance(Constructor.java:480)
at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:505)
at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:932)
at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:917)
at org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:5266)
at org.apache.calcite.sql.validate.SqlValidatorImpl.handleUnresolvedFunction(SqlValidatorImpl.java:1948)
at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:326)
at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:231)
at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6277)
at org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6264)
at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:161)
at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1862)
at org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1847)
at org.apache.calcite.sql.validate.SqlValidatorImpl.expandSelectItem(SqlValidatorImpl.java:463)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelectList(SqlValidatorImpl.java:4409)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3652)
at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:64)
at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:89)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1100)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:1071)
at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:247)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:1046)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:752)
at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:586)
at io.substrait.isthmus.SqlToSubstrait.lambda$sqlToRelNode$1(SqlToSubstrait.java:110)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at io.substrait.isthmus.SqlToSubstrait.sqlToRelNode(SqlToSubstrait.java:119)
at io.substrait.isthmus.SqlToSubstrait.executeInner(SqlToSubstrait.java:72)
at io.substrait.isthmus.SqlToSubstrait.execute(SqlToSubstrait.java:41)
at io.substrait.isthmus.PlanEntryPoint.call(PlanEntryPoint.java:59)
at io.substrait.isthmus.PlanEntryPoint.call(PlanEntryPoint.java:16)
at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
at picocli.CommandLine.access$1300(CommandLine.java:145)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
at picocli.CommandLine.execute(CommandLine.java:2078)
at io.substrait.isthmus.PlanEntryPoint.main(PlanEntryPoint.java:51)
Caused by: org.apache.calcite.sql.validate.SqlValidatorException: No match found for function signature CONCAT(<CHARACTER>, <CHARACTER>)
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.lang.reflect.Constructor.newInstance(Constructor.java:480)
at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:505)
at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:599)
... 43 more
For example in q2
has the expression: round(sun_sales1/sun_sales2,2)
Queries in tpcds having this issue: [2, 17, 24, 39, 78, 91]
How to reproduce:
Add a test to TpcdsQueryNoValidation
like this:
@ParameterizedTest
@ValueSource(ints = {2})
public void tpcdsQ2(int query) throws Exception {
testQuery(query);
}
On fixing, please move the fixed queries from the tpcdsFailure
list to the tpcdsSuccess
list
Scalar
and Aggregate
FunctionInvocation
to be made of FuncArg
sFuncArg
sSee Issue #37 for background on this issue.
Plans generated with Isthmus currently include opt_
as a prefix in function signatures.
For example
./isthmus -c "CREATE TABLE persons (zip INT)" "SELECT zip + 1 FROM persons"
creates the follow plan (elided for brevity)
{
"extensionUris": [{
"extensionUriAnchor": 1,
"uri": "/functions_arithmetic.yaml"
}],
"extensions": [{
"extensionFunction": {
"extensionUriReference": 1,
"functionAnchor": 0,
"name": "add:opt_i32_i32"
}
}],
...
}
in which the +
operators is named "add:opt_i32_i32"
The current spec for Function Signature Compound Names no longer includes opt
.
This was removed as part of substrait-io/substrait#342.
Isthmus should be updated to omit the opt_
prefix to match the spec.
I'm seeing the following task failures when running ./gradlew build
.
> Task :isthmus:spotlessJava FAILED
Step 'google-java-format' found problem in 'src/main/java/io/substrait/isthmus/TypeConverter.java':
null
java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.GeneratedMethodAccessor415.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.diffplug.spotless.java.GoogleJavaFormatStep$State.lambda$createFormat$1(GoogleJavaFormatStep.java:177)
at com.diffplug.spotless.FormatterFunc.apply(FormatterFunc.java:32)
at com.diffplug.spotless.FormatterStepImpl$Standard.format(FormatterStepImpl.java:82)
at com.diffplug.spotless.FormatterStep$Strict.format(FormatterStep.java:88)
at com.diffplug.spotless.Formatter.compute(Formatter.java:230)
at com.diffplug.spotless.PaddedCell.calculateDirtyState(PaddedCell.java:203)
at com.diffplug.spotless.PaddedCell.calculateDirtyState(PaddedCell.java:190)
at com.diffplug.gradle.spotless.SpotlessTaskImpl.processInputFile(SpotlessTaskImpl.java:102)
at com.diffplug.gradle.spotless.SpotlessTaskImpl.performAction(SpotlessTaskImpl.java:88)
at jdk.internal.reflect.GeneratedMethodAccessor458.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)
at org.gradle.api.internal.project.taskfactory.IncrementalInputsTaskAction.doExecute(IncrementalInputsTaskAction.java:32)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)
at org.gradle.api.internal.project.taskfactory.AbstractIncrementalTaskAction.execute(AbstractIncrementalTaskAction.java:25)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)
at org.gradle.api.internal.tasks.execution.TaskExecution$3.run(TaskExecution.java:242)
at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
at org.gradle.api.internal.tasks.execution.TaskExecution.executeAction(TaskExecution.java:227)
at org.gradle.api.internal.tasks.execution.TaskExecution.executeActions(TaskExecution.java:210)
at org.gradle.api.internal.tasks.execution.TaskExecution.executeWithPreviousOutputFiles(TaskExecution.java:193)
at org.gradle.api.internal.tasks.execution.TaskExecution.execute(TaskExecution.java:171)
at org.gradle.internal.execution.steps.ExecuteStep.executeInternal(ExecuteStep.java:89)
at org.gradle.internal.execution.steps.ExecuteStep.access$000(ExecuteStep.java:40)
at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:53)
at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:50)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:50)
at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:40)
at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:68)
at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:38)
at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:48)
at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:36)
at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:41)
at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:74)
at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:55)
at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:51)
at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:29)
at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:61)
at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:42)
at org.gradle.internal.execution.steps.BroadcastChangingOutputsStep.execute(BroadcastChangingOutputsStep.java:60)
at org.gradle.internal.execution.steps.BroadcastChangingOutputsStep.execute(BroadcastChangingOutputsStep.java:27)
at org.gradle.internal.execution.steps.BuildCacheStep.executeWithoutCache(BuildCacheStep.java:180)
at org.gradle.internal.execution.steps.BuildCacheStep.executeAndStoreInCache(BuildCacheStep.java:155)
at org.gradle.internal.execution.steps.BuildCacheStep.lambda$executeWithCache$4(BuildCacheStep.java:125)
at java.base/java.util.Optional.orElseGet(Optional.java:369)
at org.gradle.internal.execution.steps.BuildCacheStep.lambda$executeWithCache$5(BuildCacheStep.java:125)
at org.gradle.internal.Try$Success.map(Try.java:164)
at org.gradle.internal.execution.steps.BuildCacheStep.executeWithCache(BuildCacheStep.java:85)
at org.gradle.internal.execution.steps.BuildCacheStep.lambda$execute$0(BuildCacheStep.java:74)
at org.gradle.internal.Either$Left.fold(Either.java:115)
at org.gradle.internal.execution.caching.CachingState.fold(CachingState.java:59)
at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:73)
at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:48)
at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:36)
at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:25)
at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:36)
at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:22)
at org.gradle.internal.execution.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:110)
at org.gradle.internal.execution.steps.SkipUpToDateStep.lambda$execute$2(SkipUpToDateStep.java:56)
at java.base/java.util.Optional.orElseGet(Optional.java:369)
at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:56)
at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:38)
at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:73)
at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:44)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:37)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:27)
at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:89)
at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:50)
at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:114)
at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:57)
at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:76)
at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:50)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.executeWithNoEmptySources(SkipEmptyWorkStep.java:249)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:86)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:54)
at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:32)
at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:21)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:38)
at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:43)
at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:31)
at org.gradle.internal.execution.steps.AssignWorkspaceStep.lambda$execute$0(AssignWorkspaceStep.java:40)
at org.gradle.api.internal.tasks.execution.TaskExecution$4.withWorkspace(TaskExecution.java:287)
at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:40)
at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:30)
at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:37)
at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:27)
at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:44)
at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:33)
at org.gradle.internal.execution.impl.DefaultExecutionEngine$1.execute(DefaultExecutionEngine.java:76)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:144)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:133)
at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:77)
at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:74)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:333)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:320)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:313)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:299)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.lambda$run$0(DefaultPlanExecutor.java:143)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:227)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:218)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:140)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.google.googlejavaformat.java.FormatterException: 43:13: error: illegal start of expression
at com.google.googlejavaformat.java.FormatterException.fromJavacDiagnostics(FormatterException.java:51)
at com.google.googlejavaformat.java.Formatter.format(Formatter.java:149)
at com.google.googlejavaformat.java.Formatter.getFormatReplacements(Formatter.java:271)
at com.google.googlejavaformat.java.Formatter.formatSource(Formatter.java:247)
at com.google.googlejavaformat.java.Formatter.formatSource(Formatter.java:213)
... 142 more
> Task :core:spotlessJava FAILED
Step 'google-java-format' found problem in 'src/main/java/io/substrait/relation/files/FileOrFiles.java':
null
java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.GeneratedMethodAccessor415.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.diffplug.spotless.java.GoogleJavaFormatStep$State.lambda$createFormat$1(GoogleJavaFormatStep.java:177)
at com.diffplug.spotless.FormatterFunc.apply(FormatterFunc.java:32)
at com.diffplug.spotless.FormatterStepImpl$Standard.format(FormatterStepImpl.java:82)
at com.diffplug.spotless.FormatterStep$Strict.format(FormatterStep.java:88)
at com.diffplug.spotless.Formatter.compute(Formatter.java:230)
at com.diffplug.spotless.PaddedCell.calculateDirtyState(PaddedCell.java:203)
at com.diffplug.spotless.PaddedCell.calculateDirtyState(PaddedCell.java:190)
at com.diffplug.gradle.spotless.SpotlessTaskImpl.processInputFile(SpotlessTaskImpl.java:102)
at com.diffplug.gradle.spotless.SpotlessTaskImpl.performAction(SpotlessTaskImpl.java:88)
at jdk.internal.reflect.GeneratedMethodAccessor458.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)
at org.gradle.api.internal.project.taskfactory.IncrementalInputsTaskAction.doExecute(IncrementalInputsTaskAction.java:32)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)
at org.gradle.api.internal.project.taskfactory.AbstractIncrementalTaskAction.execute(AbstractIncrementalTaskAction.java:25)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)
at org.gradle.api.internal.tasks.execution.TaskExecution$3.run(TaskExecution.java:242)
at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
at org.gradle.api.internal.tasks.execution.TaskExecution.executeAction(TaskExecution.java:227)
at org.gradle.api.internal.tasks.execution.TaskExecution.executeActions(TaskExecution.java:210)
at org.gradle.api.internal.tasks.execution.TaskExecution.executeWithPreviousOutputFiles(TaskExecution.java:193)
at org.gradle.api.internal.tasks.execution.TaskExecution.execute(TaskExecution.java:171)
at org.gradle.internal.execution.steps.ExecuteStep.executeInternal(ExecuteStep.java:89)
at org.gradle.internal.execution.steps.ExecuteStep.access$000(ExecuteStep.java:40)
at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:53)
at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:50)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:50)
at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:40)
at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:68)
at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:38)
at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:48)
at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:36)
at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:41)
at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:74)
at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:55)
at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:51)
at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:29)
at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:61)
at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:42)
at org.gradle.internal.execution.steps.BroadcastChangingOutputsStep.execute(BroadcastChangingOutputsStep.java:60)
at org.gradle.internal.execution.steps.BroadcastChangingOutputsStep.execute(BroadcastChangingOutputsStep.java:27)
at org.gradle.internal.execution.steps.BuildCacheStep.executeWithoutCache(BuildCacheStep.java:180)
at org.gradle.internal.execution.steps.BuildCacheStep.executeAndStoreInCache(BuildCacheStep.java:155)
at org.gradle.internal.execution.steps.BuildCacheStep.lambda$executeWithCache$4(BuildCacheStep.java:125)
at java.base/java.util.Optional.orElseGet(Optional.java:369)
at org.gradle.internal.execution.steps.BuildCacheStep.lambda$executeWithCache$5(BuildCacheStep.java:125)
at org.gradle.internal.Try$Success.map(Try.java:164)
at org.gradle.internal.execution.steps.BuildCacheStep.executeWithCache(BuildCacheStep.java:85)
at org.gradle.internal.execution.steps.BuildCacheStep.lambda$execute$0(BuildCacheStep.java:74)
at org.gradle.internal.Either$Left.fold(Either.java:115)
at org.gradle.internal.execution.caching.CachingState.fold(CachingState.java:59)
at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:73)
at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:48)
at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:36)
at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:25)
at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:36)
at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:22)
at org.gradle.internal.execution.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:110)
at org.gradle.internal.execution.steps.SkipUpToDateStep.lambda$execute$2(SkipUpToDateStep.java:56)
at java.base/java.util.Optional.orElseGet(Optional.java:369)
at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:56)
at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:38)
at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:73)
at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:44)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:37)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:27)
at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:89)
at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:50)
at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:114)
at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:57)
at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:76)
at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:50)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.executeWithNoEmptySources(SkipEmptyWorkStep.java:249)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:86)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:54)
at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:32)
at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:21)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:38)
at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:43)
at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:31)
at org.gradle.internal.execution.steps.AssignWorkspaceStep.lambda$execute$0(AssignWorkspaceStep.java:40)
at org.gradle.api.internal.tasks.execution.TaskExecution$4.withWorkspace(TaskExecution.java:287)
at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:40)
at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:30)
at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:37)
at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:27)
at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:44)
at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:33)
at org.gradle.internal.execution.impl.DefaultExecutionEngine$1.execute(DefaultExecutionEngine.java:76)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:144)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:133)
at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:77)
at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:74)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:333)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:320)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:313)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:299)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.lambda$run$0(DefaultPlanExecutor.java:143)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:227)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:218)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:140)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.google.googlejavaformat.java.FormatterException: 35:71: error: ')' expected
at com.google.googlejavaformat.java.FormatterException.fromJavacDiagnostics(FormatterException.java:51)
at com.google.googlejavaformat.java.Formatter.format(Formatter.java:149)
at com.google.googlejavaformat.java.Formatter.getFormatReplacements(Formatter.java:271)
at com.google.googlejavaformat.java.Formatter.formatSource(Formatter.java:247)
at com.google.googlejavaformat.java.Formatter.formatSource(Formatter.java:213)
... 142 more
substrait-io/substrait#405 added an ExtendedExpression
message to represent standalone expressions (expressions that are not part of plans). #127 would add support for this in the Substrait Java library. It would also be nice if Isthmus could convert SQL expressions (not whole queries) to Substrait ExtendedExpression
messages. This would help to open up more applications for Substrait as an interoperable representation for expressions.
For beginners, if a user only parses a DML SQL via isthmus will meet the calcite exception, like this:
org.apache.calcite.runtime.CalciteContextException: From line 1, column 17 to line 1, column 31: Object 'XXX' not found
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.lang.reflect.Constructor.newInstance(Constructor.java:480)
at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:505)
at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:932)
at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:917)
at org.apache.calcite.sql.validate.SqlValidatorImpl.newValidationError(SqlValidatorImpl.java:5266)
at org.apache.calcite.sql.validate.IdentifierNamespace.resolveImpl(IdentifierNamespace.java:183)
at org.apache.calcite.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:188)
at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:89)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1100)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:1071)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3375)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3639)
at org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:64)
at org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:89)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1100)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:1071)
at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:247)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:1046)
at org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:752)
at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:586)
at io.substrait.isthmus.SqlToSubstrait.getBestExpRelRoot(SqlToSubstrait.java:124)
at io.substrait.isthmus.SqlToSubstrait.lambda$sqlToRelNode$1(SqlToSubstrait.java:103)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at io.substrait.isthmus.SqlToSubstrait.sqlToRelNode(SqlToSubstrait.java:104)
at io.substrait.isthmus.SqlToSubstrait.executeInner(SqlToSubstrait.java:74)
at io.substrait.isthmus.SqlToSubstrait.execute(SqlToSubstrait.java:43)
at io.substrait.isthmus.PlanEntryPoint.call(PlanEntryPoint.java:59)
at io.substrait.isthmus.PlanEntryPoint.call(PlanEntryPoint.java:16)
at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
at picocli.CommandLine.access$1300(CommandLine.java:145)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
at picocli.CommandLine.execute(CommandLine.java:2078)
at io.substrait.isthmus.PlanEntryPoint.main(PlanEntryPoint.java:51)
Caused by: org.apache.calcite.sql.validate.SqlValidatorException: Object 'XXX' not found
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
at java.lang.reflect.Constructor.newInstance(Constructor.java:480)
at org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:505)
at org.apache.calcite.runtime.Resources$ExInst.ex(Resources.java:599)
... 40 more
So, IMO, it would be better to give a friendly tip when parsing DML SQL to let the users know it could give the definition via the -c
option.
Adding python bindings for Isthmus would allow us to easily generate Isthmus substrait plans on the fly in the consumer-testing, instead of manually generating the plans and merging them into the repo to be tested against.
I noticed that both of them share almost same code path. tpch_smoke.sh
can not pass some queries(q7,q8...) while TpchQueryNoValidation
can pass all. And tpch_smoke.sh
is in CI workflow, the other one is not.
SQL create-table statements often contain key constraints, for e.g. TPC-DS table primary key declarations like below:
create table store_sales (
ss_item_sk integer not null,
...
ss_net_profit decimal(7,2) ,
primary key (ss_item_sk));
These constraint declaration are valid per SQL standard. Currently the SQL to Substrait fails when it encounters such constraints with the following error:
java.lang.ClassCastException: class org.apache.calcite.sql.ddl.SqlDdlNodes$1 cannot be cast to class org.apache.calcite.sql.ddl.SqlColumnDeclaration (org.apache.calcite.sql.ddl.SqlDdlNodes$1 and org.apache.calcite.sql.ddl.SqlColumnDeclaration are in unnamed module of loader 'app')
at io.substrait.isthmus.SqlConverterBase.parseCreateTable(SqlConverterBase.java:123)
at io.substrait.isthmus.SqlConverterBase.registerCreateTables(SqlConverterBase.java:86)
at io.substrait.isthmus.SqlToSubstrait.execute(SqlToSubstrait.java:81)
Instead of expecting users to omit constraints from input table definitions, key constraints should be ignored so that the plan generation does not fail.
In sql -> substrait plan conversion, tpc-q q7, q8, q9 would fail with error: Unable to convert call EXTRACT(string, date?).
Although substrait spec's functions_datetime.yaml
has declarations for EXTRACT
function, substrait-java's ScalarFunctionConverter
would not resolve EXTRACT
function call properly.
Part of reason it does not work properly is the handling of EnumArgument. Somehow the code handles EnumArgument as a string argument, and leads to the failure of function resolution.
Open this ticket to track this issue.
I verified all tpch queries via tpch_smoke.sh
which use graal compiled native code, found that these queries(7 8 9 12) will fail.
BTW, q15 will fail due to comments can not be parsed, remove that will work.
error message:
.......
at com.google.protobuf.util.JsonFormat$Printer.print(JsonFormat.java:381)
at io.substrait.isthmus.PlanEntryPoint.call(PlanEntryPoint.java:42)
at io.substrait.isthmus.PlanEntryPoint.call(PlanEntryPoint.java:15)
at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
at picocli.CommandLine.access$1300(CommandLine.java:145)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
at picocli.CommandLine.execute(CommandLine.java:2078)
at io.substrait.isthmus.PlanEntryPoint.main(PlanEntryPoint.java:49)
Caused by: java.lang.NoSuchMethodException: com.google.protobuf.Empty.newBuilder()
Can anyone help verify and fix it? Thanks in advance!
π hi all - having tried out Substrait and Substrait Java I'm really excited about the future of the projects! I'd love to help out if I can.
I'm currently experimenting with converting ZetaSQL parse trees into Substrait plans, but have found that it would be great to get the latest version of the Substrait protos into substrait-java.
I have a bit of experience with Gradle Java projects and saw there were a few open issues around repository automation such as auto-upgrade of the substrait specs and getting releases into Maven Central. Would it be helpful if I put up a PR to set this up? I've had good experiences with Renovate Bot in the past - you can see some of the upgrade PRs it has made to my fork of substrait-java
Error message:
[semantic-release]: node version >=18 is required. Found v16.19.0.
See https://github.com/semantic-release/semantic-release/blob/master/docs/support/node-version.md for more details and solutions.
The field Cast::failure_behavior
has no listed default and yet Isthmus does not specify a value which forces consumers to guess.
Hi, I've been trying to run the repo to get the substrait plans for the supported TPCH queries. I'm using Ubuntu for Windows.
Initially I tried the following commands:
git clone https://github.com/substrait-io/substrait-java
cd substrait-java/
git submodule update --init --recursive
./gradlew build
./gradlew nativeImage
And got a failed build error with the ./gradlew nativeImage
command (error statement attached).
As an alternative, I tried to download the isthmus-ubuntu-latest pre-compiled build artifact and ran
./isthmus-ubuntu-latest --create "cat isthmus/src/test/resources/tpch/schema.sql" "cat isthmus/src/test/resources/tpch/queries/06.sql"
but got a Calcite undefined method issue (error statement attached).
Error statements:
isthmus.txt
Please let me know if I'm missing anything that I need to do to resolve this issue. Alternatively, if anyone already has the substrait plans for all the supported TPCH queries and could directly share them with me, that'll be incredibly helpful.
Looking forward to your response, thank you very much for your support!
While testing the validator against Isthmus output for TPC-H, I found a number of issues where Isthmus disagrees with the spec as written on the website. The ones I found are:
select l_returnflag, count(*) from lineitem group by l_returnflag
. The easy solution is just to generate an emit clause for all aggregate relations.select 0.1
select -0.1
, which yields out-of-bounds value 25.5 for P=2 S=1.select 000000.000001
.decimal
is used instead of dec
, and any1
is used in place of just any
. Failing example: literally anything with a function, such as select 0.1 + 1.0
, which generates add:opt_decimal_decimal
instead of add:opt_dec_dec
.select 1
is given the schema STRUCT<STRUCT<i32>>
. The type itself is also invalid because of this, because only one field name is attached when two would be needed for that type. It should be NSTRUCT<EXPR$0: i32>
here.I haven't touched anything java for about a decade, so I'm probably not the right person to propose fixes for these things.
Here's a script to easily run Isthmus and generate a validation report for the result. Just run pip install .
at https://github.com/jvanstraten/substrait-validator/tree/initial/py to install the validator, and chuck the script in the root of substrait-java
. Call the script without arguments to generate reports for all TPC-H queries (it will try them all, but will only generate reports for successful runs), or with a SQL query as cmdline argument for a custom query based on the TPC-H database schema. The reports are written to the tpch-output
directory.
#!/usr/bin/env python3
# SPDX-License-Identifier: Apache-2.0
import os
import sys
import shutil
import subprocess
import json
import substrait_validator
def format_html_code_block(text, lang):
text = text.replace('&', '&')
text = text.replace('<', '<')
text = text.replace('>', '>')
text = text.replace('"', '"')
text = text.replace("'", ''')
text = text.replace(" ", ' ')
text = text.replace("\t", ' ')
text = text.replace("\n", '<br/>\n')
return f'<pre><code class="language-{lang}">{text}</code></pre>'
_EXTRA_HEAD = """
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.5.0/styles/default.min.css">
<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.5.0/highlight.min.js"></script>
"""
if __name__ == '__main__':
# Figure out environment.
script_path = os.path.dirname(os.path.realpath(__file__))
output_path = os.path.join(script_path, 'tpch-output')
repo_path = script_path
java_path = repo_path
tpch_path = os.path.join(java_path, 'isthmus', 'src', 'test', 'resources', 'tpch')
schema_path = os.path.join(tpch_path, 'schema.sql')
query_path = os.path.join(tpch_path, 'queries')
isthmus_path = os.path.join(java_path, 'isthmus', 'build', 'graal', 'isthmus')
# Create output directory.
if not os.path.isdir(output_path):
os.makedirs(output_path)
# Build isthmus if it hasn't already been built.
if not os.path.isfile(isthmus_path):
cur = os.curdir
os.chdir(java_path)
try:
subprocess.run(['./gradlew', 'nativeImage'], check=True)
finally:
os.chdir(cur)
# Load TPC-H schema files.
isthmus_args = [isthmus_path]
with open(schema_path, 'r') as f:
for query in filter(bool, map(str.strip, f.read().split(';'))):
isthmus_args.append('-c')
isthmus_args.append(query)
# Function to run Isthmus and the validator on some named query.
def run_isthmus(name, query):
try:
# Convert to Substrait plan with Isthmus.
result = subprocess.run(isthmus_args + [query], capture_output=True)
if result.returncode != 0:
raise Exception(result.stderr.decode('utf-8'))
plan = result.stdout.decode('utf-8')
# Convert to HTML validation result with validator.
html = substrait_validator.plan_to_html(plan)
# Unpack the HTML a bit so we can add stuff to it.
html_gen_a, remain = html.split('</head>', maxsplit=1)
html_gen_b, remain = remain.split('<body>', maxsplit=1)
html_gen_b = f'</head>{html_gen_b}<body>'
html_gen_c, html_gen_d = remain.split('</body>', maxsplit=1)
html_gen_d = f'</body>{html_gen_d}'
# Add our stuff to it.
html = []
html.append(html_gen_a)
html.append(f'<title>{name}</title>')
html.append(_EXTRA_HEAD)
html.append(html_gen_b)
html.append(f'<h1>{name} with Isthmus</h1>')
html.append(f'<h2>SQL query</h2>')
html.append(format_html_code_block(query, 'sql'))
html.append(f'<h2>Plan JSON</h2>')
html.append(format_html_code_block(plan, 'json'))
html.append(f'<h2>Validation result</h2>')
html.append(html_gen_c)
html.append('<script>hljs.highlightAll();</script>')
html.append(html_gen_d)
html = '\n'.join(html)
# Write file.
html_fname = os.path.join(output_path, f'{name}.html')
with open(html_fname, 'w') as f:
f.write(html)
except Exception as e:
print(f'{type(e).__name__} for {name}: {e}')
if len(sys.argv) <= 1:
# Run isthmus and the validator for all queries.
for query_fname in sorted(os.listdir(query_path)):
name = query_fname.split('.')[0]
with open(os.path.join(query_path, query_fname), 'r') as f:
query = f.read()
run_isthmus(name, query)
else:
# Interpret command line as a query.
run_isthmus('cmdline', ' '.join(sys.argv[1:]))
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.