Bug: Applying aggregation functions on select() to some fields and simple projection to other causes runtime errors

[imarios]

Bug: Applying aggregation functions on select() to some fields and simple projection to other causes runtime errors

[kanterov]

@imarios it seems I fixed it in #123

[OlivierBlanvillain]

That should be a compile time error, right?

I think TypedAggregateAndColumn shoudn't be <: TypedColumn, or every thing defined in AggregateFunctions becomes usable in select...

[kanterov]

Hm.. I remember it worked before, probably something was changed, and we have tests only for agg case. We need to add tests and adjust types. And if it isn't relevant anymore for any instance of TypedAggregateAndColumn, then we can remove it.

[imarios]

Maybe we shouldn't have TypedAggregateAndColumn to be both a column and an aggregated I agree. What was the initial idea of having it as both?

[kanterov]

At some point in Spark you could select(sum("a")), the problem there is a name collision between sum in select of TypedColumn and sum in agg of TypedAggregate, the idea was to create sort of union type for it :).

[imarios]

Obviously this is still an issue in Spark, if you do the same thing with datasets you get a nice runtime error. The issue here is that we are trying to give you a compiler error if things should fail at runtime. So now we need to think how we can do that here.

Running aggregations in select is useful when you don't really want any subgroups to be aggregated but you need the entire thing to be aggregated. It's like saying groupBy(no-thanks).agg(sum("i")).

What if we have a syntax like this ds.forAgg.agg( sum(ds('i)), max(ds('i)), etc. )?

This assumes that we no longer allow aggregation on select. So we might get the additional simplification of TypedAggregateAndColumn to just TypedAggregate.

[OlivierBlanvillain]

Fixed in #123/#129.