fsprojects/fsharp.linq.composablequery

Compositional Query Framework for F# Queries, based on "A Practical Theory of Language-Integrated Query"

fsharp.linq.composablequery's Issues

Incomplete normalisation due to extra terms

Terms of the form RunQueryAsValue(Quote(e)) are not properly handled/ignored in our internal format, causing the library to fail in fully normalising queries.

We should either remove or hide such terms while translating so as to not prevent the the normalisation procedure from working across the boundaries of such calls. The former may not be possible (the attempted fix in the manner of #1 did not work) and the second may be infeasible due to the large amount of cases to handle.

As this issue does not seem to impair significantly the performance of queries with nested data (or any queries whatsoever) I am marking it as an enhancement.

Expr terms missing.

There are a number of terms which are valid input to the default QueryBuilder but not to the FSharpComposableQuery builder. This happens since not all valid literals of type FSharpExpr have their equivalent as Exp.

Such terms include:

• Sequential
• PropertySet

These two can be reproduced by changing the final yield d.dpt statement in the expertise query in the Nested.fs file to yield (new Department(Dpt=d.dpt)) after implementing the changes in commit b7c5bce

Build script not generating docs

The build script uses MSBuild to build the FSharpComposableQuery.fsproj project file. Although the process builds the library, it does not generate the XML documentation file FSharpComposableQuery.XML as required by the docs generation script.

Inspecting the project file properties reveals it should create such an Xml documentation file when built, and indeed doing this from Visual Studio actually creates the file.

It is interesting why the build script does not instruct the compiler to generate docs. It is its job to parse the project file and pass all relevant options to MSBuild.

One can confirm there is a discrepancy in the parameters passed to MSBuild by examining them in both cases: the IDE passes the additional switch --doc:..\..\bin\FSharpComposableQuery.XML as compared to FAKE v3.2.3 (and v2.1.7, too).

Consider showing how this can be used with SQLProvider

See issue here: fsprojects/SQLProvider#86

"The FSharpComposableQuery nuget package is an interesting approach to normalizing F# queries to allow them to be more compositional and safe. It would be interesting and useful to check if this approach to queries can be used in conjunction with SQLProvider and assess whether it brings benefits...."

@pezipink and @colinbull are two contributors to SQLProvider.

Destructuring not supported for quoted funs?

I am using this through SQLProvider. Let me know if this issue should be posted in that repo instead.

Consider the following:

type OrgId = OrgId of int

let isProjectOwnedBy =
  <@ fun (OrgId owner) (p: MyDbType.dataContext.``dbo.ProjectEntity``) ->
    p.OrgId = Some owner
  @>

let projectsOwnedBy (orgId: OrgId) =
  let ctx = MyDbType.GetDataContext ()
  query {
    for p in ctx.Dbo.Project do
    where ((%isProjectOwnedBy) orgId)
    select p
  }

This fails at runtime with the following error (I've modified it to fit the simplified example above, so there may be errors):

System.Exception: 'Unsupported expression. Ensure all server-side objects won't have any .NET-operators/methods that can't be converted to SQL. The In and Not In operators only support the inline array syntax. (owner => (tupledArg.Item5.GetColumnOption("OrgId") == Some(owner)).Invoke(value(System.Runtime.CompilerServices.StrongBox`1[OrgId]).Value.Item))'

However, it works if I don't destructure in the quoted function:

type OrgId = OrgId of int with
  member this.Wrapped = let (OrgId x) = this in x

let isProjectOwnedBy =
  <@ fun (owner: OrgId) (p: MyDbType.dataContext.``dbo.ProjectEntity``) ->
    p.OrgId = Some owner.Wrapped
  @>

Is this intended? Am I doing things wrong?

Queries over nested structures

Queries which use intermediate nested structures are not properly normalised and may generate multiple output queries.

To reproduce, run the benchmarking code and observe the execution time of tests ex8 and ex9 in Nested.fs: while we expect ex9 to take more time to execute under the F# 3.0 query builder (as compared to ex8), both queries should take roughly similar time under the TLinq builder which is not the case.

The problem seems to be in the handling of QueryBuilder.Run invocations, as the normalisation procedure does not explicitly recognise them.

We can either remove such calls or add a new literal in the internal representation representing calls to those, and rework the normalisation rules around it.

The Travis build is failing (for Linux/OSX)

See https://travis-ci.org/fsprojects/FSharp.Linq.ComposableQuery/builds

I'm not sure what the problem is, but it would be great to make sure this builds and passes tests on OSX/Linux

yield!

The following code (run against the db.People database in the tests) fails at run time

let test3' =   query { for u in db.People do
                       if 0 <= u.Age then
                        yield! (query {yield u}) }

with the following error:

System.ArgumentException: Type mismatch when building 'cond': types of true and false branches differ. Expected 'Microsoft.FSharp.Linq.QuerySource2[People,System.Object]', but received type 'Microsoft.FSharp.Linq.QuerySource2[People,System.Linq.IQueryable]'.
Parameter name: receivedType
at Microsoft.FSharp.Quotations.PatternsModule.checkTypesSR[a](Type expectedType, Type receivedType, a name, String threeHoleSR)
at Microsoft.FSharp.Quotations.PatternsModule.mkIfThenElse(FSharpExpr e, FSharpExpr t, FSharpExpr f)
at Microsoft.FSharp.Quotations.FSharpExpr.IfThenElse(FSharpExpr guard, FSharpExpr thenExpr, FSharpExpr elseExpr)
at FSharpComposableQuery.QueryImpl.toExp@371(QueryBuilder this, Exp exp)
at FSharpComposableQuery.QueryImpl.QueryBuilder.toExpr(Exp exp)
at FSharpComposableQuery.QueryImpl.QueryBuilder.Norm[T](FSharpExpr`1 expr)
at <StartupCode$FSI_0047>.$FSI_0047.main@()
Stopped due to error

It seems that we are not treating yield! / query.yieldFrom as a recognized operator, and the rewritingperformed by Query.removeRunAs seems to break something (or perhaps, not do enough due diligence to fix up the type information).

Use a variable for the database name in the creation SQL scripts.

Instead of repeating the name of the database throughout the creation script, use SQLCMD mode to name the database. I suggest using a common prefix for FSharp.Linq.ComposableQuery to avoid name collisions with existing databases and instead of a missing db.config, set a config file that defaults to use the default instance of SQL on the local machine and uses databases with prefixed names. Prefer the use code page 1252 when saving these scripts so that diffs can be seen using source control tooling. Only people.sql has been saved in unicode. In github for windows, this shows a diff as 'binary files differ'.

:setvar DatabaseName "FCQ-People"
:setvar DatabaseName "FCQ-Org"
:setvar DatabaseName "FCQ-Xml"
:setvar DatabaseName "FCQ-Simple"

Doesn't work with SQLProvider

Description

Consider this simple database structure (I'm using MS SQL Express 2014):

create table dbo.Users(Id Int primary key, Name NVarChar(max), Role NVarChar(max))
insert into dbo.Users(Id, Name, Role) values (1, N'Peter', N'Admin')
insert into dbo.Users(Id, Name, Role) values (2, N'Peter', N'User')
insert into dbo.Users(Id, Name, Role) values (3, N'Igor', N'Admin')

I've written two very similar programs:

1. Program A using FSharp.Data.TypeProviders: https://gist.github.com/ForNeVeR/c09e21dabc840c9fd4af7600b8f363ae#file-program-fs
2. Program B using FSharp.Data.SqlProvider: https://gist.github.com/ForNeVeR/6658c6faf3719f482694c09cc922299a

Repro steps

Create the database, compile and execute Program A and Program B.

Expected behavior

Both programs should print the same output: https://gist.github.com/ForNeVeR/c09e21dabc840c9fd4af7600b8f363ae#file-output-txt

Actual behavior

Program A works, but Program B throws the following exception:

System.ArgumentException: Type mismatch when building 'args': invalid parameter for a method or indexer property. Expected 'System.Linq.IQueryable`1[FSharp.Data.Sql.Common.SqlEntity]', but received type 'System.Object'.
Parameter name: receivedType
   at Microsoft.FSharp.Quotations.PatternsModule.checkTypesWeakSR[a](Type expectedType, Type receivedType, a name, String threeHoleSR)
   at [email protected](ParameterInfo p, FSharpExpr a)
   at Microsoft.FSharp.Collections.ListModule.loop@183-27[T1,T2](FSharpFunc`3 f, FSharpList`1 list1, FSharpList`1 list2)
   at Microsoft.FSharp.Collections.ListModule.Iterate2[T1,T2](FSharpFunc`2 action, FSharpList`1 list1, FSharpList`1 list2)
   at Microsoft.FSharp.Quotations.PatternsModule.checkArgs(ParameterInfo[] paramInfos, FSharpList`1 args)
   at Microsoft.FSharp.Quotations.PatternsModule.mkInstanceMethodCall(FSharpExpr obj, MethodInfo minfo, FSharpList`1 args)
   at Microsoft.FSharp.Quotations.FSharpExpr.Call(FSharpExpr obj, MethodInfo methodInfo, FSharpList`1 arguments)
   at FSharpComposableQuery.QueryImpl.toExp@386(QueryBuilder this, Exp exp)
   at FSharpComposableQuery.QueryImpl.QueryBuilder.toExpr(Exp exp)
   at FSharpComposableQuery.QueryImpl.QueryBuilder.Norm[T](FSharpExpr`1 expr)
   at FSharpComposableQuery.QueryImpl.QueryBuilder.Run[T](FSharpExpr`1 q)
   at Program.main(String[] argv) в T:\Temp\ConsoleApplication6\ConsoleApplication6\Program.fs:строка 30

Known workarounds

Don't use SQLProvider 😿

Related information

• Operating system: Windows 10
• Branch: see my packages.config below
• .NET Runtime, CoreCLR or Mono Version: .NET 4.6.2, F# 4.4.1.0

packages.config:

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="FSharp.Core" version="4.1.17" targetFramework="net462" />
  <package id="FSharp.Data.TypeProviders" version="5.0.0.2" targetFramework="net462" />
  <package id="FSharpComposableQuery" version="1.0.4-beta" targetFramework="net462" />
  <package id="SQLProvider" version="1.1.2" targetFramework="net462" />
  <package id="System.ValueTuple" version="4.3.0" targetFramework="net462" />
</packages>

queries over data provided by ODataService type providers

Queries

For example, the following simple query over the Northwind database provided by ODataService fails. (Additional examples are in the new file NorthwindTests.)

open Microsoft.FSharp.Data.TypeProviders;

type Northwind = ODataService<"http://services.odata.org/Northwind/Northwind.svc">
let db = Northwind.GetDataContext()

// Some tests to compare 

let dbQuery =  FSharpComposableQuery.TopLevelValues.query

dbQuery { for x in db.Customers do yield x }

The error message is:

System.Exception: Unexpected table reference PropertyGet (Some (PropertyGet (None, db, [])), Customers, []) System.Data.Services.Client.DataServiceQuery`1[UnitTestProject1+Northwind+ServiceTypes+Customer]
Result StackTrace:  
at [email protected](String message)
   at Microsoft.FSharp.Core.PrintfImpl.go@523-3[b,c,d](String fmt, Int32 len, FSharpFunc`2 outputChar, FSharpFunc`2 outa, b os, FSharpFunc`2 finalize, FSharpList`1 args, Int32 i)
   at Microsoft.FSharp.Core.PrintfImpl.run@521[b,c,d](FSharpFunc`2 initialize, String fmt, Int32 len, FSharpList`1 args)
   at Microsoft.FSharp.Core.PrintfImpl.capture@540[b,c,d](FSharpFunc`2 initialize, String fmt, Int32 len, FSharpList`1 args, Type ty, Int32 i)
   at [email protected](T1 inp)
   at FSharpComposableQuery.QueryImpl.handleSpecificCall@488(QueryBuilder this, FSharpOption`1 obj, MethodInfo func, FSharpList`1 args, Type expr_ty)
   at [email protected](Type ty, Type _arg4)
   at FSharpComposableQuery.QueryImpl.QueryBuilder.Norm[T](FSharpExpr`1 expr)
   at FSharpComposableQuery.QueryImpl.QueryBuilder.Run[T](FSharpExpr`1 q)

Comparison tests failing due to IEnumerable behaviour

The default IEnumerable (also: IQueryable, IGrouping) comparer checks for referential rather than structural equality. This causes queries that output records with such elements as values to fail.

To reproduce the issue observe the comparison results of Q17 and Q42. Both queries' return values are tuples containing an IGrouping object which causes them to fail. On the other hand one can verify the result sets are the same by using a debugger.

Fixing this issue would require a custom comparer to check for element-wise equality between objects extending the IEnumerable type. This comparer must also recursively call itself in the case of nested types, such as tuples, lists, arrays or any such other custom type -- which seems infeasible.

Docs page uses mixed content and looks bad

The docs page:

https://fsprojects.github.io/FSharp.Linq.ComposableQuery/

Makes requests to http resources that are blocked in latest FF and Chrome. They should be upgraded to https.

An example where query works but dbquery doesn't using FCQ-Brands sample.

I was using query and dbquery as the standard query expression and the replacement from ComposableQuery respectively and comparing the generated SQL. I found an example where query works but dbquery doesn't. The exception is as follows.

System.InvalidOperationException: System.Linq.IQueryable`1[System.Object] is not a GenericTypeDefinition. MakeGenericType may only be called on a type for which Type.IsGenericTypeDefinition is true.

I've added the setup as a db project. The database is named FCQ-Brands. To reproduce follow the readme at FSharp.Linq.ComposableQuery\tests\FCQ-Brands\readme.md and then open the script at FSharp.Linq.ComposableQuery\tests\FCQ-Brands\FCQ-Brands.fsx. There are comments in there with the results from query and the failure from dbquery.

The query is ...

let ofBrandUserAuth = <@ fun (b : string) ->
    if b = null then query {for a in db.UserAuth do select a} else
        query {
            for a in db.UserAuth do
            join un in db.UserName on
                (a.Id = un.AuthId)
            join bu in db.Brand_User on
                (un.Id = bu.User)
            where (bu.BrandBrand.Name = b)
            select a
        } @>

<@ fun b -> query {
    for a in (%ofBrandUserAuth) b do
    select a
} @>
|> fun q -> query {yield! (%q) brand}