elasticlinq/elasticlinq

Use LINQ to query Elasticsearch with results as .NET objects

ElasticLINQ

ElasticLINQ is a free C# library for searching Elasticsearch using LINQ syntax in .NET 4.5/PCL, e.g.

var db = new ElasticContext(new ElasticConnection(new Uri("http://myserver:9200")));
var p = db.Query<People>().Where(p => p.Tags.Contains("tech") && p.State == "WA");

For information on getting started, see the Wiki.

Elasticsearch version compatibility

• 0.9 to 1.x works great
• 2.x GroupBy not supported

Beyond 2.x is unsupported.

Builds

Binary releases are available via NuGet or direct downloads

elasticlinq's Issues

Convert ElasticLINQ to portable class library

Enable support for as many platforms as possible. :)

Constant boolean within a subexpression fails with InvalidOperation

Both of these expressions fail with an InvalidOperation error:

var orTrueSub = Robots.Where(r => r.Id == 11 && (true || r.EnergyUse > 2));
var andFalseSub = Robots.Where(r => r.Id == 11 || (false && r.EnergyUse > 2));

This is because the sub-expressions resolve to constant expressions but the removal of constant expressions is done before the sub-expression is resolved within ConstantQueryFilterReducer.cs

Support enum values for terms filters

For example given the following model

public enum Status {
 on = 1,
 off = 2,
}

public class Model {
   public Status status { get;set;}
}

I should be able to have the following code

context.Query<Model>().Where(m => m.status != Status.off)

issue a query similar to this

{
  "filter" : {
    "terms" : {
      "status" : "off"
    }
  }
}

Currently the enum constant is interpreted as the value of the underlying integer and the query is "status": 2

Add support for async

It would be nice to follow in the path of EF and support async ToXxx operations (f.e., ToListAsync). Since ElasticLINQ's network protocol (HTTP) is implicitly async, it would be nice to allow better resource usage on servers by fully embracing that instead of hiding it behind LINQ's synchronous methods.

Add support for administrative APIs

Build w/ Xamarin

@SaintGimp @bradwilson @AdeMiller @aweiker @brannon & all.

I opened up the project with Xamarin (I try to build everything from this, then I know it works across all systems). The issues that popped up are reference issues with

I think there my be some solutions available here http://components.xamarin.com/view/json.net but I'm a bit time pressed to get this tested. Not sure what cycles others might have, but might be worth a go. At some point I'd be happy to roll any changes/fixes back into the library.

I know Xamarin support isn't probably a priority but I have a project I'd love to pull this into and it's 100% Linux & Xamarin stuff (they're doing a bunch of cross dev w/ Android, iOS & Ubuntu Server). ;)

Cheers,
@Adron

Update: Added a possible fix by updating the Nuget packages via Xamarin here: #10

In the testing library it however throws an issue with the accessibility of some of the variables but all the references are fixed.

Searches without an index but with a document type create incorrect URL

Based on reading the search API docs, we should have four URL patterns:

No index, no document type: http://server:9200/_search or http://server:9200/_all/_search
Index, no document type: http://server:9200/index/_search
No index, document type: http://server:9200/_all/docType/search
Index, document type: http://server:9200/index/docType/search

Add ElasticContext.QueryRaw<T>

Allow the user to specify a raw ElasticSearch query, but still leverage the type mapping for the results.

Support for querying several indices/types

Adding support for specifying indices and/or types to query.

Something like :

// query all indices and all types
context.Query().Where(x => x.Prop > 123).Select(x => new { x.Prop1, x.Prop2 })....

// query on specific indices (all types)
context.Indices("index1", "index2", "index3").Query().Where(x => x.Prop > 123).Select(x => new { x.Prop1, x.Prop2 })....

// query on specific types (all indices)
context.Types("type1", "type2", "type3").Query().Where(x => x.Prop > 123).Select(x => new { x.Prop1, x.Prop2 })....

// query on specific indices and types
context.Indices("index1", "index2", "index3").Types("type1", "type2", "type3").Query().Where(x => x.Prop > 123).Select(x => new { x.Prop1, x.Prop2 })....

Or :

var indices = new Indices("index1", "index2", "index3");
var types = new Types("type1", "type2", "type3");

// query all indices and all types
context.Query().Where(x => x.Prop > 123).Select(x => new { x.Prop1, x.Prop2 })....

// query on specific indices (all types)
context.Query(indices).Where(x => x.Prop > 123).Select(x => new { x.Prop1, x.Prop2 })....

// query on specific types (all indices)
context.Query(types).Where(x => x.Prop > 123).Select(x => new { x.Prop1, x.Prop2 })....

// query on specific indices and types
context.Query(indices, types).Where(x => x.Prop > 123).Select(x => new { x.Prop1, x.Prop2 })....

Syntax for using facets

Hi,
I'm in the process of adding facet support in my project but i fail to find an elegant syntax to add facets to a linq query.
Can you please show me how you did it in ElasticLINQ ?
Thanks

Select x.Key inside a GroupBy on a constant should select the constant

.GroupBy(x => 1).Select(x => x.Key) should return 1 and not null.

Support Count and LongCount top-level operations

Support:

• query.Count()
• query.Count(x => x.Boolean)
• query.LongCount()
• query.LongCount(x => x.Boolean)

Note that the version containing the boolean predicate overloads will also need to add a sub-filter to the facet containing the translated predicate.

Where with null test fails to apply selection criteria

Running a query like:

query.Robots.Where(r => r.Name == null);

Generates a MissingCriteria which unlike other criteria types will not satisfy the criteria selection requirements necessary to select only the right documents in Couchbase-import-style mappings (where types are not stored as ES types)

Add support for custom JsonConverter

I have several custom JsonConverters that I would like ElasitcLinq to respect as part of serialization/deserialization.

It would be great if ES would just pick up on the [JsonConverter] attributes already applied to my classes. If that isn't possible, some pluggable structure that allows me to register a JsonConverter would suffice.

Use GZip in HttpClient used to talk to ES server

The HttpClient used to talk to the server should probably be configured to use gzip compression by default. This should improve the network performance when fetching large result sets (at a small cost to CPU).

Translate GroupBy to aggregations

Right now GroupBy operations map to facets as they were the only operation available before 1.0

Since then the focus has moved to Aggregations. Facets are still supported in Elasticsearch 1.5.0 however they have already been removed in Elasticsearch master branch.

Support for .NET 4.0

How about adding support for .NET 4.0 ?
From what i see it should not be too difficult. I can take some time to look into it if you're interested.

Equals true and equals false are not translated

When writing queries like this, the translator throws an exception :

.Query(x => x.String.Contains("abc") == true)
or
.Query(x => x.String.Contains("abc") == false)

Should be easy to fix.

Add integration test suite

Should be able to point it at an ElasticSearch server, and it creates an index, populates it with data, and runs comprehensive search tests, then cleans up the index.

GroupBy followed by a Select Tuple.Create fails

The following constructor pattern on GroupBy fails:

context.Query<Product>().GroupBy(p => 1).Select(g => Tuple.Create(g.Key, g.Count()));

With an InvalidOperationException:
Additional information: variable 'g' of type 'System.Linq.IGrouping2[System.Int32,Northwind.Models.Product]' referenced from scope '', but it is not defined`

Although the following works fine:

db.Query<Products>().GroupBy(p => 1).Select(g => new { g.Key, Count = g.Count() })

GroupBy selecting a DateTime or DateTimeOffset throws

When attempting to perform an aggregation on a DateTime or DateTimeOffset such as:

db.Query<Order>()
  .GroupBy(g => g.company)
  .Select(g => new { g.Key, l = g.Max(t => t.orderedAt) })
  .ToList(); 

An exception is thrown during materialization, specifically "Can not convert Float to DateTimeOffset."

This is because Elasticsearch returns date fields as milliseconds since utc epoch when used in an aggregation.

Support Sum, Min, Max, Average top level operations

Support:

• query.Sum(a => a.Something)
• query.Min(a => a.Something)
• query.Max(a => a.Something)
• query.Average(a => a.Something)

Note that there are several overloads for each supporting different data types.

Support .Contains for strings

(From a comment on my blog post)

ElasticConnection connection = new...
ElasticContext context = new...
var r = from obj in context.Query() where obj.Text.Contains("hello");

I got NotSupportedException:
Unknown Where predicate 'obj.Text.Contains("hello")' in ElasticQueryTranslator.VisitWhere

Add support for writing/delete documents

Query with Contains and AND/OR generates invalid query

Normally Or and And queries put the two operations inside an Or or And criteria. This is not valid where the criteria is the special QueryString operation:

context.Query<Product>().Query(g => g.name.Contains("a") && g.reorderLevel > 1);

{
  "query": {
    "and": [
      {
        "query_string": {
          "query": "*a*",
          "fields": [
            "name"
          ]
        }
      },
      {
        "range": {
          "reorderLevel": {
            "gt": 1
          }
        }
      }
    ]
  }
}

We should detect this and throw a better error message that indicates for an AND the condition should be moved to the filter via Where. OR is not possible.

null values in .Contains treated incorrectly

When you write a simple null check:

query.Where(x => x.id == null)

That gets correctly translated into:

{"missing":{"field":"id"}}

However, if the null value is a member of a collection, and the .Contains feature is used:

query.Where(x => ListOfIDs.Contains(x.id))

Then the null value is placed into the terms query:

{"terms":{"id":["1","2","3",null]}}

This is not legal with Elasticsearch (and results in a 400 when the request is issued).

Add support for profiling

Profiling support should not have a hard dependency but it would be cool if it could auto-detect and wire-up to popular open source libraries such as MiniProfiler at run-time using Reflection.

Some interesting things to profile and potential data points besides memory and time:

1. Expression tree parsing (number of source expressions)
2. Request creation (number of nodes, document size)
3. Request send time (server name)
4. Response time (server name, document size)
5. Materialization time (materializer, number of nodes, projector, entity class name)

Supported .Net and Elasticsearch versions?

This looks like an interesting project. I just tried using this with .Net 4.5 and Elasticsearch 1.5.2 and went no where. A simple select operation returned no data. The code doesn't throw any exception if the uri is invalid etc.

Occasionally, I see this error on Elasticsearch

[2015-04-28 16:06:48,404][DEBUG][http.netty ] [Test Master 1] Caught exception while handling client http traffic, closing connection [id: 0x656aeb85, /0:0:0:0:0:0:0:1:50924 => /0:0:0:0:0:0:0:1:9200] java.io.IOException: An existing connection was forcibly closed by the remote host
at sun.nio.ch.SocketDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)

Should I assume that this combination of .Net and ES is not supported?

Type selection criteria is not applied to groupby/aggregate queries

A custom IElasticMapping provider that applies type selection criteria is not applied to groupby/aggregate queries. This would be a problem if you were not using the Elasticsearch type system OR fully-qualified field names.

Constant on the left side of < > <= or => operators results in reversed Range query

This is because the ConstantMember class flips the args but does not flip the operator.

GroupBy followed by a Select new initializer fails

The following constructor pattern on GroupBy fails:

context.Query<Product>().GroupBy(p => 1)
        .Select(g => new SomeClass { Key = g.Key, Count = g.Count()});

With the following exception:

Additional information: When called from 'VisitMemberInit', rewriting a node of type 'System.Linq.Expressions.NewExpression' must return a non-null value of the same type. Alternatively, override 'VisitMemberInit' and change it to not visit children of this type.

ElasticMethods.ContainsAny does not format array elements

I have an array ["FOO"] which I would have expected to get formatted as ["foo"] but it does not do the lowercase on the element.

Support OrderBy with child properties

Currently the following throws NotSupportedException:

context.Query<Order>().OrderByDescending(o => o.lines.Max(l => l.discount)).First();

It should instead generate:

{
  "sort" : [
         {
          "lines.discount" : {
             "mode" :  "max",
             "order" : "desc"
          }
       }
   ]
}

This applies to Min, Max (many types) and Sum, Avg (numeric) as described at http://www.elastic.co/guide/en/elasticsearch/reference/1.5/search-request-sort.html

Note that Count is unfortunately not supported.

Range filter does not handle two checks to the same field for the same operation

I found a bug with the range filter.

If i have more than one comparison for a given property, the generated filter groups the comparisons in the same range filter.
But if i put twice the same comparison operator, the json fails to serialize (duplicate property name).

eg : .Where(x => x.Int > 4 && x.Int > 3)

By the way, i tried to implement range grouping logic in my project but it's not an easy task, as nodes in the expression tree can be anywhere. Can you hint me on how you did it in ElasticLINQ ?

Query with multiple query_string generates invalid query

Similar to bug #42 but where AND or OR just references other contains statements we should generate a combined query_string using the enhanced rules/syntax at http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html

context.Query<Product>().Query(g => (g.name.Contains("a") && 
g.name.StartsWith("b")) || g.something == "theWord");

Would then become:

{
    "query_string": {
      "query": "(name:*a* AND name:b*) OR (something:theWord)"
    }
}

Extend TestableElasticContext to support ElasticFields etc.

Right now TestableElasticContext by way of TestableElasticQueryProvider executes the queries by compiling the expression and running against an in-memory list using LINQ to Objects.

This fails if the functionality is Elasticsearch specific and is exposed by one of the various extensions provided by ElasticFields or ElasticMethods classes.

At a minimum we should support something like Score and Id.

Allow custom field formatting via IElasticMapping

Right now values sent to ElasticSearch are ToLowered in order to work with the default analyzer however for not_indexed fields or for different languages different behavior might be required on a case-by-case basis.

Recommend:

1. Creating a NotAnalyzed attribute
2. Adding a GetFormattedFieldValue(PropertyInfo) method to IElasticMapping
3. Remove the RequestFormatter.Format method and replace it with a call to GetFormattedFieldValue earlier in the transformation

The method would just ToString the value if the PropertyInfo has NotAnalyzed or ToLower it if not.

Add administrative APIs to create and update indexes

Document why .OrderBy() with strings does not work

This is a placeholder issue to ensure that we document why .OrderBy (meaning, Elasticsearch's sort directive) typically does not work with string values.

Add ElasticMethods.ContainsAny

Since ElasticSearch supports arrays as first class citizens, there is no comparable LINQ query to verify that an array contains one or more values. The closest analogue is:

.Where(x => x.MyField.Any(y => myValues.Contains(x))

Since .Any is open-ended, we would like to implement a more restricted verions, like:

.Where(x => ElasticMethods.ContainsAny(x.MyField, myValues))

Any chance you can support Count?

Would be great if support for Count was added. Using a package that is expecting Count and ElasticLinq is throwing a NotSupportedException.

Add support for custom JsonSerializerSettings

Hi,

First off, GREAT WORK!!!

Being able to set the JsonSerializerSettings come time to deserialize a JObject in a Hit object would be great.

I found the place the settings need to go in the ElasticQueryTranslator class. The JObject.ToObject method can receive a custom JSonSerializer. Here’s the quick fix we put in place:

private Func<Hit, Object> DefaultItemProjector
{
  get
  {
    return hit => hit._source
      .SelectToken(Mapping.GetDocumentMappingPrefix(sourceType) ?? "")
      .ToObject(sourceType, Newtonsoft.Json.JsonSerializer.Create(new Dell.ComplexConfig.Formatting.V1.CcsSerializerSettings()));
  }
}

I’m guessing that passing the JsonSerializerSettings to the ElasticContext constructor and letting them trickle down would be the way to go.

Any way for us to contribute?

Thanks,

Rene

ElasticMapping.GetFieldName should respect [JsonProperty]

If a user specifies [JsonProperty(PropertyName=)] on a property or field in a class, ElasticMapping.GetFieldName should respect that attribute and use the given name (without transformation).

Support StandardDeviation operator

Add a new ElasticMethod method for supporting the Standard Deviation operation in ElasticSearch facets as LINQ does not provide a method to do this.

query.GroupBy(g => 1).Select(y => ElasticMethods.StandardDeviation(y.Something))

Not recommended to add a top-level operation for this as there is no corresponding IEnumerable/IQueryable overload and the GroupBy pattern is more efficient when selecting multiple entries.

Support LINQpad

Add support for using within LINQpad.

Lowercasing of field and type names

Hi,

I have started getting acquainted with this library as it seems ideal for what I'm trying to do. Upon trying to use it though, it seems that the object type for the query is being inferred from the type i'm trying to deserialize to. i.e. I have a class called Action, and ElasticLinq therefore thinks I want to return types of "Actions" and my URl ends up being http://stuff/actions/actions/_search. (Which is wrong. I don't even want to filter by a specific type necessarily).

In addition, all my property names are being lower-cased and therefore not matching anything. So Action has a property called "Name", but when the query is sent, the json object's property is "name" (with a small "n"). This means that it doesn't match anything, ever. Is there a way around this behavior?

Add support for complex types

Simple (flat) types work, but types with embedded complex types do not. For example, this will throw an exception today:

public class ComplexType
{
    public InnerType Inner { get; set; }
}

public class InnerType
{
    public string Value { get; set; }
}

var results = context.Query<ComplexType>()
                     .Where(x => x.Inner.Value == "foo")
                     .ToList();

ElasticFields.Score Query

Hi there,

I'm trying to create a full text query on my data and because it uses query_string, it is returning all data.

I only want the relevant results, so I want to use the ElasticFields.Score field in my queries to get all results with a score greater than 1.0, yet I can't seem to find any info on using it barring a near 2 year old video where the field gets mentioned, it doesn't work for the speaker, then he moves onto something else.

Similarly to myself, if I try to use the field, it makes my query return no results, it should return at least 1 result as the query before it contains 3 results, 1 of which has a score greater than 1.0. I'm using a standard Linq expression predicate to pass through to a function so my code looks like this:

public IQueryable<MyClassName> SearchFor(System.Linq.Expressions.Expression<Func<MyClassName, bool>> predicate)
{
    return context.Query<MyClassName>().Query(predicate);
}

This returns the results as planned. After adding in the ElasticFields.Score field:

public IQueryable<MyClassName> SearchFor(System.Linq.Expressions.Expression<Func<MyClassName, bool>> predicate)
{
    return context.Query<MyClassName>().Query(predicate).Where(p => ElasticFields.Score > 1);
}

Which returns no results. Do you know if there is an issue with the ElasticField.Score field?

The query that comes out of the above is the following:

{
  "query":{
    "query_string":{
      "query": "*{Query String}*",
      "fields": ["{Field to query on}"]
    }
  },
  "filter":{
    "range":{
      "_score":{
        "gt":1.0
      }
    }
  },
  "timeout":"10s"
}

However, it seems that elasticsearch is expecting:

{
  "min_score": "1.0",
  "query":{
    "query_string":{
      "query": "*{Query String}*",
      "fields": ["{Field to query on}"]
    }
  },
  "timeout":"10s"
}

Thanks,

Liam.

Query cannot contain AND or OR clauses

The following request generates an invalid query :

context.Query<tata>().Query(x => x.Int == 1 && x.String == "tutu")

The query generated is:

{"query":{"and":[{"term":{"int":1}},{"term":{"string":"tutu"}}]},"timeout":"10s"}

The AND and OR operators are not supported by ES at the query level (only at the filter level).

One solution is to replace AND and OR (and NOT, etc...) by bool queries.
Another one is to create a filtered query and put the criterias in the filter.

Query with locally evaluated constant fails to translate

If writing Query statement that includes either a direct boolean constant, e.g.:

var query = db.Products.Query(p => true);

or more likely something that has evaluated to true using the partial evaluator, e.g.:

Product current = null;
var query = db.Products.Query(p => current == null || p.ParentId == current.Id);

Then a translation error will occur as the binary expression can not be translated today.