werbitzky/elastix

A simple Elasticsearch REST client written in Elixir.

Elastix

A DSL-free Elasticsearch client for Elixir.

Documentation

• Documentation on hexdocs.pm
• Latest Elasticsearch documentation

Even though the documentation is pretty scarce right now, we're working on improving it. If you want to help with that you're definitely welcome. 🤗

This README contains most of the information you should need to get started, if you can't find what you're looking for, either look at the tests or file an issue!

Installation

Add :elastix to your list of dependencies in mix.exs:

def deps do
  [
    {:elastix, ">= 0.0.0"}
  ]
end

Then run mix deps.get to fetch the new dependency.

Examples

Creating an Elasticsearch index

Elastix.Index.create("http://localhost:9200", "twitter", %{})

Map, Index, Search and Delete

elastic_url = "http://localhost:9200"

data = %{
    user: "kimchy",
    post_date: "2009-11-15T14:12:12",
    message: "trying out Elastix"
}

mapping = %{
  properties: %{
    user: %{type: "text"},
    post_date: %{type: "date"},
    message: %{type: "text"}
  }
}

Elastix.Mapping.put(elastic_url, "twitter", "tweet", mapping)
Elastix.Document.index(elastic_url, "twitter", "tweet", "42", data)
Elastix.Search.search(elastic_url, "twitter", ["tweet"], %{})
Elastix.Document.delete(elastic_url, "twitter", "tweet", "42")

Bulk requests

Bulk requests take as parameter a list of the lines you want to send to the _bulk endpoint.

You can also specify the following options:

• index the index of the request
• type the document type of the request. (you can't specify type without specifying index)
• httpoison_options configuration directly passed to httpoison methods. Same options that can be passed on config file

lines = [
  %{index: %{_id: "1"}},
  %{field: "value1"},
  %{index: %{_id: "2"}},
  %{field: "value2"}
]

Elastix.Bulk.post(elastic_url, lines, index: "my_index", type: "my_type", httpoison_options: [timeout: 180_000])

# You can also send raw data:
data = Enum.map(lines, fn line -> Poison.encode!(line) <> "\n" end)
Elastix.Bulk.post_raw(elastic_url, data, index: "my_index", type: "my_type")

Configuration

Shield

config :elastix,
  shield: true,
  username: "username",
  password: "password",

Poison (or any other JSON library) and HTTPoison

config :elastix,
  json_options: [keys: :atoms!],
  httpoison_options: [hackney: [pool: :elastix_pool]]

Note that you can configure Elastix to use any JSON library, see the "Custom JSON codec" page for more info.

Custom headers

config :elastix,
  custom_headers: {MyModule, :add_aws_signature, ["us-east"]}

custom_headers must be a tuple of the type {Module, :function, [args]}, where :function is a function that should accept the request (a map of this type: %{method: String.t, headers: [], url: String.t, body: String.t}) as its first parameter and return a list of the headers you want to send:

defmodule MyModule do
  def add_aws_signature(request, region) do
    [{"Authorization", generate_aws_signature(request, region)} | request.headers]
  end

  defp generate_aws_signature(request, region) do
    # See: https://github.com/bryanjos/aws_auth or similar
  end
end

Running tests

You need Elasticsearch running locally on port 9200. A quick way of doing so is via Docker:

$ docker run -p 9200:9200 -it --rm elasticsearch:5.1.2

Then clone the repo and fetch its dependencies:

$ git clone [email protected]:werbitzky/elastix.git
$ cd elastix
$ mix deps.get
$ mix test

Copyright and License

Copyright © 2017 El Werbitzky [email protected]

This work is free. You can redistribute it and/or modify it under the terms of the Do What The Fuck You Want To Public License, Version 2, as published by Sam Hocevar. See http://www.wtfpl.net/ for more details.

elastix's Issues

Release 0.5.0 dont have Count functionality in Elastix.search module

I see count function added into Elastix.search module but release 0.5.0 didnt have that.
When are you planning to release next version?

Set request headers/support non-shield authorization header?

We're using elastic search on AWS which requires HMAC key signing as described here http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html

This elixir library implements it (https://github.com/bryanjos/aws_auth) nicely, but not sure best way to extend HTTP module to allow for auth other than shield. Happy to contribute if I can (and may have to fork library just to add it, if I can't find a clean way to work around), but I'm just getting my feet wet, and not sure if you have a strong preference for how it be done. Thanks!

Custom Headers/AWS auth.

The README describes supporting custom headers, however the same commit that adds that documentation seems to have actually removed support for custom headers. (03daf52)

I would like to be able to use this library with elasticsearch on AWS and I am happy to contribute towards a solution.

Some ideas:

• Put back the custom header support, but make the API explicit (does it expect a module? function? what? - basically no misleading .call()). Maybe just a {Module, :fun, []} tuple that gets arguments prepended an applied.
• Add an Elastix.HTTP.Authentication behaviour, which both shield and aws could implement. Could be configurable as config :elastix, auth: Elastix.HTTP.Authentication.Shield.

In the latter case AWS support could be added to this repo, added as a second hex package, or even just provide an example implementation in docs.

Update the Hex repo with latest changes

Currently the version of elastix on Hex is 0.4.0. That version seems to be missing some of the recent updates (most notably support for custom headers). Is it possible to update the hex repo to the new version that includes recent changes?

Also, I noticed all documentation seems to be missing from the Hex docs page. Is adding documentation open and feasible for new contributors?

Can not perform delete without auth options

delete from Document was not working for me without implementing a variant that includes auth options.
I would get

{:error,
 %HTTPoison.Error{
   reason: {:tls_alert,
    {:unknown_ca,
     ~c"TLS client: In state wait_cert at ssl_handshake.erl:2138 generated CLIENT ALERT: Fatal - Unknown CA\n"}},
   id: nil
 }}

After trying my variant it worked. Am I missing something or is this missing? I could open a PR for that change if somebody gives me green light. 🟢

Wrong @spec for Index.exists?

The @spec is wrong for Index.exists?.
It's specificed as.

@spec exists?(elastic_url :: String.t(), name :: String.t()) :: HTTP.resp()

But it does not return a HTTP.resp(), it returns {:ok, boolean()} | {:error, HTTPoison.Error.t()} (the :ok case is just a boolean, not the full Elasticsearch response)

Is it possible to override default HTTP lib used in elastix?

Description
Currently, we're using HTTPoison by default. This might introduce issues like we won't be able to trace this request.
If possible, can we write behavior for HTTP lib and accept that from config?
or is it possible to trace in the current situation and I am not aware of?

API to update Settings

Hey

Is there a API to update the index settings or to setup it together the mapping using Elastix?

Comparison with elasticsearch-elixir?

Thanks for this library! I'm evaluating elastix and elasticsearch-elixir for a production app and was wondering if you have any thoughts, however brief, on how these libraries might compare? Might even be worth adding to README at some point.

Support for Document Partial Updates

I'd like to use Elastic's Update API to make partial or scripted update to an existing document. Any plans on adding this? thanks in advance!

Support for bulk indexing

Would it be possible to add support for bulk indexing documents, as described in the Elasticsearch Bulk API docs? Thanks in advance!

broken with latest version of HTTPoison

In the latest version (HTTPoison 1.4.0), they have introduced a Request struct
edgurgel/httpoison#311
that makes the call here https://github.com/werbitzky/elastix/blob/master/lib/elastix/http.ex#L35 broken

Timeout on HTTP Request

Hi,

On a very long search request, I quickly get : %HTTPoison.Error{id: nil, reason: :timeout}

How to setup the timeout ?

Regards and thank you for your work !

Update poison version at the hex.pm

Hey guys, my pull request to update the poison version has been approved and merged, however the dependency version was not updated in package manager.

Can anyone update please?

Document.index_new is producing the wrong url

Calling Document.index_new/4 or Document.index_new/5 produces the wrong url because it appends a double backslash (e.g. "/movies/movie//"). This can cause problems with AWS authentication since it affects the request signature.

Seems like a minor fix so I would be glad to make a PR if needed.

Elasticsearch host with path (http://myhost/namespacedir) is stripped of path (/namespacedir)

Given an elasticsearch target, namespaced into a sub-path by nginx,
when I pass the full host+path into Elastix.Search.search (and other functions also)
then the path component of the elasticsearch url is stripped away.

I believe this code is problematic, in search.ex

def make_path(index, types, query_params, api_type \\ "_search") do
    path_root = "/#{index}"

When the index+type+queryparam path is passed to URI.merge, in prepare_url,
the preceding forwardslash in the index path replaces any existing path in the host url,
since URI.merge follows Relative Resolution of RFC 3986

Elasticsearch 7 Mapping

How is the code now to assign as mapping settings? I'm trying a few ways with a new application, but I only have this return error:

Types cannot be provided in placement mapping requests unless the include_type_name parameter is set to true. "

Throwing away errors

It would be nice to be able to pattern match on errors/ok, but it is not possible due to the error/ok atom being thrown away.

Example:
https://github.com/werbitzky/elastix/blob/master/lib/elastix/document.ex#L76

Wouldn't it be nicer to use the {:ok, res} and {:error, reason}. Or is there any other reason why this is being stripped away?

Search multiple indices

I would like to be able to search multiple indices. I suggest the search function is extended so that the index parameter can be an array.