madewithlove/elasticsearcher

Agnostic lightweight package on top of the Elasticsearch PHP client. Its main goal is to allow for easier structuring of queries and indices in your application.

elasticsearcher's Issues

Enable CircleCI for forks and their PRs

#54

Currently it is not enabled.

https://circleci.com/gh/madewithlove/elasticsearcher/edit#advanced-settings

Take null values in to account when sorting

Setting a sortBy with a null value should not result in null values in the sort body, but it should ignore it.

$this->sortBy('date', null);

// should not result in:
 0 => 
        array (size=1)
          'date' => null

// should result in
 0 => 
        array (size=1)
          'date' => 'asc' (default from that field)

Add DocumentsManager::upsert

Would be nice, as a shortcut to this:

if ($documentsManager->exists('index', 'type', $document->id)) {
    $documentsManager->update('index', 'type', $document->id, $document);
} else {
    $documentsManager->index('index', 'type', $document);
}

Use docker instead of vagrant for local dev/testing

So we can use the docker instances provided by Elastic

Error with indices manager

Hello,

I'm getting the following error with executing a search:

Index [twitter] could not be found in the register of the indices manager.' in /home/vagrant/liftmetrixapp/vendor/madewithlove/elasticsearcher/src/Managers/IndicesManager.php:87

This index does exist in Elasticsearch though.

My search class is following the docs:

protected function setup()
{
    $this->searchIn('twitter');

    $body = [
        'query' => [
            'filtered' => [
                'filter' => [
                    'term' => ['user.screen_name' => 'NicolasWidart']
                ]
            ]
        ]
    ];
    $this->setBody($body);
}

And usage:

$query = new TotalNumberOfTweets($this->elasticSearcher);
$result = $query->run();
dd($result->getResults());

Did I miss something?

Thanks

Incorrect annotation?

Just a little thing - as far as I can tell, the return of the AbstractIndex::setup() method is not used anywhere, nor does the example index class in the docs return anything. PHPStan is telling me off :)

Add exception handler that outputs the raw source query

So you can easily debug the query via Curl or other tools.

Also add a bugsnag handler which pushes this info to the service.

Allow to set querystring parameters on the query

Like: doCountOnly so it only executes aggregations and such.

Add Pagination and Sorting trait

Add some traits to simplify the setting of pagination and sorting on a query.

Example:

class Query {
    use PaginationTrait;

    function setup() {
        $this->paginate(0, 30);
    }
}

How to do AND and OR queries ? (needs documentation)

Hi, this repo looks really good, but it's hard to find out how things work: I'm trying to do AND and OR queries, but cannot figure out how to do it: Can you please gimme a hint ? I'll then kindly add this to the README. Thanks :)

    // Long notation
    $body = [
      'query' => [
        'bool' => [
          'filter' => [
            'term' => ['status' => 'active']
          ]
        ]
      ]
    ];
    $this->setBody($body);

Intermediate support for deprecation of types in ES 6+

Types have been removed as of ES 6+, and the suggestion from Elastic is to re-index and store _type in a type field, as per https://www.elastic.co/blog/index-type-parent-child-join-now-future-in-elasticsearch

Is it possible there could be an intermediate version of Elastic Searcher which kept the same API for defining types, but modified the query to add a filter for that type field (configurable would probably be nice)?

Re-organize filters to allow for other types of snippets

There are a lot of other chunks of a query that can be abstracted:

• Filters (term, terms, ...)
• Queries (match, match_all, ...)
• Function scores
• Analyzer (for the index)

Move parsing of fragments out of the query building

We should able to use fragments in different contexts (like #3), at the moment it is restricted to Queries. This should also allow for doing more complex parsing (like #13).

I would move to a FragmentParser that accepts an array and turns it into a "parsed" array. This way it can easily be re-used in other contexts.

I do see other parsers in the future (like ExceptionParser for extracting error info, or AggregationParser for simplifying aggregation results, ...). So adding a namespace Parsers sounds logical. We can move the ResultParsers under this.

Add AbstractDocument

Instead of letting the documents manager return raw arrays, let it return an AbstractDocument.

• implements array
• can be interacted with via dotted notation
• has useful getters for index, type and version number

Add helper to check if cluster is ready

Had an issue while performing a reindexing operation. The indexing was failing because the cluster was not ready to be indexed due to drop/creation of indexes. I implemented a wait until cluster is ready method, but would be nice if the package had this kind of functionality. Here is an example:

    public function waitUntilClusterIsReady()
    {
        $info = null;

        do {
            if (!is_null($info)) {
                sleep(1);
            }
            $info = $this->elasticSearcher->getClient()->cluster()->health();
        } while ($info['status'] == 'red');
    }

It could be part of the ElasticSeacher so we could use it like:

$this->elasticSearcher->isHealthy();

Allow for environment aware indices

In most cases every environment has its own elasticsearch, this is also recommended. But in some situations one elasticsearch server needs to house multiple environments. This means the same index but different data (possible different mapping depending on the feature differences).

A common solution for this is to prefix the index with the environment name (ie. staging_movies, production_movies.

This prefix should be hidden from the implementation, so the implementation should keep referring to movies. This is not possible at the moment because an index has a getName method which is used for both referring and actual index actions.

My first thought is to split this up and add a getRealName (needs better naming) method, which will return the getName by default but allows overwriting.

class MoviesIndex extends AbstractIndex {
   public function getName()
   {
       return 'movies';
   }
   public function getRealName()
   {
       return APP_ENV.'_'.$this->getName();
   }
}

The getRealName can be moved to a, for example, EnvironmentAwareTrait. This should not be part of this package. This package should only allow that the internal name and external name are different.

Weird searchInType behavior

I noticed this earlier:

The current behavior of $this->searchInType($type) is to append the $type to an array.

My use case was this:

class Stats 
{
    protected $query;

    public function __construct(Query $query) 
    {
        $this->query = $query;
    }

    public function aggregate($types = ['foo', 'bar'])
    {
        foreach ($types as $type) {
            $this->query->addData(['type' => $type]);

            $result = $this->query->run();
        }
    }
}

class Query extends AbstractQuery
{
    protected function setup()
    {
        $this->searchIn('index', $this->getData('type'));

        $this->setBody([]);
    }
}

I injected the Query into the service to make it testable.

My expectation was the following:

{
    "index": "index",
    "type": "foo"
}

and

{
    "index": "index",
    "type": "bar"
}

But instead it was:

{
    "index": "index",
    "type": "foo"
}

and

{
    "index": "index",
    "type": "foo,bar"
}

In my case I fixed it by adding a make() method to the query which does:

/**
* @return static
*/
public function make()
{
    return new static($this->searcher);
}

So you'd do: $this->query->make to get a fresh copy of the query, and in the tested mocked the return value to just return the originally mocked Query instance.

I'm wondering if this is a common enough use-case to make it the default? It's seems a bit weird to me, and not at all what I would expect.

Add support for Future's

The PHP SDK 2.0 has support for Future's (from Guzzle), which allows for queueing requests. More here: https://www.elastic.co/guide/en/elasticsearch/client/php-api/current/_future_mode.html

Will need #27

Tag the bugfix

Assuming I'm not missing something, even if I specify I want ^0.5 that won't give me the bugfix that's now in master, so I would need to require dev-master if I want to use it. That being the case, I'd like a new tag for the current commit (unless there's a valid reason to not make one).

Add support for child/parent documents

You'll have to pass a parent ID when indexing a child, this is done via a querystring parameter (see #5).

Allow for custom mapping classes in index mapping

Like the custom filters in a query. So you could abstract an analyzer for example.

Refresh index

Hi. Is there ability to refresh index immediately after updating? https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-refresh.html

Figure out how to pass client configuration per request

The PHP SDK 2.0 allows several settings to be set per request, see here: https://www.elastic.co/guide/en/elasticsearch/client/php-api/current/_per_request_configuration.html.

We can actually set these already in a Query via $this->setQueryStringParam. However that method name does not represent its full usage anymore. Its not only about query string params, but much more.

Still need to figure out how to do it for document crud requests or other requests.

Add a LICENCE

Able to configure the method to call on the elasticsearch client

I'm using the suggester API and am using query objects to configure the query it needs to perform. Configuring works but to actually perform a suggest query I have to overwrite the query's run method like this:

 /**
 * Build and execute the query.
 *
 * @return SuggestionsResultParser
 */
public function run()
{
    $query = $this->buildQuery();
    $rawResults = $this->searcher->getClient()->suggest($query);

    // Pass response to the class that will do something with it.
    $resultParser = $this->getResultParser();
    $resultParser->setRawResults($rawResults);

    return $resultParser;
}

A lot of boilerplate to configure one method, it would be nice if this was easily configurable through a method something like:

protected function executeSearch($query) 
{
   return $this->searcher->getClient()->suggest($query);
}

 /**
 * Build and execute the query.
 *
 * @return SuggestionsResultParser
 */
public function run()
{
    $query = $this->buildQuery();
    $rawResults = $this->executeSearch($query);

    // Pass response to the class that will do something with it.
    $resultParser = $this->getResultParser();
    $resultParser->setRawResults($rawResults);

    return $resultParser;
}

Laravel 5 compatibility

Hello,

When can we expect a Laravel 5 version ?

Would a PR to make it compatible be ok? I don't see anything specific for L4, so a simple change in the composer.json file should suffice.

Top level fragments

It would be interesting to be able to create fragments that aren't meant to be in an array by themselves but that merge with the parent body. Very simple example, if I want to create a PaginationFragment:

class PaginationFragment extends AbstractFragment
{
    /**
     * @param integer $page
     * @param integer $perPage
     */
    public function __construct($page, $perPage)
    {
        $this->body = [
            'size' => $perPage,
            'from' => ($page - 1) * $perPage,
        ];
    }
}

Then I'd have nowhere to put it. One solution would be to have a property for this:

class PaginationFragment extends AbstractFragment
{
    protected $merged = true;

    /**
     * @param integer $page
     * @param integer $perPage
     */
    public function __construct($page, $perPage)
    {
        $this->body = [
            'size' => $perPage,
            'from' => ($page - 1) * $perPage,
        ];
    }
}

And then in the query:

$this->body = [
    new PaginationFragment($page, $perPage),
    'query' => ['...'],
];

Prefix package namespace with vendor

Not sure how I missed this but the package namespace should be prefixed with our vendor.

So ElasticSearcher should become Madewithlove\ElasticSearcher.

To be done with a major breaking release

Fractal integration

Hi guys,

Thank you for creating this open-source project. I think it demonstrates what agnostic packages are about.

I used it in one of my projects and ended up writing Fractal transformers for all my indices. I turned it into a package and published it today. I just wanted to ask you for some basic feedback.

Thank you

Do not depend on Illuminate/Support just for Array helpers

We are only using one or 2 of the array helper functions, we could replace this with our own helper.

I wasn't seeing al the downsides of pulling in a package for this, tx to Matt Allan for making me aware in his blogpost: http://mattallan.org/2016/dont-use-illuminate-support/

The illuminate/support package pulls 52 functions in to the global namespace. It’s nice when using the framework to not have to use namespaces, but a dependency of a dependency shouldn’t pollute the global namespace.

How to use re-usable parts

Hello,

I'd like to use the re-useable parts feature, but something more complex than the 'field' & 'value' type. Even with that, does that perform a 'match' query ? What if I need something else than a match query ?

For instance a range query.

It would be nice to have re-useable classes we can create ourselves. For instance:

class MoviesYouMightLikeQuery extends AbstractQuery
{
  public function setup()
  {
    $this->searchIn('suggestions', 'movies');

     $body = array(
      'query' => array(
         new MyReUseableQueryPart(),
         'match' => ['this' => 'that'],
         [...]
      )
     );

     $this->setBody($body);
  }
}

Thanks,

Improve bulk operation

Right now the package loops over the data array and calls the index method on the DocumentsManager.

We could simply map the the dataset to so use a single HTTP call to ES by using the bulk operation describred here: https://www.elastic.co/guide/en/elasticsearch/client/php-api/current/_indexing_documents.html#_bulk_indexing

Check recursively for AbstractFragments

So you can nest your fragments. For example use the TermsFilter in your own UserFilter.