ongr-io/elasticsearchbundle

Symfony bundle for Elasticsearch with steroids

elasticsearchbundle's Issues

[_BULK] Query structure

Make sure that proper query structure is formed for each operation.

f.e. now update operation wouldn't execute.

Implement Warmers

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-warmers.html

How to customize the document_dir ?

Hi,
My symfony2 bundle is DemoBundl, but i want to customize the document_dir path set to DomoBundle/Search/Document. plz tell me how to configure?

Timestampable and Sluggable extensions

There are some documents, which need fields like createdAt or updatedAt. There are useful extensions for doctrine, which we could remake for ESB and use:

https://github.com/l3pp4rd/DoctrineExtensions/blob/master/doc/timestampable.md

https://github.com/l3pp4rd/DoctrineExtensions/blob/master/doc/sluggable.md

[_BULK] Add delete operation

SearchTest code deduplication

duplicating code at

https://github.com/ongr-io/ElasticsearchBundle/blob/master/Tests/Functional/DSL/SearchTest.php#L82
https://github.com/ongr-io/ElasticsearchBundle/blob/master/Tests/Functional/DSL/SearchTest.php#L108
https://github.com/ongr-io/ElasticsearchBundle/blob/master/Tests/Functional/DSL/SearchTest.php#L129

etc.

Calling ->current() on empty result set generates notice

$foo = $repository->findBy(['foo' => 'a']); // Generates empty result set, nothing was found.
$foo->current();

Generates following notice:

Notice: Undefined index: in /var/www/vendor/ongr/elasticsearch-bundle/Result/AbstractResultsIterator.php line 96 

There should be no notice, error or empty result is preferred.

Unexpected behaviour in ElasticsearchTestCase when querying too soon after initialization.

Consider the following code:

/**
 * Demonstrates unexpected behaviour of ElasticsearchTestCase.
 */
class ESBProblem extends ElasticsearchTestCase {

    /**
     * Test for success - succeeds.
     */
    public function testSuccess()
    {
        $repository=$this->getManager()->getRepository('AcmeTestBundle:Product');
        usleep(250000);
        $repository->execute($repository->createSearch());
    }

    /**
     * Test for fail - fails.
     *
     * @expectedException \Exception
     */
    public function testFail()
    {
        $repository=$this->getManager()->getRepository('AcmeTestBundle:Product');
        // Let's remove usleep(250000) and see what happens.
        $repository->execute($repository->createSearch());
    }
} 

add array type hint to FunctionScoreQuery $functions variable in constructor

Add method to delete document.

Add autocomplete support.

Add httpParams field to profiler

Implement Multi Get API

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-multi-get.html

Implement Index Templates

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-templates.html

consider to add open/close index and getState() methods

Tests fails with elasticsearch 1.2.4

Tests fails with elasticsearch 1.2.4 because this version does not have top_hits aggregation type.

add profiler documentation

Make index page for documentation

f.e. https://github.com/FriendsOfSymfony/FOSRestBundle/blob/master/Resources/doc/index.md

Repository::RESULTS_ARRAY does not return all fields

$repository->execute($search, Repository::RESULTS_ARRAY) returns:

Array
(
    [0] => Array
        (
            [title] => test_prod2
            [description] => test_desc2
            [price] => 7.79
        )

    [1] => Array
        (
            [title] => test_prod
            [description] => test_desc
            [price] => 0.99
        )
)

There are some fields missing, particularly id and score. Is this by design?

Refactor tests to only check for expected mapping.

This will allow us to change testing bundle more freely.

Create elasticsearch query profiler

it takes a long time to create an index for manager

es:index:create - OK
app/console es:index:create --manager="default" - takes a long time
after clearing the cache, it creates an index imediately
But when the index is dropped, the problem persist when trying to create another index.
Also sometimes it takes longer to drop indexes.

SearchTest code deduplication

duplicating code at

https://github.com/ongr-io/ElasticsearchBundle/blob/master/Tests/Functional/DSL/SearchTest.php#L82
https://github.com/ongr-io/ElasticsearchBundle/blob/master/Tests/Functional/DSL/SearchTest.php#L108
https://github.com/ongr-io/ElasticsearchBundle/blob/master/Tests/Functional/DSL/SearchTest.php#L129

etc.

change docs to rst format

[Profiler] Pretty print json

Override elasticsearch-php serializer to pretty print json.

Experiment with HHVM

Do not add new line on empty array/object in JsonFormatter

Lifecycle callbacks

Implement PrePersist and PreUpdate annotations. They should be written above methods which:

• PrePersist, should execute on document creation.
• PreUpdate, should execute on document update.

Error when a property definition has an object type mapping

For the given document mapping:

<?php
namespace Acme\AcmeDemoBundle\Document;

use ONGR\ElasticsearchBundle\Annotation as ES;
use ONGR\ElasticsearchBundle\Document\DocumentInterface;
use ONGR\ElasticsearchBundle\Document\DocumentTrait;

/**
 * @ES\Document
 */
class Content implements DocumentInterface
{
    use DocumentTrait;

    /**
     * @var string
     *
     * @ES\Property(name="name", type="string")
     */
    private $name;

    /**
     * @var string
     *
     * @ES\Property(name="meta", type="object", objectName="AcmeDemoBundle:Meta")
     */
    private $meta;

    /**
     * {@inheritdoc}
     */
    public function getName()
    {
        return $this->name;
    }

    /**
     * {@inheritdoc}
     */
    public function setName($name)
    {
        $this->name = $name;

        return $this;
    }

    /**
     * {@inheritdoc}
     */
    public function getMeta()
    {
        return $this->meta;
    }

    /**
     * {@inheritdoc}
     */
    public function setMeta(Meta $meta)
    {
        $this->meta = $meta;

        return $this;
    }
}

With the meta object definition:

<?php
namespace Acme\AcmeDemoBundle\Document;

use ONGR\ElasticsearchBundle\Annotation as ES;

/**
 * @ES\Object
 */
class Meta
{
    /**
     * @var string
     *
     * @ES\Property(name="title", type="string", index="not_analyzer")
     */
    public $key;

    /**
     * @var string
     *
     * @ES\Property(name="value", type="string", index="not_analyzer")
     */
    public $value;

    /**
     * {@inheritdoc}
     */
    public function getKey()
    {
        return $this->key;
    }

    /**
     * {@inheritdoc}
     */
    public function setKey($key)
    {
        $this->key = $key;

        return $this;
    }

    /**
     * {@inheritdoc}
     */
    public function getValue()
    {
        return $this->value;
    }

    /**
     * {@inheritdoc}
     */
    public function setValue($value)
    {
        $this->value = $value;

        return $this;
    }
}

If you create one

$manager = $this->get("es.manager");

$content = new \Acme\AcmeDemoBundle\Document\Content();
$content->setId("myCustomId");
$content->setName("Acme");

$meta = new \Acme\AcmeDemoBundle\Document\Meta();
$meta->setKey("title");
$meta->setValue("Some value");

$content->setMeta($meta);

$manager->persist($content); 
$manager->commit();

When you try to get an existing document an error is fired

$manager = $this->get("es.manager");
$repository = $manager->getRepository('AcmeDemoBundle:Content');
$document = $repository->find('myCustomId');

Fatal error: Argument 1 passed to Acme\AcmeDemoBundle\Document\Content::setMeta() must be an instance of Acme\AcmeDemoBundle\Document\Meta, instance of ONGR\ElasticsearchBundle\Result\ObjectIterator given.

I think that the problem is that the rawData must be converted to a document in this line https://github.com/ongr-io/ElasticsearchBundle/blob/master/Result/Converter.php#L90. Maybe a call to $value = $value->current(); fix that

generate virtual document classes

Create in cache virtual document classes with setters and getters. If document class already have setter or getter, use it if not create virtual.

MultiField doesn't seem to work

Hey,

I am using a multifield to have the raw version of a field available for aggregations. I annotate the field in my document like this:

    /**
     * @var string
     *
     * @ES\Property(name="manufacturer", type="string", analyzer="simple", fields={@ES\MultiField(name="raw", type="string", index="not_analyzed")})
     *
     */
    private $manufacturer;

(ES is mapped to ONGR\ElasticsearchBundle\Annotation)

When I am trying to use the manufacturer.raw field in my aggregation i get an empty result. The analyzed manufacturer field works, but of course in an analyzed way, which is not what I want.

My kibana installation doesn't seem to know anything about the manufacturer.raw field, either.

Am I missing something here?

Version of the bundle used: v0.2.0

Use virtual documents in the elasticsearch manager

Use these documents for set and get data.

Add support for 'is' getter for bool type document fields.

Add support for 'is' getter for bool type document fields (getActive() => isActive()).
Now when document have private bool type field ($active) ElasticsearchBundle check is declarated set and get methods (getActive(), setActive()) for it.
https://github.com/ongr-io/ElasticsearchBundle/blob/master/Mapping/MetadataCollector.php#L257

The date field is not persisted correctly

For the given document

<?php
namespace Acme\AcmeDemoBundle\Document;

use DateTime;
use ONGR\ElasticsearchBundle\Annotation as ES;
use ONGR\ElasticsearchBundle\Document\DocumentInterface;
use ONGR\ElasticsearchBundle\Document\DocumentTrait;

/**
 * @ES\Document
 */
class Content implements DocumentInterface
{
    use DocumentTrait;

    /**
     * @var string
     *
     * @ES\Property(name="name", type="string")
     */
    private $name;

    /**
     * @var string
     *
     * @ES\Property(name="createdAt", type="date")
     */
    private $createdAt;

    /**
     * Constructor.
     */
    public function __construct()
    {
        $this->createdAt = new DateTime();
    }

    /**
     * {@inheritdoc}
     */
    public function getName()
    {
        return $this->name;
    }

    /**
     * {@inheritdoc}
     */
    public function setName($name)
    {
        $this->name = $name;

        return $this;
    }

    /**
     * {@inheritdoc}
     */
    public function getCreatedAt()
    {
        return $this->createdAt;
    }

    /**
     * {@inheritdoc}
     */
    public function setCreatedAt(DateTime $createdAt)
    {
        $this->createdAt = $createdAt;

        return $this;
    }
}

If you create one

$manager = $this->get("es.manager");

$content = new \Acme\AcmeDemoBundle\Document\Content();
$content->setId("myCustomId");
$content->setName("Acme");

$manager->persist($content); 
$manager->commit();

When you try to get an existing document an error is fired

$manager = $this->get("es.manager");
$repository = $manager->getRepository('AcmeDemoBundle:Content');
$document = $repository->find('myCustomId');

Fatal error: Argument 1 passed to Acme\AcmeDemoBundle\Document\Content::setCreatedAt() must be an instance of DateTime, array given.

In the elasticsearch index the createAt field has a array value, not a date value.

Throwing wrong exception on getRepository when asking not existing one

It should throw Document not found.

Variable names does not follow ongr-strict-standard

Member variable "index_analyzer" is not in valid camel caps format
Member variable "saerch_analyzer" is not in valid camel caps format

ElasticsearchBundle/Annotation/MultiField.php
ElasticsearchBundle/Annotation/Property.php

Unused variable

ElasticsearchBundle/Client/IndexSuffixFinder.php

Line 36 in d5d5faa

$suffix = null;

errors in comments

/Client/Connection.php two functions have identical documentation.

    /**
     * Send refresh call to index.
     */
    public function refresh()
    {
        $this->client->indices()->refresh();
    }

    /**
     * Send refresh call to index.
     */
    public function flush()
    {
        $this->client->indices()->flush();
    }

Implement method to retrieve Elasticsearch version in TestCase

This would be useful if tests are run on older versions when some features are not supported. We could skip them by knowing from documentation if they are supported by current running elasticsearch version or not.

Related: #60

Check compatibility with Elasticsearch 1.4.x

FAILURES! Tests: 306, Assertions: 650, Failures: 6, Errors: 1.

Terms aggregation has new field in response, script filter completely fails.

Functional ORM repository test duplicated repo->execute() call

see

ElasticsearchBundle/Tests/Functional/ORM/RepositoryTest.php

Lines 235 to 239 in 1fc70e1

	$this->assertInstanceOf(
	'\ONGR\ElasticsearchBundle\Result\DocumentIterator',
	$repo->execute($search, Repository::RESULTS_OBJECT)
	);
	$this->assertCount(0, $repo->execute($search, Repository::RESULTS_OBJECT));

Replace Tests/app/fixtures with ongr/testing-bundle

$params was never been initialized in function

$params was never initialized. Although not strictly required by PHP, it is generally a good practice to add $params = array(); before regardless.

see https://scrutinizer-ci.com/g/ongr-io/ElasticsearchBundle/issues/master/files/Client/Connection.php?selectedLabels%5B0%5D=3&orderField=path&order=asc#inspectioncomment-356280567

Index import

Index import isn't working with data set sizes of 1000 and its multiples, cause of the bulk size.
With data sets of sizes like 1001 or 999 it works as expected.

 [Elasticsearch\Common\Exceptions\BadRequest400Exception]                                                                          
  {"error":"ElasticsearchParseException[Failed to derive xcontent from org.elasticsearch.common.bytes.BytesArray@1]","status":400}  

Something is wrong with this logic.

Add possibility to specify format for date type fields

Currently it is not possible to define custom format for date type fields:

@ES\Property(name="dateField", type="date", format="YYYY-MM-dd h:i:s")

Setting value to AbstractResultsIterator

This invalidation is wrong

ElasticsearchBundle/Result/AbstractResultsIterator.php

Line 113 in 3156f79

unset($this->converted[$offset]);

Converted and document arrays keys are being checked 1:1. So if converted array has values after which is being unset, they all will shift.

Missing functional tests for update command.

Handle precision strings.

Add the ability for mapping tool to compare tree levels by parsing precision.
Elasticsearch parses precision on mapping put, so the precision returned is actually tree levels.
To compare new mapping to the old one we need to parse it, since the old mapping is return by Elasticsearch and the new mapping is collected from documents. A few ideas of mine on how to solve this:

• Create new index with the new mapping and immediately delete it. This would solve various parsing, default value problems and would also simplify our mapping tool code greatly but at the cost of creating a new index on each check.
• Parse the string using a new helper class. This would add quite a lot of code complexity.

Implement Percolator

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-percolate.html

The update operation doesn't work

For example, for the given document

<?php
namespace Acme\AcmeDemoBundle\Document;

use ONGR\ElasticsearchBundle\Annotation as ES;
use ONGR\ElasticsearchBundle\Document\DocumentInterface;
use ONGR\ElasticsearchBundle\Document\DocumentTrait;

/**
 * @ES\Document
 */
class Content implements DocumentInterface
{
    use DocumentTrait;

    /**
     * @var string
     *
     * @ES\Property(name="name", type="string")
     */
    private $name;

    /**
     * {@inheritdoc}
     */
    public function getName()
    {
        return $this->name;
    }

    /**
     * {@inheritdoc}
     */
    public function setName($name)
    {
        $this->name = $name;

        return $this;
    }
}

If you create one

$manager = $this->get("es.manager");

$content = new \Acme\AcmeDemoBundle\Document\Content();
$content->setId("myCustomId");
$content->setName("Acme");

$manager->persist($content); 
$manager->commit();

When you try to update an existing document you never can update it

$manager = $this->get("es.manager");
$repository = $manager->getRepository('AcmeDemoBundle:Content');
$document = $repository->find('myCustomId');

$content->setName("Another Name");
$manager->persist($document); // The persist method never send an "update" bulk operation to elasticsearch
$manager->commit();

And look in the index

curl -XGET localhost:9200/my_index/my_type/_search?pretty

The content never change. I think that the problem is in the persist method of the Manager class https://github.com/ongr-io/ElasticsearchBundle/blob/master/ORM/Manager.php#L103-L107 that always sends the "create" operation. The Connection class never send any update option upsert, doc_as_upsert ... in the bulk method https://github.com/ongr-io/ElasticsearchBundle/blob/master/Client/Connection.php#L93-L115