divanteltd/magento1-vsbridge-indexer

This is an official, native Vue Storefront data indexer for Magento 1.9

Divante VueStorefrontIndexer Extension for Magento1

This projects is a native, Magento1 data indexer for Vue Storefront - first Progressive Web App for e-Commerce. It fills the ElasticSearch data index with all the products, categories and static information required by Vue Storefront to work.

Vue Storefront is a standalone PWA storefront for your eCommerce, possible to connect with any eCommerce backend (eg. Magento, Pimcore, Prestashop or Shopware) through the API.

Please note: This indexer is a part of magento1-vsbridge connector for Magento1.

Video demo

Sign up for a demo at https://vuestorefront.io/ (Vue Storefront integrated with Pimcore OR Magento2).

Facts

• version: beta 1.0.0
• extension key: Divante_VueStorefrontIndexer

Features

This module is in beta, however we've used it in some production sites already. Please do feel free to test it and bring Your feedback + Pull Requests :)

Full synchronization: products, categories, attribute, tax rules, cms blocks. Synchronization in real time for: products, categories, attributes, cms blocks.

Module listen on following Magento1 events:

• product save (in backend panel),
• product deletion (in backend panel)
• mass product update,
• category save,
• category deletion,
• attribute save,
• attribute deletion (after attribute is removed full product synchronization will be fired),
• cms block save,
• cms block deletion.

Requirements

• PHP >= 5.5.0
• Magento 1.9.*
• ElasticSearch 5.*
• ...

Compatibility

• Magento >= 1.9
• Vue Storefront >= 1.7

Overview

Support for aliases.

Command php -f vsf_tools.php -- --action full_reindex --store X will reindex all data to new index. It will create new index and switch aliases at the end.

If you used previous versions, you will have to delete index from ES manually. If you won't do that, you will get error when running "error":{"root_cause":[{"type":"invalid_alias_name_exception","reason":"Invalid alias name [vue_storefront_magento_1], an index exists with the same name as the alias"

Installation Instructions/Getting Stared

Install Magento Module

• Install Magento module Divante_VueStorefrontIndexer using modman or by coping code from src/
• Configure the module in Magento admin panel and run full indexation

Configuration

Go to the new ‘Indexer’ section (Stores → Configuration → Vuestorefront → Indexer), available now in the in the Magento Panel, and configure it in the listed areas:

1. General settings → List of stores to reindex

   Select stores for which data must be exported to ElasticSearch. By default stores 0 to 1 are selected. For each store view, a new, separate ElasticSearch index is created.

   

2. Elasticsearch Client

   Configure connection with ElasticSearch. Provide a host, port, and set up login and password (optionally).

   

3. Indices settings

   Batch Indexing Size → select size of packages by which you intend to send data to ElasticSrearch. Depending on the project you might need to adjust package size to the number of products, attributes, configurable products variation, etc). By default Batch, Indexing Size is set up for 1000. Indicies settings. Adjust indexing batch size to your data.

   Index Alias Prefix → define prefixes for ElasticSearch indexes. The panel allows adding prefix only to the catalog name e.g.: "vue_storefront_catalog". For each store (store view) index name is generated on the base of defined prefix and ID. Aliases cannot be created. Note: change to "vue_storefront_catalog" to make it compatible with mage2vuestorefront import.

   Index Identifier → defines the unique store identifier to append to the ElasticSearch indexes. The default value is ID which will append the Store ID to the index name e.g.: "vue_storefront_catalog_1". You can choose to change this to Store Code which will add the Store Code to the index name e.g.: "vue_storefront_catalog_storecode".

   Add Index Identifier to Default Store View → defines if we should add Index Identifier to Magento Default Store View. Select "No" - to make it compatible with mage2vuestorefront import.

   Example: When we define following indexes: "vue_storefront_catalog_1", "vue_storefront_catalog_2", "vue_storefront_catalog_3", their name will remain unchanged, and only product and category names will be updated. Important: It is crucial to update this configuration in the VSF and VSF-API (one change at the beginning of the whole configuration process).
   

   The maximum number of fields in an index -> allow to setup value for option index.mapping.total_fields.limit

4. Redis Cache Settings

   Clear cache → No/Yes (by default this option is disabled)

   VSF base Url → URL for VSF

   Invalidate Secret cache key → provide the same value as in the VSF configuration

   Connection timeout → by default set up for 10 seconds

   

5. Catalog Settings

   Use Short Catalog Urls → by default this option is disabled. The short Catalog Urls must be aligned with the VSF configuration. After any changes in the VSF configuration, the configuration in the Magento Panel must be updated and all products and categories indexed anew.

   Types of products to index → by default all product will be exported to ElasticSearch. This option allows for selecting certain product types that should be exported.

   

After updating the configuration, you can run the indexation. It is also worth query ElasticSearch using CURL, to be sure that the communication works.

Indexation

You can run full synchronization (for products, categories, attributes, cms blocks) or for specific type.

Run full synchronization for chosen store views:

cd [magento root dir]/shell
php -f vsf_tools.php -- --action full_reindex --store STORE_ID

Run full synchronization for specific type

cd [magento root dir]/shell
php -f vsf_tools.php -- --action full_reindex --store STORE_ID --type [taxrules|products|categories|attributes|cms_blocks]

php -f vsf_tools.php -- --action full_reindex --store STORE_ID --type attributes

It is worth, to begin with taxrules as it is the fastest.

Setup Cron job to update data in ElasticSearch in real time (for products, categories, attributes, cms blocks)

e.g.

*/5 * * * * cd [full path to magento directory]/shell && /usr/bin/flock -n /tmp/vsf_index.lock  /usr/bin/php vsf_tools.php --action reindex 

Support

Join our Slack channel

Licence

MIT

Copyright

(c) 2018 Divante

magento1-vsbridge-indexer's Issues

[FR] Indexing same data to more than one cluster

I have multiple test environments hooked up to single magento instance. I would like to index the same data to more than one elasticsearch instance.

Add Reviews indexer

Reviews indexer feature is already implemented as a part of #2

The thing is that we don't want to merge all the changes proposed within this PR into the core as they require some changes.

The acceptance criteria for this task is to extract the Reviews Indexer, test it and propose this feature as a separate Pull Request

Indexing products fails with Magento 1.9

When I try to run the indexing with the vsf_tools.php I get this error . This happens for all products. But all other index are inserted perfectly

In this line
https://github.com/DivanteLtd/magento1-vsbridge-indexer/blob/develop/src/app/code/community/Divante/VueStorefrontIndexer/Model/Elasticsearch/Indexer/Handler.php#L232

 [1] => Array
                        (
                            [index] => Array
                                (
                                    [_index] => vue_storefront_magento_10_1596536487
                                    [_type] => product
                                    [_id] => 6757
                                    [status] => 400
                                    [error] => Array
                                        (
                                            [type] => illegal_argument_exception
                                            [reason] => mapper [attributes_metadata.options.value] of different type, current_type [long], merged_type [text]
                                        )

                                )

                        )

Here is the print of the request that is sent

https://gist.github.com/sudarshann/793552c6d85e37d3a494e26c131bd6e7

Add verbose parameter

Verbose parameter would be very helpful while quick debugging CLI indexing.

"Can't retrieve entity config: catalog/product_link_attribute_" while full index for products

If I run php shell/vsf_tools.php --action full_reindex --action full_reindex --store 1 --type products the following exception is thrown:

PHP Fatal error:  Uncaught Mage_Core_Exception: Can't retrieve entity config: catalog/product_link_attribute_ in /var/www/magento/public/app/Mage.php:598
Stack trace:
#0 /var/www/magento/public/app/code/core/Mage/Core/Model/Resource.php(282): Mage::throwException('Can't retrieve ...')
#1 /var/www/magento/public/app/code/community/Divante/VueStorefrontIndexer/Model/Resource/Catalog/Product/Links.php(192): Mage_Core_Model_Resource->getTableName('catalog/product...')
#2 /var/www/magento/public/app/code/community/Divante/VueStorefrontIndexer/Model/Resource/Catalog/Product/Links.php(172): Divante_VueStorefrontIndexer_Model_Resource_Catalog_Product_Links->joinPositionAttribute(Object(Varien_Db_Select))
#3 /var/www/magento/public/app/code/community/Divante/VueStorefrontIndexer/Model/Resource/Catalog/Product/Links.php(129): Divante_VueStorefrontIndexer_Model_Resource_Catalog_Product_Links->prepareLinksSelect()
#4 /var/www/magento/public/app/code/community/Divante/VueStorefrontIndexer/Model/Resource/Catalog/Product/Links.php(85): Divan in /var/www/magento/public/app/Mage.php on line 598

Reason:
I found out that this happens because our catalog_product_link_attribute table is empty and the select is return false instead of an array in class method Divante_VueStorefrontIndexer_Model_Resource_Catalog_Product_Links::fetchPositionAttributeData() (line 209-220, source link).

Solution
I changed the class method joinPositionAttribute() to continue if an empty array is returned by fetchPositionAttributeData():

    /**
     * @param Varien_Db_Select $select
     *
     * @return Varien_Db_Select
     */
    private function joinPositionAttribute(Varien_Db_Select $select)
    {
        $alias = 'link_position';
        $attributePosition = $this->fetchPositionAttributeData();

        if (empty($attributePosition)) {
            return $select;
        }

        $table = $this->resource->getTableName($this->getAttributeTypeTable($attributePosition['type']));

        $joinCondition = [
            "{$alias}.link_id = links.link_id",
            $this->connection->quoteInto(
                "{$alias}.product_link_attribute_id = ?",
                $attributePosition['id']
            ),
        ];

        $select->joinLeft(
            array($alias => $table),
            implode(' AND ', $joinCondition),
            array($attributePosition['code'] => 'value')
        );

        return $select;
    }

Maybe this should be considered in the code by default if somebody isn't using upsell- or crosssell-products.

Create Attribute throw Error on Magento 1.9.x

Getting error on fresh installed vanilla magento 1.9, when the extension (VueStorefrontIndexer) is disabled the error is not appearing anymore.

Appearing on create attribute, text-field & dropdownfield

Error-Log:

a:5:{i:0;s:31:"Invalid entity_type specified: ";i:1;s:3365:"#0 /app/htdocs/app/code/core/Mage/Eav/Model/Config.php(331): Mage::throwException('Invalid entity_...')
#1 /app/htdocs/app/code/core/Mage/Eav/Model/Entity/Attribute/Abstract.php(399): Mage_Eav_Model_Config->getEntityType(NULL)
#2 /app/htdocs/app/code/core/Mage/Eav/Block/Adminhtml/Attribute/Edit/Main/Abstract.php(163): Mage_Eav_Model_Entity_Attribute_Abstract->getEntityType()
#3 /app/htdocs/app/code/core/Mage/Adminhtml/Block/Catalog/Product/Attribute/Edit/Tab/Main.php(45): Mage_Eav_Block_Adminhtml_Attribute_Edit_Main_Abstract->_prepareForm()
#4 /app/htdocs/app/code/core/Mage/Adminhtml/Block/Widget/Form.php(144): Mage_Adminhtml_Block_Catalog_Product_Attribute_Edit_Tab_Main->_prepareForm()
#5 /app/htdocs/app/code/core/Mage/Eav/Block/Adminhtml/Attribute/Edit/Main/Abstract.php(199): Mage_Adminhtml_Block_Widget_Form->_beforeToHtml()
#6 /app/htdocs/app/code/core/Mage/Core/Block/Abstract.php(937): Mage_Eav_Block_Adminhtml_Attribute_Edit_Main_Abstract->_beforeToHtml()
#7 /app/htdocs/app/code/core/Mage/Adminhtml/Block/Catalog/Product/Attribute/Edit/Tabs.php(50): Mage_Core_Block_Abstract->toHtml()
#8 /app/htdocs/app/code/core/Mage/Core/Block/Abstract.php(937): Mage_Adminhtml_Block_Catalog_Product_Attribute_Edit_Tabs->_beforeToHtml()
#9 /app/htdocs/app/code/core/Mage/Core/Block/Text/List.php(45): Mage_Core_Block_Abstract->toHtml()
#10 /app/htdocs/app/code/core/Mage/Core/Block/Abstract.php(938): Mage_Core_Block_Text_List->_toHtml()
#11 /app/htdocs/app/code/core/Mage/Core/Block/Abstract.php(656): Mage_Core_Block_Abstract->toHtml()
#12 /app/htdocs/app/code/core/Mage/Core/Block/Abstract.php(600): Mage_Core_Block_Abstract->_getChildHtml('left', true)
#13 /app/htdocs/app/design/adminhtml/default/default/template/page.phtml(58): Mage_Core_Block_Abstract->getChildHtml('left')
#14 /app/htdocs/app/code/core/Mage/Core/Block/Template.php(263): include('/app/htdocs/app...')
#15 /app/htdocs/app/code/core/Mage/Core/Block/Template.php(294): Mage_Core_Block_Template->fetchView('adminhtml/defau...')
#16 /app/htdocs/app/code/core/Mage/Core/Block/Template.php(308): Mage_Core_Block_Template->renderView()
#17 /app/htdocs/app/code/core/Mage/Adminhtml/Block/Template.php(81): Mage_Core_Block_Template->_toHtml()
#18 /app/htdocs/app/code/core/Mage/Core/Block/Abstract.php(938): Mage_Adminhtml_Block_Template->_toHtml()
#19 /app/htdocs/app/code/core/Mage/Core/Model/Layout.php(590): Mage_Core_Block_Abstract->toHtml()
#20 /app/htdocs/app/code/core/Mage/Core/Controller/Varien/Action.php(406): Mage_Core_Model_Layout->getOutput()
#21 /app/htdocs/app/code/core/Mage/Adminhtml/controllers/Catalog/Product/AttributeController.php(138): Mage_Core_Controller_Varien_Action->renderLayout()
#22 /app/htdocs/app/code/core/Mage/Core/Controller/Varien/Action.php(437): Mage_Adminhtml_Catalog_Product_AttributeController->editAction()
#23 /app/htdocs/app/code/core/Mage/Core/Controller/Varien/Router/Standard.php(262): Mage_Core_Controller_Varien_Action->dispatch('edit')
#24 /app/htdocs/app/code/core/Mage/Core/Controller/Varien/Front.php(192): Mage_Core_Controller_Varien_Router_Standard->match(Object(Mage_Core_Controller_Request_Http))
#25 /app/htdocs/app/code/core/Mage/Core/Model/App.php(381): Mage_Core_Controller_Varien_Front->dispatch()
#26 /app/htdocs/app/Mage.php(760): Mage_Core_Model_App->run(Array)
#27 /app/htdocs/index.php(72): Mage::run('', 'store')
#28 {main}";s:3:"url";s:85:"/index.php/admin/catalog_product_attribute/edit/key/fe1c97a3ffc3d2365f34dccb819de920/";s:11:"script_name";s:10:"/index.php";s:4:"skin";s:5:"admin";}

Taxrates are being omitted

Problem

Taxrates are fetched with Varien_Db_Adapter_Interface::fetchAssoc() in src/app/code/community/Divante/VueStorefrontIndexer/Model/Resource/Tax/Rates.php - this method uses the first result column as array keys. However, the first column values (tax_calculation_rule_id) will not be unique in rules with multiple country-based rates, which results in missing tax rates.

See https://framework.zend.com/manual/1.10/en/zend.db.adapter.html#zend.db.adapter.select.fetchassoc

Solution

Use Varien_Db_Adapter_Interface::fetchAll() instead.

Support for letting simple products inherit images from their parent configurable product

During the hackathon in Stockholm I noticed that product images were missing in the cart. The reason for it was that the product in Magento only had images on the configurable product, not the simple products that were linked to it. This has often been the case in many of the Magento projects I've been working with over the years, so I think it would be nice to have a feature toggle that allows simple products inherit the images of the configurable product if they are missing.

Class '\Elasticsearch\Endpoints\Bulk' not found

On PHP 5.5.9 and Magento 1.9.3.1 when I try to run

php -f shell/vsf_tools.php -- --action full_reindex --store 1

I gave this error

Full reindexing: store #1 ... 
PHP Fatal error:  Class '\Elasticsearch\Endpoints\Bulk' not found in /home/test/www/lib/vendor/elasticsearch/elasticsearch/src/Elasticsearch/ClientBuilder.php on line 529

Don't use `private` method scope in module to make class overwrites possible

While I was overwriting some classes with a module of our own to keep this module "update-able", I noticed that there is by far every function is using the scope private. This makes it not possible to simply extend a class function by another without copying the whole class.

Is there a specific reason for that?
Otherwise I would suggest to change them to the protectedfunction scope.

Missing change for new indexnaming (with storefront-api)

Setup:

M1 (1.9.4.4) + https://github.com/DivanteLtd/magento1-vsbridge-indexer + https://github.com/DivanteLtd/magento1-vsbridge
ES Index build with shell script: /shell$ php vsf_tools.php --action full_reindex --store 1

Frontend:
https://github.com/DivanteLtd/vue-storefront + https://github.com/DivanteLtd/storefront-api

Error Message:

(node:31063) UnhandledPromiseRejectionWarning: Error: {"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"vue_storefront_magento_1_attribute","index_uuid":"_na_","index":"vue_storefront_magento_1_attribute"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"vue_storefront_magento_1_attribute","index_uuid":"_na_","index":"vue_storefront_magento_1_attribute"}
(node:31063) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 47)

Add documentation for extension

I'm reading the code and it seems to me that could be feasible extending this module to supporting custom stuff.
If it's possible as I think, it could be very useful adding documentation about extending this indexer (f.i adding new entities) in order to make this module pluggable.

@afirlejczyk What do you think about?

Add Elastic 6+ support

Reflecting stock_status in is_in_stock

We're currently working on implementing VSF for a really old M1 solution. They don't show out of stock products. For some configurable products, stock_status and is_in_stock differs in Magento, stock_status is 0 and is_in_stock is 1. These products are not listed or shown in Magento frontend (as expected), but they are shown in VSF. From what I can see VSF only filters on is_in_stock which explains the different output of product listings.

As I see it this could be adjusted quite simply by one or more of the following actions:

• adding extra filters in VSF
• adjusting the logic in the indexer
• update the stock data in Magento

From a brief dialogue with @pkarw in the Slack channel, we've come to the conclusion that the best cleanest solution probably is to adjust the logic in the indexer.

issue while executing "php -f vsf_tools.php -- --action full_reindex --store STORE_ID"

Hello All,

There seems to be a bug on module configuration and execution, i had a sneek peak and saw that you had an patch on magento2-vsbridge-indexer, please do also look for the same on this repo, beneath is the issue i am faing using your develop branch

PHP Fatal error: Uncaught Elasticsearch\Common\Exceptions\BadRequest400Exception: {"error":{"root_cause":[{"type":"parse_exception","reason":"unknown key [index.mapping.total_fields.limit] for create index"}],"type":"parse_exception","reason":"unknown key [index.mapping.total_fields.limit] for create index"},"status":400} in /var/www/html/magento/lib/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php:610
Stack trace:
#0 /var/www/html/magento/lib/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php(273): Elasticsearch\Connections\Connection->process4xxError(Array, Array, Array)
#1 /var/www/html/magento/lib/vendor/react/promise/src/FulfilledPromise.php(25): Elasticsearch\Connections\Connection->Elasticsearch\Connections{closure}(Array)
#2 /var/www/html/magento/lib/vendor/guzzlehttp/ringphp/src/Future/CompletedFutureValue.php(55): React\Promise\FulfilledPromise->then(Object(Closure), NULL, NULL)
#3 /var/www/html/magento/lib/vendor/guzzlehttp/ringph in /var/www/html/magento/lib/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php on line 610

Add support for index alias

Full reindexation - create new index

Create new index while running full reindexation, mainly to automatically create new mapping for index.

Now we have to delete index, and run full reindexation to add (update) mapping in elastic.

Related #4

Use native price logic for configurable products as default

Currently the configurable_children_use_simple_price default value is set to 1 which results in a different price logic than Magento 1 normally use for configurable products. As a Magento developer quite new to Vue Storefront I rather see the default logic being used as default and let the customization be something that needs an active decision.

CMS Page/Block identifier tokenized

identifier of CMS Page and Blocks is text type therefore - seems to be interpreted as a tokenizer.

That creates a bug for cms pages with a - in its identifier such as about-us.
That's why about-us is not found at all using VSF elastic query:

GET _search
{
  "query": {
    "bool": {
      "filter": {
        "terms": {
          "identifier": [
            "about-us"
          ]
        }
      }
    }
  }
}

Whereas it is found using that elastic query:

GET _search
{
  "query": {
    "bool": {
      "filter": {
        "terms": {
          "identifier": [
            "about"
          ]
        }
      }
    }
  }
}

I suppose that identifier should be keyword type.

product_count is not indexed

Introduction

I noticed that in VSF code there is a check for excluding categories that have children_count <= 0 and product_count <= 0.
I think this is necessary in order to remove all the categories that are empty.
https://github.com/DivanteLtd/vue-storefront/blob/6cd57ab269a8a31ed8eb5a18ebc8934abe1b9b55/src/themes/default/components/core/blocks/SidebarMenu/SidebarMenu.vue#L204

Issue

The problem is that this indexer doesn't seem to take product_count in consideration so if a category haven't children categories but only products, it will be hidden by VSF.

Proposal

Index product_count field value to ES.

Check support for products "url_path"

Check if "url_path" is exported for category and product entity

Speed up exporting for products/categories.

• remove class Divante_VueStorefrontIndexer_Model_Index_Convertdatatypes and implement solution base on M2 Pull Request:
  vuestorefront/magento2-vsbridge-indexer#155

Update readme