magento-hackathon/asyncindex

asyncindex's Issues

New release tag

Can you add a new release tag to this repository?

use magento flags instead of cron messages to schedule new execution

Is that possible async index after magento's dataflow import products?

Hi,
It is great for this extension.While is that possbile put the index async after import product from magento's dataflow.
I just tried import products and reindexed all ,not async.

Compatibility to Magento 1.9.2.2 / SUPEE-6788

Admin URL has to be changed, see https://info2.magento.com/rs/318-XBX-392/images/SUPEE-6788-Technical%20Details.pdf.

cli not work

not work with 1.8.0.1?. only change status. does not reale reindexing.
checked on table core_url_rewrite

Database remove

Hello Support,

Can you please tell me where in the database I need te remove the extension?

Or do you have a run code I can use for it ?

Best regards,
Michel

Wrong order of indexers

While integrating a special indexer mode with AvS_FastSimpleImport (see branch feature/asyncindex), I had some issues with products not indexing correctly. The problem was the wrong order of Index Processes. Some processes (i.e. catalogsearch_fulltext) depend on the cataloginventory_stock process having run before (in detail: they need an entry in the db table cataloginventory_stock_status). I don't know yet why, but the core indexer seems to be using a different order (I assume it handles one event after the next, and as the stockItem uses a seperate event, this is processed first).

Partial Index from Shell is not working correctly

When running Script from shell the indexes are refreshed but the status is not correctly saved - it says they "processing"

create some sort of testing

to be sure nothing breaks we need tests, they also will make it easier to refactor and improve better.

the hard point will be the construction of testcases for situations, which could lead to deadlocks like running index process and parallel save of products.
Or what happens if multiple indices run parallel.

Question about initial setup

It is not described in the readme, but just to be sure.
You have to set all indexes to manual, otherwise this extension does not add any value at all.
Correct?

Use massupdate for updating indexes, much faster

In AvS_FastSimpleImport we use the following code to update indexes as fast as possible:
https://github.com/avstudnitz/AvS_FastSimpleImport/blob/master/src/app/code/community/AvS/FastSimpleImport/Model/Import/Entity/Product.php#L476-L532

As far as I can tell, this module runs the update per event and not as a whole, so there is room for improvement.

Async reindex collidates with reindexing from Observers

While running the async module indexes are rebuilt from cron. But e.g. sales_model_service_quote_submit_success Event forces the CatalogInventory to reindex Data as well. This colidates from time to time (depending on the amount of sales and interval for async indexing).

Magento Version where this Bug occurs: 1.9.0.1

Any suggestions ?

make parallel execution possible

should already work during the cron, as only the current executed Index gets a lock, so the others are not locked. Means we could run different Indices at the same time.

Implementation Ideas:

1:
add an additional parameter to the shell script for process code, so its easy as executiong two shell
scripts at the same time.

2:
use threading/forking offered by php extensions to execute all indices in parallel on the same call

Async Index from Backend not working

When clicking Button in Backend the schedule is created successfully and the cron is running afterwards - but the index is not correctly updated.

Make Index on save asyncron

thanks to http://www.php.net/manual/en/function.fastcgi-finish-request.php we are able to execute longer running tasks after actually sending the response.

So it would be possible to hook into the controller_front_send_response_after event, "finish" the request and then execute the index events which happened trough the current save of the request.

Possible downsides are problematic error handling.

Index inconsistent state

This issue may be related to #19

From time to time it happens, that related products "pop out" of the index and therefore are not shown at frontend anymore. To fix, I have to do a manual index of "Category Products".
I have enabled Async-Index with partial-indexing.

Anyone having a similar problem?

Should mention in README

prequisite: Magento 1.8.x and up.

Because the constant Mage_Index_Model_Process::MODE_SCHEDULE was introduced in 1.8 I think.

Indexing takes way too long

I am using asynchindex in a magento 1.9.4.1. Each night an importer pulls thousands (~11k atm) of products from another database using the default magento models (no fancy stuff here).
This has been working nicely until recently when customers started telling us they could not order due to error messages like these:

General error: 1205 Lock wait timeout exceeded; try restarting transaction, query was: SELECT si.*, p.type_idFROMcataloginventory_stock_itemASsiINNER JOINcatalog_product_entityASpON p.entity_id=si.product_id WHERE (stock_id=1) AND (product_id IN(67680, 67850, 74053, 74055, 66988, 66987, 66989, 66990, 94395, 68370, 67769, 67768, 67761, 68372, 67762, 67643, 67642, 68597, 67232, 74325, 67654, 66763, 66761, 66762, 67061, 69664, 66808, 67864)) FOR UPDATE

I must assume that the reason it has been working for a while is because the total amount of products is constantly rising.

I tried to debug and watched the database with "show processlist". The indexer kept working for hours and seemingly slow. On my dev machine it runs a lot faster although it has less power than the live server. So I killed all processes and strightened the index by using the default shell script (php indexer.php --reindexall). This worked fine and all the indexes were rebuild within minutes.

So my question is: why is there such a big difference in the performance? What could I do to get to the core of the issue here?

Thank you

Massaction is only inserted once to index_event

Hi,

I discovered a ... let's say non-intuitive behaviour ... in magento which leads to loosing index-events when using async-index.
Scince massactions naturally don't have a single primary key the key is set to null - this seems to make sense. But when another massaction occurs the event with the corresponding PK is updated instead of inserted again - which can lead to a loss of index-events when this happens right between two asyncindex - runs.
My fix is rather crude: i registered a new indexer and add time() as PK.
In config.xml you define the indexer (inside global):

        <indexer>
            <somecode>
                <model>yourmodelnamespace/indexer</model>
            </somecode>
        </indexer>
    </index>

You add the indexer to the index_process table:

/* @var $this Mage_Core_Model_Resource_Setup /
$installer = new Mage_Catalog_Model_Resource_Setup('yourmodelnamespace');
$installer->startSetup();
try {
/* @var Mage_Index_Model_Process $ip */
$ip = Mage::getModel('index/process');
$ip->setData([
'indexer_code' => 'dt_adminhtml',
'status' => Mage_Index_Model_Process::STATUS_REQUIRE_REINDEX,
'mode' => Mage_Index_Model_Process::MODE_MANUAL
]);
$ip->save();
} catch (Exception $e) {
Mage::logException($e);
}
$installer->endSetup();

You create the new Indexer-Model:

class [Placeholder]_Model_Indexer extends Mage_Index_Model_Indexer_Abstract
{
/**
* @var string[]
/
protected $_matchedEntities = array(
Mage_Catalog_Model_Product::ENTITY => array(
Mage_Index_Model_Event::TYPE_MASS_ACTION,
),
);
/*
* @param Mage_Index_Model_Event $event
* @return $this
/
protected function _registerEvent(Mage_Index_Model_Event $event)
{
if ($event->getEntity() !== Mage_Catalog_Model_Product::ENTITY) {
return $this;
}
if ($event->getType() !== Mage_Index_Model_Event::TYPE_MASS_ACTION) {
return $this;
}
$event->setEntityPk( time() ); //to prevent massactions overriding their index events
return $this;
}
/*
* @param Mage_Index_Model_Event $event
*/
protected function _processEvent(Mage_Index_Model_Event $event)
{
//do nothing, we only want to modify the event that is added
}
public function getName()
{
return 'Admin Massaction fix';
}
public function getDescription()
{
return 'Admin Massaction fix';
}
}

Sorry for the bad formatting

• Tobi

Incompatibility with JeroenVermeulen_Solarium

Hello,

looks like AsyncIndex and Solarium extension from Jeroen Vermeulen have some issues working together.

Automatic reindexing gets stuck when it comes to catalogsearch_fulltext. The process seems to run indefinitely, status is processing and tables get locked. Only server/mysql reboot helps unlocking the tables and running the reindex again.

Unfortunately there is no errors in system.log.

Manually running the indexer for catalogsearch_fulltext works well!

We have disable catalogsearch_fulltext from automatic reindexing for now but it would be great if someone can check this out.

Dependend Indexes

We have a custom indexer which manages the product prices for us, but it must run before "catalog_product_price" so it may be reindexed correctly for magentos index tables.

Here comes the tricky part: it does not work with this module as it does with the default indexer.
In the default magento indexer you could add a xml depend node to your module, so your indexer may run before another indexer:

<index>
            <indexer>
                <abc>
                    <model>custommodule/indexer</model>
                </abc>
                <cataloginventory_stock>
                    <depends>
                        <abc/>
                    </depends>
                </cataloginventory_stock>
            </indexer>
        </index>

This will work if you reindex via default shell "indexer.php" but does not work with AsyncIndex.

I tried to patch it but failed as I am unsure about how the dependend indexes get their info about the events / productids (or whatever) to index.

Lock wait timeout during import of products

We get an error message during import products that is caused by AsyncIndex. While AsyncIndex is disabled there is no error. When it is enabled the error below is shown after ~20 products.

Product: SQLSTATE[HY000]: General error: 1205 Lock wait timeout exceeded; try restarting transaction, query was: UPDATE `index_process` SET `status` = ? WHERE (process_id='2')

#0 /httpdocs/lib/Varien/Db/Statement/Pdo/Mysql.php(110): Zend_Db_Statement_Pdo->_execute(Array) 
#1 /httpdocs/app/code/core/Zend/Db/Statement.php(311): Varien_Db_Statement_Pdo_Mysql->_execute(Array) 
#2 /httpdocs/lib/Zend/Db/Adapter/Abstract.php(480): Zend_Db_Statement->execute(Array) 
#3 /httpdocs/lib/Zend/Db/Adapter/Pdo/Abstract.php(238): Zend_Db_Adapter_Abstract->query('UPDATE `index_p...', Array) 
#4 /httpdocs/lib/Varien/Db/Adapter/Pdo/Mysql.php(504): Zend_Db_Adapter_Pdo_Abstract->query('UPDATE `index_p...', Array) 
#5 /httpdocs/lib/Zend/Db/Adapter/Abstract.php(635): Varien_Db_Adapter_Pdo_Mysql->query('UPDATE `index_p...', Array) 
#6 /httpdocs/app/code/core/Mage/Index/Model/Resource/Process.php(137): Zend_Db_Adapter_Abstract->update('index_process', Array, Array) 
#7 /httpdocs/app/code/core/Mage/Index/Model/Resource/Process.php(124): Mage_Index_Model_Resource_Process->_updateProcessData('2', Array) 
#8 /httpdocs/app/code/core/Mage/Index/Model/Process.php(145): Mage_Index_Model_Resource_Process->updateStatus(Object(Mage_Index_Model_Process), 'require_reindex') 
#9 [internal function]: Mage_Index_Model_Process->register(Object(Mage_Index_Model_Event)) 
#10 /httpdocs/app/code/core/Mage/Index/Model/Indexer.php(378): call_user_func_array(Array, Array) 
#11 /httpdocs/app/code/core/Mage/Index/Model/Indexer.php(258): Mage_Index_Model_Indexer->_runAll('register', Array) 
#12 /httpdocs/app/code/core/Mage/Index/Model/Indexer.php(279): Mage_Index_Model_Indexer->registerEvent(Object(Mage_Index_Model_Event)) 
#13 /httpdocs/app/code/core/Mage/Index/Model/Indexer.php(298): Mage_Index_Model_Indexer->logEvent(Object(Varien_Object), 'catalog_product', 'catalog_reindex...', false) 
#14 /httpdocs/app/code/core/Mage/CatalogRule/Model/Rule.php(356): Mage_Index_Model_Indexer->processEntityAction(Object(Varien_Object), 'catalog_product', 'catalog_reindex...') 
#15 /httpdocs/app/code/core/Mage/CatalogRule/Model/Observer.php(60): Mage_CatalogRule_Model_Rule->applyAllRulesToProduct(Object(MGS_Abrands_Model_Product)) 
#16 /httpdocs/app/code/core/Mage/Core/Model/App.php(1358): Mage_CatalogRule_Model_Observer->applyAllRulesOnProduct(Object(Varien_Event_Observer)) 
#17 /httpdocs/app/code/core/Mage/Core/Model/App.php(1337): Mage_Core_Model_App->_callObserverMethod(Object(Mage_CatalogRule_Model_Observer), 'applyAllRulesOn...', Object(Varien_Event_Observer)) 
#18 /httpdocs/app/Mage.php(448): Mage_Core_Model_App->dispatchEvent('catalog_product...', Array) 
#19 /httpdocs/app/code/core/Mage/Core/Model/Abstract.php(344): Mage::dispatchEvent('catalog_product...', Array) 
#20 /httpdocs/app/code/core/Mage/Catalog/Model/Product.php(2083): Mage_Core_Model_Abstract->afterCommitCallback() 
#21 [internal function]: Mage_Catalog_Model_Product->afterCommitCallback() 
#22 /httpdocs/app/code/core/Mage/Core/Model/Resource/Abstract.php(110): call_user_func(Array) 
#23 /httpdocs/app/code/core/Mage/Core/Model/Abstract.php(322): Mage_Core_Model_Resource_Abstract->commit() 
#24 /httpdocs/app/code/local/Datame/Etl/lib/timeoutControllerEtl/timeoutControllerDestinationMagentoImportApi_Simple.php(455): Mage_Core_Model_Abstract->save() 
#25 /httpdocs/app/code/local/Datame/Etl/lib/timeoutController/timeoutController.php(151): timeoutControllerDestinationMagentoImportApi_Simple->execute(Array) 
#26 /httpdocs/app/code/local/Datame/Etl/Model/Etl.php(191): TimeoutController->execute(Object(TimeoutControllerSourceCsv), Object(timeoutControllerDestinationMagentoImportApi_Simple)) 
#27 /httpdocs/app/code/local/Datame/Etl/Block/Adminhtml/Etl/Import/Tablist.php(38): Datame_Etl_Model_Etl->executeNextStep() 
#28 /httpdocs/app/code/local/Datame/Etl/Block/Adminhtml/Etl/Import/Tablist.php(69): Datame_Etl_Block_Adminhtml_Etl_Import_TabList->execute(Object(stdClass)) 
#29 /httpdocs/app/code/local/Datame/Etl/Block/Adminhtml/Etl/Import/Tablist.php(18): Datame_Etl_Block_Adminhtml_Etl_Import_TabList->getSteps() 
#30 /httpdocs/app/code/core/Mage/Core/Model/Layout.php(491): Datame_Etl_Block_Adminhtml_Etl_Import_TabList->__construct(Array) 
#31 /httpdocs/app/code/core/Mage/Core/Model/Layout.php(437): Mage_Core_Model_Layout->_getBlockInstance('etl/adminhtml_e...', Array) 
#32 /httpdocs/app/code/local/Datame/Etl/controllers/Adminhtml/EtlController.php(176): Mage_Core_Model_Layout->createBlock('etl/adminhtml_e...') 
#33 /httpdocs/app/code/core/Mage/Core/Controller/Varien/Action.php(418): Datame_Etl_Adminhtml_EtlController->importAction() 
#34 /httpdocs/app/code/core/Mage/Core/Controller/Varien/Router/Standard.php(254): Mage_Core_Controller_Varien_Action->dispatch('import') 
#35 /httpdocs/app/code/core/Mage/Core/Controller/Varien/Front.php(172): Mage_Core_Controller_Varien_Router_Standard->match(Object(Mage_Core_Controller_Request_Http)) 
#36 /httpdocs/app/code/core/Mage/Core/Model/App.php(365): Mage_Core_Controller_Varien_Front->dispatch() 
#37 /httpdocs/app/Mage.php(684): Mage_Core_Model_App->run(Array) 
#38 /httpdocs/index.php(86): Mage::run('default', 'store') 
#39 {main}

I think this is related to the status updates. Maybe the updated should only be done when the value actually changed?

if ( count(Mage::getResourceSingleton('index/event')->getUnprocessedEvents($process) ) === 0 && $process->getStatus !== Mage_Index_Model_Process::STATUS_PENDING) {
    $process->changeStatus(Mage_Index_Model_Process::STATUS_PENDING);
}

What should index mode be set to?

Hi. What should the index mode be set to in Index Management? We currently keep everything manual and run indexing on a cron but we're starting to have issues with this in multi-server setup so I'm testing AsynIndex as a replacement. Thanks!

Composer package name in README is wrong

It should be magento-hackathon/async-index