bigbridge-nl / product-import Goto Github PK
View Code? Open in Web Editor NEWFast product import library for Magento 2
License: MIT License
Fast product import library for Magento 2
License: MIT License
Hi!
We have a database with around 2.500.000 products and the importer is getting quite slow. No complaining here, just trying to speed the process up again.
I looked into the Magento2DbConnection.php
class and noticed the echoSlowQueries
feature, which I used to get the following output (I dumped the $values
array as well here):
0.64743208885193:
SELECT A.`attribute_code`, O.`option_id`, V.`value`
FROM eav_attribute A
INNER JOIN eav_attribute_option O ON O.attribute_id = A.attribute_id
INNER JOIN eav_attribute_option_value V ON V.option_id = O.option_id
WHERE A.`entity_type_id` = ? AND A.frontend_input IN ('select', 'multiselect') AND V.store_id = 0
array(1) {
[0]=>
string(1) "4"
}
0.6437680721283:
SELECT A.`attribute_code`, O.`option_id`, V.`value`
FROM eav_attribute A
INNER JOIN eav_attribute_option O ON O.attribute_id = A.attribute_id
INNER JOIN eav_attribute_option_value V ON V.option_id = O.option_id
WHERE A.`entity_type_id` = ? AND A.frontend_input IN ('select', 'multiselect') AND V.store_id = 0
array(1) {
[0]=>
string(1) "4"
}
All other queries are below 0.00 seconds. I can't figure out why these queries are so slow here, because the eav_attribute
tables aren't that big.
Any help is appreciated! π
Hi,
When I'm importing products with a .XML file hardlinks er being created for the images. This causes problems when running backups on the server.
Is there any way to circumvent or fix this?
I import images over HTTPS:
<image file_or_url="https://www.example.nl/media/catalog/product/g/d/gdpr.jpg">
Thanks in advance.
Ivo
Hi,
I've been adding products and just realised I have an issue. I'm pushing products to 4 x Magento2 websites using restapi method.
3 x Magento sites only have a single website/store/storeview and they work as expected.
The 4th website has 3 website/store/storeviews setup and when I use the same import the products aren't assigned to any of the websites.
I currently haven't included the website_codes element in the XML but all the 3 single website sites are assigning correctly, I expected the 4th to assign to the main base but it does not.
Is this a bug or feature to assign automatically to single website stores?
Thanks in advance.
Hey, just wanted to toss out a quick issue in regards to importing categories. The categories are being created one level too high. The script creates categories one level too high due to the parentId of the first level being set too high (it's setting to level 3 it should be level 2).
I resolved it by changing line 253 from file ../Model/Resource/Resolver/CategoryImporter.php
$parentLevel = count($idPath);
to
$parentLevel = count($idPath) - 1;
I compared my results with a category I created using the admin page with "Default Category" being the parent category and it starts the new category at level 2.
This was an issue for anyone using ElasticSuite, it wasn't properly allowing us to navigate through category filters.
Hi Patrick,
Sorry to bother you again. I just came across this issue. When I've imported a product and I try to edit an attribute value (in my case 'Search Engine Optimization ' -> 'Meta Description') on the storeview level and hit 'Save' the product on the storeview level gets all the same values as the global attributes instead of it's own distinct values.
I've included a simple .XML that I use for my import. It consists of one product.
Screenshot before saving:
Screenshot after saving:
Maybe you can help me out. Thanks in advance.
Hi,
I was looking at a way for your module to clear Cache of a product if a change it processed for it.
I found this info that details a procedure to clear Cache for a product ID and to purge varnish.
https://stackoverflow.com/questions/42627051/how-do-i-clear-the-cache-in-magento-2-for-a-single-product-id
At the moment if I automate a change it can take up to 24 hours to show if the product page is already cached and I don't like clearing the whole cache.
When installing i got following error. Will the package be made compatible with PHP 8?
[InvalidArgumentException]
Package bigbridge/product-import has requirements incompatible with your PHP version, PHP extensions and Compos
er version:
- bigbridge/product-import 1.7.5 requires php ~7.0 which does not match your installed version 8.1.0.
Hi,
First of all. Great work on this import tool.
I'm trying to run an import with a set of five products. Two products share the same url-key. This gives a error during the import. I came across the --url-key-strategy option. This should be able to fix these errors during import right? I've tried a lot of combinations where I want to use the 'add-sku' option. I couldn't get it to work. Can you give me a clear example of how to use this?
Right now I'm running something like:
bin/magento bigbridge:product:import --url-key-strategy add-sku --auto-create-categories --dry-run test.xml
And I've tried a lot of other possibilities :)
Thanks in advance.
Hi Patrick! Awesome plugin btw.
When I do:
<store_view code="lt_spare_1">
<delete code="status"/>
</store_view>
<store_view code="lt_spare_2">
<delete code="status"/>
</store_view>
I get an error:
Error in line 10: Element 'lt_spare_2': This element is not expected.
Hi,
I have to clear all products.
I've followed sql script directly from https://www.titechglobal.com/magento-2-x-how-to-clear-test-data-before-production/ using only the first one concerning "Clearing all products".
Since then, using importing library does not work anymore. I've got :
PHP Fatal error: Uncaught Magento\Framework\Exception\RuntimeException: Type Error occurred when creating object: BigBridge\ProductImport\Model\Resource\MetaData in /var/www/vhosts/xxx/httpdocs/xxx/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php:124
I've restored a backup.
Using importing library works properly.
Then, I execute SQL
And then, importing library is not working anymore.
Did I miss something ?
Thanks for you help.
Hi,
Thanks for the great work you have done with the import module. I have it up and working as expected other than 2 issues moving forward based on how it used to work with Magmi.
Default Category/Car Parts/Brakes/Seals \/ Repair Parts
Thanks
Hello,
I'm pretty new to Magento.
Could you help me to use the programming library to import products ?
I've installed the module and run CLI successfully.
Could you give us more details on how to get started with the programming library ?
Could you give us a complete php file as an example ?
Thanks for you help.
Greetings
JeanClaude
The method setOptionTitle
on global Bundle is not working properly, the option title is empty.
Code example:
$global->setOptionTitle($option, 'capacity');
After further investigation, I have notice that the parent_product_id
column of catalog_product_bundle_option_value
table is set to 0 instead the bundle entity_id.
In createOptions
method of class BigBridge\ProductImport\Model\Resource\Storage\BundleStorage
I have solved by adding this:
original:
foreach ($product->getStoreViews() as $storeView) {
foreach ($storeView->getOptionInformations() as $optionInformation) {
$this->db->execute("
INSERT INTO `{$this->metaData->bundleOptionValueTable}`
SET
`option_id` = ?,
`store_id` = ?,
`title` = ?
", [
$optionInformation->getOption()->id,
$storeView->getStoreViewId(),
$optionInformation->getTitle()
]);
}
}
changed:
foreach ($product->getStoreViews() as $storeView) {
foreach ($storeView->getOptionInformations() as $optionInformation) {
$this->db->execute("
INSERT INTO `{$this->metaData->bundleOptionValueTable}`
SET
`option_id` = ?,
`store_id` = ?,
`title` = ?,
`parent_product_id` = ?
", [
$optionInformation->getOption()->id,
$storeView->getStoreViewId(),
$optionInformation->getTitle(),
$product->id,
]);
}
}
Hi,
I've just come across an issue doing a scan of my site using screamingfrog.
Basically, when I create products I manually set the url_key attribute and the product creates/works as expected. If I revise the product with a new url_key the attribute in the database is updated but the attribute url_path retains the old value.
This is causing some strange behaviour where the canonical URL is correct and the page loads on the correct URL, but the href in the page for the scroll to review, along with the 3rd party HTML sitemap and richsnippets are showing the old URL.
I can see there is mention of the url_path attribute in the CategoryImporter.php file but nothing in any file relating to Product import, is this an oversight? Do you have a solution?
I've used this extension for generating URL keys in the past when I used magmi (https://github.com/olegkoval/magento2-regenerate_url_rewrites) and there approach appears to be set it to null before saving the product (https://github.com/olegkoval/magento2-regenerate_url_rewrites/blob/148378eb7cbd98807d1c86e6a1262763bbf77944/Model/RegenerateProductRewrites.php#L180)
Not sure if that is of any use.
Thanks in advance.
Hi,
is it possible to import products from an php array?
I use Big Bridge product import to create configurable products. Everything works as expected. But when I try to save a configurable product, I get below prompt. I does not matter if I have changed any data or not. After that, it does not matter which option I choose, I allways get the same prompt again when I retry to save the product after choosing a prompt before.
I am not sure what is going wrong. Any idea's?
Hello,
I'm using the importer library with the help of #27 (comment) and it works like a charm.
But I don't know how to set the default stock so that salable quantity is right.
I'm doing something like that :
$product->defaultStockItem()->setQty($quantity);
$product->defaultStockItem()->setIsInStock(true);
$product->sourceItem("default")->setQuantity($quantity);
$product->sourceItem("default")->setStatus(1); // in stock
$product->sourceItem("default")->setNotifyStockQuantity(20);
In Magento dashboard / catalog / products, the column Quantity
is correctly set, but the salable quantity indicates Default stock: 0
. If I open the product and save it, the salable quantity is calculated the right way.
Did I miss something or there is an issue ?
Thanks for you help.
The product importer does not seems to work properly on Magento 2.4.1. Products are not created. Is there 2.4.1 compatibility yet?
Hi guys,
module version: 1.7.4
magento version: 2.4.2-p2
Sometimes we get these errors when using the import module:
[2021-10-06 12:26:21] main.CRITICAL: : failed! error = generated url key already exists: pharmalens-monthly-6-50-3; generated url key already exists: pharmalens-monthly-6-50-3; generated url key already exists: pharmalens-monthly-6-50-3
We always use the following logic to add a url key to the product
$global->generateUrlKey();
for global level
$productStoreView->generateUrlKey();
for every storeview.
Do you might have an idea what's going wrong here?
Thx in advance!
Hello, first of all I want to thank you for creating this library that is really useful, it improves performance considerably
My issue is the following, if I create an attribute as Text Swatch it only fills the description and not the Swatch field, then in the front it is always showing a null value but aria-label shows what is in the description field
Thanks for you help.
Hi guys,
I'm using this module to run a product import with only simple products and ran into the following issue:
Trying to get property 'path' of non-object in /var/www/store1/magento2/releases/20210416100919/vendor/bigbridge/product-import/Model/Resource/Storage/UrlRewriteStorage.php on line 376 [] []
Any idea what might be wrong?
Thx in advance!
Magento 2.4.2
bigbridge/product-import 1.7.0
Could you give working example for a-configurable.xml in example folder? I can add the product, and its visible only in admin area. Not visible in front page. I am suspecting example xml file is missing some attributes. Could you please take a look into this?
Hi,
This might be more of a feature request, where I'm not even sure if it's a good idea to do this in your importer, but maybe it would be also helpful for someone else.
I'm setting a custom_publication_date
variable based on 3 different date values, which I get from my import file. For example, if I only get publication_year
, I set the custom_publication_date
to 31.12.XXXX
, but when I also get a publication_month
the value should be 31.XX.XXXX
.
Now that works when creating new products, as I just set the value according to the given values.
But if there's an update for an existing product and I read my import file and notice I delete the publication_month
for example, I must update my custom_publication_date
value as well. The problem is I now don't have access to the previous given publication_year
value now, which the value should be based on.
I was thinking about something like $product->fetchAttributeData("publication_year")
where we then only fetch this data from the database. Of course this would probably slow down the import a lot, but with great responsibility... π
If you think this is a good idea, let me know as I would then start with the pull request.
An alternative for me would of course be a cron job to set the custom_publication_date
value based on the current other attribute values.
Regards,
Alex
So it seems I'm having an issue getting my category rewrites to work. I am running a brand new installation of Magento 2.3.4 and the first thing I did was install this extension. I am populating products just fine, they are showing up using the url_key.
The categories work using the "https://www.example.com/catalog/category/view/s/denim-jeans/id/16/". Although, when I attempt to access the category via the url_key/request_path such as "https://www.example.com/denim-jeans" it does not work.
In the url_rewrites table, categories are being generated for store_id 0 and products are being generated for store_id 1.
My solution, if I change my url category rewrites from store_id 0 to 1 from the url_rewrites table, the categories work. But it does not seem like the appropriate solution to constantly update the store_id to 1.
I've spent days researching prior to posting and just needed some clarification. My understanding is that if the only categories are store_id = 0, then it is the default category for all stores. I am only running the default store that came with my installation.
Hi,
I am facing in an issue.
When importing a value for a custom attribute it throws the following error. The attribute is a dropdown.
color is not an integer (blue) for product 'test' that starts in line 3
I am expecting the importer to just insert the value "blue". But it is not. Am i doing something wrong?
Hi,
Is it possible to export attribute values from categories like meta description, meta title and meta keywords?
Thanks in advance.
Ivo
Hi!
The code I find over here for fetching the configuration for the product url suffix, is not correct because it doesn't take all the ways into consideration for how we can determine configurations in Magento.
As far as I'm aware, the following things are checked in Magento do determine the value of a configuration setting, from highest prio to lowest:
app/etc/env.php
fileapp/etc/config.php
fileCurrently your code only supports the last two as far as I can see.
Is there a good reason why you aren't using the ScopeConfigInterface
to fetch configuration values? It supports all those scenario's I've listed above and does the right thing.
For example.
We have no value for catalog/seo/product_url_suffix
in our database, but our app/etc/config.php
file contains the following:
<?php
return [
'modules' => [
// ...
],
'system' => [
'default' => [
'catalog' => [
'seo' => [
'product_url_suffix' => '',
]
]
]
]
];
However, your module takes .html
as suffix (which is its default value in the config.xml
file of a module) instead of an empty value like we would expect it.
I would strongly recommend you to switch to using the ScopeConfigInterface
from core Magento for fetching these kind of config values, so all config values you fetch work as expected.
Thanks for considering this! π
How to generate a xml file???
Hello.
I'am trying to use your importer for periodically mass updating our about 16000 Products.
Basically, i use this code (sku is already in use):
$product = new \BigBridge\ProductImport\Api\Data\SimpleProduct($sku);
$this->importer->importSimpleProduct($product);
$this->importer->flush();
This results in:
Could you give me a hint, whats wrong here?
Magento Version: 2.3 Commerce
Hi, i have problem with set is_in stock. If I have 1 in value. Magento switch is_in_stock, but if I have 0 value Magento don't switch to out of stock.
<simple sku="xxxxxxxxx">
<stock>
<qty> =0</qty>
<manage_stock>1</manage_stock>
<is_in_stock>0</is_in_stock>
</stock>
</simple>
Hi,
First of all, thank you for sharing your great work!
So I used your importer to create a product within the global
store view and the default
store view. Unfortunately we now need to reset all the data from default
to use the default global
values.
Here is my code:
$global = $product->global();
$global->setName($this->_productData['name']);
$default = $product->storeView('default');
$default->setName($this->_productData['name']);
$importer->importSimpleProduct($product);
$importer->flush();
Here is what I tried so far, but without success:
$default->setName(NULL);
Is this even possible?
Does the module provide a possibility to 'replace' categories that already are assigned to a product. For example:
PS. Import works like a charm, big thanks!
Hi,
I've just come across an issue regarding my category URL structure.
Previously when creating/revising products with Magmi, and in combination with using Mirasvit SEO I didn't have this issue. Within the SEO extension, there is a setting to allow a flat URL structure for categories.
So previously I would have;
Top Level Category/Child Category/Child Child Category
The URLs would be;
domain.com/top-level-category
domain.com/child-category
domain.com/child-child-category
But since using the module to create/update it is now using default structure again.
domain.com/top-level-category
domain.com/top-level-category/child-category
domain.com/top-level-category/child-category/child-child-category
I appreciate that this is not default functionality, but would it be possible to add an attribute to the <category_global_names>
XML, for example <category_global_names structure="flat">
Another concern is that when using Magmi if a category already existed then no changes were made to it, so for example.
If I created a product with Magmi and it had a category called 'Special Offers', it would then create a url_key called special-offers. Now if I went into Magento and edited the key to say clearance-items and clicked save I would have a category called 'Special Offers' accessible on domain.com/clearance-items.
If I then added more items to the 'Special Offers' category the url_key would remain as the updated version. But my issue here is that looking at the behaviour and code it appears you reset the category url_key, url_path and rewrite every time the category is passed.
Maybe it could be an option to pass a custom url_key to a category also so it doesn't need to reflect the one generated from the name?
Can you give me any example of how to do that?
Can you also provide any example feed that i can import to test with my Magento store?
Can you also specify how to add dynamic xml payload to the post request sent to the server endpoint?
Hello,
I'm trying to use the importer to set the Out-of-Stock Threshold
field. Actually, I'm trying to set a negative value per product.
I know there is something like :
$stock = $product->defaultStockItem();
$stock->setBackorders(ProductStockItem::BACKORDERS_ALLOW_QTY_BELOW_0);
$stock->setUseConfigBackorders(false);
But I didn't see anything concerning this special use case.
Did I miss something or could you indicate me how to accomplish this ?
By the way, what does $stock->setUseConfigBackorders
exactly do ?
Thanks in advance for you help.
Hi guys!
As a best practise, we delete the vendor/magento/magento2-base
module when deploying a Magento shop to a server.
The code in this module is not used when running a Magento shop, its only purpose is to copy files inside it to the root of the project using a composer plugin. But when that is finished, it serves no longer a purpose anymore. So we remove this module always in order to save disk space.
However, it looks like your module relies on the composer.json
file to exist in vendor/magento/magento2-base
to figure out the current Magento version, this is done over here:
product-import/Model/Resource/MetaData.php
Line 422 in 9420451
Is there a chance you could try to revert back to using $productMetadata->getVersion()
as mentioned in the comments. It is slower indeed, but Magento seems to have fixed some slowness that was reported. This should be faster now in Magento 2.4.0 and higher by magento/magento2#24030 and magento/magento2#26001.
Would you consider trying to remove the reliance on the vendor/magento/magento2-base
module from this module? Or maybe add a fallback for when that vendor/magento/magento2-base/composer.json
file is not found to still use $productMetadata->getVersion()
? That would be appreciated π
Thanks!
Hello,
using this code the tier price is not set. If i omit the website code it works, setting the tier price for all the websites.
$pr = new SimpleProduct($sku);
$pr->setTierPrices([
new TierPrice(1,$price, 'my_customers_group', 'my_website')
]);
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.