bigbridge-nl/product-import

Fast product import library for Magento 2

BigBridge Product Import

Hi! My name is Patrick van Bergen. I created this library because I needed product import to be fast.

This library imports product data into Magento 2 via direct database queries. It has these main features

• A programming library to import products
• An CLI command for product import via XML file
• A web API service for product import via Post Request with XML
• A tool to update url_rewrites

Note

The aim of this library is speed. If you find that Magento 2's product importer is too slow, consider using this library.

Experiment with the library in a safe webshop. Make sure to create a database backup before you start.

Sample code

Let me show you a piece of code to give you an impression what it is like to use this library:

    $importer = $factory->createImporter($config);

    foreach ($lines as $i => $line) {

        $product = new SimpleProduct($line[1]);

        // global eav attributes
        $global = $product->global();
        $global->setName($line[0]);
        $global->setPrice($line[2]);

        // German eav attributes
        $german = $product->storeView('de_store');
        $german->setName($line[3]);
        $german->setPrice($line[4]);

        $importer->importSimpleProduct($product);
    }

    // process any remaining products in the pipeline
    $importer->flush();

Continue to read the full documentation

Installation

composer require bigbridge/product-import

Requirements

• For Magento 2.1+ Opensource Edition
• Requires >= PHP 7.0
• Input in UTF-8 (Magento standard)
• Unix family system

Features of the Import Library

• insert, update and delete products
• product identification based on product sku or id
• all product types (simple, configurable, grouped, bundle, downloadable, and virtual)
• automatic category generation
• automatic select and multiselect attribute option creation
• import of images from file or url
• image caching (local and HTTP cache)
• custom options
• unique url_key generation
• url_rewrite creation
• whitespace trimming (spaces, tabs, newlines) from all fields, except free field texts
• attribute deletion
• input validation (data type, requiredness, and length restrictions)
• product type changes
• importing links to products have not been imported yet
• dry run (no products are written to the database)
• multi-source inventory (msi)
• weee taxes
• removes the url_path attribute of imported products
• M2EPro notification

Continue to read about all importer features

XML file import tool

The XML import tool allows you to import product data with an XML file. It is fast and has a fixed, low memory footprint.

    bin/magento bigbridge:product:import

Continue to read about XML file import

Web API import service

The web api service performs the same service as the file import tool. But it is accessible via an XML REST call.

/rest/V1/bigbridge/products

It is a POST call and the XML is passed in the request body.

Continue to read about XML webapi import

The url_rewrite tool

Since Magento's url_rewrite table can get corrupted in many ways, it is necessary to have a tool to fix it.

This tool has the following features:

• Fast, it writes queries directly.
• It imposes no downtime for the webshop
• It updates the url_rewrite and catalog_url_rewrite_product_category per product
• It respects Magento's configuration setting "Create Permanent Redirect for URLs if URL Key Changed"
• It does not delete existing 301 url_rewrite redirects
• It does not overwrite existing non-product rewrites
• Updating single store views is possible
• Products with a visibility of "not-visible-individually" get no url_rewrite (since they would have no use; this is Magento 2 policy)

    bin/magento bigbridge:product:urlrewrite

Continue to read about the Url Rewrite Tool

Changes to Magento

The extension adds an index CATALOG_PRODUCT_ENTITY_VARCHAR_ATTRIBUTE_ID_VALUE to catalog_product_entity_varchar because it drastically speeds up checking for duplicate url_keys.

Thanks to

Thanks to Marco de Vries for telling me about the intricacies of product import.

Thanks to Martijn van Berkel for first volunteering to use the importer in production environments and for providing valuable feedback.

This project owes a great deal of ideas and inspiration from Magmi / Magento 1 Magmi

product-import's Issues

Generated Url key already exists

Hi guys,

Prerequisites

module version: 1.7.4
magento version: 2.4.2-p2

Description

Sometimes we get these errors when using the import module:

[2021-10-06 12:26:21] main.CRITICAL: : failed! error = generated url key already exists: pharmalens-monthly-6-50-3; generated url key already exists: pharmalens-monthly-6-50-3; generated url key already exists: pharmalens-monthly-6-50-3

We always use the following logic to add a url key to the product

$global->generateUrlKey(); for global level
$productStoreView->generateUrlKey(); for every storeview.

Do you might have an idea what's going wrong here?

Thx in advance!

How to use the programming library to import products

Hello,

I'm pretty new to Magento.
Could you help me to use the programming library to import products ?

I've installed the module and run CLI successfully.
Could you give us more details on how to get started with the programming library ?
Could you give us a complete php file as an example ?

Thanks for you help.
Greetings
JeanClaude

Image import creates hardlinks

Hi,

When I'm importing products with a .XML file hardlinks er being created for the images. This causes problems when running backups on the server.

Is there any way to circumvent or fix this?

I import images over HTTPS:

<image file_or_url="https://www.example.nl/media/catalog/product/g/d/gdpr.jpg">

Thanks in advance.

Ivo

Can images be replaced and not appended to? Category separator escaping?

Hi,

Thanks for the great work you have done with the import module. I have it up and working as expected other than 2 issues moving forward based on how it used to work with Magmi.

1. When I used to import products with Magmi it had the option to either overwrite images or append? Your module appears to only append and I need it to replace images. Would this be something that could be added?
2. In Magmi you can escape a slash in the category name, for example;
   Default Category/Car Parts/Brakes/Seals \/ Repair Parts
   This worked great, is there anyway to do this, at the moment it added a category called 'Seals \ ' with another child called 'Repair Parts'

Thanks

Magento 2.4.1 support

The product importer does not seems to work properly on Magento 2.4.1. Products are not created. Is there 2.4.1 compatibility yet?

Module shouldn't rely on magento2-base package to exist

Hi guys!

As a best practise, we delete the vendor/magento/magento2-base module when deploying a Magento shop to a server.
The code in this module is not used when running a Magento shop, its only purpose is to copy files inside it to the root of the project using a composer plugin. But when that is finished, it serves no longer a purpose anymore. So we remove this module always in order to save disk space.

However, it looks like your module relies on the composer.json file to exist in vendor/magento/magento2-base to figure out the current Magento version, this is done over here:

product-import/Model/Resource/MetaData.php

Line 422 in 9420451

file_get_contents(BP . '/vendor/magento/magento2-base/composer.json'), $matches)) {

Is there a chance you could try to revert back to using $productMetadata->getVersion() as mentioned in the comments. It is slower indeed, but Magento seems to have fixed some slowness that was reported. This should be faster now in Magento 2.4.0 and higher by magento/magento2#24030 and magento/magento2#26001.

Would you consider trying to remove the reliance on the vendor/magento/magento2-base module from this module? Or maybe add a fallback for when that vendor/magento/magento2-base/composer.json file is not found to still use $productMetadata->getVersion()? That would be appreciated 🙂

Thanks!

Bundle setOptionTitle empty

The method setOptionTitle on global Bundle is not working properly, the option title is empty.

Code example:

$global->setOptionTitle($option, 'capacity');

After further investigation, I have notice that the parent_product_id column of catalog_product_bundle_option_value table is set to 0 instead the bundle entity_id.

In createOptions method of class BigBridge\ProductImport\Model\Resource\Storage\BundleStorage I have solved by adding this:
original:

foreach ($product->getStoreViews() as $storeView) {
	foreach ($storeView->getOptionInformations() as $optionInformation) {

		$this->db->execute("
			INSERT INTO `{$this->metaData->bundleOptionValueTable}`
			SET
				`option_id` = ?,
				`store_id` = ?,
				`title` = ?
		", [
			$optionInformation->getOption()->id,
			$storeView->getStoreViewId(),
			$optionInformation->getTitle()
		]);
	}
}

changed:

foreach ($product->getStoreViews() as $storeView) {
	foreach ($storeView->getOptionInformations() as $optionInformation) {

		$this->db->execute("
			INSERT INTO `{$this->metaData->bundleOptionValueTable}`
			SET
				`option_id` = ?,
				`store_id` = ?,
				`title` = ?,
				`parent_product_id` = ?
		", [
			$optionInformation->getOption()->id,
			$storeView->getStoreViewId(),
			$optionInformation->getTitle(),
			$product->id,
		]);
	}
}

How to generate a xml file???

How to generate a xml file???

Issue with category levels

Hey, just wanted to toss out a quick issue in regards to importing categories. The categories are being created one level too high. The script creates categories one level too high due to the parentId of the first level being set too high (it's setting to level 3 it should be level 2).

I resolved it by changing line 253 from file ../Model/Resource/Resolver/CategoryImporter.php

$parentLevel = count($idPath);

to

$parentLevel = count($idPath) - 1;

I compared my results with a category I created using the admin page with "Default Category" being the parent category and it starts the new category at level 2.

This was an issue for anyone using ElasticSuite, it wasn't properly allowing us to navigate through category filters.

Category flat url structure

Hi,

I've just come across an issue regarding my category URL structure.

Previously when creating/revising products with Magmi, and in combination with using Mirasvit SEO I didn't have this issue. Within the SEO extension, there is a setting to allow a flat URL structure for categories.

So previously I would have;
Top Level Category/Child Category/Child Child Category

The URLs would be;
domain.com/top-level-category
domain.com/child-category
domain.com/child-child-category

But since using the module to create/update it is now using default structure again.
domain.com/top-level-category
domain.com/top-level-category/child-category
domain.com/top-level-category/child-category/child-child-category

I appreciate that this is not default functionality, but would it be possible to add an attribute to the <category_global_names> XML, for example <category_global_names structure="flat">

Another concern is that when using Magmi if a category already existed then no changes were made to it, so for example.

If I created a product with Magmi and it had a category called 'Special Offers', it would then create a url_key called special-offers. Now if I went into Magento and edited the key to say clearance-items and clicked save I would have a category called 'Special Offers' accessible on domain.com/clearance-items.

If I then added more items to the 'Special Offers' category the url_key would remain as the updated version. But my issue here is that looking at the behaviour and code it appears you reset the category url_key, url_path and rewrite every time the category is passed.

Maybe it could be an option to pass a custom url_key to a category also so it doesn't need to reflect the one generated from the name?

How to use --url-key-strategy.

Hi,

First of all. Great work on this import tool.

I'm trying to run an import with a set of five products. Two products share the same url-key. This gives a error during the import. I came across the --url-key-strategy option. This should be able to fix these errors during import right? I've tried a lot of combinations where I want to use the 'add-sku' option. I couldn't get it to work. Can you give me a clear example of how to use this?

Right now I'm running something like:

bin/magento bigbridge:product:import --url-key-strategy add-sku --auto-create-categories --dry-run test.xml

And I've tried a lot of other possibilities :)

Thanks in advance.

Fetching Magento configurations isn't working as expected

Hi!

The code I find over here for fetching the configuration for the product url suffix, is not correct because it doesn't take all the ways into consideration for how we can determine configurations in Magento.

As far as I'm aware, the following things are checked in Magento do determine the value of a configuration setting, from highest prio to lowest:

• environment variables
• setting defined in the app/etc/env.php file
• setting defined in the app/etc/config.php file
• setting stored in the database
• setting's default value

Currently your code only supports the last two as far as I can see.
Is there a good reason why you aren't using the ScopeConfigInterface to fetch configuration values? It supports all those scenario's I've listed above and does the right thing.

For example.
We have no value for catalog/seo/product_url_suffix in our database, but our app/etc/config.php file contains the following:

<?php

return [
    'modules' => [
        // ...
    ],
    'system' => [
        'default' => [
            'catalog' => [
                'seo' => [
                    'product_url_suffix' => '',
                ]
            ]
        ]
    ]
];

However, your module takes .html as suffix (which is its default value in the config.xml file of a module) instead of an empty value like we would expect it.

I would strongly recommend you to switch to using the ScopeConfigInterface from core Magento for fetching these kind of config values, so all config values you fetch work as expected.

Thanks for considering this! 🙂

Export category attribute values

Hi,

Is it possible to export attribute values from categories like meta description, meta title and meta keywords?

Thanks in advance.

Ivo

Bug importing to 2 different store views

Hi Patrick! Awesome plugin btw.

When I do:

        <store_view code="lt_spare_1">
            <delete code="status"/>
        </store_view>

        <store_view code="lt_spare_2">
            <delete code="status"/>
        </store_view>

I get an error:

Error in line 10: Element 'lt_spare_2': This element is not expected.

Two slow queries detected

Hi!

We have a database with around 2.500.000 products and the importer is getting quite slow. No complaining here, just trying to speed the process up again. ☺️
I looked into the Magento2DbConnection.php class and noticed the echoSlowQueries feature, which I used to get the following output (I dumped the $values array as well here):

0.64743208885193:
            SELECT A.`attribute_code`, O.`option_id`, V.`value`
            FROM eav_attribute A
            INNER JOIN eav_attribute_option O ON O.attribute_id = A.attribute_id
            INNER JOIN eav_attribute_option_value V ON V.option_id = O.option_id
            WHERE A.`entity_type_id` = ? AND A.frontend_input IN ('select', 'multiselect') AND V.store_id = 0

array(1) {
  [0]=>
  string(1) "4"
}
0.6437680721283:
            SELECT A.`attribute_code`, O.`option_id`, V.`value`
            FROM eav_attribute A
            INNER JOIN eav_attribute_option O ON O.attribute_id = A.attribute_id
            INNER JOIN eav_attribute_option_value V ON V.option_id = O.option_id
            WHERE A.`entity_type_id` = ? AND A.frontend_input IN ('select', 'multiselect') AND V.store_id = 0

array(1) {
  [0]=>
  string(1) "4"
}

All other queries are below 0.00 seconds. I can't figure out why these queries are so slow here, because the eav_attribute tables aren't that big.

Any help is appreciated! 👍

Please provide example of how to use post method with webapi?

Can you give me any example of how to do that?
Can you also provide any example feed that i can import to test with my Magento store?
Can you also specify how to add dynamic xml payload to the post request sent to the server endpoint?

Saving product on storeview scope overwrites storeview attribute values with global values

Hi Patrick,

Sorry to bother you again. I just came across this issue. When I've imported a product and I try to edit an attribute value (in my case 'Search Engine Optimization ' -> 'Meta Description') on the storeview level and hit 'Save' the product on the storeview level gets all the same values as the global attributes instead of it's own distinct values.

I've included a simple .XML that I use for my import. It consists of one product.

exportProducts.txt

Screenshot before saving:

Screenshot after saving:

Maybe you can help me out. Thanks in advance.

website_codes restapi

Hi,

I've been adding products and just realised I have an issue. I'm pushing products to 4 x Magento2 websites using restapi method.
3 x Magento sites only have a single website/store/storeview and they work as expected.
The 4th website has 3 website/store/storeviews setup and when I use the same import the products aren't assigned to any of the websites.

I currently haven't included the website_codes element in the XML but all the 3 single website sites are assigning correctly, I expected the 4th to assign to the main base but it does not.

Is this a bug or feature to assign automatically to single website stores?

Thanks in advance.

Choose Affected Attribute Set prompt when saving configurable product

I use Big Bridge product import to create configurable products. Everything works as expected. But when I try to save a configurable product, I get below prompt. I does not matter if I have changed any data or not. After that, it does not matter which option I choose, I allways get the same prompt again when I retry to save the product after choosing a prompt before.

I am not sure what is going wrong. Any idea's?

The setSelectAttribute method does not fill swatch input on Text Swatch

Hello, first of all I want to thank you for creating this library that is really useful, it improves performance considerably

My issue is the following, if I create an attribute as Text Swatch it only fills the description and not the Swatch field, then in the front it is always showing a null value but aria-label shows what is in the description field

Thanks for you help.

How to set out of stock threshold for backorders ?

Hello,

I'm trying to use the importer to set the Out-of-Stock Threshold field. Actually, I'm trying to set a negative value per product.
I know there is something like :

$stock = $product->defaultStockItem();
$stock->setBackorders(ProductStockItem::BACKORDERS_ALLOW_QTY_BELOW_0);
$stock->setUseConfigBackorders(false);

But I didn't see anything concerning this special use case.
Did I miss something or could you indicate me how to accomplish this ?

By the way, what does $stock->setUseConfigBackorders exactly do ?

Thanks in advance for you help.

Can't set is_in_stock = 0

Hi, i have problem with set is_in stock. If I have 1 in value. Magento switch is_in_stock, but if I have 0 value Magento don't switch to out of stock.

<simple sku="xxxxxxxxx">
            <stock>
                    <qty> =0</qty>
                    <manage_stock>1</manage_stock>
                   <is_in_stock>0</is_in_stock>
            </stock>
</simple>

Removing select attribute

Hi again!

I want to clear out a custom select attribute. I tried two things:
$global->setCustomAttribute("report_number", null); and $global->setSelectAttribute("report_number", null);.

I somehow think I already this this with the first approach some time ago, did this change maybe?

Regards,
Alex

Reset product data to use default values

Hi,

First of all, thank you for sharing your great work!

So I used your importer to create a product within the global store view and the default store view. Unfortunately we now need to reset all the data from default to use the default global values.

Here is my code:

$global = $product->global();
$global->setName($this->_productData['name']);

$default = $product->storeView('default');
$default->setName($this->_productData['name']);

$importer->importSimpleProduct($product);
$importer->flush();

Here is what I tried so far, but without success:

$default->setName(NULL);

Is this even possible?

Clear Cache after change per product ID

Hi,

I was looking at a way for your module to clear Cache of a product if a change it processed for it.

I found this info that details a procedure to clear Cache for a product ID and to purge varnish.
https://stackoverflow.com/questions/42627051/how-do-i-clear-the-cache-in-magento-2-for-a-single-product-id

At the moment if I automate a change it can take up to 24 hours to show if the product page is already cached and I don't like clearing the whole cache.

Get current product attribute before updating

Hi,

This might be more of a feature request, where I'm not even sure if it's a good idea to do this in your importer, but maybe it would be also helpful for someone else.

I'm setting a custom_publication_date variable based on 3 different date values, which I get from my import file. For example, if I only get publication_year, I set the custom_publication_date to 31.12.XXXX, but when I also get a publication_month the value should be 31.XX.XXXX.
Now that works when creating new products, as I just set the value according to the given values.

But if there's an update for an existing product and I read my import file and notice I delete the publication_month for example, I must update my custom_publication_date value as well. The problem is I now don't have access to the previous given publication_year value now, which the value should be based on.

I was thinking about something like $product->fetchAttributeData("publication_year") where we then only fetch this data from the database. Of course this would probably slow down the import a lot, but with great responsibility... 😁

If you think this is a good idea, let me know as I would then start with the pull request.
An alternative for me would of course be a cron job to set the custom_publication_date value based on the current other attribute values.

Regards,
Alex

Updating products results in database error

Hello.
I'am trying to use your importer for periodically mass updating our about 16000 Products.

Basically, i use this code (sku is already in use):
$product = new \BigBridge\ProductImport\Api\Data\SimpleProduct($sku);
$this->importer->importSimpleProduct($product);
$this->importer->flush();

This results in:

• SQLSTATE[42S22]: Column not found: 1054 Unknown column 'entity_id' in 'field list'
• duplicate entry in the database table catalog_product_entity (row_id and dates are different from the original entry)

Could you give me a hint, whats wrong here?

Magento Version: 2.3 Commerce

How to set salable quantity ?

Hello,

I'm using the importer library with the help of #27 (comment) and it works like a charm.

But I don't know how to set the default stock so that salable quantity is right.
I'm doing something like that :

$product->defaultStockItem()->setQty($quantity);
$product->defaultStockItem()->setIsInStock(true);
$product->sourceItem("default")->setQuantity($quantity);
$product->sourceItem("default")->setStatus(1); // in stock
$product->sourceItem("default")->setNotifyStockQuantity(20);

In Magento dashboard / catalog / products, the column Quantity is correctly set, but the salable quantity indicates Default stock: 0. If I open the product and save it, the salable quantity is calculated the right way.

Did I miss something or there is an issue ?

Thanks for you help.

Importing library fails after clearing all products

Hi,

I have to clear all products.
I've followed sql script directly from https://www.titechglobal.com/magento-2-x-how-to-clear-test-data-before-production/ using only the first one concerning "Clearing all products".

Since then, using importing library does not work anymore. I've got :

PHP Fatal error:  Uncaught Magento\Framework\Exception\RuntimeException: Type Error occurred when creating object: BigBridge\ProductImport\Model\Resource\MetaData in /var/www/vhosts/xxx/httpdocs/xxx/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php:124

I've restored a backup.
Using importing library works properly.
Then, I execute SQL
And then, importing library is not working anymore.

Did I miss something ?
Thanks for you help.

Url rewrites not working for categories

So it seems I'm having an issue getting my category rewrites to work. I am running a brand new installation of Magento 2.3.4 and the first thing I did was install this extension. I am populating products just fine, they are showing up using the url_key.

The categories work using the "https://www.example.com/catalog/category/view/s/denim-jeans/id/16/". Although, when I attempt to access the category via the url_key/request_path such as "https://www.example.com/denim-jeans" it does not work.

In the url_rewrites table, categories are being generated for store_id 0 and products are being generated for store_id 1.

My solution, if I change my url category rewrites from store_id 0 to 1 from the url_rewrites table, the categories work. But it does not seem like the appropriate solution to constantly update the store_id to 1.

I've spent days researching prior to posting and just needed some clarification. My understanding is that if the only categories are store_id = 0, then it is the default category for all stores. I am only running the default store that came with my installation.

PHP 8 Compatibility

When installing i got following error. Will the package be made compatible with PHP 8?

[InvalidArgumentException]                                                                                       
  Package bigbridge/product-import has requirements incompatible with your PHP version, PHP extensions and Compos  
  er version:                                                                                                      
    - bigbridge/product-import 1.7.5 requires php ~7.0 which does not match your installed version 8.1.0.      

Configurable product

Could you give working example for a-configurable.xml in example folder? I can add the product, and its visible only in admin area. Not visible in front page. I am suspecting example xml file is missing some attributes. Could you please take a look into this?

Mangeto 2.4.1 - Tier Price not created when website is specified.

Hello,
using this code the tier price is not set. If i omit the website code it works, setting the tier price for all the websites.

$pr = new SimpleProduct($sku);
$pr->setTierPrices([
    new TierPrice(1,$price, 'my_customers_group', 'my_website')
]);

Error "is not an integer"

Hi,

I am facing in an issue.

When importing a value for a custom attribute it throws the following error. The attribute is a dropdown.

color is not an integer (blue) for product 'test' that starts in line 3

I am expecting the importer to just insert the value "blue". But it is not. Am i doing something wrong?

Import from php array

Hi,

is it possible to import products from an php array?

Restapi update url issue

Hi,

I've just come across an issue doing a scan of my site using screamingfrog.

Basically, when I create products I manually set the url_key attribute and the product creates/works as expected. If I revise the product with a new url_key the attribute in the database is updated but the attribute url_path retains the old value.

This is causing some strange behaviour where the canonical URL is correct and the page loads on the correct URL, but the href in the page for the scroll to review, along with the 3rd party HTML sitemap and richsnippets are showing the old URL.

I can see there is mention of the url_path attribute in the CategoryImporter.php file but nothing in any file relating to Product import, is this an oversight? Do you have a solution?

I've used this extension for generating URL keys in the past when I used magmi (https://github.com/olegkoval/magento2-regenerate_url_rewrites) and there approach appears to be set it to null before saving the product (https://github.com/olegkoval/magento2-regenerate_url_rewrites/blob/148378eb7cbd98807d1c86e6a1262763bbf77944/Model/RegenerateProductRewrites.php#L180)
Not sure if that is of any use.

Thanks in advance.

Question: replace existing categories

Does the module provide a possibility to 'replace' categories that already are assigned to a product. For example:

• The import contains a simple product that is assigned to the category 'Default Category/Desks'.
• Product was placed in the wrong category and the category is changed in the import to 'Default Category/Chairs'.
• Import assigns product to the new category and unassigns the old one.

PS. Import works like a charm, big thanks!

Trying to get property 'path' of non-object in Model/Resource/Storage/UrlRewriteStorage.php on line 376

Issue

Hi guys,

I'm using this module to run a product import with only simple products and ran into the following issue:

Trying to get property 'path' of non-object in /var/www/store1/magento2/releases/20210416100919/vendor/bigbridge/product-import/Model/Resource/Storage/UrlRewriteStorage.php on line 376 [] []

Any idea what might be wrong?

Thx in advance!

Versions

Magento 2.4.2
bigbridge/product-import 1.7.0