blackbitdigitalcommerce/pimcore-data-director

Import Bundle for Pimcore

Data Director: Pimcore imports, exports, automation

Pimcore plugin to optimize Pimcore concerning all PIM-related features, especially imports, exports, automation, workflows

• Presentation
• YouTube videos for diferent use-cases
• Documentation

---

What does this plugin do?

Data Director can

• import XML, JSON, CSV, Excel and other data sources to Pimcore data objects, assets and documents
• export data from Pimcore to XML, JSON, CSV, Excel and other formats
• create REST APIs for writing or reading Pimcore data
• push data to external systems (shops, websites, etc.)
• create automations and workflows within Pimcore
• make daily tasks in Pimcore much more efficient

How does it work?

In the Pimcore backend GUI you can define which data is to be extracted from the import source and assign it to the object fields of your Pimcore objects. All Pimcore field types are supported for import. Also the import into assets and documents is supported (including editing metadata, tags etc.).

During the import, data can be modified and adapted to your data model.

Also exporting Pimcore data is supported - including data from relations, images / thumbnails etc. With the intelligent caching system exports are really fast as export data gets updated in the moment when an object gets saved (in the background, so saving does not take any longer) and not when the export is requested. For exports Data Director provides lots of ready-to-be-used templates:

CSV exports (file only or CSV file + assets as zip)

• JSON exports (file only or JSON file + assets as zip)
• XML exports (file only or XML file + assets as zip)
• Excel exports
• industry-standard formats like BMEcat, Datanorm via plugins

For every import or export, automatically also a REST API incl. authentication gets created without any further effort.

Besides generating such export documents, it is also possible to specify what to do with the export documents: some ready-to-be-used actions are:

• upload export document via (S)FTP, AWS S3 and other cloud storage providers
• send emails

Imports and exports can be started from backend UI, via cronjobs or event-based (as soon as an object of source data class gets saved). The latter can be used to set up automations like

• automatically send data to external systems as soon as Pimcore data object gets changed
• workflows which get executed when Pimcore data object gets changed (trigger translations, business processes, etc.)

For an overview how to set up imports and exports, please see our tutorial videos.

---

Example use-cases

Just some examples which the Data Director has been used for:

• Import / Export data to ERP systems like SAP, Navision / Business Central, Infor, Webware, JTL etc.
• Automatically export e-commerce data to online shops like BigCommerce, Shopware, Shopify, xt:commerce etc.
• Import / Export BMEcat (in general or for platforms like Mercateo, PSG, Wucato)
• Import / Export content from / to Wordpress, Hubspot, Typo3
• Import / Export customer-related data from / to CRM systems like Salesforce, Hubspot
• Export e-commerce related information to Hubspot to automate email marketing
• Create product feeds for marketplace syndicators like ProductsUp, Channable etc.
• Create exports feeds for SaaS search providers like FactFinder, Doofinder etc.
• Export product data to be used in InDesign for product catalogs, datasheets etc.
• Create product feeds for Google Shopping / Merchant Center, Facebook etc.
• Import product data from Icecat, Bidex
• Synchronize asset file folder / DAM system with Pimcore assets
• Implement data quality checks incl. notification
• Create REST APIs for reading / writing information from / to Pimcore
• Automatic translation with DeepL / AWS Translate
• Connect to translation providers like Trados
• Automatic text generation with artificial intelligence (OpenAI.com API (ChatGPT), GPT-3, GPT-4)

Advantages compared to other Pimcore import plugins

• everything is configurable in Pimcore backend user interface - no creation of PHP files or anything similar necessary, even for complex imports
• better import performance
  • streaming input files allows to also import huge import files
  • lot of optimizations to accelerate importing data to Pimcore elements - or to check previously if data actually changed (and if not skip import or saving to save time)
• auto-complete functions to set up imports very fast and to suggest how to proceed even with complex imports
• flexible import resource, you can import data from:
  • single files
  • folders
  • URLs
  • PHP script for complex requirements like importing a CSV file only if a PDF file with the same file name (but different file extension) exists in the import folder
  • cURL requests to import data from resources which need OAuth or similar authentication (but of course you can also use a PHP script in this case which authenticates and then provides the data to be imported)
• Supported import formats:
  • CSV
  • XML
  • JSON
  • Excel
  • Pimcore elements (data objects, assets, documents)
  • Pimcore reports (e.g. to import data from Google Analytics, external databases or other Pimcore database tables)
  • File system (e.g. to import assets)
• full flexibility: for common use cases of data transformation the bundle provides ready-to-be used templates. But you can also edit those and even write custom (PHP or JavaScript) functions to set up a transformation pipeline, conditions etc. for certain fields
• import object hierarchy (set parent element of imported elements)
• option to write-protect fields: only import data to certain field if field is empty
• dynamically import to object brick / classification store fields (target field can be defined dynamically based on import data)
• generate response documents, for example to send success status back to source system
• automatically start imports based on events, for example to:
  • automatically assign assets to data objects after uploading an asset
  • automatically start import for uploaded CSV, JSON, XML, Excel file
• trigger imports via push from external systems (via REST API) for live interfaces without exchanging files
• Traceability: import archive (for import files) and searchable import log history to always know when and why a certain field got a certain value
• revert imports: if data loss happened due to a mistake, only revert imported fields to certain date - much better than having to restore a database dump (because data of non-imported fields is kept)
• dataports are also stored as JSON files -> easy to trace changes via VCS (Git), easy to deploy between Pimcore instances. Also downloading and importing dataport configuration is supported.
• Optimize inheritance feature
• Permission system: for each dataport you can configure who is allowed to configure and / or execute the dataport
• import different types of elements:
  • Data Objects
  • Assets
  • Documents
  • Quantity value units (incl. conversion factors)
• built-in connectors to translation providers (DeepL, AWS Translate) to set up automatic translation with just one click (incl. caching of translations to save costs for recurring translation strings)
• artificial intelligence features, e.g. to automatically assign categories based on product name and description

Advantages compared to other Pimcore export plugins

• automatically execute or prepare exports whenever an object gets saved whose data gets exported to:
  • prepare export once the data changes, so that the data does not have to be generated in the moment when the export is requested -> very fast exports because the export document is already available in the moment of request
    • data is not extracted again and again from the Pimcore database but only when elements change
  • upload exports automatically to a target system to always have up-to-date data there
  • automatically send data the other APIs, e.g. online shops, marketplace distributors
• full flexibility for format / document structure of export document to tailor the export document structure exactly how the target system expects it
• access any data which is connected to exported object, for example when exporting products you can easily access assigned categories, images of the assigned categories and even meta data of the images of the assigned categories - you can chain that as long as you want
• full flexibility in setting up a transformation pipeline to change values to the desired format (e.g. date format conversion, convert quantity values to certain units etc.)
• predefined export templates to create CSV, XML, JSON exports with or without referenced asset files (e.g. CSV file plus assets (or thumbnails) packed in a zip archive)
• intelligent checks whether anything changed since the last export. If nothing changed, result document gets delivered from cache
• access exports via URL, for example to pull data feed in an external system
• filter elements to be exported by any field, either filter via dataport configuration or use ad-hoc filtering for one-time exports

Advantages compared to own import implementation

• Minimization of programming effort: Only data modifications from the data source require minimal programming
• Performance:
  • Data is only imported if nothing has changed since the last import
  • If an object has not changed during import, it does not need to be saved
  • Export data gets updated when objects change, so it is already prepared in the moment an export gets requested
• Flexibility:
  • Imports and exports are fully customizable to data source and your Pimcore data model
  • Supports all Pimcore elements including data objects, object bricks, field collections, assets and documents
  • Supports importing tags and properties
• Transparency:
  • with import archive and searchable import log history you can always trace why a certain field got a certain value and which import file was responsible for this
  • always see current import / export status (progress of current job, number of queued items)
• Comfort functions:
  • Automatically extract raw data fields from import resources
  • Automatically map raw data fields to Pimcore class fields
  • Optimize inheritance feature
  • Replace placeholders (for example to generate texts automatically)
  • Automatic translation (with DeepL API)
  • Error monitoring / notification of imports
  • Revert imports

One for all

Data Director is ONE bundle for:

• importing data to Pimcore elements
• checking and changing data when data objects get saved -> more transparency compared to overriding Pimcore model classes' save() method or creating event handlers
• exporting data in whatever format you want and wherever the export documents shall be sent to
• automation of data handling within Pimcore
• support UX-optimized workflows for frequent tasks

Initial effort is only necessary once, compared to when you have multiple bundles for all the different requirements which the Data Director is able to fulfill.

How to get the plugin

You can buy this plugin in the Blackbit Shop or write an email to [email protected].

Demo

You can test the Data Director at our Pimcore demo system by requesting the demo login credentials.

---

Workflow

Step 1: Import CSV, XML, JSON or Excel into Pimcore

This plugin has proven itself in many e-commerce and master data management projects and has been developed according to the needs of our customers.

The aim of the plug-in is to import information from upstream systems into the objects of Pimcore, there appropriately enrich them with further information and publish them via Pimcore itself or via a downstream system such as an online shop.

The import always takes place in two steps: In the first step, an intuitive user interface determines which columns of a CSV file or which attributes of an XML / JSON document should be imported into Pimcore.

A preview shows directly if the raw data fields are set up correctly. The import reads the data into an "intermediate table" so the user can see which data was imported from the upstream system before it is further processed in the next step.

Step 2: Mapping Attributes via Drag-and-Drop

In the second step, the information from the intermediate table is assigned to the attributes of the classes. Thise mappings get individually defined in the Pimcore backend UI via drag-and-drop.

There one or more key attributes get set so that the import process can be repeated and existing objects get updated.

The attribute mapping can also be used to modify the source data. For each field assignment, a callback function can be specified, which, for example, performs date format operations or other calculations on the basis of one or multiple imported or existing attributes of the current object.

---

Further highlights at a glance

• File-based imports (monitoring of a directory and automatic import on new files), for example for:
  • automatic asset imports into the Pimcore (e.g. from a network drive)
  • automatic mapping of assets to Pimcore objects by file name
• Import multiple files (CSV, XML, JSON, Excel) from a source directory with automatic dataset deduplication
• Import from a URL as a data source
• Import data from Pimcore objects:
  • Migrate data from one field type to another without data loss
  • dynamic mass data manipulation (currently that is not possible in the Pimcore grid (for example, increase prices by 10%))
• Import data from Pimcore reports
  • e.g. to send reports by mail (e.g. notify the responsible users when an action for the report objects is necessary)
  • e.g. to import data from Google Analytics into your Pimcore data objects to further use this data
• Import of documents (filling of the editables)
• Generate response documents
  • generate a response document including all successfully imported items and send that to the source system
  • track import errors
  • call another import which depends on the current one
  • create export documents (JSON / CSV / XML etc.) which other systems can use as import source
  • create response documents for single-page application / PWA frontend requests
• execute imports and exports with the built-in REST API
• Skipping records from the import source, e.g. if data is missing or the quality is insufficient
• easy import of relations
• automatically assign elements to relations via artificial intelligence / machine learning
• Import of object hierarchies / object trees
  • Specification of the parent element possible
  • Option to optimize inheritance (data is entered as high as possible in the object hierarchy)
• Automatic translation of texts via translation providers (via DeepL or AWS API)
• Support of all Pimcore data types including relations with metadata, object bricks, field collections etc.
• optimized performance:
  • If the source data has not changed, the import of the data record can be skipped
  • If data of an object is not changed in the import, it does not need to be saved
• generate barcode and QR code images
• Possibility to revert imports:
  • If an attribute assignment error has crept in and several thousand objects have been filled with incorrect data, the change can be reverted - in this case only the fields mapped in the import are set back, only for the objects changed in the import - a big advantage over a complete backup restore

Frequently asked questions (FAQ)

Import

• Is it poosible to create folders or an object hierarchy (import one object below another in the data object tree)?

  • Yes. You can create folders or put an imported object below another object. You can also assign a parent element dynamically based on the import data. There is also a tutorial video about how to set up an object hierarchy.

• How can existing objects be updated?

  • You can assign any field of your data object class as a key field. At the beginning of the import it is tried to find an existing object which has the imported values in this key field. It one is found, this object gets updated. For example this can be a SKU for product objects. Also combined key fields are possible, e.g. when your products have a product number and a color - only the combination of product number and color is unique, so you can set both as key fields.

    It is also possible to set the import mode to only update existing objects (and not create new ones). This is helpful when you gather data for one data object class from different sources. For example when the product data gets imported from an ERP system but the prices come from pricing system. In this case it could happen that the pricing import gets a product SKU which does not exist yet. In this case you might not want to create a new object only with the price but every other field empty.

    It is even possible to update multiple objects from one import dataset. So the used key field does not have to be unique. This can be useful if want for example to update all objects which have a certain object assigned to a many-to-many relational field. For more details, we have recorded a tutorial video about updating multiple objects from one import dataset

• Is it possible to assign the target class dynamically?

  • Every dataport has exactly one target class. But of course you can create multiple dataports which have different target classes. All of them can use the same import file. And as you can skip import items it is possible that every dataport processes the import items which it shall handle. So to summarize: Yes, you can dynamically assign the target class.

    Beside skipping it is also possible to use only some fields. For example you could use a product CSV file with a manufacturer column to create Manufacturer data objects in one dataport and to assign the maniufacturer object to the product in another dataport. It is even possible to connect multiple dataports to a pipeline - in this case first create the Manufacturer objects and afterwards assign them to products.

• How can localized fields be filled?

  • A localized field can be mapped for every language individually. This way you can assign different columns / fields from your import file to the different languages of the localized field.

    Moreover there is a built-in feature to automatically translate values using the DeepL or AWS Translate API. For more details see the tutorial video about automatic translations.

• How can other objects be assigned to relation fields?

  • You can query related objects by any field of the related object class. For example you can assign a Manufacturer object to a product by querying for a Manufacturer object which has ABC in its name field. The actual logic is implemented in callback functions within Pimcore backend (no need to create any PHP files or classes somewhere). You do not have to write complex PHP code for this task but we deliver templates for most common tasks. For example to assign above mentioned Manufacturer object, you can just select the function Assign Manufacturer object by name from the template list. We also recorded a tutorial video about assigning related objects.

• Is it possible to transform import data?

  • Yes. For every field you can implement custom logic in a callback function (inside the Pimcore backend, you do not have to create any PHP files somewhere). In this callback function you get the import data and the current object data provided and cam use any language construct like conditions, loops etc. You can use PHP or JavaScript to implement those callback functions.

• How to assign assets?

  • Assets can be assigned via URL, file system path or you can query for existing assets. You can wither create the assets during a data object import (e.g. when you have a product feed with image URLs). For more details see the tutorial video about assigning images.

    Or you import assets individually, e.g. when you connect a network drive to your Pimcore server and want to automatically import assets from this folder into Pimcore. For more details see the tutorial video about importing assets from the filesystem

• Can you import multiple files at once? What import resources are supported

  • Yes. You can import files one by one, files from a folder (including search pattern, e.g. /import/*.xml), files from remote storage ((S)FTP), Files from Pimcore Assets, data from a URL, data from a cURL request, data from a PHP script. The two latter are especially useful when you want to import data from an API which required authentication.

• How can imports be started?

  • Imports can be started:
    • manually from the Pimcore backend
    • manually via right-click on objects inside the target folder or on assets inside the source folder (when the source folder is inside Pimcore assets)
    • manually or automatically via command-line, this can also be used for periodic imports when called by a cronjob or the Process Manager Bundle
    • manually or automatically via REST API, this can be used to push data from the source system to Pimcore without exchanging files between the systems.
    • automatically when import source changes. This can for example be when a new file gets uploaded to a certain folder, you can automatically start an import (one use-case is to automatically assign assets to products by extracting the SKU from the assetfilename).
    • automatically with dependent imports / dataport pipelines

  For more details we have recorded a tutorial video about the different ways to start imports.

Documentation

To learn more about all options and how they can be configured, please read the full documentation or watch the tutorial videos.

Training

We also offer an academy course for Data Director.

pimcore-data-director's Issues

Problem with big XSL file.

We have XLS file with 10000 rows and about 75 cols.
When I click on Attribute mapping it show loading progress bar for few seconds
and then I see such error.

Is there a limit for max. rows Data Director can handle ?

Asset path not available in rawData

While exporting parent assets I want to get the path of the child asset stored in a custom property ('artbox') of the parent asset.
So in pimcore I can access the child asset's path in the following way: ParentAsset->getProperty('artbox')->getPath().

However, during the export if I run parentAsset:properties I get the following JSON

The path is null, although there is an asset in the 'artbox' property. Am I missing something? How can I access the path of the 'artbox' asset?

Thank you

Importing files with glob expression (wildcards) - does not work

My config is:

When I upload the file SupplierProduct.xml into /@DD/Supplier One/Import/ folder
DD do not start import automatically.

Is it a bug ?

matches hash of raw data values

Case

I have setup data port for update data.
Prepared XML file.

When I use this file for update data direct from PimCore Panel everything is ok.

When I use the same XML file and try to use API for update data I got error:

[INFO] Importing raw item 16984 [DEBUG] Fetching item from database [INFO] Skipping ArticleNumber=12345 because object property "importhash_5" of object #29920 matches hash of raw data values [INFO] Importing raw item 16985 [DEBUG] Fetching item from database [INFO] Skipping ArticleNumber=123456 because object property "importhash_5" of object #29921 matches hash of raw data values

Why for API import DD checks hash if dataport is not defined to do it ?

Open new window on export - why ?

I set up export data from object to XML.
Then I setup document callback function that save exported data to xml file under Asset.

When I run Complete Export, DD always open new window with some response.

Why ? Is it needed in this case ?
Will it open in all cases ?

I understand new window appear when I do not set up document callback function and DD need to display or send data to browser.
But in this case maybe there should be some message box only ?

Is it purposeful ?

Status of the import on callback function

I can check that import finished with $params['lastCall'] == true but do not see param for import status.
Can not see it in the code under
//lib/Pim/Item/Importmanager.php:384

Attempted to load class "JsonMachine" from namespace "JsonMachine"

Currently, a version "<2" of the package halaxa/json-machine is used, but a few days ago a version 1.0.0 of this package was released. Here the class "JsonMachine" has been removed / replaced.

The solution would be either to use the package version 0.8 or to make the DataDirector compatible with version 1.0.0.

The following version of DataDirector was used: 2.7.9.

In JsonParser.php line 73:
                                                                       
  Attempted to load class "JsonMachine" from namespace "JsonMachine".  
  Did you forget a "use" statement for another namespace?       

Wrong label for option ?

[NFR] in Export Dataport: allow admin to disable TWIG Lexer validation for field

If to try to EXPORT existing saved object of any class with MULTY-LANGUAGE field of type WYSIWYG

with HTML content like:
<p>{{widget hello="world" }}</p>{{media url="wysiwyg/category/500x500.jpg"}}

using raw field configuration 'fieldName#all'
then error happens which I can't process myself because it is in raw fields:
[ALERT] Parsing error for field "FieldLAbel#all": Twig\Error\SyntaxError: Unexpected character "&" in "e822858dd47cf5cdb945d5a8d75e9f8b" at line 4. in/vendor/twig/twig/src/Lexer.php:363
Stack trace:
#0 /vendor/twig/twig/src/Lexer.php(290): Twig\Lexer->lexExpression()
#1 /vendor/twig/twig/src/Lexer.php(184): Twig\Lexer->lexVar()
#2 /vendor/blackbit/data-director/lib/Pim/Item/Importer.php(3066): Twig\Lexer->tokenize(Object(Twig\Source))
#3 /vendor/blackbit/data-director/lib/Pim/Item/Importer.php(4049): Blackbit\DataDirectorBundle\lib\Pim\Item\Importer->replaceObjectIdentifier('

\n<div clas...', Object(Pimcore\Model\DataObject\MagentoCategory))
#4 /vendor/blackbit/data-director/lib/Pim/Parser/PimcoreParser.php(124): Blackbit\DataDirectorBundle\lib\Pim\Item\Importer->getObjectByIdentifier('Pimcore\Model\D...', Object(Pimcore\Model\DataObject\MagentoCategory))
#5/vendor/blackbit/data-director/lib/Pim/Parser/PimcoreParser.php(385): Blackbit\DataDirectorBundle\lib\Pim\Parser\PimcoreParser->getValue(Object(Pimcore\Model\DataObject\MagentoCategory), Array)
#6 /vendor/blackbit/data-director/lib/Pim/RawData/Importmanager.php(268): Blackbit\DataDirectorBundle\lib\Pim\Parser\PimcoreParser->current()
#7 /vendor/blackbit/data-director/Command/ImportCompleteCommand.php(169): Blackbit\DataDirectorBundle\lib\Pim\RawData\Importmanager->importDataport('10', 2, '116167', '62dac9d9cc66e5....', Object(Symfony\Component\HttpFoundation\Request))
#8 /vendor/blackbit/data-director/Controller/RestController.php(322): Blackbit\DataDirectorBundle\Command\ImportCompleteCommand::import('10', '', true, true, 'en', NULL, Object(Symfony\Component\HttpFoundation\Request), Array)
#9 /vendor/blackbit/data-director/Controller/RestController.php(378): Blackbit\DataDirectorBundle\Controller\RestController->importAction(Object(Symfony\Component\HttpFoundation\Request), '10')
#10 /vendor/symfony/http-kernel/HttpKernel.php(152): Blackbit\DataDirectorBundle\Controller\RestController->exportAction(Object(Symfony\Component\HttpFoundation\Request), 'ExportMagentoCa...')
#11 /vendor/symfony/http-kernel/HttpKernel.php(74): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1)
#12 /vendor/symfony/http-kernel/Kernel.php(202): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#13 /public/index.php(35): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request))
#14 {main}

• can you please create a raw field processor 'As Is' to send content as is without Lexer? Or to suppress Lexer errors and return original value if Lexer fails. Currently, content is processed in both raw fields and data mapping. So we tried to wrap content in 'verbose' tag. Nothing help except twig tag extenging.

• I just need html content of this field for future saving in json, I can myself do json_decode() later in callback function.

• even if error happend then why DataDirector returns empty string, not the original value, not validated twig or html?

Log Error: "The "--" option does not exist."

When I try to Execute again import nothing happend.

I checked log and I see error there:

[2022-02-10 06:52:09] console.CRITICAL: Error thrown while running command "data-director:process 2 0 --force -vv--dataport-resource-id=26 --user=2". Message: "The "--" option does not exist." {"exception":"[object] (Symfony\\Component\\Console\\Exception\\RuntimeException(code: 0): The \"--\" option does not exist. at /var/www/pimcore.dev/www/vendor/symfony/symfony/src/Symfony/Component/Console/Input/ArgvInput.php:118)","command":"data-director:process 2 0 --force -vv--dataport-resource-id=26 --user=2","message":"The \"--\" option does not exist."} []

Please send daily report with missing selectbox/measurements options instead of separate email for each item

Currently, separate email is sent for each case when no existing option found.
vendor/blackbit/data-director/lib/Pim/EmailReportingLogger.php
vendor/blackbit/data-director/Resources/views/email-report.html.twig

When you have 10 000 products to import is means 10k+ emails.

When we have this bug in DataDirector #29 it means millions of emails.

Can you please group cases in one table and send one report per day?

Thanks.

Import create two images

I have DataPort which imports some objects.
Object have one field for image (Image type).
In import file I have a column ImageUrl with URL to image.

I have setup mapping that map Object's field Image with ImageUrl field from import file and setup in settings this:

When I run import image are saved into configured asset folder but as I see it is create two the same images with different names:
One ends with sufix: -1

Is it a bug ?

Dynamic taget folder on import

Hello.

We prepared DataPort that import Supplier's Product.
Is is possible to change Target folder dynamicly ?

For example
if ABC Supplier (User) is logged the target folder will be /Products/ABC
if XYZ Supplier (User) is logged the target folder will be /Products/XYZ

Is it possible to change Setting dynamicly for example in: Initialization function of DataPort.

Bug: 'Automatically create missing options' creates doublicates

Select-boxes (Pimcore\Model\DataObject\ClassDefinition\Data\Select::class) have checkbox 'Automatically create missing options' in <>Settings on AttributeMapping tab.

That is great feature. But it creates doublicate values for some values (not all, some are ok).

Buggy output sample:

array (
  0 =>
  Pimcore\Model\DataObject\ClassDefinition\Data\Select::__set_state(array(
     'fieldtype' => 'select',
     'options' =>
    array (
      0 =>
      array (
        'key' => 'Einzelpreis',
        'value' => '0',
      ),

      // auto generated values below
      1 =>
      array (
        'key' => '0',
        'value' => '0',
      ),
      2 =>
      array (
        'key' => '0',
        'value' => '0',
      ),
      3 =>
      array (
        'key' => '0',
        'value' => '0',
      ),
      4 =>
      array (
        'key' => '0',
        'value' => '0',
      ),
   ...  

One more sample:

 array (
                      0 => 
                      Pimcore\Model\DataObject\ClassDefinition\Data\Select::__set_state(array(
                         'fieldtype' => 'select',
                         'options' => 
                        array (
                          0 => 
                          array (
                            'key' => 'Bundesrepublik Deutschland',
                            'value' => 'DE',
                          ),
                          1 => 
                          array (
                            'key' => 'Indonesia',
                            'value' => 'ID',
                          ),
                          2 => 
                          array (
                            'key' => 'Spanien',
                            'value' => 'ES',
                          ),

                         // following values are generated automatically
                          3 => 
                          array (
                            'key' => 'DE',
                            'value' => 'DE',
                          ),
                          4 => 
                          array (
                            'key' => 'DE',
                            'value' => 'DE',
                          ),
                        ),

PS: New Feature Request: add the ability to specify from which field to take the key and from wchi to take the value for new options creation.

Raw Fields IDs 1000+ dont fit the column width.

I have a dataport with source having many raw fields.
When the field ID is biggen then 1000 then the column is too narrow: I see only '1..'
And there is no ability to resize the column width.
And when I delete the field (e.g. 37), the next ID is generated not as smallest available (37) but MAX of existing fields (1256).

Filtering canceled jobs on wrong list

Canceled jobs are under

Call to a member function isAllowed() on null

On one project we use DD 2.8.7.

When we manually try to start DataPort we got message:

Call to a member function isAllowed() on null

PATH only under $params['context']['resource']['file']

In 2.8.5 under
$params['context']['resource']['file']

has full path to asset with file name.

In 3.0.6 there is only full path WITHOUT file name.

Why ?

NFR: local source path configuration for data-director:deployment:dataport-rebuild

https://www.youtube.com/watch?v=B4GqswnTYtU&t=1924s you set the local path in ''XML resource file' like /var/www/html... But that path is usually different on localhost, staging, and production.
What is the best practice to have the configuration in the repository and to have those paths different? symlinks?

Can you please provide

• the ability to provide multiple pathes in dataport-JSON-configuration
• or some local overwrites
• or at least commandline --path parameter for data-director:deployment:dataport-rebuild
  so 1 file can be working and automatically imported (data-director:deployment:dataport-rebuild) on all servers?

Empty logs

Under Result callback function I have code like this:

$params['transfer']->logs[] = $params['logs'];

if ($params['lastCall']) {
var_dump($params['transfer']->logs); die;
.....
}

Why $params['logs'] is empty ?
We use 3.0.6 and in 2.8.6 it was not empty.

Additional column for attribute mapping tab

Hello

Is it possible to add additional column "Attribute Name" and change "Attribute" to "Attribute Label" for Attribute Mapping Tab ?
We use different language for attribute Label and Name and many times it is difficult to map import fields.

Can You make such change ?

NULL under $params['context']['resource']['file']

Hi.

I am importing one xlsx file.
It was working fine but now - not.

The problem we have with this code:

On last call I want the get the name of imported file:

if ($params['lastCall']) {
    $file = $params['context']['resource']['file'];
    ....

But when I debug it it looks it has NULL - why ?

[WARNING] Additional output during run: 
array(3) {
  ["dataportId"]=>
  string(2) "55"
  ["resource"]=>
  array(2) {
    ["file"]=>
    NULL
    ["locale"]=>
    string(2) "en"
  }
  ["logs"]=>
  string(0) ""
}

[Bug] Error when clicked 'Auto Create' button

ErrorException: Warning: Undefined array key "masterDocument" in /vendor/blackbit/data-director/lib/Pim/Helper.php:226 Stack trace:
#0 /vendor/blackbit/data-director/lib/Pim/Parser/NaiveParser.php(321): Blackbit\DataDirectorBundle\lib\Pim\Helper->getFieldDefinitions(Array)
#1 /vendor/blackbit/data-director/lib/Pim/Parser/NaiveParser.php(379): Blackbit\DataDirectorBundle\lib\Pim\Parser\NaiveParser->getDataQuerySelectors(Array, 'pimCategory:cat...', 'Produkt Familie...')
#2 /vendor/blackbit/data-director/lib/Pim/Parser/NaiveParser.php(410): Blackbit\DataDirectorBundle\lib\Pim\Parser\NaiveParser->getDataQuerySelectors(Array, 'pimCategory', 'Produkt Familie')
#3/vendor/blackbit/data-director/lib/Pim/Parser/NaiveParser.php(332): Blackbit\DataDirectorBundle\lib\Pim\Parser\NaiveParser->getDataQuerySelectors(Array, 'pimCategory', 'Produkt Familie')
#4 /vendor/blackbit/data-director/lib/Pim/Parser/NaiveParser.php(260): Blackbit\DataDirectorBundle\lib\Pim\Parser\NaiveParser->getDataQuerySelectors(Array)
#5 /vendor/blackbit/data-director/Controller/ImportconfigController.php(2508): Blackbit\DataDirectorBundle\lib\Pim\Parser\NaiveParser->guessConfig()
#6 /vendor/symfony/http-kernel/HttpKernel.php(152): Blackbit\DataDirectorBundle\Controller\ImportconfigController->autoCreateRawdataFieldsAction(Object(Symfony\Component\HttpFoundation\Request))
#7 /vendor/symfony/http-kernel/HttpKernel.php(74): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1)
#8 /vendor/symfony/http-kernel/Kernel.php(202): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true)
#9 /public/index.php(35): Symfony\Component\HttpKernel\Kernel->handle(Object(Symfony\Component\HttpFoundation\Request))
#10 {main}

BUG: unique fields inheritance

If the class has some field marked as 'unique' in class configuration,
and DataDirector has checkbox to use inheritance in dataport,
then DataDirector get's error during the import:
10:57:27 ERROR [app] Object #26017 could not be saved. Reverted changes. Error: PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'PA0490:z:ab83aef5dec53bad041473916469c26c' for key 'u_index_ProAlphaID' in /vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:117

I believe it happens because DataDirector inheritance works that way: if some field value of all children-products are the same then this value is saved to the parent object and so inherited to all children.
If the child is only on then DataDirector tries to set the same value to the parent.
But for unique fields this logic should be skipped.

Delete many fields from Raw Data Field Setting

I have selected many fields and tried to remove them all with one delete click.
But DD remove only the one field where I click right mouse button.
Is it possible to add functionality for remove all selected fields ?

Currently I remove those fields direct in database: plugin_pim_rawitemFIeld

ERROR: Could not load preview panel grid's columns

Hi.

We want to setup export data for PimCore object - class Product.
Our definition is very complex - have many different relations to other objects and bricks.

When we click Auto Create button under Row Data Field section it generate almost 1000 rows.
Now when we want to Save dataport we got error:

And the Attribute mapping panel is empty:

The problem is here:

I tried to debug a little why the error occurred:

and saw something like this:

What can we do about it ?
Is it a bug or DD can not handle so much complex class definition ?

Update related object's data

Hi,

Scenario:
Have two classes. ClassA and ClassB. Both with name field.
ClassA have additional field called related which is One 2 Many relation to Class2.

I have created two objects: Object1 and Object2 with some values in their name fields and setup relation object for related field.

Now.
I want to update both name fields for those two objects from one update file.

Created update XLS file which have columns:
ClassA ID - PimCore's ID
ClassA Name
ClassB Name

I watched https://www.youtube.com/watch?v=otC8-8SYNIM&t=3124s this about dependent import.
But I want to set up dependent update.

Should I use Dependent Import to it also or there is some other solution with DD ?
I did not find any solutions for this scenario in documentation - if there is can You point me to it ?

How should I tell DD that it should update already related Object2's name field ?

Automatic import for subfolder

Hi
I have folder structure like this

I want XML file be automatically imported if someone upload file to import folder

When I setup resource folder to something like this:

and upload xml file to any of root or subfolder it starting import.

But I tried to define that files will be imported only if they uploaded to import subfolder
I setup resource folder like this:

But it is does not work.

Is it possible to do something like this with DD ?

Bug: unZIPping is impossible for StreamingXmlParser

\Blackbit\DataDirectorBundle\lib\Pim\Parser\ResourceBasedParser::extractFile

($this instanceof XmlParser && File::getFileExtension($fileName) === 'xml') ||

should be replaced with

 ($this instanceof XmlParser && File::getFileExtension($fileName) === 'xml') ||
 ($this instanceof StreamingXmlParser && File::getFileExtension($fileName) === 'xml') ||

Can linked objects be created during the import?

See title, I didn't find it in the documentation.

Removing import file after import

I can not find in documentation anything about removing import file after import.
Is it possible ?

Dependent import - clarification

Is dependent import it started for every records separately or after first import will be finished ?

Examples from video have two dataports. One for categories and products.
Both use the same file.

If we run those dataport separately the structure is not build. Products are not under categories.
But if we run it with dependent import the products are put under categories.

For example the file have two products.

<products>
<product>
 <name>First Product</name>
 <category>First Category</category>
</product>
<product>
 <name>Second Product</name>
 <category>Second Category</category>
</product>
</products>

If I setup dependent import DD will internally

1. Get first product data.
2. Create First Category object with Category dataport
3. From this created category will run Product dataport and create product under this category
4. Get second product data
5. Create Second Category
6. Create second product under second category.

Is my understanding of this process is correct ?

If it is not working like this and if in first step create two categories how is it possible that second dataport knows
which product should be placed in which category ?

No actual documentation. README

I try to setup pipe line object import.
I tried to use tutorial from youtube but it looks they are outdated.

I can not find in documentation what is it for:

Second. My setup does not work. Products are not created under Categories.
Where can I find actual documentation to setup it.

Cant use "set" as attribute because of SQL Syntax error

This is the query which has an syntax error because "set" is a reserved word.

INSERT INTO object_store_Product (oo_id,set) VALUES (1,2) ON DUPLICATE KEY UPDATE oo_id=VALUES(oo_id),set=VALUES(set)

If the query would wrap the column names in backticks it should work, no more syntax error:

INSERT INTO object_store_Product (oo_id,set) VALUES (1,2) ON DUPLICATE KEY UPDATE oo_id=VALUES(oo_id),set=VALUES(set)

How to recreate:

1. Create Pimcore class and add "set" as a checkbox (in our case)
2. Create dataport and assign any value to "set"
3. "Start import" will produce this SQL syntax error in the log

It would be great if you could look into this and create a fix for this.

Best regards Andreas

cannot access block element with data query selector "all"

I have a block element in a data object class that needs to be read by a data query selector:

I tried the documented ways to access the element, but the "all" selector does not work as expected:

Bug: --limit option is not applied

Data-director v.2.8
I put 5 files in the local folder on disk.
and execute:
bin/console data-director:extract ImportFromERP --rm
/Users/e.kaurov/htdocs/sommer/pim-pim10/bin/console data-director:process ImportFromERP --limit=1 -vv

I expect 1 file to be imported.
But all are imported instead.

messages about that are outputed, like:
19:45:15 INFO [app] Successfully saved #36830
19:45:15 INFO [app] Successfully saved #36831
etc.

BUG: implode cannot return imageGallery items pathes

I created dataDirector export dataport with raw field:
imageGallery:Items:implode:Thumbnail#somename:Path
It returns nothing '';

Athoug there is data in the gallery: if I change it to
imageGallery:Items
then I get
["/tmp/photo1.jpeg", "/tmp/photo2.jpeg"]

How can I get list of urls to asset images or/and their thumbnails?

Bug: extracting crashes if there is no 'xmlns' node.

\Blackbit\DataDirectorBundle\lib\Pim\Parser\XmlParser::getDomXPath

extracting crashes if there is no 'xmlns' node.
$root->removeAttributeNS($root->getAttributeNode('xmlns')->nodeValue, '');

Cancel import does not work.

I have many errors like this in log:

Next Doctrine\DBAL\Exception\UniqueConstraintViolationException: An exception occurred while executing 'INSERT INTO objects (o_key, o_path) VALUES (?, ?)' with params ["-Produktkatalog-Serier-Other suppliers-1000082 - SINDRE", "/"]:

SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '/--Produktkatalog-Serier-Other suppliers-1000082 - SINDRE' for key 'objects.fullpath' in /mnt/bohus-pimcore-live-volume/www/pim.bohus.no/www/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractMySQLDriver.php:74
Stack trace:

When I want cancel this import (with admin interface) - it is not working.

How can I force cancelation for running import ?

SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry for key

I try to import new Products. Some have wrong data.

Our Product class have GTIN number field which is configured to be uniqu value.
I have product with GTIN number in PimCore.

When I try to import only new products

and new product have the same GTIN number which exist in other product
i get

[ERROR] Object #1137302 could not be saved. Reverted changes. Error: PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '7030261050806' for key 'object_store_2.u_index_GTINNumber' in /www/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:115

Then I see other errors:

[ERROR] Object #1137303 could not be saved. Reverted changes. Error: Doctrine\DBAL\ConnectionException: Transaction commit failed because the transaction has been marked for rollback only.

And this error is repeated many times and DD stop working after it. It show that import is in progress but nothing happens.

Why DD just do not omite this wrong record ?

Problem with wildcards under Path to Excel File

I have a DataPort that configuration is:

When I setup
Path to Excel File: /Files/*/ProductImport/*.xlsx
And put export.xlsx file under
/Files/Daniel/ProductImport/ folder, DD do not start automaticly import. Option to do this is enabled.

But when I use XML file it is working ok.
Why ?

We use DD: 2.8.2

Big offset REST API parameter value = wronng data

There is a issue with exporting data with REST API

Examnple url:
https://someurl.loc/api/rest/export/Logistic-Hub-Products?limit=200&offset=345000&skipExtract=1&apikey=secret

if offset > totalCount it returns 1 item, but it should return empty result or get possibility getting total items for Export

Update object only for specified conditions

Is it possible to update data only when specified conditions occur ?
For example.
Imagine we have a product with some workflow status or type.
For example if workflow status is "new product" or type is 'computer' DD will allow update data.
but if workflow status is other then "new product" or type is other then 'computer' it will not allow to
update anything via DataPort.

I think about something like under initialization function add code that:

1. Load current object by some key value.
2. Check product type or workflow status.
3. If condition not pass return false.

Archive import files under different folders

I have folder structure like this:

I want to run dataport when somone upload file to Import folder but want that after import the file will be moved
from Import to Imported folder.
For example if file is uploaded to
/AutoImport/A/Import
it will be moved to /AutoImport/A/Imported after import
but
if file is uploaded to
/AutoImport/B/Import
it will be moved to /AutoImport/B/Imported after import

As I see I can specify one Archive folder path

SO is it possible with DD ?

Call to a member function getLanguage() on nul

Hello

I try to setup pipe line import.
I set up everything like in Your video and when I try to run import I get error:

[2022-01-13 11:20:39] request.CRITICAL: Uncaught PHP Exception Error: "Call to a member function getLanguage() on null" at /.../www/vendor/blackbit/data-director/Controller/RestController.php line 363 {"exception":"[object] (Symfony\Component\Debug\Exception\FatalThrowableError(code: 0): Call to a member function getLanguage() on null at /.../www/vendor/blackbit/data-director/Controller/RestController.php:363)"} []

I use: BDD 2.7.0

Data not saved into database when using DD

I have strange behaviour.

I have a class named Product which have FuturePrices field - it is FieldCollection (FututrePrice).

Product class have id=2
Table name for FuturePrice is: object_collection_FuturePrice_2

When I import data (direct from admin panel or via API call) I see that data are not saved into
database table. But when I open product in pimcore admin - I see data are there.

My DD configuration for this field looks like this:

return [
    'FuturePrice' => [
        [
            'FromDate' => $params['rawItemData']['ValidFrom']['value'],
            'NorthPrice' => $params['rawItemData']['NorthPrice']['value'],
            'SouthPrice' => $params['rawItemData']['SouthPrice']['value'],
            'BasePrice' => $params['rawItemData']['BasePrice']['value'],
        ]
    ]
];

XML:

<?xml version="1.0" encoding="UTF-8"?>
<PIMPrices>
    <Prices>
        <ArticleNumber>1234567890123</ArticleNumber>
        <SouthPrice>123.00</SouthPrice>
        <NorthPrice>123.00</NorthPrice>
        <ValidFrom>07.07.2022</ValidFrom>
    </Prices>
</PIMPrices>

logs:

[INFO] Importing object /test1
[INFO] Value for field ArticleNumber: 1234567890123
[INFO] Value for field FuturePrices: {
    "FuturePrice": [
        {
            "FromDate": "07.07.2022",
            "NorthPrice": "123.00",
            "SouthPrice": "123.00",
            "BasePrice": ""
        }
    ]
}
[INFO] Not adding field collection item because item 0 is equal
[INFO] Reason for saving: Publish state changed
[INFO] /test1 queued for saving
[INFO] Successfully saved new unpublished version for #30734 /test1

Is it a bug in DataDirector or schould I look for somewhere else.

Object hierarchy

return 'ClassName:fieldName:'.$params['value'];

What it will return if object will be not found ?
Is it possible that it will return bool value ?

Documentation say: DD try to create that object - is that right ?
But what if it can not create it ?

NFR: add DataDirector info to Class/ObjectBrick/FieldCollection Editor

Can you please implement the new feature in the next version?

All following editors have list of fields to edit:
Settings-> Data Object -> Classes
Settings-> Data Object -> ObjectBricks
Settings-> Data Object -> Fields-Collections

That is usual workplace for data manager. This is where he expects to controll if all fields are existing and imported properly.

Please display the following information in there for each field:

• Raw Data Field (SelectBox + callback function editor + data sample) (each language)
• Expression/xPath (just info, to see what means selected Raw Data Field)
• Max data length (just info, to set up Columnlength property)
• Ideally, to set automatically or to show the button to set 'Not editable = true' and columnlength = min(10, (Max data length)*1.3)

On field save, in case if updates provided, the configuration for this field should be saved to dataport. Please check if dataport is already opened for editing and show the alert.

Is it possible with DataDirector ?

Hello

I have two cases:

1. I have Customer class with some data fields and additional field called ExportFlag. Have for example 100 records but only 20 have the ExportFlag set to True.
   I want to setup export via DD but only for those records with ExportFlag == True.
   Is it possible direct in DataDirector ?

2. I have Product class and 100.000 products. I want to setup export the data for XML file. But I want to chunk this export and want that every file will have max. 1000 products.
   Is it possible with DD ?

No version on bundle list