friendsofsylius/syliusimportexportplugin

Sylius plugin to import / export data

syliusimportexportplugin's Issues

Define controllers as services

And not extend base controller, inject services instead of fetching them inside the methods.

Packagist out-of-sync with repo

On Packagist the master branch reports being last updated on March 30 with commit d8b0926 but there have been several commits since then.

refactor exporter to not fetch data redundantly

https://github.com/FriendsOfSylius/SyliusImportExportPlugin/pull/37/files#r168944399

make error handling more configurable

From @lsmith77 on October 28, 2017 19:25

skip on error, transaction batch size etc

Copied from original issue: solutionDrive/SyliusImportExportPlugin#7

refactor AbstractImporter

From @lsmith77 on October 28, 2017 19:22

https://github.com/solutionDrive/SyliusImportExport/pull/9#discussion_r146049391

We could do it differently: instead of two specific importer classes extending the main abstract importer class, we have one `ResourcesImporter` which has injected 3 parameters:
- `CsvReaderFactory`
- `ObjectManager`
- `ResourceProcessor` (or sth like this)

This `ResourceProcessor` (with `ResourceProcessorInterface`) would have factory and repository injected, and do all *update or create* logic inside. Even more - we could pass there the whole `$csvReader` iterator (or use something like `iterator_to_array` to pass array, it doesn't matter at all to be honest 😄 ) and operate on this multiple rows inside, and then we could get rid of `ObjectManager` here. There are few benefits of such an approach.
- we make importer class much more easier - it just reads data from csv file and pass it to processor
- we get rid of this smelly `createOrUpdateObject` function
- if we want, the resource processor could not even operate on doctrine or even database - it just needs to implement specific method (`public function process(iterable $dataRows): void`) and inside it can write to database, queue, redis... anything 🚀 

Copied from original issue: solutionDrive/SyliusImportExportPlugin#4

RFC: Define per importer/exporter whether they are CLI only, UI only or both

Probably through configuration or maybe through tagging. E.g. I want an Order exporter in UI, but I do not want an Order Importer. I do want a Countries importer on CLI & UI. Etc :-)

Dependency in TestApplication has security issues

jquery is installed for the test-application in an outdated version.

It is recommended to update to the version 3.0.0.
current:
"jquery": "^2.2.0",

recommended:
jquery ~> 3.0.0

Referenced vulnerabilities:
https://nvd.nist.gov/vuln/detail/CVE-2015-9251
https://nvd.nist.gov/vuln/detail/CVE-2016-10707

add importer for product stock

From @lsmith77 on October 29, 2017 9:46

Copied from original issue: solutionDrive/SyliusImportExportPlugin#17

Some dependency broke behat-suite

After updating the composer packages, the behat suite does not run any longer. I also restarted the last successfull build in travis and the build was broken with the same error.

The error is:

In ParameterBag.php line 102:

The service "test.behat.context.cli_countries_context" has a dependency on a non-existent parameter "symfony.kernel.project_dir".

The output of the composer update on my machine were:

Package operations: 0 installs, 8 updates, 0 removals

• Updating twig/twig (v2.4.4 => v2.4.6): Downloading (100%)
• Updating sylius/sylius (v1.1.0 => v1.1.1): Downloading (100%)
• Updating symfony/symfony (v3.4.4 => v3.4.5): Downloading (100%)
• Updating doctrine/orm (v2.6.0 => v2.6.1): Downloading (100%)
• Updating sensiolabs/security-checker (v4.1.7 => v4.1.8): Downloading (100%)
• Updating friendsofsymfony/rest-bundle (2.3.0 => 2.3.1): Downloading (100%)
• Updating slevomat/coding-standard (4.4.6 => 4.5.0): Downloading (100%)
• Updating nikic/php-parser (v3.1.4 => v3.1.5): Downloading (100%)

add importer for customers

From @lsmith77 on October 29, 2017 9:49

Copied from original issue: solutionDrive/SyliusImportExportPlugin#18

README config suggestion is not working

From README:

fos_sylius_import_export:
    importer:
      # set to false to not add an upload form to the entity overview pages
        web_ui:               true
        # set to an integer value to flush the object manager in regular intervals
        batch_size:           false
        # if incomplete rows (ie. missing required fields) should be considered failures
        fail_on_incomplete:   false
        # if to stop the import process in case of a failure
        stop_on_failure:      false

Using this out of the box results in:

Fatal error: Uncaught Symfony\Component\Config\Definition\Exception\InvalidTypeException: Invalid type for path "fos_sylius_import_export.importer.batch_size". Expected int, but got boolean. in /var/www/webshop/vendor/symfony/symfony/src/Symfony/Component/Config/Definition/IntegerNode.php:29

Leaving batch_size out of there, makes it work again. I think we should either suggest an integer or maybe use ~ if that's possible to revert back to the default if users are simply copy pasting the defaults. They are often read as suggested settings I think.

add importer for promotions

From @lsmith77 on October 29, 2017 9:50

Copied from original issue: solutionDrive/SyliusImportExportPlugin#19

support XML for import/export

From @lsmith77 on October 29, 2017 9:45

Copied from original issue: solutionDrive/SyliusImportExportPlugin#13

add order export

Resource name in Import Form is singular

https://github.com/FriendsOfSylius/SyliusImportExportPlugin/blob/master/src/Resources/views/Crud/import_form.html.twig#L7

It's now: Country import, while Import countries would be nicer. Currently the 'country' term is injected, not the plural version of it - countries.

can export order？

change Bundle class and extension to use FOS prefix

https://github.com/Sylius/Sylius/pull/8903/files

convert import form to use the form component

From @lsmith77 on November 2, 2017 16:1

see https://github.com/solutionDrive/SyliusImportExportPlugin/blob/master/src/Resources/views/Crud/import_form.html.twig

Copied from original issue: solutionDrive/SyliusImportExportPlugin#25

make commands services

https://symfony.com/doc/current/console/commands_as_services.html

re-use doctrine metadata to determine fields to map

From @lsmith77 on October 28, 2017 19:27

this will make it possible to add new properties to an entity and automatically supporting it via an importer without adjustments

portphp could be used here

relates to portphp/doctrine#9

Copied from original issue: solutionDrive/SyliusImportExportPlugin#8

setup scrutinizer

From @lsmith77 on October 28, 2017 19:30

Copied from original issue: solutionDrive/SyliusImportExportPlugin#11

add importer for (product) images

From @lsmith77 on October 29, 2017 9:46

optionally overwrite existing images

Copied from original issue: solutionDrive/SyliusImportExportPlugin#16

add importer for product stock

From @lsmith77 on October 29, 2017 9:46

Copied from original issue: solutionDrive/SyliusImportExportPlugin#17

Badges are invalid

The packagist badge in README.md contains invalid badge references. Can you please fix it? :)

Export requires write permissions

Running into this because my filesystem is read-only (Platform.SH) besides some pre-defined folders:

ErrorException: fopen(order-2018-03-30.csv): failed to open stream: Read-only file system
#14 vendor/friendsofsylius/sylius-import-export-plugin/src/Writer/CsvWriter.php(37): handleError
#13 vendor/sentry/sentry/lib/Raven/ErrorHandler.php(0): fopen
#12 vendor/friendsofsylius/sylius-import-export-plugin/src/Writer/CsvWriter.php(37): setFile
#11 vendor/friendsofsylius/sylius-import-export-plugin/src/Exporter/ResourceExporter.php(59): setExportFile
#10 vendor/friendsofsylius/sylius-import-export-plugin/src/Controller/ExportDataController.php(72): exportData
#9 vendor/friendsofsylius/sylius-import-export-plugin/src/Controller/ExportDataController.php(49): exportAction
#8 vendor/symfony/symfony/src/Symfony/Component/HttpKernel/HttpKernel.php(151): handleRaw
#7 vendor/symfony/symfony/src/Symfony/Component/HttpKernel/HttpKernel.php(68): handle
#6 vendor/symfony/symfony/src/Symfony/Component/HttpKernel/Kernel.php(202): handle
#5 vendor/symfony/symfony/src/Symfony/Component/HttpKernel/HttpCache/HttpCache.php(462): forward
#4 vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/HttpCache/HttpCache.php(57): forward
#3 vendor/symfony/symfony/src/Symfony/Component/HttpKernel/HttpCache/HttpCache.php(416): fetch
#2 vendor/symfony/symfony/src/Symfony/Component/HttpKernel/HttpCache/HttpCache.php(313): lookup
#1 vendor/symfony/symfony/src/Symfony/Component/HttpKernel/HttpCache/HttpCache.php(188): handle
#0 web/app.php(28): null

exporting xlsx via cli creates a file with zero bytes

Export keys need to be defined twice

Per example, the keys (id, code & enabled) are defined twice. What's the reason to do this? And could we maybe extract them away in another service that's managing the keys?

Further, shouldn't they be case sensitive? Field names can also have different casing.

services:
    sylius.exporter.countries.csv:
        class: FriendsOfSylius\SyliusImportExportPlugin\Exporter\ResourceExporter
        arguments:
            - "@sylius.exporter.csv_writer"
            - "@sylius.exporter.pluginpool.countries"
            - ["Id", "Code", "Enabled"]
        tags:
            - { name: sylius.exporter, type: country, format: csv }

    sylius.exporter.pluginpool.countries:
        class: FriendsOfSylius\SyliusImportExportPlugin\Exporter\Plugin\PluginPool
        arguments:
            - ["@sylius.exporter.plugin.resource.country"]
            - ["Id", "Code" ,"Enabled"]

    sylius.exporter.plugin.resource.countries:
        class: FriendsOfSylius\SyliusImportExportPlugin\Exporter\Plugin\ResourcePlugin
        arguments:
            - "@sylius.repository.country"
            - "@property_accessor"
            - "@doctrine.orm.entity_manager"

Export RFC: application level export format instead of per importer

Source: https://github.com/FriendsOfSylius/SyliusImportExportPlugin/pull/37/files#r168944314

Composer is required for test

add importer for orders

From @lsmith77 on October 29, 2017 9:50

Copied from original issue: solutionDrive/SyliusImportExportPlugin#20

add support for enqueue to support messages queues

From @lsmith77 on October 28, 2017 19:24

see also https://github.com/php-enqueue

see for some inspiration https://github.com/Sylake/SyliusConsumerPlugin/tree/master/src

Copied from original issue: solutionDrive/SyliusImportExportPlugin#5

add importer for price data

From @lsmith77 on October 29, 2017 9:46

Copied from original issue: solutionDrive/SyliusImportExportPlugin#15

add ability to set bulk data batch size

Suggests outdated?

    "suggest": {
        "portphp/excel": "To support importing excel files, use dev-master",
        "portphp/csv": "To support importing csv files, use dev-master"
    },

Is it still advised to use dev-master?

setup travis ci

RFC: Config setting to tell CLI exporter to create folders if not existing

WDYT?

add product importer

From @lsmith77 on October 28, 2017 19:20

optionally overwrite existing product data

Copied from original issue: solutionDrive/SyliusImportExportPlugin#1

Cannot install plugin with Sylius 1.1.4

When doing composer require friendsofsylius/sylius-import-export-plugin --dev, composer throw this error:

It seems that this plugin conflicts with Symfony 3.4.5+, which is required by Sylius 1.1.4 (symfony/symfony locked at 3.4.8), but I don't know why.

Export RFC: no coupling to PortCsvWriter

Source: https://github.com/FriendsOfSylius/SyliusImportExportPlugin/pull/37/files#r168944522

add category importer

From @lsmith77 on October 28, 2017 19:20

Copied from original issue: solutionDrive/SyliusImportExportPlugin#2

Export memory usage

My custom order export currently simply exports all orders to either CSV or Excel. On the first run in production (without grid filters) trying to export to CSV, I'm running into memory issues already with a memory limit of around 130mb:

Symfony\Component\Debug\Exception\OutOfMemoryException: Error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes)
  File "vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php", line 2780
    $this->originalEntityData[$oid][$field] = $pColl;

Probably the grid filters are going to help, but memory consumption is an issue anyway probably.

collect metadata during import

From @lsmith77 on October 28, 2017 19:24

like number of items imported, failures, time consumed

Copied from original issue: solutionDrive/SyliusImportExportPlugin#6

add exporter infrastructure

From @lsmith77 on October 28, 2017 19:21

or should export be moved to a different plugin? I think there is likely some overlap in code needed but not a lot either. of course any overlapping code could also be moved to a separate lib.

of course if we drop export from this plugin, we would need to rename the namespace

Copied from original issue: solutionDrive/SyliusImportExportPlugin#3

also support Excel files

From @lsmith77 on October 29, 2017 9:45

Copied from original issue: solutionDrive/SyliusImportExportPlugin#14

support json for import/export

From @lsmith77 on October 29, 2017 10:19

Copied from original issue: solutionDrive/SyliusImportExportPlugin#21

Import form is shown while no importer is defined

This is my config, but I haven't defined a 'country' importer. The form is still shown in my admin, which is not really something I would like in this case. Related to #69. Should we make it configurable?

services:
    sylius.exporter.countries.csv:
        class: FriendsOfSylius\SyliusImportExportPlugin\Exporter\ResourceExporter
        arguments:
            - "@sylius.exporter.csv_writer"
            - "@sylius.exporter.pluginpool.countries"
            - ["Id", "Code", "Enabled"]
        tags:
            - { name: sylius.exporter, type: country, format: csv }

    sylius.exporter.pluginpool.countries:
        class: FriendsOfSylius\SyliusImportExportPlugin\Exporter\Plugin\PluginPool
        arguments:
            - ["@sylius.exporter.plugin.resource.country"]
            - ["Id", "Code" ,"Enabled"]

    sylius.exporter.plugin.resource.country:
        class: FriendsOfSylius\SyliusImportExportPlugin\Exporter\Plugin\ResourcePlugin
        arguments:
            - "@sylius.repository.country"
            - "@property_accessor"
            - "@doctrine.orm.entity_manager"

use service locator for the export command

https://symfony.com/doc/current/service_container/service_locators.html

follow up to #52

support deleting/disabling items via import

From @lsmith77 on October 28, 2017 19:29

Copied from original issue: solutionDrive/SyliusImportExportPlugin#9

Export based on grid filters

Improve fetching count in order exporter

#107 (review)

cc @enesaktay