semaio/magento2-configimportexport

Import/Export configuration data in Magento 2 via CLI.

ConfigImportExport

This extension provides CLI commands for Magento 2 to import/export system configuration data.

Why this extension?

While Magento 2 offers the app:config:dump / app:config:import commands, they're limited to one environment and a little bit hard to manage, especially in CI/CD setups.

This extension supports managing configuration values for multiple environments (production, staging, etc.) out-of-the-box, defining and overwriting base configuration values in a specific environment. It also allows different file formats (e.g., YAML, JSON). In general, it makes that process a bit more streamlined and easier to manage, with the added benefit of generalization and cross-compatibility with other agencies/merchants using this.

Installation

Add the package to your composer.json

composer require semaio/magento2-configimportexport

Enable and install the module

php bin/magento module:enable Semaio_ConfigImportExport
php bin/magento setup:upgrade

Facts

• Supported Magento versions are 2.3 and 2.4.
• Supported PHP versions are 7.2, 7.3, 7.4, and 8.1.

Functionality

This module is a work in progress and will be extended in the future with more functionality and support for other file formats.

File formats

This module currently supports the following file formats:

• YAML (default)
• JSON

See docs/file-formats.md for more information and examples.

Import config data

See docs/config-import.md for more information.

Export config data

See docs/config-export.md for more information.

Usage with Deployer

When using a push approach in CI/CD, the environment variables should be replaced on the runner and not on the initiating host or target host. For this, you can use this Deployer task to do this and import the config settings;

task('magento:config:set', function () {
    $resolver = new \Semaio\ConfigImportExport\Model\Resolver\EnvironmentVariableResolver();
    $glob = glob('config/store/**/*.yaml');
    array_walk($glob, function ($file) use ($resolver) {
        if (file_put_contents($tempFile = tempnam('/tmp', get('clientslug')), $resolver->resolveValue(file_get_contents($file)))) {
            upload($tempFile, get('release_path') . '/' . $file);
        }
    });
    run('{{bin/php}} {{release_path}}/bin/magento config:data:import {{release_path}}/config/store ' . get('environment'));
});

Support

If you encounter any problems or bugs, please create an issue on GitHub.

Contribution

Any contribution to the development of MageSetup is highly welcome. The best possibility to provide any code is to open a pull request on GitHub.

Licence

Open Software License (OSL 3.0)

Contributors

Thanks to all contributors who invested their valuable time to contribute to this module. Much appreciated!

Inspiration

This module is inspired by the awesome n98-magerun plugin "HarrisStreet ImpEx" by @SchumacherFM for Magento 1 which you can find here.

Copyright

(c) 2016-2022 semaio GmbH / Rouven Alexander Rieker

magento2-configimportexport's Issues

Environment variables that include numbers are not parsed

General

• PHP version: 8.1
• Magento version:2.4.5
• Module version: 3.10.1

Issue description

Environment variables that include numbers are not parsed.

Steps to reproduce

1. Load environment variable INCLUDES_NUMBER_1=somevalue
2. Use this for a configuration like: '%env(INCLUDES_NUMBER_1)%'
3. Run setup:upgrade
4. Expected result for the configuration should be somevalue
5. Actual result: %env(INCLUDES_NUMBER_1)% (literal)

Suggested solution

Update the regex in Semaio\ConfigImportExport\Model\Resolver\EnvironmentVariableResolver
from
'/\%env\((?!PHP_|HTTP_|SERVER_|SCRIPT_|QUERY_|DOCUMENT_)([A-Z\_]{3,})\)\%/'
to
'/\%env\((?!PHP_|HTTP_|SERVER_|SCRIPT_|QUERY_|DOCUMENT_)([A-Z\_0-9]{3,})\)\%/'

And also create a testcase for this
Setup:
putenv('INCLUDES_NUMBER_1=testvalue5');

Validate:
$this->assertEquals($this->environmentVariableResolver->resolveValue('%env(INCLUDES_NUMBER_1)%'), 'testvalue5');

Return value of "Semaio\ConfigImportExport\Command\ImportCommand\Interceptor::execute()" must be of the type int, "null" returned

General

• PHP version: 8.3
• Magento version: 2.4.7
• Module version: 3.10.1

Issue description

execute() return must be of type int, null returned

Steps to reproduce

1. Use Symfony console 6.4.7
2. Execute config command
3. Errors occurs

Symfony has a check for the correct return type:

            if (!\is_int($statusCode)) {
                throw new \TypeError(sprintf('Return value of "%s::execute()" must be of the type int, "%s" returned.', static::class, get_debug_type($statusCode)));
            }

We need to add a "return Command::SUCCESS;" in the Import and Export Command to resolve this problem.

Fatal Error on fully commented YAML file

If a YAML file does not parse to an array there is a type-mismatch error passing the data along into the normalizer.

This for example happens if the parse result is NULL b/c the config.yaml file only consists of comments.

Compatibility for Magento 2.3?

General

• Magento version: 2.3
• PHP version: 7.2

Issue description

Any idea if everything works in magento 2.3? any support here?

Add support for PHP 7.3

General

• Module version: dev-develop
• Magento version: 2.3.3
• PHP version: 7.3.11

Issue description

Magento supports PHP 7.3 since Version 2.3.3

Steps to reproduce

1. Have PHP 7.3 running
2. run composer install
3. This package requires php ~7.0.0 || ~7.1.0 || ~7.2.0 but your PHP version (7.3.11) does not satisfy that requirement.

Deprecated Functionality: preg_replace_callback(): Passing null to parameter

Please make sure that you provide as much information as possible. If you have an example files, which reflect the issue, please make sure to attach them to the issue - thank you!

General

• Module version: 3.10.0
• Magento version: 2.4.5-p1
• PHP version: 8.1.11

Issue description

In ErrorHandler.php line 62:

  Deprecated Functionality: preg_replace_callback(): Passing null to parameter #3 ($subject) of type array|string is deprecated in /var/www/html/ve
  ndor/semaio/magento2-configimportexport/Model/Resolver/EnvironmentVariableResolver.php on line 31

Steps to reproduce

1. run config import with a value (that was exported before) like:

tax/defaults/postcode:
  default:
    0:

Config Paths with no Scope-Values

There might be no values to set (e.g. due to commenting out some settings) for scopes. In that case there is no need to iterate over NULL. Not even needless, it's also crashing the import.

Greater control of export output directory via --filename parameter

First of all, thanks for this module; Zookal/HarrisStreet-ImpEx was one of my favourite and most-used M1 magerun tools so it is great to see it recreated and re-imagined for M2.

One thing I wonder is whether we can have greater control of the export directory using the --filename parameter. Currently it will always export to var/Ymd_His followed by either config or the value of the --filename parameter and then optionally the namespace.

This isn't very flexible. For example, for my workflow I like to keep a copy of the dumped export in git, but currently it always dumps to a new directory because we can't remove the Ymd_His part of the export path so this then requires a manual step to copy or move everything into a consistent directory.

Also, a lesser issue than the timestamping is that I might not want the data in var at all but possibly somewhere else. Setting --filename on HarrisStreet-ImpEx provided complete run-time override of the full output directory path.

I appreciate that emulating the HarrisStreet-ImpEx behaviour and allowing to completely override the path using --filename would be a breaking change for existing workflows so this may not be a popular idea.

However, a simple solution to the timestamping with no breaking changes would be to add a --timestamp (-t) parameter which defaults to true and if set to false will remove the timestamp portion of the export path. For me personally this would be a huge improvement in usability and come with no cost that I can see.

What do you think? I could do a PR for the latter if it would be useful?

Export: when using filePerNameSpace don't force config_ prefix or add underscore to filename

General

• PHP version: 8.1
• Magento version: 2.4.5
• Module version: 4.0.0

Issue description

When using export with filePerNameSpace the user should have the choice to decide whether they wish to add any prefix to the filename and also to be able to declare exactly what they wish that prefix to be.

Currently:

• if a filename is not provided then "config_" is automatically added as a prefix to the output filename
• if a filename is provided then an underscore "_" is automatically added to that filename before being used as a prefix

Proposed:

• if a filename is not provided then no prefix should be added to the output filename
• if a filename is provided then that filename should be used as the prefix without any additional underscore

With these proposed changes in place the user now has complete control over the output filename prefix. If they wish to have "config_" prefix then they can provide that as the filename argument. If they don't wish to have any prefix then they can just omit the filename argument. If they wish to have a prefix but without any underscore then they can do that too.

[Question] Differences with Magento 2 standard export / import

With Magento2, I can export the configuration with
bin/magento app:config:dump

and I can import it with
bin/magento setup:upgrade

Excluding the choice of the format, what are the main features that this module brings in comparaison to those commands ?

import fails from yaml file

Please make sure that you provide as much information as possible. If you have an example files, which reflect the issue, please make sure to attach them to the issue - thank you!

General

• Module version: 3.0.0
• Magento version: 2.2.7
• PHP version: 7.1.25

Issue description

At import, Yaml File Reader > yaml::parse return filepath instead of array. looks like wrong symfony/yaml version.
symfony/yaml version: 3.2.18
...

Steps to reproduce

1. bin/magento config:data:export to export config
2. copy to base and dev folder (dev only custom parts)
3. bin/magento config:data:import config/store dev

Empty directory throws error and breaks deployment

General

• PHP version: 8.1
• Magento version: 2.4.6
• Module version: 4.1.0

Issue description

During deployment we run run('{{release_or_current_path}}/bin/magento config:data:import {{release_or_current_path}}/.config/store {{environment}}');

The environment variable is set through GitLab CI Variables.

Following scenario:
If you have an empty base directory and yaml files inside prod directory, the deployment for any other environment, besides prod, without config files will fail.

What do you think about changing this behaviour?
https://github.com/semaio/Magento2-ConfigImportExport/blob/main/Model/Processor/ImportProcessor.php#L89

Upgrade to PHP 7.4 with Magento 2.4

The extension needs to be upgraded to support PHP 7.4 with framework 103.
This males it compat with Magento 2.4

Allow encryption of core config data

Hi there,

I stumbled over a use case, where I need to import config data to a config field with \Magento\Config\Model\Config\Backend\Encrypted as a backend model.
While I could encrypt this field locally (same encryption key needed) there are situations where you do not want to do so, especially in a kubernetes situation.

So I had the idea to resolve the data just like an env variable and just extended the module in a way to support substitution like:

'0': '%encrypt(%env(PAY_PAL_SECRET)%)%'

If you are interested, I could share...

Best regards,

Willi

Importing fully exported config result in error when run cron

General

• Module version: 2.0.0
• Magento version: EE 2.1.5
• PHP version: 7.0.15

Issue description

Magneto use a config with path crontab/default/jobs/magento_scheduled_import_export_log_clean/schedule/cron_expr to store the cron schedule of import/export log cleaning schedule options (See \Magento\ScheduledImportExport\Model\System\Config\Backend\Logclean\Cron::CRON_STRING_PATH). This is exported as crontab/default/jobs. Once the exported file is imported, it will basically discard all crontab configuration (defined in crontab.xml of modules) and result in error when attempt to run cron.

Steps to reproduce

1. In Magento backend, Go to Store > Configuration > Advanced > System
2. Click save
3. Export config
4. Import exported files
5. Try to run cron (bin/magento cron:run)
6. See error invalid argument supplied to foreach

Thoughts on support for importing secrets via environment variables?

Has there been any thought or desire to include the ability to import secret config (api user/password, merchant ids, etc) via environment variables? I might pick this up unless there has already been some work done somewhere else, or opinions on how to handle this differently.

Magento 2.4.4 and php 8.1 support

Please make sure that you provide as much information as possible. If you have an example files, which reflect the issue, please make sure to attach them to the issue - thank you!

General

• Module version: 3.x
• Magento version: 2.4.4
• PHP version: 8.1.5

Issue description

Any plans to update this module for magento 2.4.4 and php 8.1.x?

Lock configuration

Lock configuration options in system configuration when settings are maintained via this module.

Add no-cache option to config import

Dataloss due to faulty encoding of multi-line string values

Making use of string values in Yaml that contain line-feeds, these are mangled due to faulty encoding in https://github.com/semaio/Magento2-ConfigImportExport/blob/develop/Model/Processor/ImportProcessor.php#L125-L127

Changing the original data in these lines damages the data leading to data-loss on LF/CR ascii characters that are part of the string.

Next to the flaw in the data-processing, the comment ("// Valid scope Write output") is wrongly worded, this is not valid scope write output, the output is broken when written into the database via the config writer.

Refs:

• ed911c5 @therouv [FEATURE] Initial module commit
• http://www.yaml.org/spec/1.2/spec.html#id2760844

Possibility to filter export also by scope ID

Not actually an issue, but an enhancement which I could need, an which - if wanted - I could provide an PR for:

I would add an Option --includeScopeId which allows to export e.g. only the config data for website number 42 by typing

bin/magento c:d:e --includeScope=websites --includeScopeId=42

Would that make sense in your eyes?

Best regards,

Willi

Magento 2.4.7 - symfony/yaml ^7.0 dependency

General

• PHP version: 8.3
• Magento version: 2.4.7
• Module version: 4.3.0

Issue description

Required update of symfony/yaml dependency to "^3.4|^4.0|^5.0|^6.0|^7.0"
Module is currently incompatible with Magneto 2.4.7. (symfony/yaml v7.1)

I will update this issue with a pull request soon.

No Password

I'm trying to use the migration tool but am getting SQLSTATE[HY000] [1045] Access denied for user (u sing password: NO). i don't see a password field in the settings or config for migration

Set up Github actions (especially unit tests)

Set up Github actions and run unit tests for all supported Magento / PHP versions to provide early feedback to developers if everything still works as expected.

Unit tests should run on push to main branch and every pull request:

• https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onpushbranchestagsbranches-ignoretags-ignore
• https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onpull_requestpull_request_targetbranchesbranches-ignore

Discovered https://github.com/extdn/github-actions-m2 during a quick research. Maybe a good starting point?

Getting issue with Magento 2.3 and PHP 7.2

General

Module version: 3.0.0
Magento version: 2.3.0
PHP version: 7.2.12

Issue description

Integrated this module with above mentioned versions of Magento and php it shows incompatibility issue while installing module.

Please do needful for the same to integrate with mentioned versions.

Default Magento Export file

In the Export of an clean Magento installation, there are only an few settings included. I have an older installation of Magento 2, where I installed several plugins. These also contain settings that are exported. I would like to start over but import only the core Magento values. Is there an way (with an little hack) to generate an Magento file with all core config fields included? Than I can do an file compare and generate an file with only those core config settings but then from my previous installation.

Thanks in advance.

Tag version for Magento 2.1

Could you please tag a version 2.2.3 on commit 920a169? The latest version 3.0 isn't compatible with Magento 2.1, and the last tagged version before that is missing my YAML patch from PR #18.

Can't install package due to composer conflicts.

General

• Module version: 2.2
• Magento version: 2.2.0
• PHP version: 7.0.23

Issue description

Can't install due to composer conflicts. See dump:

Using version ^2.2 for semaio/magento2-configimportexport
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Conclusion: don't install semaio/magento2-configimportexport 2.2.2
    - Conclusion: don't install semaio/magento2-configimportexport 2.2.1
    - Conclusion: remove symfony/dependency-injection v3.3.10
    - Installation request for semaio/magento2-configimportexport ^2.2 -> satisfiable by semaio/magento2-configimportexport[2.2.0, 2.2.1, 2.2.2].
    - Conclusion: don't install symfony/dependency-injection v3.3.10
    - semaio/magento2-configimportexport 2.2.0 requires symfony/yaml ~2.0 -> satisfiable by symfony/yaml[2.0.4, 2.0.5, 2.0.6, 2.0.7, v2.0.10, v2.0.12, v2.0.13, v2.0.14, v2.0.15,

Commands are not listed by bin/magento for M2.4.1

General

• Module version: 3.3.0
• Magento version: 2.4.1
• PHP version: 7.4

Issue description

I created a new Magento 2.4.1 project and installed the semaio/magento2-configimportexport package afterward as described in the README file. When invoking bin/magento the commands exposed by the module are not listed and I cannot invoke them.

I could track things down to this:
Currently, the module is registering the commands with type Magento\Framework\Console\CommandList. When changing the type to Magento\Framework\Console\CommandListInterface the commands appear when invoking bin/magento .

This is also mentioned in the README file of the Console package: https://github.com/magento/magento2/blob/2.4-develop/lib/internal/Magento/Framework/Console/README.md

Happy to provide a PR if changing the type makes sense for you.

Add the ability to resolve values

It would be a nice option to be able to add a way to resolve certain values, for example to resolve attribute id's based on the attribute_code, since attribute_id's change and attribute codes don't.

the value in the yaml file could easily be prefixed by some tag to indicate it should be resolved, something like #attr:some_attribute_code.