bobdenotter/conimex

🍛 Content Importer and Exporter (as YAML) for Bolt 4

Conimex: Content Importer and Exporter

Author: Bob den Otter [email protected]

Extension to export your bolt content as YAML, and import it again.

Installation:

composer require bobdenotter/conimex

Usage:

Exporting:

bin/console conimex:export websitename.yaml

To export a single Content Type, for example pages, use:

bin/console conimex:export websitename.yaml --contenttype=pages

To export a single Content Type as a CSV, use:

bin/console conimex:export websitename.csv --contenttype=pages

Importing:

bin/console conimex:import websitename.yaml

Running PHPStan and Easy Codings Standard

First, make sure dependencies are installed:

COMPOSER_MEMORY_LIMIT=-1 composer update

And then run ECS:

vendor/bin/ecs check src

conimex's Issues

Undefined variable

Hi,

I recently used the extension to import some data and I got the following error in the console:

Bolt version: Bolt 5 beta 3
Extension version: 1.3.2
PHP version: 7.3

ErrorException: Trying to access array offset on value of type null

I have exported my data from v3.7.0 and i get the following error when trying to import.

Issue seems to be here: https://github.com/bobdenotter/conimex/blob/master/src/Import.php#L134

If I change to

$slug = $record->get('slug');

it's working, but i guess it will break other setups.

My guess is that the second part is still called even if get('slug') succeed.

Unexpected output for relations

When exporting using conimex version 2.0.7 (but also earlier versions) in a Bolt 5.1 (upgraded from 4.x) project I'm seeing unexpected output for relations:

        relations:
            "\0*\0items": {  }
            "\0*\0escapeWhenCastingToString": {  }

Unable to import custom content with null slug

If we have a content type with no slug field defined, and we use conimex to export that content, the slug field will be null.

    -
        id: 80
        contentType: reviews
...
        fields:
            name: PuzzleBoxWorld
            content: 'An amazing sequence of steps to open and somewhat difficult to even put back in the original position. Very imaginative!'
            featured: true
            slug: null

Attempting to import this content fails, even though the content worked fine in the source Bolt website.

Error:

8/72 [=====>--------------------------------------------]  11%TypeError {#4502
  #message: "Argument 1 passed to Bolt\Common\Str::slug() must be of the type string, null given, called in /home/user/mySite/vendor/bobdenotter/conimex/src/Import.php on line 143"
  #code: 0
  #file: "./vendor/bolt/common/src/Str.php"
  #line: 235
  trace: {
    ./vendor/bolt/common/src/Str.php:235 { …}
    ./vendor/bobdenotter/conimex/src/Import.php:143 { …}
    ./vendor/bobdenotter/conimex/src/Import.php:118 { …}
    ./vendor/bobdenotter/conimex/src/Import.php:87 { …}
    ./vendor/bobdenotter/conimex/src/Command/ImportCommand.php:62 { …}
    ./vendor/symfony/console/Command/Command.php:298 { …}
    ./vendor/symfony/console/Application.php:1042 { …}
    ./vendor/symfony/framework-bundle/Console/Application.php:96 { …}
    ./vendor/symfony/console/Application.php:299 { …}
    ./vendor/symfony/framework-bundle/Console/Application.php:82 { …}
    ./vendor/symfony/console/Application.php:171 { …}
    ./bin/console:48 {
      › $application = new Application($kernel);
      › $application->run($input);
      › 
    }
  }
}

Taxonomies are not imported

Prerequisites:

• Data is exported from Bolt 3.7.4
• The result file is imported into Bolt 5.0.6

Steps to reproduce:

• export the data with nut database:export --file boltcms.yml
  • file is exported successfully and all taxonomies are in place for each item that has taxonomy
• import the file with bin\console conimex:import boltcms.yml

Expected:
All items are imported including taxonomies.

Actual:
Taxonomies are not imported. DB tables bolt_taxonomy and bolt_taxonomy_content are empty.

Bolt\Entity\Field::setCurrentLocale() must be of the type string, null given

Import for 3.7.x, following error happened:

Seems like the code to handle select fields is triggering an error by using "null" for the locale.

https://github.com/bobdenotter/conimex/blob/master/src/Import.php#L224

If I change the code to:

$this->contentEditController->updateField($field, $item, 'en');

it is ok, but I'm not sure how to get the main locale properly.

Post status ends up being wrong if "date" exists

While importing from v3.7 I found out some entries end up with the wrong status.

Here are some cases:

1. Depublish date maybe "wild" with old posts

Most of my old posts prior to 2018 (probably on a old 3.x version or 2.x) have a weird depublish date in the yaml file

datedepublish: "1900-01-01 00:00:00"

or

datedepublish: "-0001-11-30 00:00:00"

It makes the entries go to unpublished status during import, maybe we can add a check for the validity of this date.

2. Draft are published if publish date exist no matter the status

On 3.x entries, the publish date was filled even if the status is "draft".
During import the Importer follows the publish date and not the status.

Fun fact, I also found out that there is the same wrong date issue for draft:

datedepublish: "-0001-11-30 00:00:00"
ownerid: 1
status: draft

Fix broken export because $selectFieldDefinitionKey is undefined

On Bolt 5 the export command is broken.

Error on importing posttypes with collection/set fields

I install a fresh new bolt installation (and a fresh new conimex) and add a posttype like this:

tests:
    name: tests
    singular_name: test
    fields:
        tittel:
            type: text
        things:
            type: collection
            fields:
                thing:
                    type: set
                    fields:
                        thing_id:
                            type: text
        slug:
            type: slug
            uses: tittel

I fill out a post with some test content and export it with bin/console conimex:export websitenametest.yaml tests and get this file:

__bolt_export_meta:
    date: '2021-04-05T16:20:17+02:00'
    version: 4.1.17
    platform:
        client_version: 'mysqlnd 7.4.15'
        driver_name: mysql
        connection_status: '127.0.0.1 via TCP/IP'
        server_version: 5.5.5-10.4.11-MariaDB
__users:
    -
        [...]
content:
    -
        id: 1
        contentType: tests
        author:
            id: 1
            username: magnus
        status: published
        createdAt: 2021-04-05T14:29:06+02:00
        modifiedAt: 2021-04-05T14:19:49+02:00
        publishedAt: 2021-04-05T12:29:06+02:00
        depublishedAt: null
        fields:
            things: [{ name: thing, type: set, value: { thing_id: tresters } }, { name: thing, type: set, value: { thing_id: test2 } }]
            slug: soteroen
            tittel: Soteroen
        taxonomies: {  }
        twig: null
        relations: {  }

I then try to import it again with bin/console conimex:import websitenametest.yaml -vvv and get the following output in the console:

 ! [NOTE] Importing Bolt database contents from: websitenametest.yaml

^ array:3 [
  "date" => "2021-04-05T16:20:17+02:00"
  "version" => "4.1.17"
  "platform" => array:4 [
    "client_version" => "mysqlnd 7.4.15"
    "driver_name" => "mysql"
    "connection_status" => "127.0.0.1 via TCP/IP"
    "server_version" => "5.5.5-10.4.11-MariaDB"
  ]
]
 // Importing ContentType content

 0/1 [>-------------------------------------------------]   0% < 1 sec/< 1 sec 34.5 MiB
In CollectionField.php line 64:

  [Error]
  Call to a member function setParent() on array


Exception trace:
  at /mnt/c/xampp/htdocs/test/vendor/bolt/core/src/Entity/Field/CollectionField.php:64
 Bolt\Entity\Field\CollectionField->setValue() at /mnt/c/xampp/htdocs/test/vendor/bolt/core/src/Entity/Content.php:500
 Bolt\Entity\Content->setFieldValue() at /mnt/c/xampp/htdocs/test/vendor/bobdenotter/conimex/src/Import.php:184
 BobdenOtter\Conimex\Import->importRecord() at /mnt/c/xampp/htdocs/test/vendor/bobdenotter/conimex/src/Import.php:108
 BobdenOtter\Conimex\Import->importContentType() at /mnt/c/xampp/htdocs/test/vendor/bobdenotter/conimex/src/Import.php:78
 BobdenOtter\Conimex\Import->import() at /mnt/c/xampp/htdocs/test/vendor/bobdenotter/conimex/src/Command/ImportCommand.php:57
 BobdenOtter\Conimex\Command\ImportCommand->execute() at /mnt/c/xampp/htdocs/test/vendor/symfony/console/Command/Command.php:256
 Symfony\Component\Console\Command\Command->run() at /mnt/c/xampp/htdocs/test/vendor/symfony/console/Application.php:989
 Symfony\Component\Console\Application->doRunCommand() at /mnt/c/xampp/htdocs/test/vendor/symfony/framework-bundle/Console/Application.php:96
 Symfony\Bundle\FrameworkBundle\Console\Application->doRunCommand() at /mnt/c/xampp/htdocs/test/vendor/symfony/console/Application.php:290
 Symfony\Component\Console\Application->doRun() at /mnt/c/xampp/htdocs/test/vendor/symfony/framework-bundle/Console/Application.php:82
 Symfony\Bundle\FrameworkBundle\Console\Application->doRun() at /mnt/c/xampp/htdocs/test/vendor/symfony/console/Application.php:166
 Symfony\Component\Console\Application->run() at /mnt/c/xampp/htdocs/test/bin/console:39

conimex:import <filename>

If I remove the collection it imports just fine. If I try with a single set instead of a collection I get a very similar error.

This time with just a set

tests:
    name: tests
    singular_name: test
    fields:
        tittel:
            type: text
        sett:
            type: set
            fields:
                thing_id:
                    type: text
        slug:
            type: slug
            uses: tittel

__bolt_export_meta:
    date: '2021-04-05T16:38:49+02:00'
    version: 4.1.17
    platform:
        client_version: 'mysqlnd 7.4.15'
        driver_name: mysql
        connection_status: '127.0.0.1 via TCP/IP'
        server_version: 5.5.5-10.4.11-MariaDB
__users:
    -
        [...]
content:
    -
        id: 1
        contentType: tests
        author:
            id: 1
            username: magnus
        status: published
        createdAt: 2021-04-05T14:29:06+02:00
        modifiedAt: 2021-04-05T14:38:25+02:00
        publishedAt: 2021-04-05T12:29:06+02:00
        depublishedAt: null
        fields:
            slug: soteroen
            tittel: Soteroen
            sett: { thing_id: test }
        taxonomies: {  }
        twig: null
        relations: {  }

 ! [NOTE] Importing Bolt database contents from: websitenametest.yaml

^ array:3 [
  "date" => "2021-04-05T16:38:49+02:00"
  "version" => "4.1.17"
  "platform" => array:4 [
    "client_version" => "mysqlnd 7.4.15"
    "driver_name" => "mysql"
    "connection_status" => "127.0.0.1 via TCP/IP"
    "server_version" => "5.5.5-10.4.11-MariaDB"
  ]
]
 // Importing ContentType content

 0/1 [>-------------------------------------------------]   0% < 1 sec/< 1 sec 34.5 MiB
In SetField.php line 45:

  [Error]
  Call to a member function setParent() on string


Exception trace:
  at /mnt/c/xampp/htdocs/test/vendor/bolt/core/src/Entity/Field/SetField.php:45
 Bolt\Entity\Field\SetField->setValue() at /mnt/c/xampp/htdocs/test/vendor/bolt/core/src/Entity/Content.php:500
 Bolt\Entity\Content->setFieldValue() at /mnt/c/xampp/htdocs/test/vendor/bobdenotter/conimex/src/Import.php:184
 BobdenOtter\Conimex\Import->importRecord() at /mnt/c/xampp/htdocs/test/vendor/bobdenotter/conimex/src/Import.php:108
 BobdenOtter\Conimex\Import->importContentType() at /mnt/c/xampp/htdocs/test/vendor/bobdenotter/conimex/src/Import.php:78
 BobdenOtter\Conimex\Import->import() at /mnt/c/xampp/htdocs/test/vendor/bobdenotter/conimex/src/Command/ImportCommand.php:57
 BobdenOtter\Conimex\Command\ImportCommand->execute() at /mnt/c/xampp/htdocs/test/vendor/symfony/console/Command/Command.php:256
 Symfony\Component\Console\Command\Command->run() at /mnt/c/xampp/htdocs/test/vendor/symfony/console/Application.php:989
 Symfony\Component\Console\Application->doRunCommand() at /mnt/c/xampp/htdocs/test/vendor/symfony/framework-bundle/Console/Application.php:96
 Symfony\Bundle\FrameworkBundle\Console\Application->doRunCommand() at /mnt/c/xampp/htdocs/test/vendor/symfony/console/Application.php:290
 Symfony\Component\Console\Application->doRun() at /mnt/c/xampp/htdocs/test/vendor/symfony/framework-bundle/Console/Application.php:82
 Symfony\Bundle\FrameworkBundle\Console\Application->doRun() at /mnt/c/xampp/htdocs/test/vendor/symfony/console/Application.php:166
 Symfony\Component\Console\Application->run() at /mnt/c/xampp/htdocs/test/bin/console:39

conimex:import <filename>

My system

 Bolt version: 4.1.17

 * Install type: Packaged distribution
 * Database: mysql 5.5.5-10.4.11-MariaDB - 127.0.0.1 via TCP/IP (with JSON)
 * PHP version: 7.4.15
 * Symfony version: v5.2.6
 * Operating System: Linux - 4.4.0-19041-Microsoft

Conimex version: 1.3.2

Error: Attempted to call function "keys" from namespace "BobdenOtter\Conimex

When trying to import fields which have had localisation enabled an error occurs:

Symfony\Component\ErrorHandler\Error\UndefinedFunctionError^ {#2819
  #message: "Attempted to call function "keys" from namespace "BobdenOtter\Conimex"."
  #code: 0
  #file: "/app/vendor/bobdenotter/conimex/src/Import.php"
  #line: 394
  trace: {
    /app/vendor/bobdenotter/conimex/src/Import.php:394 { …}
    /app/vendor/bobdenotter/conimex/src/Import.php:298 { …}
    /app/vendor/bobdenotter/conimex/src/Import.php:122 { …}
    /app/vendor/bobdenotter/conimex/src/Import.php:91 { …}
    /app/vendor/bobdenotter/conimex/src/Command/ImportCommand.php:62 { …}
    /app/vendor/symfony/console/Command/Command.php:299 { …}
    /app/vendor/symfony/console/Application.php:996 { …}
    /app/vendor/symfony/framework-bundle/Console/Application.php:96 { …}
    /app/vendor/symfony/console/Application.php:295 { …}
    /app/vendor/symfony/framework-bundle/Console/Application.php:82 { …}
    /app/vendor/symfony/console/Application.php:167 { …}
    /app/bin/console:48 {
      › $application = new Application($kernel);
      › $application->run($input);
      ›
      arguments: {
        $input: Symfony\Component\Console\Input\ArgvInput {#2 …}
      }
    }
  }

It appears to be calling keys() which is undefined rather than array_keys().

1.x `availableLocales` undefined variable exception

bin/console conimex:import institutions.json 
 ! [NOTE] Importing Bolt database contents from: institutions.json                   
^ array:3 [
  "date" => "2021-06-14T12:28:00+00:00"
  "version" => "3.7.2"
  "platform" => "sqlite"
]
 // Importing ContentType institutions                                               
   0/103 [>-------------------------------------------------]   0%
In Import.php line 149:
  Notice: Undefined variable: availableLocales  

^ As reported by @evertalbers on Slack.

Already fixed in 2.0 beta, but needs to also be fixed in 1.x since only 1.x works with Bolt 5

ContentType not found by name

Here's my contenttypes.yaml definition for the articles content type:

articles:
    name: Blog
    slug: blog
    singular_name: Article
    singular_slug: blog
    fields:
        title:
...

Now, I have an export from a Bolt 3 site with the above content type ☝️ , and I get the error Requested ContentType articles is not defined in contenttypes.yaml.

Here's the relevant bits from the export:

articles:
    -
        slug: new-site-update
        title: New Site Update!
 ...

Is the export wrong? (Not a conimex issue) Or should conimex look for contenttypes by the definition name, not by the slug?

Bolt\Repository\RelationRepository::findRelations() must be of the type int or null, bool given

Importing from 3.7 i get the following error:

Blame is on this line: https://github.com/bobdenotter/conimex/blob/master/src/Import.php#L348

Indeed the constructor for findRelations is

public function findRelations(Content $from, ?string $name, ?int $limit = null, bool $publishedOnly = true)

and we have

$currentRelations = $this->relationRepository->findRelations($content, null, true, null, false);

so moving to

$currentRelations = $this->relationRepository->findRelations($content, null, null, false);

solves it.

Slow import

When content is link with taxony, import process is very slow.
The import time increase with the number of items in the DB.
import can take more than 1s for an item. the DB have only 3000 items.

the bottleneck is in the function addContent of the entity Taxonomy.
It's seem to be a cache issue because $this->content contains all items of Taxonomy table.

Maybe bolt need this cache to optimize the website but do we really need the cache when we import data ?

Bolt 4.1.x
last conimex version
PHP 7.3.23

Error during import of data structure with collection type

Running an import of a data structure containing a collection results in the following error message:

Argument 1 passed to Bolt\Repository\FieldRepository::factory() must be an instance of Tightenco\Collect\Support\Collection, null given, called in ... cdu.asbach/vendor/bolt/core/src/Controller/Backend/ContentEditController.php on line 371

The export has been created with app/nut database:export. The import is done by conimex version 1.3.5. However the same error occurs with conimex version 2.0.

Enclosed you will find the contenttypes.yaml defintion file for the custom datatype "eingangsseite" and the
conimex.support.tar.gz
eingangsseite.export.yaml file.

Thank you in advance.

-Eric

Simple select field are not importing from Bolt 3 to Bolt 5

Currently the select field which are being imported are those which have values as referenced content.

Select fields like this are not importing:

        cms:
            label: CMS
            type: select
            values:
                'bolt': "Bolt 3"
                'bolt4': "Bolt 4"
                'drupal7': "Drupal 7"
                'drupal8': "Drupal 8"
                'wordpress': "Wordpress"
                'civicrm': "CiviCRM"
                'other': "Other"

Import of select fields referenced to records from Bolt 3 export

When exporting the data from a Bolt 3 site, make the import into a Bolt 4 site set the right values for the select field with reference to other records.

Importing a collection with one field results in an error

When I try to import a repeater field with one sub-field from Bolt 3.7 into a collection field in a Bolt 5 database using conimex, I get the following error:

In FieldRepository.php line 96:
                                                                                                                                                                     
  [TypeError]                                                                                                                                                        
  Argument 1 passed to Bolt\Repository\FieldRepository::factory() must be an instance of Tightenco\Collect\Support\Collection, null given, called in vendor/bolt/core/src/Controller/Backend/ContentEditController.php on line 453   

Looking at the controller where the error is thrown, I see that $collection->getDefinition()->get('fields')->get($name); is returning an empty result, perhaps because the name being passed to it is 'timetabl' not 'timetable' (which is the name of the field). I can see that the repeater name is always being truncated by one letter (so if I shorted the field name to 'time' in the contenttypes.yaml' file, it becomes 'tim' in the import script), but I have not idea whether this is intentional or not.

TaxonomyRepository->factory() expects second argument to be string

I recently bumped into this issue when running an import. Apparently if the value of the taxonomy slug is set to null in the exported data (.yaml files) the import will fail when running because the expected type of data is a string and not null.

Import users with the id they have been exported

Users are being imported and related to records where they were not related to.

This doesn't work properly because in cases were users have been deleted and added the exported list of users is no longer a list with ids: 1, 2, 3, 4 but ids: 1, 4, 5, 8.

It seems that the import is importing the users and assigning an auto increment id and not taking the id the user has been exported with as the PRIMARY_KEY in the bolt_user table.

exporting and importing from bolt 3 to bolt 4

The import and export files are incompatible.
Even with minor tweaks in the yaml file it is impossible to import everythingto bolt 4
Making this extentsion support the export from bolt 3 would resolve this isseu

Import ignores status

When importing a yaml file the status is always set to 'published' instead of taking the value in the yaml file into account.

Float and Integer fields are imported as generic

Hi!

It seems floatand integer fields are not being imported with the proper field type.

Float and integer fields is known as number field in Bolt 4 and the mode can be set either to be an integer or a float. After the import has run this field is imported as generic type (See screenshot)

Bolt version: Bolt 5 beta 3
Extension version: 1.3.2
PHP version: 7.3

Bolt 3 user roles do not translate to Bolt 5 roles

One thing I noticed on my import of Bolt 3 data to a Bolt 5 site is my user seems to be lacking the admin permissions I was used to on the Bolt 3 site.

Here's my import data, obfuscated of course. 😄

__users:
    -
        id: 1
        username: bornm
        password: pbfuscated
        email: ...
        displayname: bornm_master
        enabled: true
        roles:
            - root
            - everyone
    -
        id: 2
        username: mnawrocki
        password: obfuscated
        email: ...
        displayname: Mike Nawrocki
        enabled: true
        roles:
            - chief-editor
            - root
            - editor
            - admin
            - developer
            - everyone

And the assignable roles are (by default) the following:

assignable_roles: [ROLE_DEVELOPER, ROLE_ADMIN, ROLE_CHIEF_EDITOR, ROLE_EDITOR, ROLE_USER]

This isn't a huge deal - I think I can go through my import file and replace root with ROLE_ADMIN, editor with CHIEF_EDITOR, etc. The point is this is a "gotcha" from the Bolt 3 upgrade, and it should either be 1) handled seamlessly, or 2) noted in the upgrade docs.

Error Call to private method Bolt\Controller\Backend\ContentEditController::getFieldToUpdate()

• Bolt 4.2.2
• Local setup with SQLite, PHP 7.3.28 on a Mac
• Trying to import a json file from a Bolt 3.2.latest site
• bobdenotter/conimex 1.3.5
• Will attach the imported json file and the 'target' Bolt 4.2 Contenttypes file, with only the relevant contenttype, the "institutions"

institutions.json.zip
contenttypes_institutionsonly.yaml.zip

This is what happens at import:

evertalbers@MacBook-Pro-2 PCAF-Multilan % bin/console conimex:import importfiles/institutions.json 

 ! [NOTE] Importing Bolt database contents from: importfiles/institutions.json                                          

^ array:3 [
  "date" => "2021-08-20T14:48:36+00:00"
  "version" => "3.7.4"
  "platform" => "sqlite"
]
 // Importing ContentType institutions                                                                                  

   0/146 [>-------------------------------------------------]   0%
In Import.php line 223:
                                                                                                                                      
  Call to private method Bolt\Controller\Backend\ContentEditController::getFieldToUpdate() from context 'BobdenOtter\Conimex\Import'  

Any help will be apreciated. If it helps to e.g. drop one or two fields, no problem.