kontent-ai/cli

Kontent.ai command line interface supporting migrations scripts processing written by Management SDK JS.

cli's Issues

Time stamp support for migrations

Motivation

Include the timestamp in the migration data contract.

Proposed solution

Add an option to use this timestamp for orders.

Additional context

Q: Validate to include the timestamp in the file name. (maybe as a custom extension and then use the filename instead of module property.
Q: Validate how to work with time ranges.

See: https://github.com/digital-detox/kontent-migration-cli/blob/main/src/lib/run.ts#L19-L33

Readme contains a lot of information about boilerplate

Motivation

Readme is too long and can get lost easily. The migration example describes the whole process of using a boilerplate. Wouldn't it be better to only reference the boilerplate repo and say to follow instructions there?

Proposed solution

Reduce the migration example paragraph to only navigate to the migration-boilerplate. It has a fully covered readme by itself.

Additional context

Add any other context, screenshots, or reference links about the feature request here.

How to manage content groups?

I noticed there's no documentation on using content groups and adding elements into them.
I used the following syntax but I get the following error:

code:

const BuildResourceModuleTypeData = (builder: ContentTypeElementsBuilder): ContentTypeModels.IAddContentTypeData => {
  let titleRef = builder.textElement({
    name: 'Title',
    codename: 'title',
    type: 'text'
  });

  return {
    name: 'Resource Module Test',
    codename: 'resource_module_test',
    elements: [
      titleRef,
      builder.urlSlugElement({
        name: 'URL Slug',
        codename: 'url_slug',
        type: 'url_slug',
        depends_on: {
          element: titleRef
        }
      }),
      builder.linkedItemsElement({
        name: 'Configuration',
        type: 'modular_content',
        allowed_content_types: [
          {
            codename: 'ui_configuration'
          }
        ],
        item_count_limit: {
          condition: 'at_most',
          value: 1
        },
        content_group: {
          codename: 'settings'
        }
      }),
      builder.contentGroup({
        name: 'settings'
      })
    ]
  };

Error:

An error occurred while running migration: 02_createResourceModule.js see the output from running the script.
Error details
  Message: The provided request body is invalid. See the 'validation_errors' attribute for more information and specify a valid JSON object.
  Code: 5
  Validation Errors: [
    ValidationError {
      message: "The object must contain the property 'type'."
    }
  ]

builder constructor method contentGroup does not allow a property named type as its not defined in its interface

Could you please elaborate and include some examples on usage.

Also it would be nice that errors have some more detailed references.Sometimes its very difficult to find where the problem is in a long script with current given feedback

Decouple the status information

Motivation

status.json could be decoupled.

Proposed solution

Default implementation should stay the same.
Introduce a customization interface to be able to store data in i.e. local DB/maybe Kontent storage.

Additional context

The plugin functionality would work -> i.e. module with specific Data.
Think about the testability -> the test could be prepared for the interface for better DX.

See https://github.com/digital-detox/kontent-migration-cli

Migration run with specific name does not check if migration was already executed in status.json

Brief bug description

What went wrong?
Running the already executed migration is aborted by error from the management client.

Repro steps

1. Run some migration with for example in range (migration run --range 3:5)
2. Run migration run -n "alreadyExecutedMigration" and it gives the error from the ManagementClient.

Expected behavior

Should skip the already executed migration.

Test environment

• Platform/OS: [e.g. .NET Core 2.1, iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Additional context

Add any other context about the problem here.

Screenshots

Add links to screenshots, if possible.

Status management does not work correctly

Brief bug description

The last migration is saved into a status file, however, the tool skips only this last migration and tries to run all migrations before this one.

Feat: Update @kentico/kontent-management

Motivation

Update @kentico/kontent-management dependency to version 0.3.2 which adds support for following endpoints:

• list/add/modify asset folders
• add/get/list webhooks endpoints
• project information endpoint
• single taxonomy item

and

• adds ability to automatically retrieve all items from paging queries
• contains few improvements to types and fixes certain models.

For list of changes see changelog at https://github.com/Kentico/kontent-management-sdk-js/blob/master/CHANGELOG.md

Replace status.json with content items in Kontent Project

Motivation

When keeping the state of the migrations not in the status.json file, it would be easier to implement migrations in a CICD pipeline. My suggestion would be to keep this state as close as possible to the project / environment and store it as separate content-type(s).

Proposed solution

If the CLI could figure out based on projected & environment what the current state of the migrations is it would be really easy to run the migrations in an automatic stateless deployment process without telling the CLI exactly what to migrate.
With the current implementation we need to keep the status.json in sync between different deployment-agents in our CICD pipeline which makes it really hard to implement

Additional context

The proposed solution is heavily inspired by the EF Core Code first migration

Add codecov

Motivation

Add codecov to know how much of the code by tests

Proposed solution

Find out how to add codecov to readme and implement it.

Additional context

Add any other context, screenshots, or reference links about the feature request here.

Backup command accepts projectId argument instead of environmentId

Brief bug description

The kbm backup command should accept --environmentId according to documentation, but instead it accepts --projectId

Expected behavior

The backup command should accept --environmentId

Rollback function does not execute

Brief bug description

There is a bug that rollback function does not execute

Repro steps

1. Create a migration with rollback function defined
2. Try to run migration with --rollback switch
3. See error

Expected behavior

migration rollback should be executed and status.json updated

Test environment

Additional context

Add any other context about the problem here.

Screenshots

Add links to screenshots, if possible.

Feat: add support for backing & restoring via backup manager

Motivation

Extends & unifies use of kontent CLI actions so that devs are able to easily backup & restore their projects using this repo without the need of using backup manager separately.

Proposed solution

Add set of CLI commands that will make use of the https://github.com/Kentico/kontent-backup-manager-js

Introduce up and down migrations

Motivation

To be able to use it as rollback from: https://github.com/digital-detox/kontent-migration-cli#commands

Proposed solution

Think about the state information setup.

Additional context

• There wouldn't be any fail-safe for that.

apiClient.modifyTaxonomy is not a function

Brief bug description

Attempting to update a taxonomy group in a CLI script. All other migration scripts are working as intended so my overall setup is fine.

What went wrong?
Check out the test script below. Adding the taxonomy works. I get an error that "apiClient.modifyTaxonomy is not a function" when I try to update it.

An error occurred while running migration: update_taxonomy_test.js see the output from running the script.
Message: apiClient.modifyTaxonomy is not a function

Repro steps

Run the following:

const migration = {
    order: 01,
    run: async (apiClient) => {
        // Create a test taxonomy - works
        const addTaxonomyResponse = await apiClient
        .addTaxonomy()
        .withData(
              {
                name: "Test",
                codename: "test_group",
                terms: [{
                    name: "Some Term",
                    codename: "some_term",
                    terms: []
                }],
              },
        )
        .toPromise();    
        
        // Try to modify taxonomy - error 'apiClient.modifyTaxonomy is not a function'
        await apiClient
        .modifyTaxonomy()
        .byTaxonomyCodename('test_group')
        .withData(
            [
              {
                op: 'replace',
                property_name: 'codename',
                value: 'updated_test_group'
              },
            ]
        )
        .toPromise();
    }
};

module.exports = migration;

Expected behavior

The function should exist and the taxonomy should be updated.

Test environment

• Platform/OS: MacOS Big Sur 11.0.1
• Package Dependencies
  • "@kentico/kontent-cli": "0.0.9",
  • "dotenv": "^8.2.0",
  • "faker": "^5.1.0",
  • "rxjs": "^6.6.3"
  • I also have "@kentico/kontent-delivery": "^10.1.0" for some Preview API work
  • In an attempt to fix the issue (and suspecting the management API might need an update - I tried adding "@kentico/kontent-management": "^0.4.7" when I hit the error to see if I needed a newer version of that. This didn't fix my issue.

Change backup from backup-manager to data-ops

Brief bug description

The backup manager is deprecated

Repro steps

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

What is the correct behavior?

Test environment

• Platform/OS: [e.g. .NET Core 2.1, iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Additional context

Add any other context about the problem here.

Screenshots

Add links to screenshots, if possible.

[FR] Add option to disable error logging

Motivation

The CLI logs verbose errors to the command line and adds a lot of extra noise to the CLI output when errors are handled properly. This can be disabled easily when making requests using the JavaScript SDK.

const managementClient = new ManagementClient({
   apiKey: 'x',
   projectId: 'y',
   httpService: new HttpService({
      logErrorsToConsole: false
   })
});

The CLI configures it’s own httpService and doesn't allow additional options to be configured. The default info logged out is easy to confuse with an error in the migration script when infact the error is safely handled in the migration.

Proposed solution

Add the ability to disable logging errors to the console through a param on the run command or when invoking apiClient so users can opt in to their own error handling.

Additional context

A case where this is useful is when updating all language variants of an item with a new value and some items are translated and others aren’t, so they don't need to be updated and will return a 404 if you attempt to get data from the variant.

If logging were disabled and the following code were in a migration:

for (const language of languages.data.items) {
    try {
        const languageVariant = await apiClient.viewLanguageVariant()
            .byItemCodename("item_codename")
            .byLanguageId(language.id)
            .toPromise()

        // Item isn’t translated in this language
        if (!languageVariant) {
            return
        }
       
       // Do stuff and upsert language variant
    } catch(error) {
        console.log(error.message)
    }
}

Output would be:

    The requested variant '{id: cd571471-a7ec-4f9c-b5e5-421f10becf13}' of content item '{codename: item_codename}' was not found.

Instead of the verbose log containing the full response.

Update Kontent Backup manager

Motivation

Once Kontent Backup Manager is released in v3.0, upgrade the dependency and deal with the breaking changes.
Ideally think of some more elegant way, how to set up this dependency future-proof for new version of backup manager.

Proposed solution

See related draft pull request.

Additional context

Go through // TODO comments before merge

Enable generating new migrations with shortcuts 'js' and 'ts'

Motivation

Usually, when deciding between javascript and typescript, you can write shortcuts 'js' or 'ts' instead of full names 'javascript' and 'typescript'. Slight improvement, but it can reduce the annoyance of always getting errors when specifying only shortcuts.

Proposed solution

Add shortcuts into src/cmds/migration.add.ts.

Additional context

Add any other context, screenshots, or reference links about the feature request here.

Create a helper function for common Kontent.ai changes

Expected result

Allow users to make more complicated operations by single and easy call i.e. add content type to allowed content type of another content type's element.

Additional context

• Decide whether these functions should be accessible only programmatically, or if it makes sense to provide them via the CLI as well.

Resources

Examples - will be @tommarhall

Model TS classes need to be held in separate folders inside or outside Migrations folder

Motivation

The Kentico model generator (https://github.com/Kentico/kontent-model-generator-js) allows users to generate TS classes to create strictly typed objects to receive data from the Content Delivery APIs (https://github.com/Kentico/kontent-delivery-sdk-js/blob/master/DOCS.md). However there is no way to maintain a subfolder inside or outside the Migrations folder (https://github.com/Kentico/kontent-migrations-boilerplate/tree/master/Migrations) without running into the following errors during execution of the migration CLI:

Proposed solution

Modify the CLI to allow TS files (typically model classes) to be maintained outside or inside the Migrations folder. There should be a way to set the compiler to allow folders with model files during Transpiling. This will allow easier maintenance of code.

Change to projectId to environmentId and update to management-sdk v5

Motivation

Currently the CLI uses management-sdk 3.1 but you already have v5 with new options in it.

Proposed solution

Update dependency to version 5 and change project-id usage to environment-id

Can't run migration by following the tutorial

Brief bug description

What went wrong?

Try following the instructions https://github.com/Kentico/kontent-cli/#-migration-example and run a migration, but get a error:

echuvelev@chuvelev-mbp  ~/Code/kontent-migrations-boilerplate   master ● ?  npm run migrate "01_sample_init_createBlogType"                                  1 ↵  20:40:55

> kontent-migrations-boilerplate@1.1.1 migrate
> tsc && kontent migration run --environment DEV -n "01_sample_init_createBlogType"

kontent migration run

Runs a migration script specified by its name, or runs multiple migration scripts in the specified order.

Options:
  --version                Show version number  [boolean]
  -h, --help               Show help  [boolean]
  --name, -n               Migration name  [string]
  --project-id, -p         Project ID to run the migration script on  [string]
  --api-key, -k            Management API key  [string]
  --environment, -e        Environment name  [string]
  --all, -a                Run all migration scripts in the specified order  [boolean]
  --range, -r              Run all migration scripts in the specified range, eg.: 3:5 will run migrations with the "order" property set to 3, 4 and 5  [string]
  --force, -f              Enforces run of already executed scripts.  [boolean] [default: false]
  --continue-on-error, -c  Continue executing migration scripts even if a migration script fails.  [boolean] [default: false]
  --debug, -d              Run in debug mode  [boolean] [default: false]

TypeError: Cannot read properties of undefined (reading 'match')
    at Object.exports.getRange (/Users/echuvelev/Code/kontent-migrations-boilerplate/node_modules/@kentico/kontent-cli/lib/cmds/migration/run.js:180:25)
    at /Users/echuvelev/Code/kontent-migrations-boilerplate/node_modules/@kentico/kontent-cli/lib/cmds/migration/run.js:120:34
    at Generator.next (<anonymous>)
    at /Users/echuvelev/Code/kontent-migrations-boilerplate/node_modules/@kentico/kontent-cli/lib/cmds/migration/run.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (/Users/echuvelev/Code/kontent-migrations-boilerplate/node_modules/@kentico/kontent-cli/lib/cmds/migration/run.js:4:12)
    at Object.handler (/Users/echuvelev/Code/kontent-migrations-boilerplate/node_modules/@kentico/kontent-cli/lib/cmds/migration/run.js:115:24)
    at Object.runCommand (/Users/echuvelev/Code/kontent-migrations-boilerplate/node_modules/yargs/build/lib/command.js:196:48)
    at Object.parseArgs [as _parseArgs] (/Users/echuvelev/Code/kontent-migrations-boilerplate/node_modules/yargs/build/lib/yargs.js:1043:55)
    at Object.runCommand (/Users/echuvelev/Code/kontent-migrations-boilerplate/node_modules/yargs/build/lib/command.js:157:36)

Repro steps

1. Clone repo kontent-migrations-boilerplate
2. Follow the instruction - https://github.com/Kentico/kontent-cli/#3-run-a-migration
3. See the error

Expected behavior

Running the 01_sample_init_createBlogType.js migration.
Status file was updated see /Users/echuvelev/Code/kontent-migrations-boilerplate/status.json
The "01_sample_init_createBlogType.js" migration on a project with ID "7bc88b7e-80d2-0087-4b4e-a33e80420328" executed successfully.

What the correct behavior is?

• Migration has run successfully.

Test environment

• Platform/OS: MacOS Monterey 12.3.1

Additional context

Add any other context about the problem here.

Screenshots

Add links to screenshots, if possible.

Upgrade to new JS SDK

Motivation

New SDK has a new version with a bunch of new features. We need to accommodate the news.
https://github.com/Kentico/kontent-management-sdk-js/blob/master/CHANGELOG.md

Proposed solution

New JS SDK does not need rxjs - Update readme.
Test out that that setting latest version as a peer dependency works fine.

Verify Migration consistency

Motivation

Currently, you can run all migrations (and possibly a range of them #32). It would be great t have some verification mechanism, that would notify you you have order property in migration properly in place - From 1 to X.

Proposed solution

Raise warning/exception if the migrations do not have a range being set properly.

items_count_limit does not work

Brief bug description

I am attempting to modify an existing content type and add a linked items with a items_count_limit restriction but it does not work.

Repro steps

1. Make sure you have a content type you want to create a migration to add a new linked items item.
2. For the value of the modification, add items_count_limit. Specify the condition and value.
3. Compile the TypeScript migration into JavaScript.
4. Run the migration with kontent migration run -n test -e DEV.
5. Check your Kentico project to see the results.
6. Notice the items_count_limit had no effect.

Expected behavior

I expect the items_count_limit restriction to apply.

Additional context

Below is my migration code.

const migration: MigrationModule = {
  order: 1,
  run: async apiClient => {
    await apiClient
      .addContentType()
      .withData(BuildChecklistItemType)
      .toPromise()

    await apiClient
      .modifyContentType()
      .byTypeCodename('checklist')
      .withData(ChecklistModification)
      .toPromise()
  },
}

const BuildChecklistItemType = (
  builder: ContentTypeElementsBuilder,
): ContentTypeModels.IAddContentTypeData => ({
  codename: 'checklist_item_1',
  elements: [
    builder.textElement({
      codename: 'text',
      is_required: true,
      name: 'Text',
      type: 'text',
    }),
  ],
  name: 'Checklist Item 1',
})

const ChecklistModification: ContentTypeModels.IModifyContentTypeData[] = [
  {
    op: 'addInto',
    path: '/elements',
    value: {
      allowed_content_types: [
        {
          codename: 'checklist_item_1',
        },
      ],
      codename: 'items_1',
      is_required: true,
      items_count_limit: { // LOOK HERE
        condition: 'exactly',
        value: 4,
      },
      name: 'Items 1',
      type: ElementModels.ElementType.modularContent,
    },
  },
]

Here is the result in Kentico:

Everything worked as expected except the items_count_limit.

Please let me know if I can provide any additional information. Thank you! :)

Unable to run a TypeScript migration

Bug Description

There is a way to create both JavaScript and TypeScript migrations but I can only run JavaScript migrations. When I try to run my new TypeScript migration, it says Couldn't import the migration script from C:\CFAS\KenticoKontentCli\Migrations\test.ts.js due to an error: "Cannot find module C:\CFAS\KenticoKontentCli\Migrations\test.ts.js".

It looks like it is looking for a JavaScript file by default. It assumes the migration is JavaScript.

Repro Steps

1. Run kontent migration add --name test --template-type "typescript".
2. Create your migration.
3. Run kontent migration run --name test --environment DEV.
4. Observe error.

Expected Behavior

It should run the TypeScript migration file.

Additional Context

I attempted to run the TypeScript migration file provided by the boilerplate. I deleted the JavaScript file and left the TypeScript file alone to see what it would do. It gave me an error because it is looking for a JavaScript file.

I was looking at the code for run and I have a feeling this might have something to do with it.

Perhaps there could be a flag to specify if the migration is JavaScript or TypeScript. If not, have run determine it automatically. If it cannot find the .js file, then look for a .ts file.

Screenshots

Please let me know if I can provide any further details. Thank you! :)

Upgrade @kentico/kontent-management

Motivation

kontent-cli has a dependency of kontent-management locked at version 0.3.13. The kontent-management version is now at 0.3.17. The kontent-management upgrade will fix linting issues for the kontent-cli.

Proposed solution

Perhaps upgrade the kontent-management dependency or maybe it should be a peer dependency.

Please let me know if I can provide any more details. Thank you!

Use transpiled JS files with static HTML

Motivation

This feature would allow people who use the https://github.com/Kentico/kontent-migrations-boilerplate to be able to reference the transpiled JS files using the <script> tag in HTML files. Thus API calls could be tested in the browser.

The transpiled JS files also need to be created in a seperate folder (could be named 'dist') by the CLI. The "outDir" config in TSCONFIG could be used, but it creates the error that "all source files must be in the same directory".

Additional context

The framework now supports just using the kontent cli to perform the migrations. This makes the user heavily dependant on the CLI to test all scenarios.

Make Kontent MAPI SDK a peer dependency

Expected result

For the projects that are using Kontent CLI for managing their migrations, it is difficult to set the version of MAPI SDK since it is hardcoded in the dependencies.

Let's verify whether setting this dependency as a peer dependency would fix the issue.

Additional context

Also, it is hard to verify alpha/beta versions in the kontent migration boilerplate - specifically this verification is blocked: kontent-ai/migrations-boilerplate#3

Resources

Also it would fix the issues similar to #17 kontent-ai/management-sdk-js#47