roadiehq/backstage-entity-validator

Validate properties and well known annotations in your Backstage catalog-info.yaml files.

Backstage entity validator

This package can be used as a GitHub action or a standalone node.js module

GitHub action

Inputs

path

Optional Path to the catalog-info.yaml file to validate. Defaults to catalog-info.yaml at the root of the repository. It also can be a glob like services/*/catalog-info.yaml or a list of files separated by comma users.yaml,orgs/company.yaml.

verbose

Optional Specify whether the output should be verbose. Default true.

validationSchemaFileLocation

Optional Specify the location of the validation schema file.

Outputs

None. Prints out the validated YAML on success. Prints out errors on invalid YAML

Example usage

- uses:  RoadieHQ/[email protected]
  with:
    path: 'catalog-info-1.yaml'

- uses:  RoadieHQ/[email protected]
  with:
    path: 'catalog-info-1.yaml,catalog-info-2.yaml,catalog-info-3.yaml'

- uses:  RoadieHQ/[email protected]
  with:
    path: 'catalog-info-*.yaml,services/**/*/catalog-info.yaml'

- uses:  RoadieHQ/[email protected]
  with:
    path: 'catalog-info-*.yaml,services/**/*/catalog-info.yaml'
    validationSchemaFileLocation: 'custom-validation-schema.json'

CircleCI Orb

Inputs

path

Optional Path to the catalog-info.yaml file to validate. Defaults to catalog-info.yaml at the root of the repository.

Outputs

None. Prints out the validated YAML on success. Prints out errors on invalid YAML

Example config

description: >
  Sample catalog-info.yaml validation
usage:
  version: 2.1
  orbs:
    entity-validator: "roadiehq/[email protected]"
  workflows:
    use-entity-validator:
      jobs:
        - entity-validator/validate:
            path: catalog-info.yaml

Using the CLI

Usage

Usage: validate-entity [OPTION] [FILE]

Validates Backstage entity definition files.  Files may be specified as
arguments or via STDIN, one per line.

OPTION:
-h  display help
-q  minimal output while validating entities
-i  validate files provided over standard input
-l  location of custom validation schema file

Examples:

# in a shell

# validate all entities contained in the "catalog" and subfolders
validate-entity catalog/**/*.yaml

# list of files produced by a script to validate
find-relevant-yaml-files.sh | validate-entity -i 

Installing and running

As a global tool

# install
npm install --global @roadiehq/backstage-entity-validator

# run
validate-entity file1.yaml file2.yaml

In an existing node project

# install
npm install --save-dev @roadiehq/backstage-entity-validator

# run
npx validate-entity file1.yaml file2.yaml

When working on this tool

# install
npm install

# run
npm run validate file1.yaml file2.yaml

# or
bin/bev file1.yaml file2.yaml

backstage-entity-validator's Issues

GitHub Action: Allow multiple files as input

Hi, thank you so much for creating this validator. I've noticed that the input (in a Github Action context) can be only a single path. In our monorepo context this doesn't scale very well. Ideally I would like to be able to define a glob or pass in a list of files like

- uses: RoadieHQ/backstage-entity-validator@@0.3.0
  with:
    path: './a/service-a.yaml ./b/service-b.yaml ./c/service-c.yaml'

or as a glob

- uses: RoadieHQ/backstage-entity-validator@@0.3.0
  with:
    path: './*/service-*.yaml'

Would you accept such a contribution?

techdocs default mkdoc.yaml

Version:

v0.3.11

What happened:

Starting from version 1.11.0, Backstage now includes a default mkdoc.yaml file. backstage/backstage#15417

https://github.com/backstage/backstage/blob/9e826a3f8796b5f5bfd2f2eb31406c9bec327bdd/docs/releases/v1.11.0-changelog.md?plain=1#L833

When the following annotation is set, the following error message is displayed Failed to validate catalog-info.yaml: Error: Techdocs annotation specifies "dir" but file under mkdocs.yaml|mkdocs.yml not found

apiVersion: backstage.io/v1alpha1
metadata:
  annotations:
    backstage.io/techdocs-ref: dir:.

Expected behavior:

The validation should pass when the above annotations are set, as the default mkdocs.yaml file will be utilised.

Additional Information:

Handle nunjucks syntax for template files

Describe Request:

Running this against the cli generated example template that contains a contents folder with templated catalog-info file where the name parameter uses nunjucks to dynamically create the name, fails as the validator expects a string matching a limited regex for the name parameter.

Examples:

Adding https://github.com/backstage/software-templates/tree/main/scaffolder-templates/pull-request fails validation due to:

Error: "metadata.name" is not valid; expected a string that is sequences of [a-zA-Z0-9] separated by any of [-_.], at most 63 characters in total but found "${{ values.name | dump }}". To learn more about catalog file format, visit: https://github.com/backstage/backstage/blob/master/docs/architecture-decisions/adr002-default-catalog-file-format.md

Supporting Documentation Links:

Circle CI Orb build not executing integration tests

What happened:

I was looking to make some enhancements to the Circle CI Orb, to make it a little easier to use however on looking at the build I realise that the Orb tests defined in the integration-test-1 job are never invoked

Expected behavior:

The CircleCI build should run the integration tests to verify the Orb works before publishing.

Additional Information:

Due to limitations in Orb testing it might not be possible to test out unhappy paths, but it should be straightforward to run through the happy path examples.

Looks like this orb is based on a slightly older orb template, but you can see how the tests should be invoked by looking at the template here

Allow to pass custom spec -> definition for API entity

Describe Request:

I would like to pass custom definitions (like Kafka topics) to a catalog-info API entity definition but because of hardcoded validator rule it's impossible to do that and pass validation.
Could you please add the possibility to pass definition options?

Examples:

apiVersion: backstage.io/v1alpha1
kind: API
metadata:
  namespace: my-namespace
  name: api-name
  annotations:
    gitlab.com/project-id: '12345678'
spec:
  type: api-type
  owner: group:owner-group
  lifecycle: experimental
  definition:
    $topic: topic.area.name.1

Validation of this catalog info will fail because of hardcoded tule here:

function modifyPlaceholders(obj) {
  for (const k in obj) {
    if (typeof obj[k] === 'object') {
      try {
        if (obj[k].$text || obj[k].$openapi || obj[k].$asyncapi) {
          obj[k] = 'DUMMY TEXT';
          return;
        }
      } catch (e) {
        throw new Error(
          `Placeholder with name '${k}' is empty. Please remove it or populate it.`,
        );
      }
      modifyPlaceholders(obj[k]);
    }
  }
}

But with the following update, it'll pass:

function modifyPlaceholders(obj, condition) {
  for (const k in obj) {
    if (typeof obj[k] === 'object') {
      try {
        for (const c of condition) {
          if (obj[k][c]) {
            obj[k] = 'DUMMY TEXT';
            return;
          }
        }
      } catch (e) {
        throw new Error(
          `Placeholder with name '${k}' is empty. Please remove it or populate it.`,
        );
      }
      modifyPlaceholders(obj[k], condition);
    }
  }
}

Where const condition = ['$text', '$openapi', '$asyncapi', '$topic']

Supporting Documentation Links:

https://github.com/RoadieHQ/roadie-backstage-plugins/blob/0b7f60228b6de7c7d0baa5033aff6b105d258d5b/utils/roadie-backstage-entity-validator/src/validator.js#L39C1-L55C2

Auto-discover metadata files

Describe Request:

Instead of requiring users to manually specify the list of Backstage YAML files, we should check all YAML files for apiVersion: backstage.io/v1alpha1 and syntax check them if they have changed.

This would make it easier to automatically lint Backstage metadata files when running the validator as a hosted check solution.

Support System, Domain and the v1beta2 templates

System, Domain and the v1beta2 template kinds should be validated.

They currently don't exist in the list:

backstage-entity-validator/src/validator.js

Line 24 in b92b6eb

const VALIDATORS = {

No such file or directory error during CI build

Version:

Version 0.3.8

What happened:

We are trying to use the entity validator in an Azure DevOps build and are hitting this error:

/usr/bin/env: 'node\r': No such file or directory

The call we are making is: validate-entity 'catalog/**/*.yaml' -q

We store all our catalog entities as subfolders under catalog. This works fine on a local machine but fails as soon as it runs in the CI.

Expected behavior:

It should validate the entities without error.

Additional Information:

I'm honestly just looking for some advice as to what we might be doing wrong. Thanks for making this tool!

[Question] Entity validator source

Hi there,

I was having a look at the entity validator and I notice that it points to a library hosted on NPM here: https://www.npmjs.com/package/@roadiehq/roadie-backstage-entity-validator

The actual source code doesn't seem to be available from what I can tell though unless I'm missing something?

I can understand that RoadieHQ may want to keep it closed source as it's useful for their business but just wanted to confirm that I haven't overlooked another repo 🙂

Thanks!

DependsOn resource syntax error not detected

Version: 0.3.11

What happened:

I created a catalog-info.yml file with a service that declared a dependency on a resource. However, I had a syntax error in it and backstage actually complained about it when importing. However, the validator did not.

The mistake I made was writing this:

spec:
  type: service
  lifecycle: production
  owner: team-name
  dependsOn:
    - resource/default:kinesis-stream # wrong
    - resource:default/kinesis-stream # correct

Expected behavior:

I would expect the same error message that backstage gives:
Entity reference "resource/default:kinesis-stream" had missing or empty kind (e.g. did not start with "component:" or similar)

Additional Information:

Follow links in Location files

Describe Request:

A good way to represent a Monorepo in Backstage is to have a Location file at the root of the repo and include links in it to point to other files in the subdirectories where the metadata actually lives.

The validator should follow the links in these Location files to discover and validate the metadata in the subdirectories.

Update action to use Node 16

Describe Request:

Github has announced they will be deprecation of Node 12 in actions and recommending upgrading to use Node 16. Using this action currently results in the following warning.

Node.js 12 actions are deprecated. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/. Please update the following actions to use Node.js 16

Supporting Documentation Links:

https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/

Support `$yaml` and `$json` fields in API Kind definitions

Describe Request:

When validating entries with Kind: API, the validator only checks for a string type on the spec.definition field at the moment. The backstage model however has added limited support for object types on the definition field when the keys are one of the following:

• $json
• $yaml
• $text

Examples:

From the Backstage Documentation

apiVersion: backstage.io/v1alpha1
kind: API
metadata:
  name: petstore
  description: The Petstore API
spec:
  type: openapi
  lifecycle: production
  owner: [email protected]
  definition:
    $text: https://petstore.swagger.io/v2/swagger.json

Supporting Documentation Links:

https://backstage.io/docs/features/software-catalog/descriptor-format#substitutions-in-the-descriptor-format

Update action to use Node 20

Github actions has been giving deprecation notices for actions using node 16 and recommends they upgrade to node 20.

Verboseness of verbose flag

Describe Request:

With the verbose flag set to true (default) the output includes a dump of every yaml file the validator is processing. This becomes quite noisy in a monorepo. I wonder what the discussion process was behind adding this output and if you would be open to remove it or put it behind another flag.

Support Gitlab Pipelines

Describe Request:

I would love to use the validator in my GitLab pipelines.

At the moment I have to maintain my own image, maybe you could support this by adding a Docker build to the pipeline and provide it as an image in your existing Dockerhub org?

I would be happy to create a PR for a Dockerfile and the initial action but of course can't configure the credentials for the org.

Examples:

validate:
  stage: test
  image: roadiehq/backstage-entity-validator
  script:
  - validate-entity file1.yaml

Supporting Documentation Links:

• https://github.com/marketplace/actions/build-and-push-docker-images

Possible unnecessary entry in schema

This is new to me, so it is possible that I have misunderstood something, but I believe that this line should not be here:

backstage-entity-validator/src/schemas/annotations.schema.json

Line 16 in 68841ee

"uniqueItems": true,

I've been seeing this error when I run this validator:

strict mode: missing type "array" for keyword "uniqueItems" at "Entity metadata annotations#/properties/metadata/properties/annotations" (strictTypes)

Based on my read of the json schema spec, uniqueItems should only be present for array types, not objects - relevant doc

Validation fails when using `backstage.io/techdocs-ref: dir:.` annotation

Version:

0.3.10

What happened:

When validating a catalog-info.yml file with backstage.io/techdocs-ref: dir:. set, validation fails with

Failed to validate repositories/api-gateway.yaml: Error: Techdocs annotation specifies "dir" but file under repositories/mkdocs.yaml|repositories/mkdocs.yml not found

We use a monorepo for all our catalog files, so the app code and mkdocs.yml file are not in the same repo where we want to run the validator.

Expected behavior:

Backstage claims that is the default value when using external storage for techdocs (like S3 in our case). Ideally this would pass validation.

Additional Information:

The schema should be extendable

Describe Request:

In our org we have decided that we want pull requests to fail if the catalog-info.yaml has set spec.lifecycle to anything other than "production" or "experimental". We would like to enforce this via the entity validator.

Add support for YAML files with multiple documents

Describe Request:

Examples:

Supporting Documentation Links:

Referential integrity should be checked

There are cases where a Backstage metadata file will fail to import because of a group or other dependency that doesn't exist.

For example, if Group:engineering doesn't exist in the Backstage instance we try to import this YAML into, the import will fail.

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: sample-service-1
  description: A service for testing Backstage functionality
  annotations:
    github.com/project-slug: RoadieHQ/sample-service
spec:
  type: service
  owner: Group:engineering
  lifecycle: experimental

The validator should detect these failures as part of its standard suite of checks.

Locations fail without `spec` attribute which is inconsistent with docs

Version:

0.3.7

What happened:

validator fails with:

Your catalog file(s) is/are not valid:

There was an error verifying entity-templates/catalog-info-2.yml:
TypeError: <root> should have required property 'spec' - missingProperty: spec

Expected behavior:

the validator should succeed (no errors should arise). This should happen because of these backstage docs which show spec as optional

Additional Information:

here is a sample Location (catalog-info.yaml)

apiVersion: backstage.io/v1alpha1
kind: Location
metadata:
  name: todo-example-project

a fix was to add a dummy spec:

apiVersion: backstage.io/v1alpha1
kind: Location
metadata:
  name: todo-example-project
spec:
  type: dummy

Orb usage examples are wrong - recommend adding a job to the orb to make them right

Version:

0.4.2

What happened:

The usage examples for the Circle Orb suggest a job is provided by the orb, when it is in fact a command. This means the examples don't actually work.

Instead of doing this:

description: >
  Sample catalog-info.yaml validation
usage:
  version: 2.1
  orbs:
    entity-validator: "roadiehq/[email protected]"
  workflows:
    use-entity-validator:
      jobs:
        - entity-validator/validate:
            path: catalog-info.yaml

You actually need something like this:

description: >
  Sample catalog-info.yaml validation
usage:
  version: 2.1
  orbs:
    entity-validator: "roadiehq/[email protected]"
   
jobs:
  validate:
    docker:
      - image: cimg/node:lts
    steps:
      - checkout
      - entity-validator/validate:
          path: catalog-info.yaml
      
workflows:
  use-entity-validator:
    jobs:
      - validate

Expected behavior:

Whilst this could just be a documentation update, ideally the orb would provide both a job and command. The job would minimise the amount of config required to get something up and running quickly, and also mean the examples would work

Additional Information:

I'm happy to make a PR with the changes to add a job, but I'm a bit nervous because the build doesn't actually test the orb is working (#46)

Is this maintained?

Hi RoadieHQ, is this stilling being maintained to keep up with backstage schema changes etc?

Fix automatic NPM release via Github action

Looks to be unable to find the correct packages in npmjs.org and responds with 404 making it necessary to release that package manually.