checkupjs/checkup

A health checkup for your project.

checkup's Issues

[task] mixin migrations

Write a task that returns the number mixins remaining to migrate, with links to documentation on the "dangers of mixins". For each mixin, we should also return the number of time the mixin is being used (to help approximate scope/effort to remove)

[task] number of skipped tests

Write a task that returns the number of skipped tests

[enhancement]: Allow outdated dependencies task to work with custom registries

Evaluate error handling across the CLI

We should have a consistent pattern for handling errors across the CLI. Errors that occur should follow the following rules:

1. Error output should be visually consistent
2. Errors should make a distinction between internal and user errors
3. The error should have a short, descriptive message
4. There should be a call to action if the error requires user action to resolve
5. There should be no exceptional error stacks propagated to stdout/html (expected error cases can output stacks)
6. Additional information should be written to a log file in the working directory

[task] number of stateless services

Write a task that returns the number of stateless services (aka the number of services that shouldn't be services)

Add missing dependencies of @checkup/core

Currently @checkup/core is missing the direct dependency on a few libs.

• globby
• cli-ux
• cosmicconfig

[task] Functions without JSDocs

Write a task that returns the number of functions without JSDocs. This can be extended to also check the number of Components without JSDocs defining the interface, etc

Improvements on `file-searcher`

The following improvements/changes need to be made to file searching primitives:

1. Remove file-searcher-task
2. Refactor file-searcher.ts to just export functions, and remove base class
3. Refactor the two functions to have the same APIs

Add support to allow checkup to be run programmatically via Node API

Use case: if developers want to run checkup as part of a script, they shouldn't need to shell out to do so. There should be a way to programmatically invoke checkup directly to make this process cleaner

[task] Linting Summary

Write a task that runs all of the linting defined in the project being "checked up on" and returns a summary of the current violations.

There should be multiple breakdowns of data here - number of violations by filetype (eslint v templatelint v stylelint), breakdown of violations by rule, etc.

Proposed change to aggregated result structure

I'd like to propose a change to the aggregated result structure - the structure we ultimately end up with when we merge task results.

First, I propose we change the categories to the following:

1. Meta - gathers meta data to be output in an unstructured format, such as project name, version, repo info, checkup version and config info, etc.
2. Insights - tasks such as dependency info, types, tests, etc.
3. Migrations - this one is obvious, migrations that are being tracked

When we process the task results, we'd 'lift' the meta tasks out to a top level meta bucket, leaving the remaining tasks that were categorized as either Insights or Migrations in a sibling results bucket. This would allow us greater control over the meta output, as it pertains to reporting.

Here's the proposed format:

{
  "meta": {
    "project": {
      "name": "my-project",
      "version": "0.0.1",
      "repository": {
        "activeDays": "0 days",
        "age": "0 days",
        "totalCommits": 0,
        "totalFiles": 0
      }
    },
    "checkup": {
      "configHash": "9b74c9897bac770ffc029102a200c5de",
      "version": "0.0.1"
    }
  },
  "results": [
    {
      "meta": {
        "taskName": "mock-task",
        "friendlyTaskName": "Mock Task",
        "taskClassification": {
          "category": "insights",
          "priority": "medium"
        }
      },
      "result": {
        "mockTask1": 5
      }
    },
    {
      "meta": {
        "taskName": "mock-task2",
        "friendlyTaskName": "Mock Task 2",
        "taskClassification": {
          "category": "insights",
          "priority": "high"
        }
      },
      "result": {
        "mockTask2": 10
      }
    },
    {
      "meta": {
        "taskName": "mock-task3",
        "friendlyTaskName": "Mock Task 3",
        "taskClassification": {
          "category": "insights",
          "priority": "low"
        }
      },
      "result": {
        "mockTask2": 10
      }
    },
    {
      "meta": {
        "taskName": "mock-task4",
        "friendlyTaskName": "Mock Task 4",
        "taskClassification": {
          "category": "migrations",
          "priority": "low"
        }
      },
      "result": {
        "mockTask2": 10
      }
    }
  ]
}

We need to figure out a generic way to output the meta data in a nice way for reporting, but it shouldn't be too complicated.

Please comment with your thoughts.

Plugins should be identifiable by a keyword

Oclif plugins are identifiable via the oclif-plugin keyword added to the plugin's package.json. We should add a similar designation for checkup plugins.

checkup-plugin

When loading plugins, we should validate that the keyword is present. We should also update the plugin generator to include this keyword.

[task] Dependency freshness

Write a task that breaks down dependency freshness for all deps in the repo, and organizes them by most -> least out of date

[bug] can not generate plugin, task or config.

When running checkup generate plugin foo ./lib I get the following error:

 ›   Error: No valid generator found for plugin. Valid generators are base-generator.d, 
 ›   base-generator.d.ts.map, base-generator.js, base-generator.js.map, config.d, config.d.ts.map, 
 ›   config.js, config.js.map, plugin.d, plugin.d.ts.map, plugin.js, plugin.js.map, task.d, 
 ›   task.d.ts.map, task.js, task.js.map

When I log out what this.validGenerators I get:

[
  'base-generator.d',
  'base-generator.d.ts.map',
  'base-generator.js',
  'base-generator.js.map',
  'config.d',
  'config.d.ts.map',
  'config.js',
  'config.js.map',
  'plugin.d',
  'plugin.d.ts.map',
  'plugin.js',
  'plugin.js.map',
  'task.d',
  'task.d.ts.map',
  'task.js',
  'task.js.map'
]

Rename tasks to be more descriptive of what they _do_

I feel like we should start thinking about naming tasks more what they do, like what eslint does. In that sense, we should think about naming this FindTodosInCommentsTask and the file would be find-todos-in-comments-task.ts. This makes it much easier to understand what the task does just by reading the filename. We should also go through each current task and make sure its name makes sense.

Rename all tasks to be verbs that describe what the tasks are doing.

[task] Ember Test type and breakdown

Write a task that returns the types of tests in the application, with some information about module length (average module length, number of modules over x lines long, etc)

Update plugin generator to standardize naming

• if you generate using checkup generate plugin foobar, it will create checkup-plugin-foobar as your plugin name
• if you generate using checkup generate plugin checkup-plugin-foobar, it will generate as is
• if you generate into a directory that's named checkup-plugin-*, it will generate the files inside that directory instead of creating a new top level dir

Simplify config generation and loading

The current config setup is quite complicated. We can simplify the overall loading and generation of configs by using some simple functions:

• for generators: utilize yeoman style generator files for both JSON and JS
• for loading: provide simple load function that will load either a JSON or JS config, or throw if no config found.

Basic setup TODO's

• Move linting configuration (deps, eslintrc, etc) into the project root
• Move gitignore into project root
• Add testing for multiple node versions
• Add testing for windows?
• Change the linting config to be less annoying (closer to a thin layering on top of eslint:recommended + eslint-plugin-node + prettier)

[proposal] - Remove HTML reporting in favor of external report dashboard

The current plan with the CLI included the ability to output to an HTML generated report. While the use of an HTML report is still a viable path, I propose we disconnect it from the CLI itself.

This will do the following:

1. Simplify the CLI options and formats. The CLI will only concern itself with simplified console output and JSON. The JSON will be writable to a file, and consumable via external sources.
2. Simplify the path to creating schemas for tasks/task results. The idea would be to provide simple, generic result objects that can be populated by tasks, and output by the system. Defining a type in the result object will allow you to hint to the system how you want your result output. This is effectively the path to marrying the structure we were moving towards with result data, and the looseness with which we currently allow outputting to the console/json. The end result should be simpler for consumers in the frequent case (most people just want to gather some data and have it output in a consistent fashion), but allow for extension for custom/complex cases (you want to create a task result yourself, and specify the output).
3. Allow us to focus, separately, on a better, more robust HTML output that better allows formatting and traversing verbose report content.

I propose we

1. completely remove all HTML report functionality
2. create a separate, sibling repository in this org to house an HTML reporter app
3. work on simplifying the task result schemas and output to make it easy for plugin and task authors to output data

dotfiles nested inside exclude paths arent being excluded

walksync (what we use to get paths) uses minimatch under the hood, which by default will not match dotfiles. there is an option dot, which is false by default. Spoke to @rwjblue, and will surface dot as a top level option to walksync.

Discussed potentially surfacing all minimatch options as top level to walksync instead, waiting on response from @rwjblue about what he prefers in terms of next steps.

Once that option is surfaced to walksync top level (one way or another), set as true to calls to walksync in get-paths, so dotfiles get excluded appropriately.

Example of the problem :
Right now node_modules/.bin is being included even though we are passing in **/node_modules/** as a path to be excluded

[task] Ember addon/engines count

Need a total of ember addons/engines

• should gather names of addons/engines

Add "Action Items" as a construct to checkup

Summary

At the bottom of the checkup report in console (also be pulled into a separate section for JSON results) we want to show a list of Action Items so a dev running checkup can know what requires immediate attention. Tasks can be highlighted as urgently actionable, phrased in a way that makes it clear to a dev what they are meant to do to fix. Once an Action Item is complete, it will no longer show up in the Action Items list (unlike tasks, which are purely informational, not prescriptive, and will show up with zero results). For example, if you have zero skipped tests, that result would show up. But if there was an Action Item configured to reduce number of skipped tests, and you had zero, it wouldn't show up as an Action Item for you - since you have already done it.

We want to make it possible for both task authors and those configuring checkup to add items to the Action Items list.

Examples

Examples of tasks that would benefit from having Action Items configured by authors:

1. Using yarn
2. Having proper linting set up
3. Using volta

^^^^ The theme emerging here is maybe "Project Prerequisites" or things related to the setup/toolchain of your app.

Examples of why those configuring checkup may want to set up custom Action Items:

1. We are migrating away from pattern foo and it is blocking an important upgrade. We need everyone to work on prioritizing moving off of foo
2. There is a cleanup initiative within an app to remove lint-disables, we may want to configure a threshold such that prioritizing removing lint-disables would come up as an Action Item - so if your app has more than x lint-disables, it would show up as an Action Item
3. For a task that breaks down the testing pyramid, we may have there be a conditional Action Item, such that if you have fewer than x percent of your tests as unit tests, show an Action Item to prioritize refactoring your test suite

^^^^^ theme emerging here is problematic patterns (potentially blocking other work) and overall health of an area of the app.

Requirements

1. Action Items are conditional (by default in a boolean manner - i.e number of problems > 0, but can be configured more explicitly as defined by the task)
2. Action Items can be turned on by default in a task by the task author
3. Tasks that are turned on as Action Items by default can be overridden by config
4. Tasks that aren't turned on as Action Items by default can still have an ActionItemMessage, in the event that they do become Action Items. That message is also overridable by config
5. Tasks that aren't Action Items by default can become Action Items by turning them on as such in the config.
6. There is a max # of Action Items (maybe 5?). If more than that are turned on by default given the plugins a user has enabled, 5 are chosen at random (we should also supply a warning message saying that we dropped Action Items because there are more than 5 enabled by default). Config will validate that no more than 5 can be passed in.

Changes to be made

export interface TaskMetaData {
  taskName: TaskName;
  friendlyTaskName: TaskName;
  taskClassification: TaskClassification;
}

will become

export interface TaskMetaData {
  taskName: TaskName;
  friendlyTaskName: TaskName;
  taskClassification: TaskClassification;
  actionItemData?: ActionItemData;
}

export interface ActionItemData {
  isActionItem: boolean;
  actionItemMessage: string;
  actionItemConditions?: ActionItemCondition[];
}

export interface ActionItemCondition { 
  key: string;
  condition: Function;
  value?: number; 
}

Task authors can choose to expose any number of actionItemConditions to be configured to turn on/off.

Examples:
For the eslint-disable task, an ActionItemCondition could be:

let condition: ActionItemCondition = {
	key: 'numEslintDisables';
    condition: (numEslintDisables, value = 0) => numEslintDisables > value;
    value: 20; 
}

^^ so if there are more than 20 'numEslintDisables', it would show up as an action item.

The config to make eslint-disables an Action Item with its default message would look like this:

{
	actionItems: { 
		'eslint-disables': true,
	}

}

The config to make eslint-disables an Action Item with a custom error message would look like this:

{
	actionItems: { 
		'eslint-disables': {
			message: 'Stop disabling eslint!!!!!!!!!'
		}
	}

}

^^^ if you supply a custom message, it is turned on as an Action Item by default. If this is confusing, we can also add an enabled property, but i dont see why someone would supply a custom message without turning it on.......

The config to conditionally make eslint-disables an Action Item with a custom error message would look like this:

{
	actionItems: { 
		'eslint-disables': {
			message: 'Stop disabling eslint!!!!!!!!!',
			condition: 'numEslintDisables',
			value: 20
		}
	}

}

This change would require verifying config upfront to ensure that the conditions being used in the config are in fact exposed by a given task.

I am open to the idea that adding this level of flexibility may be more complexity that it is worth, and would only be used by power users. I think it is quite powerful and can see many uses for it in my app, but I can ack that it is complicated. We can potentially reduce the scope here by just making action items toggleable on/off in a boolean manner, and maybe adding the custom message as well, without the conditional piece (basically removing ActionItemCondition from ActionItemData).

Looking forward to feedback :)

Idea: new generator command

Adding a new checkup plugin to configuration can be handled via generators.

$ checkup config:add plugin checkup-plugin-ember

This would do the following:

1. Add the checkup-plugin-ember entry to the plugins array in the config
2. Add the checkup-plugin-ember package to the package.json's devDependencies
3. Run yarn/npm install

[docs] Write "Authoring a Task"

Should include:

1. generating a task
2. Task, TaskResult classes structure
3. Testing

[task] Project info

Write a task that returns the project info

[docs] Write getting started doc

Should include:

1. installing checkup
2. generating a config
3. adding plugins
4. running checkup

Config generation workflow issues

Following the setup process defined at https://github.com/checkupjs/checkup/blob/master/packages/cli/README.md it notes to generate a config file via running

$ checkup generate config

However that produces an error

So I did this instead:

$ checkup generate config.js

Following the prompts, I selected Ember and JSON yet the resulting file is .checkuprc which does not match any of the options that I chose when generating the file.

I would like to see the docs be updated to match the correct syntax for file generation (or the file generation to be fixed to match the docs) and then the resulting file to match the options passed .checkuprc.json in this case or .checkuprc.js if I chose JS or .checkuprc.toml if I choose TOML

Update plugin generator

• Add checkup-plugin keyword to package.json

• REmove src/types/index.ts

• Update @checkup/core and @checkup/test-helpers to ^0.0.1

• Update tsconfig.json to not inherit

  {
      "compilerOptions": {
        "declaration": true,
        "importHelpers": true,
        "module": "commonjs",
        "strict": true,
        "target": "es2017",
        "allowSyntheticDefaultImports": true,
        "sourceMap": true,
        "outDir": "lib",
        "rootDir": "src",
        "types": ["jest", "node"]
      },
      "include": ["src/**/*"]
    }
  

• Update package.json with @types/jest and @types/node

[task] Ember Core Dependency Versions

Write a task that defines what are "Ember core deps" and returns the version of each. Potential enhancement - return latest of each of these deps so people can see the difference between where they are and where they should be

[docs] Write "Authoring a plugin"

Should include:

1. generating a plugin
2. how to build and test
3. how to publish
4. directory structure

Add ability to provide options to tasks via config

Similar to how eslint provides the ability to provide options to rules, we need to be able to provide options to tasks.

Based on the current config format, we should allow options to be passed to Tasks via the task object.

type CheckupConfig = {
    plugins: string[];
    tasks: {
        [fullyQualifiedTaskName: string]: "on" | "off" | ["on" | "off", unknown];
    };
}

• The key format is

  ember/ember-test-task

  ember - short plugin name (the plugin name with the checkup-plugin segment removed)
  / - delimiter between plugin name and task name
  ember-test-task - task name

• The value format is

boolean - indicates whether a task is enabled or disabled. When enabled, this key/value will not be present. When disabled, will be false.
object - when an object is provided, this constitutes the options passed to the task.

The checkup config will be passed to each task, which should have a getter in the base task to read the specific tasks' config values from the checkup config.

We may want to consider allowing tasks to be skipped up front - TBD

A nice feature would be some analysis of the available git information

None of these are my ideas, they all come directly from the book Software Design X Rays: Fix Technical Debt with Behavioral Code Analysis by Adam Tornhill :)

But it would be super cool to have analysis of what are the “hot” files, or where do we have lots of different people touching the same files which can be broken down to more granular modules, where do we have related changes always happening in tandem indicating tight coupling, among other things.

ember-es6-class/no-object-extend rule

checkup/packages/checkup-plugin-ember-octane/src/utils/task-configs.ts

Line 12 in 8f88c8b

rules: [

any reason this chunk isn't using the no-object-extend rule defined here?
https://github.com/scalvert/eslint-plugin-ember-es6-class

this is what our team used to track our native class migration

[task] Project/Repository meta info

Need to collect simple project repo info

• name
• version
• repo age
• active days
• total commits
• total files

[task] Dependency Security Issues

We should consider exposing security issues via https://www.npmjs.com/package/@npmcli/arborist and https://www.npmjs.com/package/npm-audit-report#basic-usage-example

[ui][stdout] Update stdout output infra to be consistent

The current stdout infra via the ui module is fairly disjointed and doesn't lead to consistency for task authors. We should add some standardized ui ouput functions that provide consistency when outputting a task.

Known corner cases/bugs for plugin-ember-octane

• NaN being returned for migration tasks when there are no files found in association to ESLint migration tasks
• [ ] Erroneously counting files as migrated via ESLint when file extends a non-ember construct. Example: a component definition that extends something other than @ember/component.
• Refactor types out into proper file structure
• [ ] requirement for babel-eslint to be install via host app (might be something to do with linking)
• [ ] requirement for specific version of ember-template-lint to be installed via host app (might be something to do with linking)

[enhancement] ember-template-lint additional types

I am trying to add a local plugin for ember-template lint that looks something like the following;

const TEMPLATE_LINT_CONFIG: TemplateLintConfig = {
  plugins: [
    {
      name: 'local-rules',
      rules: {
        'hello-world': require(path.resolve(__dirname, '../../template-lint-rules/hello-world')),
      },
    },
  ],
  rules: {
    'hello-world': 'error'
  },
};

Looking at the types for ember-template-lint config it looks like it is missing some fields.

checkup/packages/core/src/types/ember-template-lint.ts

Lines 29 to 33 in 9fe60a9

	export interface TemplateLintConfig {
	rules?: {
	[name: string]: RuleLevel | RuleLevelAndOptions;
	};
	}

ember-template-lint doesn't expose types you can see the top level properties that are allowed here:

https://github.com/ember-template-lint/ember-template-lint/blob/b7195d0b6ae8051aff904a49bf9969741cfbf877/lib/get-config.js#L119-L126

I am using @checkup/[email protected]

[task] Octane migrations

plugin-ember-octane

• ESLint JSON
• ESLint Console
• Template Lint JSON
• Template Lint Console

Change "Eslint Overrides" to "eslint-disable Usages"

It's hard to understand what Eslint Overrides" means. Does it mean config overrides, rule overrides? I would honestly not have attributed this to eslint-disable` annotations. I think we should rename this to make it more clear.

[task] Number of lint overrides

Write a task that returns the number of lint overrides (broken down by type of override - eslint, stylelint, templatelint, etc)

Plugin generator: Generate with latest version of packages

The package.json should generate the latest version of the @checkup packages.

• @checkup/core
• @checkup/test-helpers

Can use yarn info <package> version --json for this

[task] Checkup meta info

Need a meta task that outputs

• checkup version
• checkup config (hashed value for comparison purposes)

Update task generator

• Add meta category (once #174 is merged)
• Remove src/tasks/index.ts (it's unnecessary)
• Related to above, change register-task.ts import to single line import from direct task file (no longer importing from index)

Adding support to checkup to operate on specific paths

Currently, checkup can only be run on an entire application, beginning at the root which is passed in via (the only) command line args. We need to know the root for a number of reasons, one of which being it is where checkups goes to look for the checkup config.

Checkup needs to be able to operate on a subset of the files in a directory, primary use case being large applications with shared ownership. In order to provide detailed and targeted insights to the right people, we need to account for shared ownership of an application as a first class citizen, and provide separate checkup reports for each of the separate ownership "groups".

To accomplish this, the following changes need to be made:

1. The paths command line arg will be used to pass in globs in the same manner eslint does.
2. The root directory will be passed in via a --cwd flag, defaulting to ..
3. All tasks that will operate on these globs directly, instead of assuming they operate on everything inside root.

Implementation details of task related changes:
Right now there are 4 types of tasks we support - the following changes will need to be made to each type to allow for this functionality.

1. Meta tasks (provide meta info about the application such as name, days active, etc.)
   • Meta tasks will still operate on the root
2. Tasks that read and analyze JSON
   • Tasks that operate on package.json (dependency related tasks) will take in a flag that determines whether they operate on the root package.json, or follow the globs. Open question - what happens for dependency tasks when multiple package.jsons are found? Tasks need to account for this.
3. Tasks that invoke lint rules
   • All tasks that invoke lint rules simply need to pass in the globs to the parsers, as eslint and templatelint (the only parsers we have right now), support this in a first class manner. Worth noting, any new parser that that developers intend to add in the future must have this functionality.
4. String search/directory search
   • file-searcher.ts will need to be modified to account for this functionality.

Extensions:

1. Enable the configuration of paths to operate on in the checkup config. Note - if there are paths specified in the config, and paths specified via command line, command line will be prioritized and paths specified in the config will be overriden by command line

Improvements for config generator

• Provide output indicating what file was produced
• Show the file name of the file generated

[task] Type breakdown (Ember)

Write a task that breaks down the different types of constructs in an Ember app

[task] Number of TODOs

Need to determine number of TODOs (TODO, todo) in comments

[feature] more stack info when error is thrown

Motivation

When an error is thrown in a checkup task the resulting output is the following;

It would be great to see additional stack information about where the error is coming from or a full stack that can be referenced in an external file.