sparshithnr/doc-tester

Test your markdown file code samples.

DocTester

This library is used to run code samples in markdown documents as acceptance tests, using simple comment-based assertions. By testing code snippets in documentation, you can have greater confidence that code in your documentation works as advertised. doc-tester expects code snippets in the markdown to follow a certain assertion language. Please rewrite markdown file to adhere to doc-tester's format.

Table of Contents

1. Installation
2. Usage
   • From Commandline
   • From Code
3. How to use import
4. Assertions in Doc

Installation

npm install doc-tester

If you are using yarn

yarn add doc-tester

Usage

From commandline

doc-tester

options

1. -f (--file)

Path of file to be tested (default: ./README.md).

2. -c (--cleanup)

Setting this to false will not remove the test file generated by parsing the documentation file (default :true).

3. --inspect (--inspect-brk)

Runs tests w/ the node --inspect option, allowing a debugger to be attached (default: false).

4. -o (--output)

Path where generated test file will be written. (default: ./test.js)

From code

import { runTest } from 'doc-tester';
await runTest({
  codeArray: ['add(3,4) // equals: 7;'],
  importsArray: [`import { add } from './add'`]
} /* , options */) // equals: true;

Options

1. testName

Name for the test block. Defaulted to Doc Test.

2. cleanup
3. inspect
4. output

How to use import

In order for absolute imports to work, make sure what you're importing is present in your project's node_modules/ folder.

Relative imports are generally a bad idea in documentation examples -- create examples as your users would need to think about them -- consuming a library as a dependency, using absolute imports.

Please keep in mind that some import-related issues can be solved by symlinking a project's root directory (or build output) into its own node_modules/ folder, effectively simulating what your users might see in their own node_modules when depending on your library.

You can use the code below sample code get the test working, replace doc-tester with your module name.

const fs = require('fs-extra');
try {
  fs.symlinkSync(__dirname, `./node_modules/doc-tester`);
} catch(err) {
  if (err.code == 'EEXIST') {
    console.log('doc-tester already symlinked to node_modules');
  } else {
    throw err;
  }
}

Check the package.json of this project how I do it.

Assertions in Doc.

The doc-tester expects code block to be marked with language, ex: js, ts etc. It decides whether to generate tests for the given code block or not based on this. This library also expects to follow the below syntax in the doc.

statement // assertionType: expectedResult

Assertion types equals, not-equals, throws and deep-equals are supported as of now.

// Let's assume README has snippet Math.sqrt(4) // 2
// statement is Math.sqrt(4), expected value is 2 and assertionType is equals. We rewrite it to,
Math.sqrt(4); // equals: 2

Math.sqrt(4) // not-equals: 4
new Error('3'); // throws
{ a : 5, b: 6}; // deep-equal: { a : 5, b: 6 }

Note: The statement should be a non-assignment statement.

ex: a=3; //equals: 3 can not be tested.

doc-tester's Issues

README should describe comment assertion language

Please add to the README: a list (and examples) of the special comments we can use in code blocks

4; // equal: 4

Add mocha support

Add CI

I would recommend travis-ci linux, and ideally also windows.

document public API from library in readme

Once this API stabilizes, could this API be documented in the readme? That way:

1. the library can dog-food itself more
2. other folks can easily integrate programatically

A line that does not end with `;` causes some issues:

A small modification to the example in the readme reveals new lines are ignored:

import {
  subtract
} from './add';
const numbers = {
  a: 4,
  b: 3
};
console.log('hello')
subtract(numbers.a, numbers.b) // equals: 1;
add( {
  a: 4
}.a, 4) // equals: 8;

Specifically I added a console.log( without a trailing ; (which is valid JS)

Result:

node ./bin/doc-test -f ./README.md                                                                                        (127)
Not all tests passed:
SyntaxError: unknown: Unexpected token, expected "," (12:59)

  10 | assert.notEqual(add(4, 5), 6);
  11 | assert.throws(() => { throw add(2, 3) }, 'Throws error');
> 12 | const numbers = {  a: 4,  b: 3};assert.equal(console.log(1)subtract(numbers.a, numbers.b), 1);
     |                                                            ^
  13 | assert.equal(add( {  a: 4}.a, 4), 8);
  14 | ;assert.expect(5);
  15 |     });

Readme should describe how inline `imports` work

Specifically, they should explain that they abide by the node resolution algorithm, based on the doc file they are written in.

On that note, should we provide the accurate __dirname and __filename variables to these test files?

Add debugging/inspecting support

Given this slightly modified example from the readme (debugger inserted).

import {
  subtract
} from './add';
const numbers = {
  a: 4,
  b: 3
};
debugger;
subtract(numbers.a, numbers.b) // equals: 1;
add( {
  a: 4
}.a, 4) // equals: 8;

I would expect (or similar):

./bin/doc-test --inspect -f README.md
./bin/doc-test --inspect-brk -f README.md

To break within the executed tests, and be debuggable from chromes about:inspect

Add support using existing assertions w/ typescript code snippets

It should be relatively easy to handle this by installing the appropriate babel 7 plugins. I think it's just

{
    "presets": [
        "@babel/env",
        "@babel/typescript"
    ],
    "plugins": [
        "@babel/proposal-class-properties",
        "@babel/proposal-object-rest-spread"
    ]
}

At which point this test case

```ts
function add(a: number, b: number) { return a + b; }
add(3, 4); // equals: 7
```

should work just as easily as your JS examples. You may need to write the code w/ tests to a file instead of evaluating it as a string, in order to make @babel/preset-typescript happy

Minimum viable README

• #6 document public API in README
• #8 document how inline imports work
• #19 document comment assertion language

Only print TAP output to console if run through the CLI tool

This is the output I'm seeing from the documentation tool

  📦  pkg-js-single-file
TAP version 13
ok 1 test > 3 + 4
1..1
# pass 1
# skip 0
# todo 0
# fail 0
    ✅ SECRET_STRING

I'm going to be invoking runTests many times (each file has many symbols, and each symbol can have many doctest blocks) and would like to avoid extra stuff being logged to the console

This is what I'm looking for:

  📦  pkg-js-single-file
    ✅ foo
    ✅ bar
    ❌ baz
        < some data on expected vs. encountered values >