networkservicemesh/gotestmd

gotestmd

Tool to generate go tests based on markdown files.

Usages

Generate suites with default bash runner:

gotestmd INPUT_DIR OUTPUT_DIR

Generate suites that using a custom runner:

gotestmd INPUT_DIR OUTPUT_DIR BASE_PKG

Makrdown syntax

• #Run - OPTIONAL - Contains any text and bash steps. Can be any level, should be used once in a file.
• #Cleanup - OPTIONAL - Contains bash steps. Can be any level, should be used once in a file.
• #Requires - OPTIONAL - Contains a list of required dependencies in format markdown links.
• #Includes - OPTIONAL -Contains a list of using examples in context of this example in format markdown links.

To generate minimal suite required one of sections: Run or Cleanup or Requires.

Examples

See at examples

gotestmd's Issues

Cleanup in bash scripts is not reversed.

Cleanup should happen in reverse order.

For example:

Setup is:

1. Loadbalancer
2. DNS
3. Spire
4. NSM

Then cleanup should be:

1. NSM
2. Spire
3. DNS
4. Loadbalancer

Currently cleanup order is the same as setup.

How to reproduce

Generate any suite with more than 1 dependency.
For example, 2-cluster interdomain suite:

gotestmd /path/to/deployments-k8s/examples ./result-folder --bash --match interdomain

Bash runner doesn't work with multiline output

Expected Behavior

Getting a normal pods description if we run kubectl get pods -n ... on runner:

NAMESPACE            NAME                                         READY   STATUS    RESTARTS   AGE
kube-system          coredns-74ff55c5b-86bbt                      1/1     Running   0          3h20m
kube-system          coredns-74ff55c5b-pjzpk                      1/1     Running   0          3h20m
kube-system          etcd-kind-control-plane                      1/1     Running   0          3h20m
...

Current Behavior

Currently we have only part of unformatted output:

time="2021-05-04T15:29:56+07:00" level=info msg="3h24m\nlocal-path-storage   local-path-provisioner-78776bfc44-vrcdc      1/1     Running   0          3h24m"

Additionally

The same problem occurs if we call similar commands that has multiline output ( e. g. kubectl describe nodes)

Allow to gotestmd parse bidirectional examples

Currently, if the child has a few parents and it declared Requries section then it cannot be correctly parsed by gotestmd.

Solution

In case if we have bidirectional examples(when the child points to multiple parents and parents point to the child) then we should ignore Requires section in the leaf.

Folder names aren't handled correctly

Expected Behavior

Symbols that are valid for golang source tree are preserved, invalid symbols are replaced where needed, these replacements are consistent across result files.

Current Behavior

When a folder, specified in Requires section, contains - symbol in name

1. package name is generated with _ symbol
2. import statements also have _ instead of -
3. but the physical folder where this package is located still contains - symbol
   So import statements are invalid.

Also, beginning of the name isn't preserved for some reason. See "Steps to Reproduce" section for details.

Steps to Reproduce

1. Create a directory where a file depends on folder with - symbol in the name
   Here is an archive with such dependency: gtm-dash-example.zip
2. Run gotestmd . ./gen inside the folder from archive

[if you downloaded the archive] you got folder ./gen/ash-example. First symbol in the name is stripped. Number of symbols removed correlates with source path length (. is 1 symbol, if you use ./ then first 2 symbols are stripped, and so on)

./gen/ash-example/suite.gen.go contains the following import: dash-example/gen/ash_example/dir_with_dashes.
But the folder nearby is dir-with-dashes, so the import must be dash-example/gen/ash_example/dir-with-dashes instead

Generate code for running commands manually

Motivation

Currently gotestmd generates only automatic tests.
A test has requirements, the main body and a cleanup.
User is expected to run tests using go test.
There is no support to run custom commands without integrating them into the testing system.
Generated commands could be re-used in performance testing to keep performance testing up-to-date.

Proposal

Add commands to run the main body and cleanup as separate entities.
For example:

1. Generate test suite
2. Take a test suite A
3. We call A.setup and only seup commands are executed
4. Call some custom commands, which aren't a part of the original tests
5. Manually call the A.cleanup

The difference from current behavior is that setup and cleanup are not linked.
We can take an already generated code and integrate it into some custom program, without using the go test.

Possible issues

1. What if the test relies on the fact that setup and cleanup share the same shell?
2. Would it be possible to get the same effect without changes in gotestmd code, but only using external configuration?

Rework gotestmd to work correctly with remote examples

Motivation

We can use gotestmd with only remote github readme.md examples. So in this case we can remove bash folder moving logic.

this

gotestmd/pkg/suites/shell/suite.go

Line 39 in 739f614

func (s *Suite) Runner(dir string, env ...string) *Runner {

should be reworked to

func (s *Suite) Runner(env ...string) *Runner

Update to integration-tests is not working properly

Actual behaviour

https://github.com/networkservicemesh/gotestmd/runs/4226391214?check_suite_focus=true

Expected behaviour

gotestmd should update the version for integration-tests on merge into main branch.

gotestmd hangs without visible errors when invoked in a dir without go.mod

Expected Behavior

When gotestmd in invoked in an invalid directory, it should print descriptive error message and exit.

Current Behavior

When gotestmd in invoked in an invalid directory, it silently hangs for indefinite amount of time.

Steps to Reproduce

1. Create empty directory
2. Run gotestmd . .

gotestmd panics on generating suites from https://github.com/networkservicemesh/deployments-k8s

Steps to reproduce:

1. Clone https://github.com/networkservicemesh/deployments-k8s
2. Make sure that you have patch: #15
3. Try to generate suites via gotestmd from deployments-k8s

Actual: https://github.com/networkservicemesh/deployments-k8s/runs/2005755642

It's impossible to get output of hanged commands

Expected Behavior

If some problematic command produces output that helps to fix it, it should be possible to get the output from logs.

Current Behavior

Command output is logged only when command has finished. So, if a command never finishes, there are no logs from this command.

Steps to Reproduce

1. Go to Hello World example and add sleep infinity after the echo command.
2. Compile and run the Hello World example.
3. You will never see the output that was produced before infinite sleep command.

Here are logs of running this modified example, with a manual Ctrl+C after some time.

=== RUN   TestRunHelloWorld
time=2021-07-28T14:43:45+07:00 level=info msg=# Hello world!
echo "Hello world!"
sleep infinity TestRunHelloWorld=stdin
^CFAIL  github.com/networkservicemesh/gotestmd  10.011s

Bash runner doesn't handle stderr correctly

Bash runner expects that we either only have some output or only an error of some sort.
But when a command writes something into stderr, we have at least 2 objects somewhere in the b.errCh and b.outCh.
Bash runner takes only one of them:

gotestmd/pkg/suites/shell/bash.go

Lines 168 to 175 in 36a9ba9

	select {
	case err = <-b.errCh:
	return "", err
	case output = <-b.outCh:
	return output, nil
	case <-b.ctx.Done():
	return "", b.ctx.Err()
	}

So the other one can potentially be taken by consequent commands. This creates issues both with logs and with determining whether the command has actually succeeded or not.
Also if we don't take the result from the channel, the goroutine that writes into the channel will be blocked on channel send operation. It won't read its corresponding exec pipe, and due to pipes having a fixed size the pipe can potentially be overflown, thus blocking all the programs that tries to write into it. This results in commands not being run at all after some point.

Buffer size is limited that can produce leaks of messages in output

Example output

See at
describe-ns-hcsmb.log
as you can see describe is not full.

Steps to reproduce

1. Create a new unit test
2. Create a bash instance
3. Write a file with text that has size more than 10000 bytes
4. Verify that we don't lose any byte in the output