shivansh / smoketestsuite Goto Github PK

View Code? Open in Web Editor NEW

4.0 4.0 2.0 367 KB

Smoke testing infra for all utilities in the FreeBSD base system (made as a part of Google Summer of Code '17)

Home Page: https://wiki.freebsd.org/SummerOfCode2017/SmokeTestingOfBaseUtilities

License: BSD 2-Clause "Simplified" License

Makefile 1.26% C 10.29% Shell 24.39% C++ 64.05%

freebsd base-system smoke-testing infrastructure

smoketestsuite's Introduction

Smoke testing of base utilities (FreeBSD)

Test generation tool made as a part of Google Summer of Code '17 with FreeBSD.
Refer the FreeBSD wiki for an overview and updates.

Project layout

├── deprecated_tests
│   └── ............................:: Tests pertaining to initial test plan
└── src ............................:: Automation tool pertaining to new test plan
    ├── annotations
    │   └── ........................:: Annotation files (generated/user-defined)
    ├── generated_tests
    │   └── ........................:: Generated atf-sh test scripts
    ├── scripts
    │   └── ........................:: Helper scripts
    ├── add_testcase.cpp ...........:: Testcase generator
    ├── generate_license.cpp .......:: Customized license generator
    ├── generate_test.cpp ..........:: Test generator
    ├── logging.cpp ................:: Logger
    ├── read_annotations.cpp .......:: Annotation parser
    └── utils.cpp ..................:: Index generator

Automation tool

An implementation of the automation tool briefly described here [1][2].
The following diagram summarizes how different components fit with the testcase-generator -

Dependencies

Boost C++ libraries : The tool was tested to work with the port boost-all-1.64.0.
Kyua

Instructions

NOTE: The tool is yet to be merged in the FreeBSD src, and the status can be tracked here. In case using it before the merge, the contents of src should be copied under <local_FreeBSD_src>/tools/tools/smoketestsuite before proceeding further.

The tool needs to know about the utilities in src which don't already have tests. The list of all such utilities can be generated via -
```
make fetch_utils
```
This will generate the file scripts/utils_list.
For generating the tests, execute the following -
```
make clean
make && make run
```

A few demo tests are located in src/generated_tests.

smoketestsuite's People

Contributors

Stargazers

Watchers

Forkers

aniketp gaomeitao

smoketestsuite's Issues

Update definition of exec() to return output/error at stdout/stderr

Dynamically generate environment dependent "successful" testcases

Design an interface to facilitate test creation

Some good features -

Base utilities of choice should be selected on the fly
Not all tests need to be produced at once

Add functionality to ignore options which depend on user input

Specifically avoid these kind of tests.
For e.g. ln -i depends on user input.

Add support for long options

Adding support for this feature will also fix the options which are missed due to the following kind of definitions (taken from date(1)) -

.It Fl r Ar seconds

Fix test generation for tset(1)

terminate called after throwing an instance of 'std::out_of_range'
  what():  basic_string::substr: __pos (which is 7) > this->size() (which is 6)
Generating test for: tset(1) ...*** Signal 6

Fix test for domainname(1)

Generate a list of all base utilities currently without tests

Running in Batch mode barely passes any tests

Run in 'batch mode' ? [y/N] y
Number of utilities to select for test generation: 5

           Utility | Progress
         ----------+-----------
        fingerd(8) | 5/5
echo '#! /usr/libexec/atf-sh' > fingerd_test.tmp
cat fingerd_test.sh >>fingerd_test.tmp
chmod +x fingerd_test.tmp
mv fingerd_test.tmp fingerd_test
install  -o root  -g wheel -m 555  fingerd_test  /usr/tests/libexec/fingerd/fingerd_test
install  -o root  -g wheel -m 444  Kyuafile  /usr/tests/libexec/fingerd/Kyuafile
fingerd_test:invalid_usage  ->  failed: atf-check failed; see the output of the test for details  [0.031s]
fingerd_test:no_arguments  ->  failed: atf-check failed; see the output of the test for details  [0.029s]

Results file id is usr_tests_libexec_fingerd.20180130-202309-467799
Results saved to /root/.kyua/store/results.usr_tests_libexec_fingerd.20180130-202309-467799.db

0/2 passed (2 failed)
          talkd(8) | 0/0
echo '#! /usr/libexec/atf-sh' > talkd_test.tmp
cat talkd_test.sh >>talkd_test.tmp
chmod +x talkd_test.tmp
mv talkd_test.tmp talkd_test
install  -o root  -g wheel -m 555  talkd_test  /usr/tests/libexec/talkd/talkd_test
install  -o root  -g wheel -m 444  Kyuafile  /usr/tests/libexec/talkd/Kyuafile
talkd_test:no_arguments  ->  failed: atf-check failed; see the output of the test for details  [0.056s]

Results file id is usr_tests_libexec_talkd.20180130-202309-795414
Results saved to /root/.kyua/store/results.usr_tests_libexec_talkd.20180130-202309-795414.db

0/1 passed (1 failed)
         rbootd(8) | 3/3
echo '#! /usr/libexec/atf-sh' > rbootd_test.tmp
cat rbootd_test.sh >>rbootd_test.tmp
chmod +x rbootd_test.tmp
mv rbootd_test.tmp rbootd_test
install  -o root  -g wheel -m 555  rbootd_test  /usr/tests/libexec/rbootd/rbootd_test
install  -o root  -g wheel -m 444  Kyuafile  /usr/tests/libexec/rbootd/Kyuafile
rbootd_test:invalid_usage  ->  failed: atf-check failed; see the output of the test for details  [0.014s]
rbootd_test:no_arguments  ->  failed: atf-check failed; see the output of the test for details  [0.014s]

Results file id is usr_tests_libexec_rbootd.20180130-202309-992132
Results saved to /root/.kyua/store/results.usr_tests_libexec_rbootd.20180130-202309-992132.db

0/2 passed (2 failed)
           ftpd(8) | 26/26
echo '#! /usr/libexec/atf-sh' > ftpd_test.tmp
cat ftpd_test.sh >>ftpd_test.tmp
chmod +x ftpd_test.tmp
mv ftpd_test.tmp ftpd_test
install  -o root  -g wheel -m 555  ftpd_test  /usr/tests/libexec/ftpd/ftpd_test
install  -o root  -g wheel -m 444  Kyuafile  /usr/tests/libexec/ftpd/Kyuafile
ftpd_test:invalid_usage  ->  failed: atf-check failed; see the output of the test for details  [0.013s]
ftpd_test:no_arguments  ->  failed: atf-check failed; see the output of the test for details  [0.014s]

Results file id is usr_tests_libexec_ftpd.20180130-202310-186159
Results saved to /root/.kyua/store/results.usr_tests_libexec_ftpd.20180130-202310-186159.db

0/2 passed (2 failed)
        rlogind(8) | 4/4
echo '#! /usr/libexec/atf-sh' > rlogind_test.tmp
cat rlogind_test.sh >>rlogind_test.tmp
chmod +x rlogind_test.tmp
mv rlogind_test.tmp rlogind_test
install  -o root  -g wheel -m 555  rlogind_test  /usr/tests/libexec/rlogind/rlogind_test
install  -o root  -g wheel -m 444  Kyuafile  /usr/tests/libexec/rlogind/Kyuafile
rlogind_test:invalid_usage  ->  failed: atf-check failed; see the output of the test for details  [0.013s]
rlogind_test:no_arguments  ->  failed: atf-check failed; see the output of the test for details  [0.013s]

Results file id is usr_tests_libexec_rlogind.20180130-202310-311523
Results saved to /root/.kyua/store/results.usr_tests_libexec_rlogind.20180130-202310-311523.db

0/2 passed (2 failed)

All the tests are failing on running the script in Batch mode. I tried this with multiple values "Number of Utilities" but to no benefit.

[Batch mode] Update utility makefile

Fix test for rm(1)

rm -f exits with a zero status, unlike in the corresponding testcase.
The same behavior is repeated for -

sync
domainname

make && make run fails on generate-annot.sh

On running tests using make && make run as mentioned in README, I get this error

Generating annotations...
sh /root/smoke/src/scripts/generate_annot.sh
kyua: E: No previous results file found for test suite usr_tests_bin_date.

Fix usage message collation

brandelf_test.sh#49

Remove global variable "opt_list"

Escape single quotes in inline messages

Missing usage variable in mkdir_test.sh

The commit which broke this - afe9f92 ; The assumption made is false.

Exit status in mkdir_test.sh is supposed to be 64

System reboots on running the tests in Normal mode

Run in 'batch mode' ? [y/N] N

           Utility | Progress
         ----------+-----------
        fingerd(8) | 5/5
          talkd(8) | 0/0
         rbootd(8) | 3/3
           ftpd(8) | 26/26
        rlogind(8) | 4/4
    rpc.rquotad(8) | 0/0
     rpc.rwalld(8) | 1/1
      bootptest(8) | 3/3
    revnetgroup(8) | 2/2
     rpc.rstatd(8) | 0/0
      addr2line(1) | 11/11
             nm(1) | 41/41
        strings(1) | 7/7
        cxxfilt(1) | 6/6
          mtree(8) | 34/34
          bmake(1) | 0/0
        smbutil(1) | 1/1
         loader(8) | 0/0
         pxeldr(8) | 0/0
        zfsboot(8) | 0/0
        gptboot(8) | 0/0
           init(8) | 0/0
           ping(8) | 30/30
     fdisk_pc98(8) | 10/10
        rcorder(8) | 2/2
         comsat(8) | 0/0
   mount_cd9660(8) | 9/9
        hastctl(8) | 5/5
       sunlabel(8) | 9/9
           clri(8) | 0/0
       shutdown(8) | 6/6
          pfctl(8) | 67/67
      adjkerntz(8) | 2/2
    conscontrol(8) | 0/0
           ipfw(8) | 14/14
       bsdlabel(8) | 0/0
           natd(8) | 31/31
    nvmecontrol(8) | 0/0
          devfs(8) | 2/2
         nfsiod(8) | 1/1
          route(8) | 7/7
         bootpd(8) | 6/6
  mount_unionfs(8) | 2/2
         swapon(8) | 7/7
        elfcopy(1) | 15/15
          mknod(8) | 0/0
     zfsbootcfg(8) | 0/0
        rtquery(8) | 8/8
         ggatel(8) | 6/6
         ggated(8) | 8/8
         ggatec(8) | 11/11
          dmesg(8) | 4/4
         umount(8) | 8/8
       savecore(8) | 8/8
       ldconfig(8) | 10/10
           fsdb(8) | 3/3
            md5(1) | 7/7
        badsect(8) | 0/0
         dumpfs(8) | 0/0
      kldunload(8) | 4/4
   fsck_msdosfs(8) | 6/6
         dumpon(8) | 0/0
    newfs_msdos(8) | 23/23
            ddb(8) | 0/0
        fsirand(8) | 3/3
     rpc.sprayd(8) | 0/0
sh: Syntax error: Unterminated quoted string
sh: Syntax error: Unterminated quoted string
 etherswitchcfg(8) | 3/3
     comcontrol(8) | 0/0
        kldload(8) | 3/3
        nos-tun(8) | 0/0
      mount_nfs(8) | 26/26
          fdisk(8) | 13/13
        restore(8) | 17/17
    recoverdisk(1) | 4/4
          unvis(1) | 4/4
          ping6(8) | 31/31
  mount_msdosfs(8) | 12/12
        openssl(1) | 0/0
         crypto(1) | 0/0
         gencat(1) | 0/0
          lsvfs(1) | 1/1
         ktrace(1) | 9/9
       procstat(1) | 15/15
           from(1) | 3/3
             nl(1) | 11/11
        elfdump(1) | 16/16
            tip(1) | 2/2
           size(1) | 12/12
          truss(1) | 11/11
Connection to localhost closed by remote host.
Connection to localhost closed.

To test whether some specific utilities like truss or rpc.sprayd were causing the issue, I removed them from the utils_list. But even then this problem persists.

atf_test_case invalid_usage
invalid_usage_head()
{
        atf_set "descr" "Verify that an invalid usage with a supported option " \
                        "produces a valid error message"
}

invalid_usage_body()
{
        atf_check -s not-exit:0 -e inline:"kill: option requires an argument -- s
usage: kill [-s signal_name] pid ...
       kill -l [exit_status]
       kill -signal_name pid ...
       kill -signal_number pid ...
" kill -s
        atf_check -s not-exit:0 -e inline:"HUP INT QUIT ILL TRAP ABRT EMT FPE KILL BUS SEGV SYS PIPE ALRM TERM URG
STOP TSTP CONT CHLD TTIN TTOU IO XCPU XFSZ VTALRM PROF WINCH INFO USR1 USR2
" kill -l
        atf_check -s not-exit:0 -e inline:"kill: unknown signal Ar; valid signals:
HUP INT QUIT ILL TRAP ABRT EMT FPE KILL BUS SEGV SYS PIPE ALRM TERM URG
STOP TSTP CONT CHLD TTIN TTOU IO XCPU XFSZ VTALRM PROF WINCH INFO USR1 USR2
" kill -Ar
        atf_check -s not-exit:0 -e inline:"kill: unknown signal Ar; valid signals:
HUP INT QUIT ILL TRAP ABRT EMT FPE KILL BUS SEGV SYS PIPE ALRM TERM URG
STOP TSTP CONT CHLD TTIN TTOU IO XCPU XFSZ VTALRM PROF WINCH INFO USR1 USR2
" kill -Ar
}

atf_test_case no_arguments
no_arguments_head()
{
        atf_set "descr" "Verify that kill(1) fails and generates a valid usage " \
                        "message when no arguments are supplied"
}

no_arguments_body()
{
        atf_check -s not-exit:0 -e match:"$usage_output" kill
}

atf_init_test_cases()
{
        atf_add_test_case invalid_usage
        atf_add_test_case no_arguments
}