trailofbits/manticore

Symbolic execution tool

⚠️ Project is in Maintenance Mode ⚠️

This project is no longer internally developed and maintained. However, we are happy to review and accept small, well-written pull requests by the community. We will only consider bug fixes and minor enhancements.

Any new or currently open issues and discussions shall be answered and supported by the community.

Manticore

Manticore is a symbolic execution tool for the analysis of smart contracts and binaries.

Features

• Program Exploration: Manticore can execute a program with symbolic inputs and explore all the possible states it can reach
• Input Generation: Manticore can automatically produce concrete inputs that result in a given program state
• Error Discovery: Manticore can detect crashes and other failure cases in binaries and smart contracts
• Instrumentation: Manticore provides fine-grained control of state exploration via event callbacks and instruction hooks
• Programmatic Interface: Manticore exposes programmatic access to its analysis engine via a Python API

Manticore can analyze the following types of programs:

• Ethereum smart contracts (EVM bytecode)
• Linux ELF binaries (x86, x86_64, aarch64, and ARMv7)
• WASM Modules

Installation

Note: We recommend installing Manticore in a virtual environment to prevent conflicts with other projects or packages

Option 1: Installing from PyPI:

pip install manticore

Option 2: Installing from PyPI, with extra dependencies needed to execute native binaries:

pip install "manticore[native]"

Option 3: Installing a nightly development build:

pip install --pre "manticore[native]"

Option 4: Installing from the master branch:

git clone https://github.com/trailofbits/manticore.git
cd manticore
pip install -e ".[native]"

Option 5: Install via Docker:

docker pull trailofbits/manticore

Once installed, the manticore CLI tool and Python API will be available.

For a development installation, see our wiki.

Usage

CLI

Manticore has a command line interface which can perform a basic symbolic analysis of a binary or smart contract. Analysis results will be placed into a workspace directory beginning with mcore_. For information about the workspace, see the wiki.

EVM

Manticore CLI automatically detects you are trying to test a contract if (for ex.) the contract has a .sol or a .vy extension. See a demo.

Click to expand:

$ manticore examples/evm/umd_example.sol 
 [9921] m.main:INFO: Registered plugins: DetectUninitializedMemory, DetectReentrancySimple, DetectExternalCallAndLeak, ...
 [9921] m.e.manticore:INFO: Starting symbolic create contract
 [9921] m.e.manticore:INFO: Starting symbolic transaction: 0
 [9921] m.e.manticore:INFO: 4 alive states, 6 terminated states
 [9921] m.e.manticore:INFO: Starting symbolic transaction: 1
 [9921] m.e.manticore:INFO: 16 alive states, 22 terminated states
[13761] m.c.manticore:INFO: Generated testcase No. 0 - STOP(3 txs)
[13754] m.c.manticore:INFO: Generated testcase No. 1 - STOP(3 txs)
...
[13743] m.c.manticore:INFO: Generated testcase No. 36 - THROW(3 txs)
[13740] m.c.manticore:INFO: Generated testcase No. 37 - THROW(3 txs)
[9921] m.c.manticore:INFO: Results in ~/manticore/mcore_gsncmlgx

Manticore-verifier

An alternative CLI tool is provided that simplifies contract testing and allows writing properties methods in the same high-level language the contract uses. Checkout manticore-verifier documentation. See a demo

Native

Click to expand:

$ manticore examples/linux/basic
[9507] m.n.manticore:INFO: Loading program examples/linux/basic
[9507] m.c.manticore:INFO: Generated testcase No. 0 - Program finished with exit status: 0
[9507] m.c.manticore:INFO: Generated testcase No. 1 - Program finished with exit status: 0
[9507] m.c.manticore:INFO: Results in ~/manticore/mcore_7u7hgfay
[9507] m.n.manticore:INFO: Total time: 2.8029580116271973

API

Manticore provides a Python programming interface which can be used to implement powerful custom analyses.

EVM

For Ethereum smart contracts, the API can be used for detailed verification of arbitrary contract properties. Users can set the starting conditions, execute symbolic transactions, and then review discovered states to ensure invariants for a contract hold.

Click to expand:

from manticore.ethereum import ManticoreEVM
contract_src="""
contract Adder {
    function incremented(uint value) public returns (uint){
        if (value == 1)
            revert();
        return value + 1;
    }
}
"""
m = ManticoreEVM()

user_account = m.create_account(balance=10000000)
contract_account = m.solidity_create_contract(contract_src,
                                              owner=user_account,
                                              balance=0)
value = m.make_symbolic_value()

contract_account.incremented(value)

for state in m.ready_states:
    print("can value be 1? {}".format(state.can_be_true(value == 1)))
    print("can value be 200? {}".format(state.can_be_true(value == 200)))

Native

It is also possible to use the API to create custom analysis tools for Linux binaries. Tailoring the initial state helps avoid state explosion problems that commonly occur when using the CLI.

Click to expand:

# example Manticore script
from manticore.native import Manticore

m = Manticore.linux('./example')

@m.hook(0x400ca0)
def hook(state):
  cpu = state.cpu
  print('eax', cpu.EAX)
  print(cpu.read_int(cpu.ESP))

  m.kill()  # tell Manticore to stop

m.run()

WASM

Manticore can also evaluate WebAssembly functions over symbolic inputs for property validation or general analysis.

Click to expand:

from manticore.wasm import ManticoreWASM

m = ManticoreWASM("collatz.wasm")

def arg_gen(state):
    # Generate a symbolic argument to pass to the collatz function.
    # Possible values: 4, 6, 8
    arg = state.new_symbolic_value(32, "collatz_arg")
    state.constrain(arg > 3)
    state.constrain(arg < 9)
    state.constrain(arg % 2 == 0)
    return [arg]


# Run the collatz function with the given argument generator.
m.collatz(arg_gen)

# Manually collect return values
# Prints 2, 3, 8
for idx, val_list in enumerate(m.collect_returns()):
    print("State", idx, "::", val_list[0])

Requirements

• Manticore requires Python 3.7 or greater
• Manticore officially supports the latest LTS version of Ubuntu provided by Github Actions
  • Manticore has experimental support for EVM and WASM (but not native Linux binaries) on MacOS
• We recommend running with increased stack size. This can be done by running ulimit -s 100000 or by passing --ulimit stack=100000000:100000000 to docker run

Compiling Smart Contracts

• Ethereum smart contract analysis requires the solc program in your $PATH.
• Manticore uses crytic-compile to build smart contracts. If you're having compilation issues, consider running crytic-compile on your code directly to make it easier to identify any issues.
• We're still in the process of implementing full support for the EVM Istanbul instruction semantics, so certain opcodes may not be supported. In a pinch, you can try compiling with Solidity 0.4.x to avoid generating those instructions.

Using a different solver (Yices, Z3, CVC4)

Manticore relies on an external solver supporting smtlib2. Currently Z3, Yices and CVC4 are supported and can be selected via command-line or configuration settings. If Yices is available, Manticore will use it by default. If not, it will fall back to Z3 or CVC4. If you want to manually choose which solver to use, you can do so like this: manticore --smt.solver Z3

Installing CVC4

For more details go to https://cvc4.github.io/. Otherwise, just get the binary and use it.

    sudo wget -O /usr/bin/cvc4 https://github.com/CVC4/CVC4/releases/download/1.7/cvc4-1.7-x86_64-linux-opt
    sudo chmod +x /usr/bin/cvc4

Installing Yices

Yices is incredibly fast. More details here https://yices.csl.sri.com/

    sudo add-apt-repository ppa:sri-csl/formal-methods
    sudo apt-get update
    sudo apt-get install yices2

Getting Help

Feel free to stop by our #manticore slack channel in Empire Hacking for help using or extending Manticore.

Documentation is available in several places:

• The wiki contains information about getting started with Manticore and contributing

• The API reference has more thorough and in-depth documentation on our API

• The examples directory has some small examples that showcase API features

• The manticore-examples repository has some more involved examples, including some real CTF problems

If you'd like to file a bug report or feature request, please use our issues page.

For questions and clarifications, please visit the discussion page.

License

Manticore is licensed and distributed under the AGPLv3 license. Contact us if you're looking for an exception to the terms.

Publications

• Manticore: A User-Friendly Symbolic Execution Framework for Binaries and Smart Contracts, Mark Mossberg, Felipe Manzano, Eric Hennenfent, Alex Groce, Gustavo Grieco, Josselin Feist, Trent Brunson, Artem Dinaburg - ASE 19

If you are using Manticore in academic work, consider applying to the Crytic $10k Research Prize.

Demo Video from ASE 2019

Tool Integrations

• MATE: Merged Analysis To prevent Exploits
  • Mantiserve: REST API interaction with Manticore to start, kill, and check Manticore instance
  • Dwarfcore: Plugins and detectors for use within Mantiserve engine during exploration
  • Under-constrained symbolic execution Interface for symbolically exploring single functions with Manticore

manticore's Issues

CPUID with EAX=d not implemented

2017-03-11 22:30:52,647: [21627] MAIN:INFO: Loading program: ['./logmein']
2017-03-11 22:30:52,648: [21627] MAIN:INFO: Workspace: ./mcore_nOMGzP
2017-03-11 22:30:54,362: [21677] CPU:INFO: CPUID with EAX=d not implemented @ 7ffffffd39ec
2017-03-11 22:30:54,363: [21677] EXECUTOR:ERROR: THIS SHOULD NOT REACHABLE! Exception in user code: CPUID with EAX=d not implemented @ 7ffffffd39ec
Traceback (most recent call last):
  File "/root/CTF/HackUCF/manticore/manticore/core/executor.py", line 877, in run
    if not current_state.execute():
  File "/root/CTF/HackUCF/manticore/manticore/core/executor.py", line 157, in execute
    result = self.model.execute()
  File "/root/CTF/HackUCF/manticore/manticore/models/linux.py", line 1642, in execute
    self.current.execute()
  File "/root/CTF/HackUCF/manticore/manticore/core/cpu/abstractcpu.py", line 376, in execute
    implementation(*instruction.operands)
  File "/root/CTF/HackUCF/manticore/manticore/core/cpu/abstractcpu.py", line 610, in new_method
    return old_method(cpu,*args,**kw_args)
  File "/root/CTF/HackUCF/manticore/manticore/core/cpu/x86.py", line 902, in CPUID
    raise NotImplementedError(errormsg)
NotImplementedError: CPUID with EAX=d not implemented @ 7ffffffd39ec

root@kali:~# file logmein 
logmein: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=97397baa37fa328d7a05de07b1779badd8cc948a, not stripped

Fix examples/linux/*c headers

Fix pdb support in hooks

It would be useful to be able to drop into pdb inside a hook, especially for a new user. Right now, we get an error. It's related to our use of multiprocessing somehow.

2017-03-06 12:02:25,464: [94843][0] EXECUTOR:ERROR: THIS SHOULD NOT REACHABLE! Exception in user code:
Traceback (most recent call last):
  File "/mnt/hgfs/code/manticore/manticore/core/executor.py", line 875, in run
    self.will_execute_pc(current_state, current_state.cpu.PC)
  File "/mnt/hgfs/code/manticore/manticore/utils/event.py", line 31, in __call__
    return self.emit(*args, **kwargs)
  File "/mnt/hgfs/code/manticore/manticore/utils/event.py", line 48, in emit
    results.append(f(obj, *args, **kwargs))
  File "/mnt/hgfs/code/manticore/manticore/manticore.py", line 617, in _hook_callback
    cb(state)
  File "statemachine_eval.py", line 35, in check_tainted_cmp
    insn = state.cpu.instruction
  File "statemachine_eval.py", line 35, in check_tainted_cmp
    insn = state.cpu.instruction
  File "/usr/lib/python2.7/bdb.py", line 49, in trace_dispatch
    return self.dispatch_line(frame)
  File "/usr/lib/python2.7/bdb.py", line 68, in dispatch_line
    if self.quitting: raise BdbQuit
BdbQuit

Integrate linter into CI

Add example scripts showing support for symbolic execution, taint analysis, and binary instrumentation

Since these are the three main analysis types we advertise support for

Add other callback types

Gracefully error if z3 is not installed

This is in solver.py. We currently let the OSError bubble up from the Popen.

Add install.sh

write a quick opinionated install.sh script for installing manticore

• checks if in venv and does --user install or not based on that? (http://stackoverflow.com/questions/15454174/how-can-a-shell-function-know-if-it-is-running-within-a-virtualenv)
• does the pip install command
• installs z3
• basic checks that install worked

Move core.mappings to utils

Add a 'context' to State

Provide the ability to add arbitrary data that an analyst would like to keep track of to the State structure. For instance, dereferencing count, heap information, and other data that you'd like to use to guide exploration. (cc @montyly)

Support loading ELF shared objects

use case: analyzing a function in a .so

Run linter and fix surface level bugs

unbreak unicorn support for unsupported instructions

This helps in situations where we encounter a binary with an instruction we don't support yet.

Initial support for macOS

we've gotten a few requests from users to be able to have Manticore run on OS X. this shouldn't be too hard as nothing we do inherently ties us to linux. we'd need to

• eliminate all uses of /dev (i think there are a few)
• make sure all python stdlib calls we make work uniformly on linux/osx
• make sure python dependencies work on both
• add features to allow users to specify locations for ld.so and libraries (#93)

add note to docs about required pip version

Fix all spelling and grammar issues

Specifically in error messages

Move State to separate file

It resides in executor.py right now, which can make it hard to read the code.

Bugs in dynamic linker loader

There is at least 1 bug in our handling of dynamically linked binaries, and loading them into the emulated address space. Function: Linux.load.

Expand testing infrastructure

Current tests do not cover some behavior of the executor. We also lack integration tests.

Fix test_integration_basic_stdin test

relevant: #50 #51

enabling this test seems to break following z3 related tests, possibly crashing z3

• do all tests talk to the same instance of z3?
• do we need to have a reset somewhere in between each test?

Move handleInvalidPC from linux to arm

this is an arm specific function for handling kernel user mode helpers for linux arm, and shouldn't be in the general linux class

Add Armv7Operand.size support for memory operands

we only support reg operands right now

Move stats.py to scripts

Refactor solver logic to centralize error handling

Centralize spawning of z3 subprocess and checking of z3 validity in a single function, rather than having the check in the ctor, and spawn elsewhere.

Improve README

• multiple installation sections are confusing
• directly put --no-binary stuff into commands to run
• add keystone engine to dev install command

Installation feedback

• --no-binary is not available commonly it seems. maybe we should have people use the old no-use-wheel arg?

Add LD_LIBRARY_PATH type feature

Something that ties us to Linux right now is fetching libc and the dynamic linker, for example, directly from the filesystem on the machine. To allow mcore to run on more platforms like OS X, add a way to specify directories for mcore to look in for things like ld-linux.so, so you could analyze a dynamically linked ELF on OS X.

Remove old license header

Implement SMT optimizations

Add an FAQ

You know what to answer

Port to Python3

--log is broken

Command line flags should be better documented

Right now many of the command line flags have no usage examples or thorough explanation of their functionality, this can make it hard for users to effectively use them

Greater control of analysis entry point

use case: analyzing just a particular function, or even just a few instructions of a program

Add benchmarking facilities

Users should ideally have the ability to gain insight on manticore's performance via a simple script. Implementing this probably has two major components:

• Choose/create a workload to benchmark on that effectively emulates "real life" workloads

• Add an option to manticore to save performance information (e.g. time spent in the solver, time spent saving/recovering states, etc.)

Then, our script simply runs manticore on the aforementioned workload and displays the performance information.

x86: Implement symbolic LSL

add example scripts for new_symbolic_buff/value and symbolicate_buffer APIs

Graceful fallback if Z3 doesn't support optimization queries

So we can lessen our Z3 requirements and make it easier to install

Use consistent naming in API

State.symbolicate_buffer uses the label parameter to tag symbolic values, where as State.new_symbolic_buffer uses name.

Fix ARMv7 dynamic binary errors

When a dynamic binary is run on my machine, mcore doesn't even reach main, but errors a message to stderr.

Clean CGC artifacts from linux model

Implement ARM Linux intrinsics differently.

Currently, ARM intrinsics support is done via handling InvalidPCException. This is how it's done in qemu for performance reasons. This complicates our code for both the Linux model and for the Unicorn emulation.

We should do it the way it's done in hardware: Map an extra code of page (which we have) when loading an ARM binary. We won't even need to add atomic support because we're guaranteed to never get hardware interrupts.

Made platforms object oriented

There is a lot of shared code between the models (decree/linux/windows) right now, that could be made more manageable with a parent abstract model.

AttributeError: 'NoneType' object has no attribute 'address'

This issue was automatically created by Allstar and refers to trailofbits/manticore.

Security Policy Violation
Project is out of compliance with Binary Artifacts policy: binaries present in source code

Rule Description
Binary Artifacts are an increased security risk in your repository. Binary artifacts cannot be reviewed, allowing the introduction of possibly obsolete or maliciously subverted executables. For more information see the Security Scorecards Documentation for Binary Artifacts.

Remediation Steps
To remediate, remove the generated executable artifacts from the repository.

First 10 Artifacts Found

• examples/linux/binaries/corruption
• examples/linux/binaries/multiple-styles
• examples/script/aarch64/basic
• examples/script/aarch64/hello42
• tests/native/binaries/arguments
• tests/native/binaries/arguments_linux_amd64
• tests/native/binaries/arguments_linux_armv7
• tests/native/binaries/basic_linux_amd64
• tests/native/binaries/basic_linux_armv7
• tests/native/binaries/basic_state_merging
• Run a Scorecards scan to see full list.

Additional Information
This policy is drawn from Security Scorecards, which is a tool that scores a project's adherence to security best practices. You may wish to run a Scorecards scan directly on this repository for more details.

---

⚠️ There is an updated version of this policy result! Click here to see the latest update

---

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

Require argument to add_hook to be an int or None

If they accidentally pass in a str (if they forgot to convert sys.argv[1] to an int for example) it's bad UX because their hook inexplicably won't run. This because how we look up the hooks in _hook_callback

SLinux.__init__() ignores the constraints argument

We should either remove the argument, or initialize constraint set correctly

CTRL+C not always manage to interrupt when in multiprocessing.

^C2017-04-21 10:29:34,425: [20040][4048] EXECUTOR:ERROR: Interrupted!
2017-04-21 10:29:34,503: [20038][4068] EXECUTOR:ERROR: Interrupted!
2017-04-21 10:29:34,513: [20036][4041] EXECUTOR:ERROR: Interrupted!
2017-04-21 10:29:34,528: [20037][4043] EXECUTOR:ERROR: Interrupted!
^C

^[[B^CTraceback (most recent call last):
File "/usr/bin/manticore", line 9, in
load_entry_point('Manticore', 'console_scripts', 'manticore')()
File "/Projects/manticore/manticore/main.py", line 117, in main
m.run(args.timeout)
File "/Projects/manticore/manticore/manticore.py", line 567, in run
# and execute those that are?
File "/Projects/manticore/manticore/manticore.py", line 464, in _join_workers
def _model_hook_callback(self, state, pc):
File "/Projects/manticore/manticore/core/executor.py", line 98, in shutdown
with self._lock:
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 991, in enter
return self._callmethod('acquire')
File "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod
kind, result = conn.recv()
KeyboardInterrupt
Process Process-3:
Process Process-4:
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Process Process-5:
Traceback (most recent call last):
File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
self.run()
File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
self.run()
self._target(*self._args, **self._kwargs)
self._target(*self._args, **self._kwargs)
File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
File "/Projects/manticore/manticore/core/executor.py", line 837, in run
File "/Projects/manticore/manticore/core/executor.py", line 837, in run
self._target(*self._args, **self._kwargs)
File "/Projects/manticore/manticore/core/executor.py", line 837, in run
self._stats.append(self._profile.stats.items())
self._stats.append(self._profile.stats.items())
File "/Projects/manticore/manticore/utils/nointerrupt.py", line 17, in exit
File "/Projects/manticore/manticore/utils/nointerrupt.py", line 17, in exit
self._stats.append(self._profile.stats.items())
File "/Projects/manticore/manticore/utils/nointerrupt.py", line 17, in exit
self.old_handler(*self.signal_received)
self.old_handler(*self.signal_received)
self.old_handler(*self.signal_received)
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt

symbolic argv[1]

argv is symbolicated accurately, but the resulting expression isn’t written back into the address space

https://github.com/trailofbits/manticore/blob/master/manticore.py#L53

SolverException: [Errno 32] Broken pipe

(manticore) root@kali:~/CTF/HackTheVote/pwn# manticore ./irs 
2017-03-13 16:17:59,590: [48618] MAIN:INFO: Loading program: ['./irs']
2017-03-13 16:17:59,591: [48618] MAIN:INFO: Workspace: ./mcore_LV0KP1
2017-03-13 16:18:01,437: [48668] CPU:ERROR: CPUID with EAX=d not implemented @ f7ff4826
 
2017-03-13 16:22:56,061: [48668] SMT:WARNING: Found an unknown core, probably a solver timeout
2017-03-13 16:24:29,324: [48668] MEMORY:INFO: Reading 4 bytes from symbolic address <manticore.core.smtlib.expression.BitVecAnd object at 0x7f4a32aed750>
2017-03-13 16:25:00,200: [48668] SMT:WARNING: Found an unknown core, probably a solver timeout
2017-03-13 16:25:35,299: [48668] SMT:WARNING: Found an unknown core, probably a solver timeout
2017-03-13 16:26:07,874: [48668] SMT:WARNING: Found an unknown core, probably a solver timeout
2017-03-13 16:26:40,069: [48668] EXECUTOR:ERROR: Failed an internal assertion: 
Traceback (most recent call last):
  File "/root/.envs/manticore/local/lib/python2.7/site-packages/manticore/core/executor.py", line 923, in run
    assert se.message == 'Max number of different solutions hit'
AssertionError

2017-03-13 16:26:40,100: [48668] EXECUTOR:ERROR: Traceback (most recent call last):
2017-03-13 16:26:40,100: [48668] EXECUTOR:ERROR:   File "/root/.envs/manticore/local/lib/python2.7/site-packages/manticore/core/executor.py", line 923, in run
2017-03-13 16:26:40,101: [48668] EXECUTOR:ERROR:     assert se.message == 'Max number of different solutions hit'
2017-03-13 16:26:40,101: [48668] EXECUTOR:ERROR: AssertionError
Process Process-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/root/.envs/manticore/local/lib/python2.7/site-packages/manticore/core/executor.py", line 1018, in run
    if solver.check(current_state.constraints):
  File "/root/.envs/manticore/local/lib/python2.7/site-packages/manticore/core/smtlib/solver.py", line 58, in check
    return self.can_be_true(constraints, True)
  File "/root/.envs/manticore/local/lib/python2.7/site-packages/manticore/core/smtlib/solver.py", line 319, in can_be_true
    self._reset(temp_cs.related_to(expression))
  File "/root/.envs/manticore/local/lib/python2.7/site-packages/manticore/core/smtlib/solver.py", line 206, in _reset
    self._stop_proc()
  File "/root/.envs/manticore/local/lib/python2.7/site-packages/manticore/core/smtlib/solver.py", line 172, in _stop_proc
    self._send("(exit)")
  File "/root/.envs/manticore/local/lib/python2.7/site-packages/manticore/core/smtlib/solver.py", line 222, in _send
    raise SolverException(e)
SolverException: [Errno 32] Broken pipe

And attached is the binary
ultrababy.zip

Note: This is using the dev-fix-cpuid branch

Command line flags are misleading

Some of them are broken and don't work (--timeout, --env). Need to verify that all advertised functionality in --help works

rename test/ to tests/ to be idiomatic