themperek/cocotb-test

Unit testing for cocotb

cocotb-test's Issues

Continuous integration

Use Azure Pipelines

Compile libraries during setup individual for every available simulator?

Allow file specific compile arguments

As far as I understood, there is no possibility to have file specific compile arguments. My source files should be compiled into different libraries, like:

ghdl -a --work=lib_foo file1.vhd
ghdl -a --work=lib_bar file2.vhd

At first, is my assumption correct?
Do you think it would be suitable to add another argument (for example file_specific_compile_args) to the Simulator class, which must have the same length as vhdl_sources, respectively verilog_sources?

Simulation pseudo-fails when using parallelized tests (xdist)

I am using parametric tests to re-run a simulation with different inputs, in parallel using pytest xdist.

In some cases, the simulation will report as failed, this is the stacktrace:

>       assert os.path.isfile(results_xml_file), "Simulation terminated abnormally. Results file not found."
E       AssertionError: Simulation terminated abnormally. Results file not found.
E       assert False
E        +  where False = <function isfile at 0x7fecdc42ea60>('/tmp/tmpd5zd6huo')
E        +    where <function isfile at 0x7fecdc42ea60> = <module 'posixpath' from '/usr/lib/python3.6/posixpath.py'>.isfile
E        +      where <module 'posixpath' from '/usr/lib/python3.6/posixpath.py'> = os.path

../../cocotb-test/cocotb_test/run.py:36: AssertionError

I don't really have a reliable way to reproduce it, However I assume its some race condition caused by the parallelized tests, because I don't see that in case I don't use parallelization.

Testcase support

It would be a great idea to add testcase parameter so we can start forgetting environmens variables in python scripts.

tkinter dependency

Hello,

I have had a problem with tkinter dependency in Linux Mint 19 Cinnamon:

E   ModuleNotFoundError: No module named 'tkinter'

I have solved it with:

sudo apt install -y python3-tk

Maybe you could specify it in the README.md.

Thank you, nice software ;)

DeprecationWarning for getiterator in Python 3.8

../../../.pip-modules/lib/python3.8/site-packages/cocotb_test/plugin.py:43
  /workspace/.pip-modules/lib/python3.8/site-packages/cocotb_test/plugin.py:43: DeprecationWarning: This method will be removed in future versions.  Use 'tree.iter()' or 'list(tree.iter())' instead.
    for testcase in testsuite.getiterator("testcase"):

-- Docs: https://docs.pytest.org/en/stable/warnings.html

GHDL waveform

Hi,

I'm having problems to generate the waveform with cocotb-test and GHDL:

• GHDL:

GHDL 0.36 (tarball) [Dunoon edition]
 Compiled with GNAT Version: 7.4.0
 GCC back-end code generator

• pytest (5.3.5)
• cocotb-test (0.1.0)
• cocotb (1.4.0)

I can generate the .vcd waveform with a simple Makefile:

TOPLEVEL_LANG=vhdl
VHDL_SOURCES =  $(PWD)/../../RTL_code/sample.vhd
TOPLEVEL=sample
MODULE=sample_cocotb
SIM=ghdl
COMPILE_ARGS=--ieee=synopsys
SIM_ARGS=--vcd=func.vcd
include $(shell cocotb-config --makefiles)/Makefile.inc
include $(shell cocotb-config --makefiles)/Makefile.sim

But no with cocotb-test:

from cocotb_test.simulator import run
import os

CURRENT_PATH = os.path.dirname(os.path.abspath(__file__)) + '/'


def test_modulator():
    run(
        vhdl_sources=[CURRENT_PATH +
                      "/../../RTL_code/sample.vhd"],
        toplevel="sample",
        module="sample_cocotb",
        toplevel_lang="vhdl",
        sim_args=["--vcd=func.vcd"],
        compile_args=["--ieee=synopsys"]
    )

SIM=ghdl pytest -v -s --junitxml=test-results.xml --cocotbxml=test-cocotb.xml

Recompile if change only

Based on timestamps: https://stackoverflow.com/questions/22630756/how-does-makefile-know-that-a-file-changed-and-then-recompile-it

Python 3.9.2: subprocess.py - RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used

On Python 3.9.2, the bufsize=1 on the following line of file cocotb_test/simulator.py:

101: p = Popen(command, stdout=PIPE, stderr=STDOUT, shell=True, bufsize=1)

causes this warning:

.../python/3.9.2/lib/python3.9/subprocess.py:941: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  self.stdout = io.open(c2pread, 'rb', bufsize)

Removing bufsize=1 fixes this issue.

Opening `vvp` subprocess fails after large number of tests in the same test run.

I am currently (ab?)using cocotb-test to run a simulation repeatedly, with different test vectors for each run.

With this I am running into an issue, that at some point the subprocess call here fails:

cocotb-test/cocotb_test/simulator.py

Line 76 in e9ae218

process = subprocess.check_call(cmd, cwd=self.sim_dir)

With OSError: [Errno 12] Cannot allocate memory or OSError: [Errno 7] Argument list too long (The latter is a bit misleading, I think it is indeed caused by some failure to allocate memory for the subprocess).

This happens repeatably on my system roughly after about 3000 tests, but I would assume this is highly system(-memory) dependent.

I tried to work around this by calling gc.collect() after each call to run(...), in hope that this would free the allocated memory, however without success.

Maybe one way to fix this could be: Instead of launching a process every time, a shell could be launched, which is kept open and wrapped in some sort of runner object. This runner object is then used to launch vvp, but the process itself is reused.

I must admit this is just a quick thought, and might cause trouble later when using parallized tests (the runner cannot be used by multiple test runs at the same time).

Maybe relevant SO questions:
https://stackoverflow.com/questions/1367373/python-subprocess-popen-oserror-errno-12-cannot-allocate-memory

Add support for verilog plusargs for icarus.

The verilog standard defines plusargs (command line arguments that start with a +), that can be passed to the simulator. These plusargs can be retrieved in the verilog code with the $test$plusargs and $value$plusargs system tasks.

Icarus verilog requires these plusargs to be passed after the design file name (in the icarus man page the plusargs are called extended-args):

vvp [-inNsvV] [-Mpath] [-mmodule] [-llogfile] inputfile [extended-args...]

It might make sense to allow another argument to the run(...) method that takes these additional plusargs, and then appends them after [self.sim_file] here (without removing the possibility of adding extra_simulation_args):

cocotb-test/cocotb_test/simulator.py

Lines 122 to 127 in 32b8014

	def run_command(self):
	return (
	["vvp", "-M", self.lib_dir, "-m", "gpivpi"]
	+ self.extra_simulation_args
	+ [self.sim_file]
	)

I am not sure how other simulators handle these plusargs. Would be good to keep compatibility.

Accept simulator as a parameter to cocotb_test.simulator.run

It would be nice if the simulator could be selected without setting an env var. Ideally, cocotb_test.simulator.run should be updated to accept a simulator argument that could select the simulator. Not sure what precendence would make sense with respect to the SIM env var.

Return error on cocotb error

Parse results.xml?

Separate stderr stream for logging

stderr stream should show as an error in the log file.

The cocotb logs only show up in the terminal, and not dumped to file.

This is with xcelium, but potentially applies to other simulator.
All the log messages from cocotb, such as:
self.dut._log.info("Doing something")
do not appear in the compilation/simulation logs.

Back compatibility mode for Makefiles

One can create command ex. cocotb-setup -run that would parse all needed environment variables and call run from cocotb-test. There would stay only one Makefile for backword compatibility.

AttributeError: 'ElementTree' object has no attribute 'getiterator' when using --cocotbxml=... with Python 3.9.1

With Python 3.9.1 I get the following error without

Traceback (most recent call last):
  ...
  File ".../cocotb_test/plugin.py", line 40, in pytest_sessionfinish
    for testsuite in tree.getiterator("testsuite"):
AttributeError: 'ElementTree' object has no attribute 'getiterator'

Based on the following bugfix:

• What’s New In Python 3.9
  Methods getchildren() and getiterator() of classes ElementTree and Element in the ElementTree module have been removed. They were deprecated in Python 3.2. Use iter(x) or list(x) instead of x.getchildren() and x.iter() or list(x.iter()) instead of x.getiterator(). (Contributed by Serhiy Storchaka in bpo-36543.)
  • bpo-36543 Removed methods Element.getchildren(), Element.getiterator() and ElementTree.getiterator() and the xml.etree.cElementTree module.

Element.getiterator() should be replaced with .iter(x). Changing these two lines in cocotb_test/plugin.py:

40:                for testsuite in tree.getiterator("testsuite"):
...
44:                     for testcase in testsuite.getiterator("testcase"):

to:

40:                for testsuite in tree.iter("testsuite"):
...
44:                     for testcase in testsuite.iter("testcase"):

fixes the issue.

Parse logging from cocotb

Parse and pass cocotb log and not only forwards (recognize log level).

Potentially can make a logger that serializes the log (json/yaml) on cocotb side and properly display in cocotb-test.

Cannot set verilator 'timescale'

From the cocotb doc, Verilator handles timescale a bit differently than other simulators.

Looking at the verilator Makefile
we ultimately need to append the -DVL_TIME_PRECISION_STR switch to the CPPFLAGS env var.

ifdef COCOTB_HDL_TIMEPRECISION
  SIM_BUILD_FLAGS += -DVL_TIME_PRECISION_STR=$(COCOTB_HDL_TIMEPRECISION)
endif
SIM_BUILD_FLAGS += -std=c++11
# Compilation phase
$(SIM_BUILD)/$(TOPLEVEL): $(SIM_BUILD)/Vtop.mk
	CPPFLAGS="$(SIM_BUILD_FLAGS)" make -C $(SIM_BUILD) -f Vtop.mk

In cocotb=test, the Verilator class doesn't allow for adding more options to CPPFLAGS

self.env["CPPFLAGS"] = "-std=c++11"

I'm not sure what is the best way to support this while keeping things consistent with other simulators.

Error ghdl + python 2.7

I am running a simple example and I have this problem:

==================================================================================== test session starts =====================================================================================
platform linux2 -- Python 2.7.15+, pytest-3.3.2, py-1.5.2, pluggy-0.6.0
rootdir: /home/cruiz/Descargas/cocotbExamples/simple, inifile:
collected 1 item                                                                                                                                                                             

pytest_adder.py ghdl -i /home/cruiz/Descargas/cocotbExamples/simple/adder.vhd
ghdl -m adder
analyze /home/cruiz/Descargas/cocotbExamples/simple/adder.vhd
elaborate adder
ghdl -r adder --vpi=/usr/local/lib/python2.7/dist-packages/cocotb_test/libs/ghdl/libvpi.so
loading VPI module '/usr/local/lib/python2.7/dist-packages/cocotb_test/libs/ghdl/libvpi.so'
/usr/local/lib/python2.7/dist-packages/cocotb_test/libs/ghdl/libvpi.so: cannot open shared object file: No such file or directory
./adder:error: cannot load VPI module
F                                                                                                                                                                      [100%]

========================================================================================== FAILURES ==========================================================================================
______________________________________________________________________________________ test_adder_vhdl _______________________________________________________________________________________

    @pytest.mark.skipif(os.getenv("SIM") == "icarus", reason="VHDL not suported")
    def test_adder_vhdl():
>       run(vhdl_sources=["adder.vhd"], toplevel="adder", module="test_adder", toplevel_lang="vhdl")

pytest_adder.py:7: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/lib/python2.7/dist-packages/cocotb_test/run.py:35: in run
    results_xml_file = sim.run()
/usr/local/lib/python2.7/dist-packages/cocotb_test/simulator.py:105: in run
    self.execute(cmds)
/usr/local/lib/python2.7/dist-packages/cocotb_test/simulator.py:133: in execute
    process = subprocess.check_call(cmd, cwd=self.sim_dir, env=self.env)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

popenargs = (['ghdl', '-r', 'adder', '--vpi=/usr/local/lib/python2.7/dist-packages/cocotb_test/libs/ghdl/libvpi.so'],)
kwargs = {'cwd': '/home/cruiz/Descargas/cocotbExamples/simple/sim_build', 'env': {'CINNAMON_VERSION': '4.0.10', 'COCOTB_RESULTS_FILE_NAME': '/tmp/tmpg1hkuX', 'COCOTB_SIM': '1', 'COLORTERM': 'truecolor', ...}}
retcode = 1, cmd = ['ghdl', '-r', 'adder', '--vpi=/usr/local/lib/python2.7/dist-packages/cocotb_test/libs/ghdl/libvpi.so']

    def check_call(*popenargs, **kwargs):
        """Run command with arguments.  Wait for command to complete.  If
        the exit code was zero then return, otherwise raise
        CalledProcessError.  The CalledProcessError object will have the
        return code in the returncode attribute.
    
        The arguments are the same as for the Popen constructor.  Example:
    
        check_call(["ls", "-l"])
        """
        retcode = call(*popenargs, **kwargs)
        if retcode:
            cmd = kwargs.get("args")
            if cmd is None:
                cmd = popenargs[0]
>           raise CalledProcessError(retcode, cmd)
E           CalledProcessError: Command '['ghdl', '-r', 'adder', '--vpi=/usr/local/lib/python2.7/dist-packages/cocotb_test/libs/ghdl/libvpi.so']' returned non-zero exit status 1

/usr/lib/python2.7/subprocess.py:190: CalledProcessError

My OS is: Linux Mint 19.1
cocotb-test: 0.0.4.dev1
ghdl:

GHDL 0.36-dev (v0.35-271-g5cd3de46) [Dunoon edition]
 Compiled with GNAT Version: 7.3.0
 GCC back-end code generator
Written by Tristan Gingold.

Copyright (C) 2003 - 2015 Tristan Gingold.
GHDL is free software, covered by the GNU General Public License.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Checkpointing

Does it make sense to include checkpointing functionality as the part of this package?
If cocotb-test is going to be merged with core cocotb, it is probably too much complexity to be added (also including parallel regression specific functions).

Cleanup script

• pytest plugin pytest -clean?
• custom script cocotb-clean?

Simulator class API

Maybe split for every simulator into 2 parts and make something like

get_simulator(sources=..., ... ):

class Simulator:

    def compile(self, force=False): #compile and elaborate 
        pass

    def run(self):
       pass

    def compile_and_run(self, force_complie=True):
       pass

or

class Simulator:

    def run(self, compile=True, simulate=True, force=False):
       pass

Would allow better control and ability to precompile only once per session/test.

Python 2.7 regex problem

setup.py lists Python 2.7 support, but re.ASCII is not a valid flag in Python 2.

cocotb-test/cocotb_test/simulator.py

Line 11 in 1dc351d

_space_re = re.compile(r"([\s])", re.ASCII)

Support for module parameters

Support for setting module parameters should be added. Different simulators need module parameters passed through in different ways - different command line switches are used, and some require them in SIM_ARGS, some in COMPILE_ARGS.

Icarus verilog accepts module parameters in COMPILE_ARGS as -P toplevel.PARAM=value. Verilator accepts module parameters in COMPILE_ARGS as -GPARAM=value. No idea about other simulators.

Having a standardized way to support this in cocotb-test would be very convenient, perhaps via a module_params dict passed to Simulator.__init__().

Current simulator support:

• Icarus Verilog
• Questa
• Ius
• Xcelium
• Vcs
• Ghdl
• Riviera
• Verilator

Argument on Xcelium build forces pre-step setup

Hi,

could you please remove the default argument "-elaborate" on Xcelium/Ius simulator, to use that you need to force work lib creation previously, what doesn't happen at time 0 when you're running the sims and by default xrun will know what to call first depending the stage you're. I was wondering why I couldn't run the same tests with cocotb-test that I successfully run with standard makefile flow and removing this argument made it work.

cocotb-test/cocotb_test/simulator.py

Line 607 in 3ed5aac

"-elaborate",

SIM_BUILD var not found at run time

Hey,
I'm trying to run the following code below using Cocotb-test and when I run through the makefile, it finishes without no issues, but when I launch with pytest, it complains about the not finding the env SIM_BUILD var that's dynamically build for each case of test.

https://github.com/aignacio/async_gp_fifo/blob/master/test_fifo_async.py#L33

INFO     cocotb:simulator.py:255     40.00ns ERROR    cocotb.regression                         regression.py:399  in _score_test                     Test Failed: run_test_001 (result was KeyError)
INFO     cocotb:simulator.py:255                                                                                                                      Traceback (most recent call last):
INFO     cocotb:simulator.py:255                                                                                                                        File "/usr/local/lib/python3.9/site-packages/cocotb/regression.py", line 599, in _my_test
INFO     cocotb:simulator.py:255                                                                                                                          await function(dut, *args, **kwargs)
INFO     cocotb:simulator.py:255                                                                                                                        File "/Users/aignacio/projects/asyng_gp_fifo/test_fifo_async.py", line 115, in run_test
INFO     cocotb:simulator.py:255                                                                                                                          ff_driver = AFIFODriver(signals=dut,debug=True)
INFO     cocotb:simulator.py:255                                                                                                                        File "/Users/aignacio/projects/asyng_gp_fifo/test_fifo_async.py", line 33, in __init__
INFO     cocotb:simulator.py:255                                                                                                                          sim_build     = os.environ['SIM_BUILD']
INFO     cocotb:simulator.py:255                                                                                                                        File "/usr/local/Cellar/[email protected]/3.9.1_8/Frameworks/Python.framework/Versions/3.9/lib/pytho
n3.9/os.py", line 679, in __getitem__
INFO     cocotb:simulator.py:255                                                                                                                          raise KeyError(key) from None
INFO     cocotb:simulator.py:255                                                                                                                      KeyError: 'SIM_BUILD'

What might be the case to check before launching the test in addition of SIM_NAME?

Missing documentation

Hello there, I'm using cocotb for quite a while now and I recently discovered this package. I can finally get rid of the makefiles!
Yet I was wondering whether there is any plan to document the methods defined by this module.

I went through the code, despite it is often clear, there are some kwargs that it is not so trivial to figure out what their purpose is. Perhaps it would be handy to have a quick look to documentation to make sense of everything available to the user.

Let me know if I can help in this task.

Windows support issues

This is not so easy considering how setuptools work.
The problem is this: https://stackoverflow.com/questions/34689210/error-exporting-symbol-when-building-python-c-extension-in-windows
The workarounds are ugly and does not seem to work well here (c++).

One solution is to call gcc directly. At the end, this will be cleanest. Since we will need some dependency function for compiling Verilog this can be reused.

No mutable objects as default values

Hi!

I found that Simulator's arguments are initialized with '[]. It's not a good idea because it can produce an unexpected behavior.

Example:

In [271]: class MyClass: 
     ...:     def __init__(self, arg=[]): 
     ...:         arg.append('bleee') 
     ...:         print(arg) 
     ...:                                                                          

In [272]: a = MyClass()                                                            
['bleee']

In [273]: b = MyClass()                                                            
['bleee', 'bleee']

GHDL integration

Hi,
This is exactly what i was looking for. Is there anyway it can be integrated into GHDL. I don't mind helping out im just not that experienced with python.
Kind regards,
Ben

Cannot specify a command file for Icarus from compile_args - Needed for setting timescale

To set the icarus timescale, a cmds.f file with the following content is created:

+timescale+1ns/1ns

Then compile_args is set to:
compile_args = ["-f /home/cmds.f"]

However, icarus complains that the file cannot be read:

INFO cocotb: Running command: iverilog -o /home/sim_build/toplevel.vvp -D COCOTB_SIM=1 -s toplevel -g2012 -f /home/cmds.f  /home/toplevel.sv 
INFO cocotb: iverilog: cannot open command file  /home/cmds.f for reading.
ERROR cocotb: Command terminated with error 1
AssertionError: Simulation terminated abnormally. Results file not found.

Interestingly, if I then manually go in /sim_build and type the exact same command, there are no errors and subsequent run work fine...
iverilog -o /home/sim_build/toplevel.vvp -D COCOTB_SIM=1 -s toplevel -g2012 -f /home/cmds.f /home/toplevel.sv

unrecognized arguments: --cov=cocotb --cov-branch

When I create the test_dff.py as described in readme and run pytest I get the following error:

ERROR: usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --cov=cocotb --cov-branch
inifile: /mnt/d/git/cocotb/setup.cfg
rootdir: /mnt/d/git/cocotb

I am using python 3.8, Ubunt 18, cocotb and cocotb-test from pip

ghdl with latest release

Hi i have just updated my test enviroment to the latest version of cocotb-test. In the requirements of cocotb-test is a dependancy on cocotb 1.3.1. which as far as i can tell has a issue with GHDL where i get the error "Couldn't find makefile for simulator: "ghdl"! Available simulators: " - issue #1699 in cocotb. Is there any way we can change the dependency to 1,4.0 dev as i know that works

Simulator generates wrong number of bits for parameters

My code looks like this

tests_dir = os.path.abspath(os.path.dirname(__file__))
rtl_dir = os.path.abspath(os.path.join(tests_dir, '..', 'hdl'))

@pytest.mark.parametrize("input_width_a", [16, 32])
@pytest.mark.parametrize("input_width_b", [16, 32])
@pytest.mark.parametrize("output_width", [8, 16, 32])
@pytest.mark.parametrize("blocking", [1])
@pytest.mark.parametrize("truncate", [1])
def test_complex_multiplier(request, blocking, input_width_a, input_width_b, output_width, truncate):
    dut = "complex_multiplier"
    module = os.path.splitext(os.path.basename(__file__))[0]
    toplevel = dut

    verilog_sources = [
        os.path.join(rtl_dir, f"{dut}.v"),
    ]

    parameters = {}

    parameters['INPUT_WIDTH_A'] = input_width_a
    parameters['INPUT_WIDTH_B'] = input_width_b
    parameters['OUTPUT_WIDTH'] = input_width_a
    parameters['BLOCKING'] = blocking
    parameters['TRUNCATE'] = truncate

    extra_env = {f'PARAM_{k}': str(v) for k, v in parameters.items()}

    sim_build = os.path.join(tests_dir, "sim_build",
        request.node.name.replace('[', '-').replace(']', ''))

    cocotb_test.simulator.run(
        python_search=[tests_dir],
        verilog_sources=verilog_sources,
        toplevel=toplevel,
        module=module,
        parameters=parameters,
        sim_build=sim_build,
        extra_env=extra_env,
    )

when I run it with pytest I get the following errors:

INFO     cocotb:simulator.py:244 Running command: perl /usr/local/bin/verilator -cc --exe -Mdir /mnt/d/git/complex_multiplier/tests/sim_build/test_complex_multiplier-1-1-32-32-32 -DCOCOTB_SIM=1 --top-module complex_multiplier --vpi --public-flat-rw --prefix Vtop -o complex_multiplier -LDFLAGS -Wl,-rpath,/usr/local/lib/python3.8/dist-packages/cocotb/libs -L/usr/local/lib/python3.8/dist-packages/cocotb/libs -lcocotbvpi_verilator -GINPUT_WIDTH_A=32 -GINPUT_WIDTH_B=32 -GOUTPUT_WIDTH=32 -GBLOCKING=1 -GTRUNCATE=1 /usr/local/lib/python3.8/dist-packages/cocotb/share/lib/verilator/verilator.cpp /mnt/d/git/complex_multiplier/hdl/complex_multiplier.v
INFO     cocotb:simulator.py:255 %Warning-WIDTH: /mnt/d/git/complex_multiplier/hdl/complex_multiplier.v:14:21: Operator VAR 'BLOCKING' expects 1 bits on the Initial value, but Initial value's CONST '32'h1' generates 32 bits.
INFO     cocotb:simulator.py:255                                                                             : ... In instance complex_multiplier
INFO     cocotb:simulator.py:255    14 |       parameter bit BLOCKING /*verilator public_flat_rd*/  = 0,
INFO     cocotb:simulator.py:255       |                     ^~~~~~~~
INFO     cocotb:simulator.py:255                 ... Use "/* verilator lint_off WIDTH */" and lint_on around source to disable this message.
INFO     cocotb:simulator.py:255 %Warning-WIDTH: /mnt/d/git/complex_multiplier/hdl/complex_multiplier.v:15:21: Operator VAR 'TRUNCATE' expects 1 bits on the Initial value, but Initial value's CONST '32'h1' generates 32 bits.
INFO     cocotb:simulator.py:255                                                                             : ... In instance complex_multiplier
INFO     cocotb:simulator.py:255    15 |       parameter bit TRUNCATE /*verilator public_flat_rd*/  = 1)
INFO     cocotb:simulator.py:255       |                     ^~~~~~~~
INFO     cocotb:simulator.py:255 %Warning-WIDTH: /mnt/d/git/complex_multiplier/hdl/complex_multiplier.v:55:20: Operator ASSIGNDLY expects 4 bits on the Assign RHS, but Assign RHS's REPLICATE generates 3 bits.
INFO     cocotb:simulator.py:255                                                                             : ... In instance complex_multiplier
INFO     cocotb:simulator.py:255    55 |             tvalid <= {{(STAGES){1'b0}}};
INFO     cocotb:simulator.py:255       |                    ^~
INFO     cocotb:simulator.py:255 %Warning-WIDTH: /mnt/d/git/complex_multiplier/hdl/complex_multiplier.v:58:19: Operator ASSIGNDLY expects 16 bits on the Assign RHS, but Assign RHS's REPLICATE generates 32 bits.
INFO     cocotb:simulator.py:255                                                                             : ... In instance complex_multiplier
INFO     cocotb:simulator.py:255    58 |             ai_bi <= {OUTPUT_WIDTH{1'b0}};
INFO     cocotb:simulator.py:255       |                   ^~
INFO     cocotb:simulator.py:255 %Warning-WIDTH: /mnt/d/git/complex_multiplier/hdl/complex_multiplier.v:59:19: Operator ASSIGNDLY expects 16 bits on the Assign RHS, but Assign RHS's REPLICATE generates 32 bits.
INFO     cocotb:simulator.py:255                                                                             : ... In instance complex_multiplier
INFO     cocotb:simulator.py:255    59 |             ai_br <= {OUTPUT_WIDTH{1'b0}};
INFO     cocotb:simulator.py:255       |                   ^~
INFO     cocotb:simulator.py:255 %Warning-WIDTH: /mnt/d/git/complex_multiplier/hdl/complex_multiplier.v:60:19: Operator ASSIGNDLY expects 16 bits on the Assign RHS, but Assign RHS's REPLICATE generates 32 bits.
INFO     cocotb:simulator.py:255                                                                             : ... In instance complex_multiplier
INFO     cocotb:simulator.py:255    60 |             ar_bi <= {OUTPUT_WIDTH{1'b0}};
INFO     cocotb:simulator.py:255       |                   ^~
INFO     cocotb:simulator.py:255 %Warning-WIDTH: /mnt/d/git/complex_multiplier/hdl/complex_multiplier.v:61:19: Operator ASSIGNDLY expects 16 bits on the Assign RHS, but Assign RHS's REPLICATE generates 32 bits.
INFO     cocotb:simulator.py:255                                                                             : ... In instance complex_multiplier
INFO     cocotb:simulator.py:255    61 |             ar_br <= {OUTPUT_WIDTH{1'b0}};
INFO     cocotb:simulator.py:255       |                   ^~
INFO     cocotb:simulator.py:255 %Error: Exiting due to 7 warning(s)
ERROR    cocotb:simulator.py:258 Command terminated with error 1

My HDL file looks like this

module complex_multiplier
    #(parameter int INPUT_WIDTH_A `VL_RD = 16, // must be multiple of 8
      parameter int INPUT_WIDTH_B `VL_RD = 16, // must be multiple of 8
      parameter int OUTPUT_WIDTH `VL_RD = 32,  // must be multiple of 8
      parameter int STAGES `VL_RD = 3,  // minimum value is 2
      parameter bit BLOCKING `VL_RD = 0,
      parameter bit TRUNCATE `VL_RD = 1)
    (   

thanks for helping!

Linking to cocotb Make definitions + improved pytest discovery

Hello!

I've been pursuing (for quite some time tbh) a 100% FOSS RTL dev workflow/toolchain based on cocotb and Verilator which integrates (more or less) seamlessly with modern Python (i.e. arbitrary virtualenvwrapper, pytest, tox integration). I came across cocotb-test quickly after having the need to pipe cocotb TBs into pytest, but a few outstanding issues prevented that from working out of the box:

• we're using cocotb==1.4.* (in part because Verilator support is quite recent) which was broken (that is, control_test.simulator.Verilator's build flags) as of cocotb-test==0.0.6
• we use cocotb.regression.TestFactory to parameterize our TBs, and naturally if you use pytest then you want to have the ability to detect individual test instances (as generated by these test factories) rather than manually replacing vanilla cocotb Makefiles with Simulator.run() calls
  • For context, I'm under the impression that these sort of verification components are subject to change (e.g. with further refactoring of the cocotb codebase and consolidation of separately maintained cocotbext-* repos), so I'm keen to avoid adding excess code to maintain

This being said, I expanded on what existed under cocotb_test.simulator.Verilator, adding code to link + parse the Makefiles (specifically, their variables and targets) provided in the current cocotb module. All of the relevant build invocations/definitions for the simulator are piped into cocotb_test.Simulator.__init__(), so in principle there is no need to separately maintain those details. I also added some code to automatically inspect and add proper pytest hooks (each with a single simulator.run() call) to @cocotb.test instances in a vanilla cocotb test module.

Of course, I've only ironed out these details when using Verilator, so if you think this approach is valuable I can plan on open-sourcing it once we're satisfied internally

ghdl error

Hi,
Thanks for putting in ghdl support. I have probably missed something, but when i run the dff test i am getting a error

E CalledProcessError: Command '['ghdl', '-r', 'dff', '--vpi=/home/ben/tools/cocotb-test/tests/build/libvpi.so']' returned non-zero exit status 1

when i run the command separately i get the following issue
loading VPI module '/home/ben/tools/cocotb-test/build/libvpi.so'
libcocotbutils.so: cannot open shared object file: No such file or directory
ghdl:error: cannot load VPI module

obviously you wrote the test with a verilog dut but i changed it to vhdl

Any advice on how to get it working as i assume i am just missing something

Thanks

Ben

Support VHDL with VCS

See cocotb/cocotb#1103

marge results xml of all test in some smart way

List of tests

Hi!
Is there a way to list the tests? Similar to VUnit:
https://vunit.github.io/python_interface.html

get_tests(pattern='*')

    Get a list of tests
    Parameters
        pattern – A wildcard pattern matching the test name
    Returns
        A list of Test objects

Specifying compile_only always fails the build

Since the run function always asserts that a results XML file exists, the compile_only function of cocotb-test effectively always fails the build.

SIM env needed when simulator is hardcoded

Hi @themperek

When i'm trying to hardcode the simulator (no SIM env) i get the following error:

run(simulator=Icarus,
    toplevel=name,
    module=module,
    verilog_sources=[verilog_file])

Traceback (most recent call last):
  File "test.py", line 15, in <module>
    test_module()
  File "test.py", line 12, in test_module
    run(design, get_current_module(), ports=ports, vcd_file='output.vcd')
  File "/home/ademski/Documents/opensource/nmigen-cocotb/nmigen_cocotb.py", line 73, in run
    compile_args=compile_args_waveforms if vcd_file else [])
  File "/home/ademski/Documents/opensource/nmigen-axi/venv/lib/python3.7/site-packages/cocotb_test/run.py", line 22, in run
    sim = simulator(**kwargs)
  File "/home/ademski/Documents/opensource/nmigen-axi/venv/lib/python3.7/site-packages/cocotb_test/simulator.py", line 179, in __init__
    super(Icarus, self).__init__(*argv, **kwargs)
  File "/home/ademski/Documents/opensource/nmigen-axi/venv/lib/python3.7/site-packages/cocotb_test/simulator.py", line 37, in __init__
    self.lib_dir = os.path.join(libs_dir, os.getenv("SIM"))
  File "/usr/lib/python3.7/posixpath.py", line 94, in join
    genericpath._check_arg_types('join', a, *p)
  File "/usr/lib/python3.7/genericpath.py", line 149, in _check_arg_types
    (funcname, s.__class__.__name__)) from None
TypeError: join() argument must be str or bytes, not 'NoneType'

I workaround it with a simple os.environ['SIM'] = 'icarus' before the run(...). I would be great that if simulator argument is present, it won't need the SIM env.

cocotb-test sim_build directory location

I just ran in to an issue with cocotb-test that needs to be addressed. I have a directory structure that looks something like this

root
  +- rtl
  |   +- mod1.v
  |   +- mod2.v
  +- tb
      +- test_mod1
      |   +- Makefile
      |   +- test_mod1.py
      +- test_mod2
          +- Makefile
          +- test_mod2.py

If I run one of the makefiles or if I run py.test from within test_mod1 or test_mod2, everything works fine, and the sim_build_* directories are created inside the test_mod1/test_mod2 directories alongside the test scripts. However, if I run py.test from the tb directory that's up one level, all of the sim_build directories are created there and there are all sorts of conflicts.

The problem is the first line of Simulator.__init__:

self.sim_dir = os.path.join(os.getcwd(), sim_build)

There are two main issues here: first, if somebody provides an absolute path for sim_build, things will break, so detecting that and preserving an absolute path would be a good idea. Second, using os.getcwd like that is causing the sim_build directories from the tests in the subdirectories to all end up in the same location. I'm not sure what the best solution would be for that. One option is to do the same thing as the lib directory:

self.lib_dir = os.path.join(os.path.dirname(cocotb.__file__), "libs")

There may be other options as well, perhaps there is a way to set up pytest to change the cwd. However, I do not like passing any command line arguments to pytest - everything should run correctly if pytest is run with no arguments.

[Feature] Output object return

It might be interesting if run() returns an object similar to results.xml.

USER API

Requirements:

• simple and compatible with all tools for basic usage
• possible to derive custom configuration/custom simulation for advanced usage

Possible idea: A generic simulation class with attributes for settings/parameters and build()/run() functions that can be derived and customized. Can be used as an input parameter to test function.
Need to provide dependency tracking.

Use the exixting API?

• https://github.com/olofk/edalize
• https://vunit.github.io/
  ..

Support for waveform dumps

Support for enabling waveform dumping should be added. Different simulators need module parameters passed through in different ways - different command line switches are used, and some require them in SIM_ARGS, some in COMPILE_ARGS.

Verilator accepts --trace and hopefully soon --trace-fst in COMPILE_ARGS. Icarus Verilog requires a little bit more work, the best option I think is to write out a module that contains $dumpfile("toplevel.fst"); and $dumpvars(0, toplevel), add this to the sources, and add a -s mod_name to COMPILE_ARGS to select it as an additional top-level module. It's also necessary to select the dump file format in plusargs (e.g. -fst). No idea about other simulators, although it seems some of them can look for a WAVES env var.

Having a standardized way to support this in cocotb-test would be very convenient, perhaps via an argument to Simulator.__init__().

run_filename is not what you want when running in virtualenv

I noticed that when I run pytest with pipenv (which is fairly standard), the run_filename passed to the simulator is the parent directory of the Python in the virtual environment (e.g. /home/<user>/.local/share/virtualenvs/<foldername>-ffNzxzuc/lib/python3.7/site-packages/_pytest/python.py). Since this is later used to resolve relative verilog sources, includes, etc. it is probably not a good idea to use the inspect module to figure out the path of the running file.

Way to pass parameters to the cocotb test

Is there a way to pass parameters to the cocotb test? Typically I'll have a large number of parameters that I use to generate my vhdl. The cocotb test needs access to these parameters to determine to properly test the core.

One option would be to add a generic parameter to the top level module that is just a string pointing at a file that contains all the parameters that the test will need (i.e. dut.TEST_PARAMETERS_FILENAME), but that seems pretty ugly.

Support passing a working directory to the call to cocotb

Currently the cocotb build is run with a working directory of sim_build. This is different than when cocotb is run with make -C working_dir. It would be good to be able to pass the working directory to the spawned cocotb process as an argument.