cornell-brg/pymtl

Python-based hardware modeling framework

PyMTL is an open-source, Python-based framework for multi-level hardware modeling. It was introduced at MICRO-47 in December, 2014.

DEPRECATION NOTE: We will not be maintaining this PyMTL version 2 repo. We are instead working very hard on the new PyMTL3 version 3 repo pymtl3. PyMTLv3 has been significantly rewritten to improve both the performance and productivity of FL, CL, and RTL modeling. While we will still provide limited support for PyMTLv2, we encourage new users to start experimenting with PyMTLv3. Also note that PyMTLv2 only works with Python 2.7 (which will not be maintained past January 1, 2020). PyMTLv3 takes advantage of many new and exciting Python 3.6+ features.

Tutorials

If you are interested in learning more about the PyMTL framework, we recommend you take a look at two tutorials that have been developed for Cornell ECE 4750. This is a course on computer architecture targeting seniors and first-year graduate students. Throughout the semester, students gradually design, implement, test, and evaluate a basic multicore system capable of running simple parallel applications at the register-transfer level. This year, students are using the PyMTL framework for all functional-level modeling and testing. Students have the option of using PyMTL or Verilog for the RTL design portion of the lab assignments. The first tutorial focuses on the PyMTL framework, while the second tutorial illustrates how PyMTL's Verilog import feature can enable applying PyMTL's powerful functional-level and testing features to RTL designs written in Verilog.

• PyMTL Hardware Modeling Framework Tutorial (github)
• Verilog Hardware Description Language Tutorial (github)

We have also developed tutorials specifically on PyMTL CL modeling, integrating PyMTL with the Xilinx Vivado High-Level Synthesis (HLS) tool, and using PyMTL to drive a Synopys-based ASIC EDA toolflow.

• PyMTL CL Modeling Tutorial (github)
• PyMTL/HLS Tutorial (github)
• PyMTL-Based ASIC Toolflow Tutorial

License

PyMTL is offered under the terms of the Open Source Initiative BSD 3-Clause License. More information about this license can be found here:

• http://choosealicense.com/licenses/bsd-3-clause
• http://opensource.org/licenses/BSD-3-Clause

Publications

If you use PyMTL in your research, please cite our MICRO'14 paper:

  @inproceedings{lockhart-pymtl-micro2014,
    title     = {PyMTL: A Unified Framework for Vertically Integrated
                 Computer Architecture Research},
    author    = {Derek Lockhart and Gary Zibrat and Christopher Batten},
    booktitle = {47th IEEE/ACM Int'l Symp. on Microarchitecture (MICRO)},
    month     = {Dec},
    year      = {2014},
    pages     = {280--292},
    doi       = {10.1109/MICRO.2014.50},
  }

Installation

PyMTL requires Python2.7 and has the following additional prerequisites:

• verilator, pkg-config
• git, Python headers, and libffi
• virtualenv

The steps for installing these prerequisites and PyMTL on a fresh Ubuntu distribution are shown below. They have been tested with Ubuntu Trusty 14.04.

Install Verilator

Verilator is an open-source toolchain for compiling Verilog RTL models into C++ simulators. PyMTL uses Verilator for both Verilog translation and Verilog import. You can install Verilator using the standard package manager but the version available in the package repositories is several years old. This means you will need to build and install Verilator from source using the following commands:

 % sudo apt-get install git make autoconf g++ libfl-dev bison
 % mkdir -p ${HOME}/src
 % cd ${HOME}/src
 % wget http://www.veripool.org/ftp/verilator-4.008.tgz
 % tar -xzvf verilator-4.008.tgz
 % cd verilator-4.008
 % ./configure
 % make
 % sudo make install

Verify that Verilator is on your path as follows:

 % cd $HOME
 % which verilator
 % verilator --version

PyMTL uses pkg-config to find the Verilator source files when performing both Verilog translation and Verilog import. Install pkg-config and verify that it is setup correctly as follows:

 % sudo apt-get install pkg-config
 % pkg-config --print-variables verilator

If pkg-config cannot find information about verilator, then you can also explicitly set the following special environment variable:

 % export PYMTL_VERILATOR_INCLUDE_DIR="/usr/local/share/verilator/include"

Install git, Python headers, and libffi

We need to install the Python headers and libffi in order to be able to install the cffi Python package. cffi provides an elegant way to call C functions from Python, and PyMTL uses cffi to call C code generated by Verilator. We will use git to grab the PyMTL source. The following commands will install the appropriate packages:

 % sudo apt-get install git python-dev libffi-dev

Install virtualenv

While not strictly necessary, we strongly recommend using virtualenv to install PyMTL and the Python packages that PyMTL depends on. virtualenv enables creating isolated Python environments. The following commands will install virtualenv:

 % sudo apt-get install python-virtualenv

Now we can use the virtualenv command to create a new virtual environment for PyMTL, and then we can use the corresponding activate script to activate the new virtual environment:

 % mkdir ${HOME}/venvs
 % virtualenv --python=python2.7 ${HOME}/venvs/pymtl
 % source ${HOME}/venvs/pymtl/bin/activate

Install PyMTL

We can now use git to clone the PyMTL repo, and pip to install PyMTL and its dependencies. Note that we use pip in editable mode so that we can actively work in the PyMTL git repo.

 % mkdir -p ${HOME}/vc/git-hub/cornell-brg
 % cd ${HOME}/vc/git-hub/cornell-brg
 % git clone https://github.com/cornell-brg/pymtl.git
 % pip install --editable ./pymtl
 % pip install --upgrade pytest

Testing

Before running any tests, we first create a build directory inside the PyMTL repo to hold any temporary files generated during simulation:

 % mkdir -p ${HOME}/vc/git-hub/cornell-brg/pymtl/build
 % cd ${HOME}/vc/git-hub/cornell-brg/pymtl/build

All Python simulation tests can be easily run using py.test (warning: there are a lot of tests!):

 % py.test ..

The Verilog simulation tests are only executed if the --test-verilog flag is provided. For Verilog testing to work, PyMTL requires that Verilator is on your PATH and that the PYMTL_VERILATOR_INCLUDE_DIR environment:

 % py.test .. --test-verilog

When you're done testing/developing, you can deactivate the virtualenv::

 % deactivate

pymtl's Issues

Add ability to generate s.combinational/s.posedge blocks in a for loop

It is currently not possible to do the following:

  def elaborate_logic( s ):
    for i in range( 4 ):
      @s.combinational
      def logic():
        s.out[i] = s.in_ + i

Instead a user must do:

  def elaborate_logic( s ):
    @s.combinational
    def logic():
      for i in range( 4 ):
        s.out[i] = s.in_ + i

The former is preferable in cases where you want to make sure the generated Verilog is explicitly synthesizable. The latter is a bit unclear as to how the synthesized hardware will look.

C Translation Python harnesses don't use .value/.next

Replace @capture_args decorator with metaclass magic

In order to support Verilog generation of PyMTL modules with different parameters (ie. a 3 port vs 4 port mux), we need to annotate the constructor with the @capture_args decorator:

  class MyHardware( Model ):
    @capture_args
    def __init__( s, some_param ):
      ...

This will ensure the generated Verilog has a unique name determined by the value of some_param. Unfortunately this isn't really documented, and it's easy to forget which modules have had @capture_args added to them until Verilog testing fails.

Shorten parameterized module name mangling when generating Verilog

Fix manual cpp harnesses in no_python to match updated generated cpp code

PyMTL wrapper generated by verilator_cffi writes both .value and .next for output ports

Verilog: Implement PortBundle Translation

Fix verilator_cffi harness generation to handle lists of PortBundles properly

Add a way to access a PortBundle's attribute name, rather than full name (ie. out instead of bundle.out)

connecting a model OutPort to a submodel InPort using slices does not simulate correctly

The following PyMTL model will not simulate correctly:

class CombOutputToRegInputSlice( Model ):
  def __init__( s, nbits ):
    s.nbits  = nbits
    s.in_    = InPort  ( nbits )
    s.out    = OutPort ( nbits )
    s.other  = OutPort ( nbits )

  def elaborate_logic( s ):
    s.reg = Register( s.nbits )
    s.connect( s.in_, s.other )
    for i in range( s.nbits ):
      s.connect( s.reg.in_[i], s.other[i] )
      s.connect( s.reg.out[i], s.out[i]   )   

The same module using @s.combinational logic also fails:

  def elaborate_logic( s ):
    s.reg = Register( s.nbits )

    @s.combinational
    def comb():
      s.other.value = s.in_

    for i in range( s.nbits ):
      s.connect( s.reg.in_[i], s.other[i] )
      s.connect( s.reg.out[i], s.out[i]   ) 

Replacing the output connection to not use slices fixes the problem:

  def elaborate_logic( s ):
    s.reg = Register( s.nbits )
    s.connect( s.in_, s.other )
    for i in range( s.nbits ):
      s.connect( s.reg.in_[i], s.other[i] )
    s.connect( s.reg.out, s.out   )

Create PortList class, subclasses Python list, but can have attributes (eg. .name)

Enable translation of slice constants

We sometimes use slice contants in the following way:

   data_field = slice(0,32,0)
   @s.combinational
   def logic():
     s.mem_req[data_field].value = 5

Would be nice for this to be translatable.

C Translation Python harnesses exposes PortBundles as separate wires

Fix pypy error when translating verilog

Add linting to check for assigning to InPort within module (issue only appears after verilog translation)

Add check to determine proper PortBundle directionality

Given a PortBundle Pair:

  InputBundle, OutputBundle

It's possible to use them incorrectly when connecting to modules:

   # modelA.out = OutputPortBundle( 8 )
   # modelB.in_ = OutputPortBundle( 8 ), but SHOULD be InputPortBundle( 8 )
   s.connect( modelA.out, 

This simulates completely fine in Python, but it fails when translating to Verilog (the port directions are incorrect).

Add linting detection of invalid dependencies in behavioral code

A few PyMTL users have written combinational RTL code that has a format similar to:

  @s.combinational
  def logic():
    s.status.value = s.enq.rdy and s.enq.val
    s.enq.rdy.value = some_state
    s.output.value = s.status

This code is difficult to understand and has unclear implications for synthesis (ie. what does the hardware look like that Verilog would generate from this?).

Additionally, this code may only work if certain Simulator optimizations are disabled. If some_state modifies the value of s.enq.rdy; then this logic() block would need to execute twice instead of once. This prevents the optimization which prevents a combinational block from being put on the event queue if it is currently executing.

It would be nice to have a way to detect these sorts of issues by analyzing dependencies in the AST, and notifying the user of the problem.

concat() function is not translatable

Cannot infer temporary bitwidth when a previously inferred temporary on RHS

PyMTL allows the creation of temporaries in a number of scenarios when the right-hand-side (RHS) of an assignment allows for easy bitwidth inference:

  some_temp = s.port_a and s.port_b
  s.out.value = some_temp

However, if the RHS of a temporary assignment contains another temporary, this fails:

  some_temp = s.port_a and s.port_b
  some_temp2 = some_temp and s.port_c  # FAILS due to some_temp on RHS

Add translation type inference for BinOp nodes

Add .name field to PortBundles

Make list/bit indexing with negative one (signal[-1]) translatable

Fix name $ to __24 mangling in verilator_cffi

Flattened portlists are mangled the following way during Verilog translation:

in_[0] becomes in_$000

Verilation further mangles these flattened ports to:

in_$000 becomes in___24000

This mangling propagates to the generated cpp_cffi wrapper since the cffi interface generated cannot expose names with $ in them (not a valid python name).

Verilog signals used before reg declaration fail on VCS due to one pass compilation

Code snippets like the following pass testing in verilator and iverilog, but fail on VCS since it only uses one pass compilation:

  assign mul$a          = reg_a;
  assign mul$b          = reg_b;
  reg    [  31:0] reg_a;
  reg    [  31:0] reg_b;

Changing the code to the following fixes the issue:

  reg    [  31:0] reg_a;
  reg    [  31:0] reg_b;
  assign mul$a          = reg_a;
  assign mul$b          = reg_b;

Provide a way to generate Verilog ready to use Synopsys DesignWare components

Add mechanism to detect registers that aren't properly reset (initialize to X)

verilog translation should generate localparam rather than param for constants scoped within a module

Rename pmlib to librtl and libcas

Shared libraries created during cpp translation should start with 'lib'

Add reduce_or and reduce_and operators

There currently is no good way to perform a reduction operation on a bitvector, or a list of single bits. In verilog its possible to do:

  assign nonzero = |a[3:0];
  assign allset  = &b[3:0];

In PyMTL, theres no good way to do this.

Verilog: Implement BitStruct Translation

Fix exception in GetRegsIntsParamsTempsArrays when LHS is not a Name/Attribute node

Add @requires_verilator decorator for pytest

Synopsys DesignCompiler requires inputs/outputs to have reg/wire labels

Currently the generated verilog does not explicitly declare wire/reg type for input and output ports:

  module MyModule                       
  (                                                            
    input  [  34:0] from_cpu_msg,                                                  
    output [   0:0] from_cpu_rdy
  )

Synopsys DesignCompiler always warned about this, but it was not required to do so. However, the addition of the `default_nettype none flag causes DC to error out.

We could simply remove the default_nettype flag, but the right thing to do is to add the reg/wire type labels:

  module MyModule                       
  (                                                            
    input  wire [  34:0] from_cpu_msg,                                                  
    output wire [   0:0] from_cpu_rdy
  )

executing 'py.test --test-verilog' with pypy results in libffi segfaults

Running the following command with a patched version of pypy (2a86e6f):

• py.test ../mvmult/MatrixVecLaneRTL_test.py --test-verilog

Results in a segfault with the following output:

../mvmult/MatrixVecLaneRTL_test.py ............*** glibc detected *** /home/graduate/dml257/venvs/pypyHG/bin/pypy-c: double free or corruption (!prev): 0x000000000513f9b0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x34fb676166]
/lib64/libc.so.6[0x34fb678c93]
./MatrixVecLaneRTL_0x71fa47b90d216111.so(destroy_model+0x20)[0x7fe4b5240ec0]
/usr/lib64/libffi.so.5(ffi_call_unix64+0x4c)[0x34fce05dac]
/usr/lib64/libffi.so.5(ffi_call+0x214)[0x34fce05b34]
/home/graduate/dml257/venvs/pypyHG/bin/pypy-c[0x63474b]

pypy version: https://bitbucket.org/pypy/pypy/commits/2a86e6faaf0f8ff057cf49a5c240d10d6739dd79

Add support for PortBundles containing PortLists

Add py.test check to skip tests dependent on iverilog if it is not installed

The following tests fail if iverilog is not installed:

• ../new_pymtl/translation_tools/verilog_behavioral_test.py
• ../new_pymtl/translation_tools/verilog_structural_test.py

Modify concat to take varargs, not list!

Add checks to prevent use of Verilog keywords!

Known keywords that signal names cannot include:

• reg
• wire
• wait
• const
• type

multiple assignments to a temporary result in 'duplicate declaration' errors in verilog

Ensure deterministic ordering of port/signal declarations in verilog translation to assist caching

Signal list in a for loop not added to sensitivity list for aug_assign

The following fails to trigger execution of a combinational block when a signal in the list s.done is updated:

  is_done = 1
  for i in range( s.nlanes ):
    is_done &= s.done[0]

The problem seems to be the augmented assign (&=). The following two alternatives work:

  is_done = s.done[0]
  for i in range( s.nlanes ):
    is_done &= s.done[i]

and

  is_done = 1
  for i in range( s.nlanes ):
    is_done = is_done & s.done[i]

Add VCD tracing

Add translation type inference for subscript nodes

../new_pymtl/translation_tools/visitors.py:325: Exception

Exception: Cannot infer type from <_ast.Subscript object at 0x10a638550> node

Translation cannot handle True and False

Should be able to easily convert these to 1 and 0...

Modify Verilog translation of temporaries to include function name (add scoping)

Currently two blocks that use the same variable name will cause problems because of name aliasing:

  @s.combinational
  def logic0:
    temp = a & b

  @s.combinational
  def logic2:
    temp = a & b

A possible fix in verilog is to attach the function name as a scoping technique, so the two temps become logic0_temp and logic2_temp.

Add translation type inference for _ast.Compare nodes

Fix verilog_cffi to use datatypes defined by verilatedos.h, currently not portable!

Add `default_nettype none to the top of generated verilog files

variables containing Bits are not converted into verilog params