firesim/firesim

FireSim: Fast and Effortless FPGA-accelerated Hardware Simulation with On-Prem and Cloud Flexibility

FireSim: Fast and Effortless FPGA-accelerated Hardware Simulation with On-Prem and Cloud Flexibility

We held the First FireSim and Chipyard User/Developer Workshop at ASPLOS 2023 on March 26, 2023! This workshop featured a full-day of talks from users and developers in the FireSim and Chipyard community. YouTube videos of the talks are available on the 2023 Workshop Page!

Contents

1. Using FireSim
2. What is FireSim?
3. What can I simulate with FireSim?
4. Need help?
5. Contributing
6. Publications

Using FireSim

To get started with FireSim, you can find our extensive documentation and getting-started guide at docs.fires.im. The FireSim codebase is open-source at github.com/firesim/firesim and we welcome pull requests and issues. You can also get help from the FireSim user community on our User Forum. Additionally, we frequently run tutorials at various conferences and events; for overview purposes, you can find the most recent slide decks at fires.im/tutorial-recent (you should still follow docs.fires.im for the most up to date getting-started guide).

Another good overview from a recent event (in video format) can be found on YouTube.

What is FireSim?

FireSim is an open-source FPGA-accelerated full-system hardware simulation platform that makes it easy to validate, profile, and debug RTL hardware implementations at 10s to 100s of MHz. FireSim simplifies co-simulating ASIC RTL with cycle-accurate hardware and software models for other system components (e.g. I/Os). FireSim can productively scale from individual SoC simulations hosted on on-prem FPGAs (e.g., a single Xilinx Alveo board attached to a desktop) to massive datacenter-scale simulations harnessing hundreds of cloud FPGAs (e.g., on Amazon EC2 F1).

Who's using and developing FireSim? FireSim users across academia and industry (at 20+ institutions) have published over 40 papers using FireSim in many areas, including computer architecture, systems, networking, security, scientific computing, circuits, design automation, and more (see the Publications page). FireSim has also been used in the development of shipping commercial silicon. FireSim was originally developed in the Electrical Engineering and Computer Sciences Department at the University of California, Berkeley, but now has industrial and academic contributors from all over the world.

You can learn more about FireSim in the following places:

• FireSim website: https://fires.im
• FireSim ISCA 2018 Paper: Paper PDF | IEEE Xplore | ACM DL | BibTeX | Selected as one of IEEE Micro’s “Top Picks from Computer Architecture Conferences, 2018”.
• FireSim ISCA-50 Retrospective: Paper PDF
• FireSim documentation: https://docs.fires.im
• Scala API Documentation: https://fires.im/firesim/latest/api/
• LatchUp 2023 Update Talk: YouTube
• FireSim (+Chipyard) Tutorial: https://fires.im/tutorial-recent/
• ASPLOS 2023 Tutorial Videos: YouTube Playlist
• Updates/News: Changelog | FireSim Blog | Twitter

What can I simulate with FireSim?

FireSim can simulate arbitrary hardware designs written in Chisel or Verilog. With FireSim, users can write their own RTL (processors, accelerators, etc.) and run it at near-FPGA-prototype speeds on cloud or on-prem FPGAs, while obtaining performance results that match an ASIC implementation of the same design. Depending on the hardware design and the simulation scale, FireSim simulations run at 10s to 100s of MHz. Users can also integrate custom software models for components that they don't need or want to write as RTL. To help construct a closed and deterministic simulated environment around a design, FireSim includes validated and high-performance HW/SW models for I/Os like DRAM, Ethernet, Disks, UART, and more. The User Publications page links to a selection of papers written by FireSim users.

FireSim was originally developed to simulate datacenters by combining open RTL for RISC-V processors with a custom cycle-accurate network simulation. By default, FireSim provides all the RTL and models necessary to cycle-exactly simulate from one to thousands of multi-core compute nodes, derived directly from silicon-proven and open target-RTL (RISC-V Rocket Chip, BOOM, and Chipyard), with an optional cycle-accurate network simulation tying them together. FireSim also provides a Linux distribution that is compatible with the RISC-V systems it simulates and automates the process of including new workloads into this Linux distribution. These simulations run fast enough to interact with Linux on the simulated system at the command line, like a real computer. Users can even SSH into simulated systems in FireSim and access the Internet from within them.

Head to the FireSim Website to learn more.

Need help?

• Join the FireSim Mailing list: https://groups.google.com/forum/#!forum/firesim
• Post an issue on this repo
• Follow on Twitter for project updates: @firesimproject

Contributing

• See CONTRIBUTING.md

Publications

ISCA 2018: FireSim: FPGA-Accelerated Cycle-Exact Scale-Out System Simulation in the Public Cloud

You can learn more about FireSim in our ISCA 2018 paper, which covers the overall FireSim infrastructure and large distributed simulations of networked clusters. This paper was selected as one of IEEE Micro’s "Top Picks from Computer Architecture Conferences, 2018" and for "ISCA@50 25-year Retrospective 1996-2020".

Sagar Karandikar, Howard Mao, Donggyu Kim, David Biancolin, Alon Amid, Dayeol Lee, Nathan Pemberton, Emmanuel Amaro, Colin Schmidt, Aditya Chopra, Qijing Huang, Kyle Kovacs, Borivoje Nikolic, Randy Katz, Jonathan Bachrach, and Krste Asanović. FireSim: FPGA-Accelerated Cycle-Exact Scale-Out System Simulation in the Public Cloud. In proceedings of the 45th International Symposium on Computer Architecture (ISCA’18), Los Angeles, CA, June 2018.

Paper PDF | IEEE Xplore | ACM DL | BibTeX

FPGA 2019: FASED: FPGA-Accelerated Simulation and Evaluation of DRAM

Our paper from FPGA 2019 details the DRAM model used in FireSim:

David Biancolin, Sagar Karandikar, Donggyu Kim, Jack Koenig, Andrew Waterman, Jonathan Bachrach, Krste Asanović, FASED: FPGA-Accelerated Simulation and Evaluation of DRAM, In proceedings of the 27th ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, Seaside, CA, February 2018.

Paper PDF | ACM DL | BibTeX

IEEE Micro Top Picks of 2018: FireSim: FPGA-Accelerated, Cycle-Accurate Scale-Out System Simulation in the Public Cloud

This article discusses several updates since the FireSim ISCA 2018 paper:

Sagar Karandikar, Howard Mao, Donggyu Kim, David Biancolin, Alon Amid, Dayeol Lee, Nathan Pemberton, Emmanuel Amaro, Colin Schmidt, Aditya Chopra, Qijing Huang, Kyle Kovacs, Borivoje Nikolic, Randy Katz, Jonathan Bachrach, and Krste Asanović. FireSim: FPGA-Accelerated Cycle-Exact Scale-Out System Simulation in the Public Cloud. IEEE Micro, vol. 39, no. 3, pp. 56-65, (Micro Top Picks 2018 Issue). May-June 2019.

Article PDF

ICCAD 2019: Golden Gate: Bridging The Resource-Efficiency Gap Between ASICs and FPGA Prototypes

Our paper describing FireSim's Compiler, Golden Gate:

Albert Magyar, David T. Biancolin, Jack Koenig, Sanjit Seshia, Jonathan Bachrach, Krste Asanović, Golden Gate: Bridging The Resource-Efficiency Gap Between ASICs and FPGA Prototypes, In proceedings of the 39th International Conference on Computer-Aided Design (ICCAD '19), Westminster, CO, November 2019.

Paper PDF

ASPLOS 2020: FirePerf: FPGA-Accelerated Full-System Hardware/Software Performance Profiling and Co-Design

Our paper that discusses system-level profiling features in FireSim:

Sagar Karandikar, Albert Ou, Alon Amid, Howard Mao, Randy Katz, Borivoje Nikolić, and Krste Asanović, FirePerf: FPGA-Accelerated Full-System Hardware/Software Performance Profiling and Co-Design, In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2020), Lausanne, Switzerland, March 2020.

Paper PDF

IEEE MICRO 2021: Accessible, FPGA Resource-Optimized Simulation of Multi-Clock Systems in FireSim

In this special issue, we describe the automated instance-multithreading optimization and support for multiple clock domains in the simulated target.

David Biancolin, Albert Magyar, Sagar Karandikar, Alon Amid, Borivoje Nikolić, Jonathan Bachrach, Krste Asanović. Accessible, FPGA Resource-Optimized Simulation of Multi-Clock Systems in FireSim. In IEEE Micro Volume: 41, Issue: 4, July-Aug. 1 2021

Article PDF

ISCA@50 Retrospective: 1996-2020: FireSim: FPGA-Accelerated Cycle-Exact Scale-Out System Simulation in the Public Cloud

This retrospective paper, included in the "ISCA@50 Retrospective: 1996-2020" collection, provides an update and retrospective on FireSim's development and evolution since the original ISCA 2018 paper.

Sagar Karandikar, Howard Mao, Donggyu Kim, David Biancolin, Alon Amid, Dayeol Lee, Nathan Pemberton, Emmanuel Amaro, Colin Schmidt, Aditya Chopra, Qijing Huang, Kyle Kovacs, Borivoje Nikolic, Randy Katz, Jonathan Bachrach, and Krste Asanović. FireSim: FPGA-Accelerated Cycle-Exact Scale-Out System Simulation in the Public Cloud. In ISCA@50 Retrospective: 1996-2020, Edited by José F. Martínez and Lizy K. John, June 2023.

Retrospective PDF

You can find other publications, including publications that use FireSim on the FireSim Website.

firesim's Issues

Don't use a global gitconfig setting to speed up recursive submodule init

For two reasons:

• It pollutes a user's gitconfig
• It can collide with simultaneous invocations of build-setup.sh

Add support for simulation with EDMA/XDMA in XSim

This should be possible once #36 is done.

Make infrasetup complain more obviously when stuff it wants to copy doesn't exist on the manager

Or, just run ./build.sh in firesim-software in the build-setup.sh script.

Make launchrunfarm complain and exit if runfarm with the specified name already exists

Fix IP address conflict between simulations and hosts

This is only a bug for users trying to SSH into simulated nodes. Doesn't matter for anyone else.

If a user used the aws-setup.py script to setup their account (which they should be doing), host instances get IPs from 192.168.0.0/24 and in all of our scripts, we default simulated nodes to also getting 192.168.0.0/24 addresses. This is a problem when bridging together the VPC network and the simulated network (e.g. when enabling ssh into simulated nodes).

This was unnoticed because my account was not set-up with our aws-setup.py because it didn't exist when I made my account :)

I think the fix is as explained here, but we will also need to propagate the IP address changes to all of our experiment scripts. I haven't had a chance to look, but IIRC the manager does not need to change.

https://groups.google.com/d/msg/firesim/zBu-a2A3xHM/jCaHTkd9AgAJ

Request build farm instances only after replace_rtl succeeds

It'll save some money if the user isn't watching the console and replace_rtl fails for some reason, and pre-launching the instances doesn't materially improve compile time.

Deprecate sbi-disk in pk/fesvr/spike, add generic-blkdev model to spike

This is part of unifying software configs across spike/qemu/firesim.

Right now, spike uses sbi-disk (which requires changing kernel cmdline), firesim has generic-blkdev, and qemu has virtio disks. As a first step, it would be nice to have a generic-blkdev model in spike.

Alternatively, we could deprecate spike and add a generic-blkdev model to qemu (or do both).

@zhemao @davidbiancolin @NathanTP

Update to upstream aws-fpga 1.4.0

This adds support for Vivado 2017.4.

https://github.com/aws/aws-fpga/releases/tag/v1.4.0

[UART] Use a pty instead of taking over stdin/stdout

This only requires modifying the UART simulation driver (C++).

See @a0u's example: https://gist.github.com/a0u/2d533ca4ea91764a0fa8

Remove block device driver messages from dmesg in Linux

There are too many.

test

On an error during buildafi, the build instances are not terminated

I was trying to build a mega boom afi image and ran into a timing error during the routing phase. The build exited with an error but it left 8 c4.4xlarge instances in running state instead of terminating them.

Need to handle asynchronous set in AsyncReset pass.

See chipsalliance/rocket-chip#1483

Will need to inspect the blackbox parameter to do the right thing.

This could silently break MIDAS-level simulation on the next bump.

Killing simulation doesn't work if executable name is too long

When running "firesim runworkload" or "firesim kill", killing the simulation executable won't work because the name is too long.

Add sifive uart driver to firesim linux version

Adding the sifive uart driver to the firesim linux version will allow us to boot fedora cleanly without the small patch we currently have

Inconsistent naming between manager/json

Not a correctness problem, but confusing when writing the json - the manager calls individual benchmarks "jobs", while the json calls them "workloads".

Fix how sourceme scripts deal with ssh-agent when user is using ssh forwarding

Give target-land device tests to their own recipes in the generated makefrag

We'd like to be easily able to test things like the blockdev, uart, even NIC in midas level simulation, without having to go through a bunch of manual steps.

Take per-port flag to enable/disable throttling in the switch models

This is a hacky solution which I'm going to merge for now, but this needs to be addressed properly too:
https://github.com/firesim/firesim/pull/161/files

add ctags target to Makefile

[workloads] include scripts to build SPEC benchmarks for RISC-V systems from user's SPEC install

Test setup-fixes branch on a clean AWS account

The fix staged on the https://github.com/firesim/firesim/tree/setup-fixes branch should address: https://groups.google.com/d/msg/firesim/_bk8O2_Z5Xk/X-pwJDI4AAAJ

C++ compilation error with synthesized assertions

make TARGET_PROJECT=midasexamples DESIGN=GCD verilator

Error Log: gcd-error.log

automatically prefix runfarms with some kind of user ID

makes sharing runtime configs easier if you need to run everything on one account

Re-enable non-timing-accurate network support

We previously supported this (e.g. https://aws.amazon.com/blogs/compute/bringing-datacenter-scale-hardware-software-co-design-to-the-cloud-with-firesim-and-amazon-ec2-f1-instances/ ), but it is currently not supported since nobody uses it. It would be nice to add it back, especially for cool demos (although you can still SSH into cycle-accurate-networked sims by instantiating an SSHPort in the switch model).

Fast setup requires setting LD_LIBRARY_PATH to include $RISCV/lib

This came up before but i assumed i just did something wrong. When running the assembly tests i get:

/home/centos/firesim-bump-firrtl/riscv-tools-install/bin/spike-dasm: error while loading shared libraries: libspike_main.so: cannot open shared object file: No such file or directory

Its easy to workaround this by setting LD_LIBRARY_PATH=$RISCV/lib But this isn't an issue for @sagark -- what gives?

Non-specific debug output

Output from running infrasetup when there is a chisel compile-time error does not display any information about the error itself.

Linux networking driver performance debugging

Add deterministic timing model to block device.

This is perhaps the largest remaining source of non-determinism in our sims.

Add support to the manager for booting QEMUs instead of FPGA simulations

For testing distributed software before launching FPGA sims. @NathanTP should specify needed features, will be assigned to Sean Luchen

[docs] Doc updates for release

• add a page that lists out all manager commands, including runcheck @sagark
• Tutorial: "add peripheral to FireChip -> simulate on FPGA" @zhemao @sagark
• running assembly tests/using vcs/verilator/XSim @davidbiancolin
• Workload docs
  • docs for running SPECInt17 Ref, also see #10 @davidbiancolin
  • improve docs for running ISCA experiments @sagark

[supernode] Cache address-map lookups in the sw driver for endpoints

Will probably improve perf a bit.

Fix subnetting for 256+ node networked simulations

In 256+ node simulations, we should be using /16 instead of /24 addresses.

This was fixed in scripts during ISCA, but didn't make it into the scripts here.

Unable to build integer only binaries with firesim toolchain

I am not sure if this is the right place to report this.

I am trying to build an integer only binary using -march=rv64i -mabi=lp64 using the firesim built toolchain. I am getting an error as follows:

riscv64-unknown-linux-gnu-gcc -c qht.c -std=c11 -ggdb3 -O2 -march=rv64ima -mabi=lp64 -fPIC
In file included from /home/centos/firesim/riscv-tools-install/sysroot/usr/include/features.h:447:0,
                 from /home/centos/firesim/riscv-tools-install/sysroot/usr/include/pthread.h:21,
                 from qht.h:4,
                 from qht.c:7:
/home/centos/firesim/riscv-tools-install/sysroot/usr/include/gnu/stubs.h:17:11: fatal error: gnu/stubs-lp64.h: No such file or directory
 # include <gnu/stubs-lp64.h>
           ^~~~~~~~~~~~~~~~~~
compilation terminated.

Use low-latency host networking stack instead of regular TCP/IP in switch models

This should help large-networked simulation performance.

E.g. http://www.f-stack.org/development.html

This requires knowledge of C++/network programming, no hardware dev.

Support for non-tethered systems

This will require re-thinking how we handle program load -- if we want to use loadmem, we should keep the target under reset, or write to DRAM at simulation time 0 -- and how we detect simulation termination.

FireBoom config throws SIGILL for mcf from Spec2017

Were the spec17 benchmarks tested for FireBoom? I am getting a SIGILL error while running mcf.

Starting speed 605.mcf_s run with 1 threads
[    0.336000] mcf_s_base.risc[96]: unhandled signal 4 code 0x1 at 0x00000000000101cc in mcf_s_base.riscv-64[10000+84000]
[    0.336000] CPU: 0 PID: 96 Comm: mcf_s_base.risc Not tainted 4.15.0-rc6-31580-g9c3074b5c2cd #1
[    0.336000] sepc: 00000000000101cc ra : 000000000001479c sp : 0000003fffe60c20
[    0.336000]  gp : 0000000000097100 tp : 0000000000099700 t0 : 0000000000000000
[    0.336000]  t1 : 2f2f2f2f2f2f2f2f t2 : 000000000000000c s0 : 0000000000014c04
[    0.336000]  s1 : 0000000000014cdc a0 : 0000000000000002 a1 : 0000003fffe60d68
[    0.336000]  a2 : 0000003fffe60d80 a3 : 0000000000000000 a4 : 0000003fffe60c48
[    0.336000]  a5 : 00000000000101cc a6 : 0000000000097320 a7 : 7efefefefefefeff
[    0.336000]  s2 : 0000000000000000 s3 : 00000000000d6008 s4 : 00000000000d6a88
[    0.336000]  s5 : 00000000000d4b88 s6 : 00000000000d4d68 s7 : 0000000000000000
[    0.336000]  s8 : 00000000000d6a48 s9 : 0000000000000000 s10: 000000000009c000
[    0.336000]  s11: 0000000000000000 t3 : 8101010101010100 t4 : 00000000000966b8
[    0.336000]  t5 : 00000000000956b8 t6 : 0000000000000000
[    0.336000] sstatus: 8000000200006020 sbadaddr: 00000000d5634785 scause: 0000000000000002
[    0.364000] reboot: Power down

[Workloads] gen-benchmark-rootfs.py improvements.

• Unmount the temporary fs if the script fails
• Resolve race condition between mounting and unmounting temporary fs
• Don't poweroff the instance -- let the user specify that in their command if necessary

[manager] Copy manager log into output directory

For commands like buildafi and runworkload, which produce their own output directory, it would be nice to have a copy of the log stored in that directory also.

Proper support for 200Gbps+ BW

Will be assigned to @abejgonzalez

Update riscv-gcc to fix SPEC2017 regressions

Recent riscv-tools bumps to track upstream gnu-toolchain causes regressions in SPEC2017 gcc and perlbench. Previously, we were tracking a version of riscv-gcc which resolved these issues. (see riscv-collab/riscv-gcc#100)

On the next riscv-tools bump, check to make sure the lra-contraints bugfix is present (riscv-collab/riscv-gcc@36e932c) in the version of riscv-gcc we are using, or revert to using our own submodule pointer for gcc (in a fork of the gnu-toolchain). Any version greater than 7.2.0 should suffice.

@donggyukim @cookiestuf @zhemao

Try XDMA, which is now supported with 1.4.0 Shell

[BOOM] Add support for BOOM

BOOM Support has been added to FireSim in the 1.2 release. However, there are still target-level bugs that need to be resolved in BOOM before it can be used fully for Linux-based software (Linux currently boots most of the way, but then hangs, which can be reproduced in both regular VCS (no sim components) and on the FPGA). See the log in the next post.

Documentation regarding custom AFI/AGFI is unclear

I tried following the documentation to create an agfi image for my modified BOOM core, but the instructions are not clear on what to do after successful building of the AGFI.

Section 4.3 ends by mentioning how to create a new fpga build, but stops short of mentioning how to deploy/run that custom built fpga and defers to the Advanced Docs section.

In the Advanced Docs section, we have instructions using make but no mention on how to customize and deploy the modified RTL.

Can you please add instructions on modifying and running the custom FPGA AGFI?

Running spec2017 does not work

I tried the commands as listed in the guide. I trimmed the benchmark list to just run mcf. When I run the benchmark I get the following error in the output:

[email protected]:~/firesim/deploy/results-workload$ cat 2018-08-31--23-56-42-spec17-intspeed/605.mcf_s/output/605.mcf_s_.err 
./run.sh: line 4: ./: Is a directory
[email protected]:~/firesim/deploy/results-workload$ cat 2018-08-31--23-56-42-spec17-intspeed/605.mcf_s/output/605.mcf_s_.out 
Running: ./ inp.in 

Looks like the run.sh script is not picking up the benchmark name properly?

[supernode] Integrate supernode into master

It currently lives on the supernode-beta branch. Want to add this as generic support for multiple copies of a target-design.

Fix deploy/workloads/bw-test.sh

Howie divided the bw-tests into one and two instance versions breaking this script. Im not sure what the desired behavior is so i'm punting to you @zhemao.

This also breaks run-all.sh.

Remove poweroff -f from gen-benchmark rootfs script

Workloads should supply it manually in their command

https://github.com/firesim/firesim/blob/master/deploy/workloads/gen-benchmark-rootfs.py#L26

Renable openocd; bump automake to 1.14 in the machine-launch-script

@zhemao once this is done, we don't have to have a special riscv-tools, right?

[Manager] Add support for MIDAS examples

I don't think will require an API changed -- just need the ability to optionally specify TARGET_PROJECT in build configs and hwdbs.