alephao/solidity-benchmarks

Benchmarks of popular contract implementations in solidity

Solidity Benchmarks

Benchmarks for popular implementations of contract standards.

⚠️ The gas usage shown in the benchmarks doesn't take into account the 21k gas added to every ethereum transaction

• ERC20 0.8.24
• ERC721 0.8.24
• ERC1155 0.8.24

You can see benchmarks for different compiler versions on benchmarks/

The file data.json provides a json format of all the data used in the benchmarks. That file is generated by running the command make json.

Method used

We create a minimal implementation of each contract that uses the specific implementation as a base. Then for each of the methods we want to benchmark, we create a test contract that set the environment in the setUp function and each test only runs the specific function we're benchmarking, trying to reduce the noise as much as possible.

The gas usage shown here is not 100% accurate, but it's good enough to be able to compare the gas usage between the implementations.

All tests are generated using the template files in templates

All tables in the readmes are generated using the scripts in scripts

Contributing

There are many ways to contribute to this project

• Add a snapshot for the latest solc version
• Add or suggest a contract implementation (the instructions for adding are below)
• Update a contract implementation
• Enhance the codegen scripts

Setup for local development

• Install foundry http://getfoundry.sh
• You'll need python 3 installed to run the scripts under the scripts folder
• Install the python dependencies pip install -r requirements.txt
• git clone --recurse-submodules https://github.com/alephao/solidity-benchmarks.git

How to add a contract

1. Create a minimal implementation on src/, the contract name and file name should follow the convention <Contract Type>_<Variation>.
2. Implement the common interface that is in other files of the same contract type (for ERC721 for example, it's mint and safeMint functions)
3. Add an entry to the contracts.<contract type>.variations property on test-cases.yml, following the examples there.
4. Add an entry to scripts/.py's variants var following the examples there. It should map the variant name you used in the contract like ERC721_<Variant> to the name you want to appear on the table. E.g.:
5. In case you added an ERC721 that's also ERC2309 compliant, add a another contract in the same file following the convention <Contract Type>_<Variation>_ERC2309, and add an entry to the contracts.ERC721.ERC2309Variations in the test-cases.yml

variations = {
    "OZ": "OpenZeppelin",
    "OZEnumerable": "OpenZeppelin Enumerable",
    "OZConsecutive": "OpenZeppelin Consecutive",
    "Solady": "Solady",
    "Solmate": "Solmate",
    "A": "ERC721A",
    "B": "ERC721B",
    "K": "ERC721K",
}

5. Run the following commands:

make codegen
make snapshot
make readme
make json

6. Add the contract to the list at the top of the <Contract Type>.md. (If you added a new ERC721, update the list on top of ERC721.md)

Quick links

ERC20

• 0.8.20 or 0.8.20-ir
• 0.8.21 or 0.8.21-ir
• 0.8.22 or 0.8.22-ir
• 0.8.23 or 0.8.23-ir
• 0.8.23 or 0.8.24-ir

ERC721

• 0.8.20 or 0.8.20-ir
• 0.8.21 or 0.8.21-ir
• 0.8.22 or 0.8.22-ir
• 0.8.23 or 0.8.23-ir
• 0.8.24 or 0.8.24-ir

ERC1155

• 0.8.20 or 0.8.20-ir
• 0.8.21 or 0.8.21-ir
• 0.8.22 or 0.8.22-ir
• 0.8.23 or 0.8.23-ir
• 0.8.24 or 0.8.24-ir

solidity-benchmarks's Issues

[Feature request] benchmark deployment costs

feat: add ERC721 burn

Mostly implemented on my fork based on transferFrom logic. N/A for ERC721B.

Track ERC2309 compliant ERC721s

Description

I've recently updated both OpenZeppelin contracts (#12), ERC721A (#13), and both now support a consecutive minting according to ERC2309. However, the current way of adding a new variation and test its implementation is not compatible with this test case.

The reason behind this is that when the ConsecutiveTransfer event is emitted outside of the constructor, the contract is not ERC721 compatible anymore as stated in the ERC721A comments and also restricted on OpenZeppelin Contracts.

I think both ERC2309 implementations are worth reviewing and benchmarking. I can take care by myself and send a PR, however, I'd love some feedback before moving forward.

Context

I tried following the README steps in order to add a new variant, this is an example of what I was thinking for an ERC721_OZConsecutive:

// SPDX-License-Identifier: UNLICENSED
pragma solidity >=0.8.4;

import {ERC721} from "@openzeppelin/contracts/token/ERC721/ERC721.sol";
import {ERC721Consecutive} from "@openzeppelin/contracts/token/ERC721/extensions/ERC721Consecutive.sol";

contract ERC721_OZConsecutive is ERC721Consecutive {
    uint256 currentSupply;

    constructor() ERC721("Name", "Sy") {}

    function mint(address to, uint96 amount) external payable {
        uint256 startingIndex;
        unchecked {
            startingIndex = currentSupply + 1;
            currentSupply += amount;
        }

        _mintConsecutive(to, amount);
    }
    
    ...
}

However, it fails because the _mintConsecutive(to, amount) is called outside the constructor, and I don't see an easy workaround.

Right now my best approach is to add an alternative test_deploy with the consecutive minting for ERC2309-compliant tokens, but I wanted to open a discussion before sending the PR.

I'd love feedback @alephao

chore: hide/ignore inapplicable tests

Use-case: Some contracts may not support all tests, for instance ERC721B currently doesn't support burning. This may be the case for other contracts/tests in the future.

feat: benchmark by compiler version

explicitly state the features of a contract for a more fair comparison

Many of the contracts here have some extra functionality, for example if we look at the ERC20s, Solmate and Maple implements ERC-2612 permits, while the basic OZ's ERC20 doesn't.

Comparing Solmate vs Maple is fair, but OZ vs Solmate is not.

I think it might be a good idea to categorize them somehow, current idea is:

On READMES

Provide a table with features. Example for ERC20:

Contract/Features	ERC2612	Another Feature
Solmate	✅	✅
Maple	✅	❌
OZ	❌	✅
OZ with Permit	✅	✅

On JSON

ERC20
  Solmate
    features: [ERC2612]
  Maple
    features: [ERC2612]
  OZ:
    features: []

ERC721
  OZ:
    features: []
  A
    features: [Enumerable]
  OZEnumerable
    features: [Enumerable]

(sorry for being lazy and not writing json by hand, but you got the idea)

Provide a JSON file with all the data

There is a tool I want to build on top of this. Writing this down before I forget.

For that tool I need a more structured data source, I think a simple JSON file would work well.

I want to be able to:

• Compare gas cost of a function across compiler versions
• Filter results, for example, compare only ERC721A vs ERC721B
• Sort by column
• Get diff between columns on raw values and percentage, between same implementation or other implementations

My initial thought of the JSON structure:

{
  "ERC20": {
    "v0.8.10": {
      "deployment": {
        "Open Zeppelin": "568857",
        "Open Zeppelin Permit": "926489",
        "Solmate": "656844",
        "Maple": "672309"
      },
      "transfer": {
        "Open Zeppelin": "20917",
        "Open Zeppelin Permit": "20873",
        "Solmate": "20590",
        "Maple": "20657"
      }, 
      "approve": {
        "Open Zeppelin": "32649",
        "Open Zeppelin Permit": "32672",
        "Solmate": "32547",
        "Maple": "32598"
      }
    },
    "v0.8.11": {
      "deployment": {
        "Open Zeppelin": "568857",
        "Open Zeppelin Permit": "926489",
        "Solmate": "656844",
        "Maple": "672309"
      },
      "transfer": {
        "Open Zeppelin": "20917",
        "Open Zeppelin Permit": "20873",
        "Solmate": "20590",
        "Maple": "20657"
      }, 
      "approve": {
        "Open Zeppelin": "32649",
        "Open Zeppelin Permit": "32672",
        "Solmate": "32547",
        "Maple": "32598"
      }
    }
  },
  "ERC721": {
    "v0.8.10": {
      "deployment": {
        "Open Zeppelin": "568857",
        "Open Zeppelin Permit": "926489",
        "Solmate": "656844",
        "Maple": "672309"
      },
      "mint": {
        "Open Zeppelin": "20917",
        "Open Zeppelin Permit": "20873",
        "Solmate": "20590",
        "Maple": "20657"
      }
    },
    "v0.8.11": {
      "deployment": {
        "Open Zeppelin": "1111",
        "Solmate": "3333",
        "ERC721A": "4444"
      },
      "mint": {
        "1": {
          "Open Zeppelin": "1111",
          "Open Zeppelin Enumerable": "2222",
          "Solmate": "3333",
          "ERC721A": "4444",
          "ERC721B": "5555",
          "ERC721K": "6666"
        },
        "2": {
          "Open Zeppelin": "1111",
          "Open Zeppelin Enumerable": "2222",
          "Solmate": "3333",
          "ERC721A": "4444",
          "ERC721B": "5555",
          "ERC721K": "6666"
        },
        "3": {
          "Open Zeppelin": "1111",
          "Open Zeppelin Enumerable": "2222",
          "Solmate": "3333",
          "ERC721A": "4444",
          "ERC721B": "5555",
          "ERC721K": "6666"
        }
      }
    }
  }
}

use gyb instead of stencil for codegen

What is GYB: https://nshipster.com/swift-gyb/
Source code: https://github.com/apple/swift/blob/main/utils/gyb.py

Why?

• stencil-cli only works on macos, already had people wanting to contribute but couldn't because of codegen
• I like GYB, it is very easy to use and it's python, we're already using python in the codebase

What to do?

• Copy gyb to the repo
• Update template files to use GYB syntax
• Create a script that read the yml config and runs GYB using the config and template files
• Add to makefile

Add benchmarks for solidity 0.8.14~0.8.17

1. Install foundry and solc-select
2. With solc-select install solidity versions from 0.8.10 up to 0.8.17
3. Add the missing commands to Makefile

   solidity-benchmarks/Makefile

   Lines 5 to 10 in 2a5d94e

   	.PHONY: snapshot
   	snapshot:
   	forge snapshot --force --optimize --use 0.8.10 --snap 0-8-10.gas-snapshot
   	forge snapshot --force --optimize --use 0.8.11 --snap 0-8-11.gas-snapshot
   	forge snapshot --force --optimize --use 0.8.12 --snap 0-8-12.gas-snapshot
   	forge snapshot --force --optimize --use 0.8.13 --snap 0-8-13.gas-snapshot

4. Add the missing versions to the python script

   solidity-benchmarks/scripts/main.py

   Lines 104 to 109 in 2a5d94e

   	versions = [
   	"0.8.10",
   	"0.8.11",
   	"0.8.12",
   	"0.8.13",
   	]

5. Run make readme and make json
6. Send a PR referencing this issue