Group related tests into one file by making json an array of vectors

As test vectors begin to grow, would be nice to group tests in a single file instead of having a new file for every new test and grouping based on the prefix.

This will also help reduce file bloat especially if fuzz testing is added later. This will require the test runners to be updated to deserialize a slice/vector of the test vector struct, but is a change more easily made earlier than later.

Nothing necessary to change, just opening issue for discussion as this seemed odd to me.

thanks :)

update Lotus driver + CI to deal with incorrect test vectors

filecoin-project/lotus#3081

Test vectors marked as incorrect should be t.SkipNow()ed by the current CI test-conformance job filecoin-project/lotus#3310.
~~We should add a second job test-conformance-incorrect to exercise only the incorrect vectors, performing a negation assert.~~

ignore commits touching only corpus directory in github action

This means that when a PR for a vector update created by the action is merged it doesn't trigger the action again, only to find no changes.

https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#onpushpull_requestpaths

Selector for lotus specific tests

Within the chaos actor, there are boolean flags from parameters that set address and Cid to address.Undef and cid.Undef which can sometimes lead to Lotus specific checks occurring. Since these types should never in practice arise (or it's a bug in Lotus and they cannot be deserialized as that) would be nice to have some indication that it is testing a Lotus specific code path/ sanity check.

Nbd if not changed, but would be appreciated if non-protocol defined functionality tests would be denoted in some way so it's more easily identified by any other client using the submodule

Test that actor creation fails and gas is deducted when passing in bad/invalid arguments to actor constructor via init actor

Suite: actor_creation.

v.Messages.Sugar().CreatePaychActor() creates a correct payment channel actor constructor message.

To test this case, we can create a custom message using v.Messages.Typed() and pass in a from address that doesn't match the sender. That should make the paych constructor fail, and should bubble up the failure to us.

Test current balance behaviour in recursive sends

Actor A invokes a method on Actor B and supplies a nonzero value. Before Actor B’s execution is complete, it invokes a method on Actor A, but does NOT supply value. Within this context, CurrentBalance should factor in the amount initially sent to Actor B.

Finally, Actor B’s execution finishes successfully and Actor A continues execution where it left off. CurrentBalance should factor in the amount sent to Actor B.

Via @wadealexc

ci in specs-actors: build test vectors and fail if different

figure out preconditions and Lotus entrypoint for tipset, chain and blockseq vectors

Copied from the Chain Validation - Layer 3 requests:

Some things to be tested for a block:

Parents not in ticket order

ReceiptRoot doesn’t match StateRoot

ParentWeight doesn’t match

Signature doesn’t match

After a ChangeWorkerSignature, before/after that crosses lookback

WinningPoSt uses a faulty sector

Sector was ok at lookback but faulty now

Sector was faulty at lookback but recovered now

Miner doesn’t have enough power (check the boundary exactly)

Miner has power now but didn’t at the lookback

Miner had power at lookback but has less now

Miner has been consensus-slashed since the lookback

Miner doesn’t meet the consensus minimum

But neither do any other miners

But one other miner does

But three other miners do

Before/after lookback

Randomness lookback lands on empty tipset

Ticket is invalid

ElectionProof is invalid

BeaconEntries array is wrong length

BeaconEntries don’t form sequence

For sequences of blocks in a chain:

Chain weight evaluation

Tie-break for equally-heavy chains

With equal min-tickets

For ordering & finality:

Before finality, order does not affect chain selection. Test every permutation of block arrival across many forks with different weights.

Re-org to heavier chain, switch back if chain becomes relatively lighter

Test boundaries of long chain vs high power,

Switching to shorter but heavier chain

Switching away from a massively slashed chain (e.g. selfish mining)

Impact of null tipsets on weight

When one chain reaches finality, a heavier branch prior to finality is rejected while a heavier branch later is switched

Tests for system exit codes

We should test the VM returns the correct system exit codes in certain scenarios. Following is the list of system error codes, scenarios and tests that invoke them. For each exit code we need to define at least one scenario that invokes it and write a corresponding test.

(From https://github.com/filecoin-project/specs-actors/blob/a73a22a48b668c2f5a1cc6a31caef03bcaec5f70/actors/runtime/exitcode/reserved.go#L3-L58)

Note: coverage screenshots above are from the conformance test coverage data, these paths may be exercised in other tests.

coverage analysis for test vector corpus

During a call yesterday, @alanshaw asked how we can can identify which areas are untested by our test vectors (spoiler alert: many!)

Ideally we'd be able to trace how much of the reference implementation's codebase (Lotus+specs-actors) is exercised via our test vector generation scripts/corpus, so we can easily identify which test vectors need to be written to capture untested areas.

We can use go cover profiles for this, targeting the Lotus VM package and all of specs-actors for the cover profile.
We could hook in Coveralls or Codecov to give us a live new.

Note that our coverage analysis will overlap with the cover analysis of [specs-actors] tests (https://codecov.io/gh/filecoin-project/specs-actors), but the context is different, so I think it's justified.

Ideally we'd be able to measure not just binary yes/no coverage, but also coverage density, so that we know how many times a particular line/path has been hit.

Actions

Experiment.

Try to set up coverage analysis in CI, filtering by the above packages.
Visualise in Coveralls or Codecov.

[spike] full coverage of all actor methods

Explore calling each actor and their corresponding methods. Can help avoid situations like: filecoin-project/venus#4134

cc @icorderi

Test behaviour of DeleteActor and beneficiary

Test the behaviour as documented on the runtime DeleteActor method, including payment of residual balance to the beneficiary, and failure if the beneficiary is invalid.

This can use the payment channel actor, which deletes after settlement, or by extending the chaos actor.

support expressing that a vector is known to be broken in an implementation

With the increased test coverage, we are finding some actual Lotus VM behaviours to not match expectations. Currently our toolkit aborts generating the test vector when an assertion fails.

To deal with this situation and proceed with vector generation anyway, we should:

Introduce a v.MarkBroken(impl string, msg string) method to acknowledge that a vector is broken in a given implementation.
When that method is called, flip a flag in the Asserter to make it not abort.
Capture the known breakages in a "broken_in" test vector field.

Context: #49

Tipset-class vectors: null rounds and reward policy

Problem

When writing tipset-class vectors that include null rounds and verify proper reward application, we are confronted with a catch-22.

Say my starting epoch is 1, and I’m modelling the following chain history:

epoch 1: tipset A.
epochs 2-10: null rounds.
epoch 11: tipset B.

To verify the rewards applied at epoch 11 (tipset B), one needs to load the state of the reward actor at epoch 10, which is a null round. This state is an intermediate state, but is not explicitly linked to a block. It must exist in the CBOR store, but it is not referenceable or discoverable.

The state of the actor at that point is generated by the StateManager ticking cron for null rounds, and the cron jobs updating the reward policy in the reward actor.

Not even the StateGetActor RPC call allows you to access the state of an actor in a null round.

There are ways to work around this, e.g. intercept calls to CBORStore#Put and track the ones that carry a reward.State argument. But ideally they'd be a higher-level solution here.

Impact

Until this is solved, it is impossible to test proper reward application in a tipset that immediately succeeds a null round.

Test unencodable return value (PuppetActor#ReturnMarshalCBORFailure)

Implement a test for an unencodable return value

Test that CreateActor aborts if provided an address that already exists

Test that empty params must be empty

Test that if a method declares adt.EmptyValue params, then the params must be an empty array. Extra bytes should not be ignored.

Note that this is different to a plain value send (method 0), which explicitly ignores params and which can be non-empty.

Capture `ExecutionTrace`s as diagnostics in test vectors

Add a new top-level diagnostics field in the JSON schema and the go entity.
Add a diagnostics.execution_traces field which is an array of base64, gzipped strings.
Save the execution traces of every VM message into that array, in the same position as the message that generated it.
Enhance the test runner (filecoin-project/lotus#3081) with an env variable EXEC_TRACE_MISMATCH_FAIL that can be 0 or 1. By default it's 0. If a 1 value is passed, tests should fail if the execution trace mismatches.
Add a new CI job in that PR that enables this flag (that way we can monitor failures and decide if this is too much of an implementation detail to assert against).

Separate vectors from generator/ go code

Would be nice if the json vectors are put into a submodule or separate repo so that clients that use the vectors as a submodule do not need to pull in all repos recursively.

The main issue personally is that my go tools show errors in my editor (VSCode). The issue was first that because submodules are not pulled recursively by default, ffi was not pulled in, but even after doing that the makefile commands and go code linting still errors so I can't get rid of the errors easily.

Side benefit is also having to pull in a smaller repo as a submodule, but nothing is functionally wrong, so this can be low priority for sure :)

Thanks!

controlled randomness

Right now, the randomness we inject into the VM (via vm.Randomness#GetRandomness) is always a fixed 32-byte value. We will probably need to control randomness for more complex test vectors.

ci here: build test vectors and fail if different

Let's add support for being alerted when consensus-breaking changes are made upstream. Add a periodic CircleCI job (hourly) that:

clones the repo.
upgrades the specs-actors and lotus dependencies to whichever branch we're tracking (next).
runs make (generates test vectors).
fails if git diff returns differences.
notifies us on Slack when that happens.

consolidate the Puppet and the Chaos actor

Test that actor creation fails and gas is deducted when passing in unparseable constructor params via init actor

Failure case in actor_creation suite.

Use msg := v.Messages.Sugar().CreatePaychActor() to stage a good paych construction message. Do not apply it.
Directly mangle the parameters to form an invalid CBOR payload by prepending a bunch of zero bytes to msg.Message.Params.
CommitApplies() and expect a failure.

Test that StateHandle.Create is disallowed outside constructor methods

An actor's constructor is always method id 1. Use of the state handle Create method should be disallowed from other methods.

Test caller validation requirements

Check that:

a method must invoke at least one caller validation method, or abort
ValidateImmediateCallerIs() and ValidateImmediateCallerType() each abort if given no arguments
validating twice aborts the message

readme: document chaos actor in README

incorrect vectors: negate result on Lotus CI

Extracted from #72.

For incorrect vectors, if running the vector causes the implementation to derive the postconditions captured in the test vector, this indicates a failure. (i.e. incorrect behaviour is still present).

A non-match suggests a success or a skip.

analyse and define schema for chain-class test vectors

populate the version of lotus and spec actors under `_meta.gen`

I think we can extract them using the build package and iterating through the imported modules: https://golang.org/pkg/go/build/.

introduce block sequence test vector class

Block sequence class: model a sequence of {block_arrival_time_offset, full_block} that captures the timeline in which blocks are to be received.

This is to test chain selection and reorgs.
Requires:
- An implementation that is able to mock time.
- Oni introduced that feature in Lotus already.
- In Lotus, targeting the syncer#ValidateBlock method, not the VM API.
- Crafting blocks programmatically.
- Computing the correct postcondition state tree that indicates the test passed.
The test driver needs to feed blocks to the syncer, advancing time respectively as the test vector indicates.

Test enforcement of state readonly semantics

If an actor loads state with ReadOnly, it should abort if it modifies that state object before either (a) starting a transaction, or (b) returning.

Test that aborting before validating caller preserves the abort exit code

An actor method must invoke caller validation, else the call will abort with SysErrIllegalActor. But what it it aborts before the validation (e.g. it needs to load state to do so)? The exit code that is an argument to Abortf should be preserved.

(I believe Lotus already does this as expected).

prototype a chaos actor that triggers violations

This is similar to the puppet actor in specs-actors, but it lives in the test-vectors repo. That allows us to evolve its functionality and eventually standardise and spec out its ABI. We want other other implementors to implement this actor and deploy it with a standard CodeCID, and under a standard singleton address, so that test vectors will be able to leverage it.

Test vectors requiring the chaos actor will carry selector chaos_actor=true.

Test that actor creation fails and gas is deducted when providing an unknown actor code CID

Failure case in actor_creation suite. Known actor IDs are here: https://github.com/filecoin-project/specs-actors/blob/master/actors/builtin/codes.go

Test VerifySignature syscall

From the docs:

// Verifies that a signature is valid for an address and plaintext.
// If the address is a public-key type address, it is used directly.
// If it's an ID-address, the actor is looked up in state. It must be an account actor, and the
// public key is obtained from it's state.

test that a pubkey address is used directly, even if the corresponding actor doesn't exist
test that an ID address is resolved to the corresponding pubkey address
test that an address corresponding to a non-account actor results in error (not panic/abort)
test that a bad signature results in error

analyse and define schema for tipset-class test vectors

Note: Miners earn rewards for mining a block, and those rewards get applied as a balance bump on the state tree. We will likely need a miner_addr field on the Preconditions object in test vectors, so we can associate the reward accordingly.

Test that a panic in actor code results in undefined execution

If an actor implementation panics directly (rather than calling Abortf) then the evaluation is undefined. There is no exit code corresponding to this. The result should not go on chain. A panic (which could also come from some actor dependency) may indicate a transient state or error that cannot be replicated by other nodes and thus cannot form part of consensus. E.g. an out-of-memory.

Test that when an actor panics, the VM indicates this by some kind of failure that is distinguished from an Abortf and has no risk of going on chain.

This could be a little tricky, and implementation specific.

analyse and define schema for block sequence-class test vectors

Chaos actor CallerValidationBranch doesn't need to be bigint

Since it is just denoting an enum/ number (currently 0 to 3) it can be replaced with using CborInt to avoid the allocation. (not sure if there is a type for unsigned integer if that's what you want, I'm sure there must be)

Not a big deal if this doesn't change, we could avoid the allocation on our end if we wanted to, but just for future reference, or if someone wanted to change sooner than later.

Test serialization failure in actor storage put()

See test cases to implement in #54 (comment).

Actors can attempt to store data that cannot be serialized, which should result in ~~a SysErrSerialization~~ an error that the actor can handle by aborting with an appropriate exit code.

One way of exercising this is to cause an actor to attempt to store a bitfield with > 8192 RLE-encoded bytes.
Another way might be an explicit call from the puppet actor that stores an object that always fails to MarshalCBOR.

E.g. see filecoin-project/lotus#1627

Remove reward state initialisation

Once this issue is fixed, we can remove the hacky thing in test.go that creates the reward state with an initial per reward message of 1e17.

Test that message send, and all other side-effecting functions, are disallowed inside state transactions

selectors as json

given existing json serialization having the extra level of selector serialization seems un-needed. lets encode it as a json map or array of strings

Test Runtime#ResolveAddress

If the address is an ID address, it should be returned directly, even if no such actor exists
If the {pubkey|actor} address is not in the InitActor's table, return (Undef, false)
If the {pubkey|actor} address is in the InitActor's table, the corresponding ID address should be returned

Test serialization failure of actor state in transaction

Test that if an actor mutates its state object into an unserializable form during a transaction, the message aborts with ErrSerialization.

Note that it's not a system error code, despite originating in the VM. Serialization logically is part of the actor code.

verification syscalls must be able to fail (VerifyPoSt, VerifySeal, VerifySignature)

Right now they return unconditional success. However, this won't allow us to test failing scenarios, whether extracted from chains or generated.

(re)Generate test vectors on merge to master in Github Action

This will save people from forgetting to add the test vector to the corpus for newly added tests.

Tags for vectors in _meta

Update the schema to add a tags: string[] to _meta. This can be used, for example, to tag areas of the spec that vectors cover. This would allow vectors to be processed by the filecoin specs website so that indicators and styling can be added to spec sections as appropriate.

The work described by this issue is to enable this functionality, @hugomrdias and @yiannisbot to decide and document the format and content and actual tags can be added at a later date.

Test that an actor cannot Abort with a system exit code

Test that if an actor calls Abortf with an ExitCode within the range [0, 15] the runtime should abort with SysErrorIllegalActor.

Test also that a negative value also results in SysErrorIllegalActor

filecoin-project / test-vectors Goto Github PK

test-vectors's Introduction

[DEPRECATED] VM and chain test vectors for Filecoin implementations

License

test-vectors's People

Contributors

Stargazers

Watchers

Forkers

test-vectors's Issues

Actions

Problem

Impact

Recommend Projects

Recommend Topics

Recommend Org