CLI Feature: Test suite

[onflow]

CLI Feature: Test suite

[sideninja]

Hi 👋
my name is Gregor, and as a software engineer working on developer tools, I will be your point of contact for this task, so good luck, and don't hesitate to reach out to me with any problems or questions.

I'll keep an eye on your progress and will be reviewing your code.

You can comment here or find me on Discord (I'm sideninja#1970). Join the Flow Discord server if you're not there already!

[jrkhan]

Hello!

My team (currently just myself) would be interested in tackling this issue! Writing tests seems to me to be a great way to learn about an API.

Team:
https://www.hackerearth.com/challenges/hackathon/flip-fest/dashboard/f541fdb/team/view/
Members:
@jrkhan
Timeline:
Expect to make steady progress adding tests over the 2-3 weeks.

Initial observations/questions:
Currently, the command layer/internal packages do not appear to be picked up by the code coverage report:
https://codecov.io/github/onflow/flow-cli/commit/dc04cc327163db18a0705562fdd8dd70b9618522
I would assume we'll also want to track /internal here?

Are other contributors running through vscode/using gopls language server? If so, I'd be curious what settings you use for the equivalent of goimports -local github.com/onflow/flow-cli - currently running into import order related lint errors more often than I'd like.

What are your thoughts on table driven tests?
https://github.com/jrkhan-bot/flow-cli/pull/1/files#diff-38c58eb68c7d16ea13df9ba3c3375d34995d5b5d394159bbbcf2937305f216cdR29-R56

[sideninja]

Hi, great to have you work on this.

Currently, the command layer/internal packages do not appear to be picked up by the code coverage report:
https://codecov.io/github/onflow/flow-cli/commit/dc04cc327163db18a0705562fdd8dd70b9618522
I would assume we'll also want to track /internal here?

Yes, that is the package that needs testing.

Are other contributors running through vscode/using gopls language server? If so, I'd be curious what settings you use for the equivalent of goimports -local github.com/onflow/flow-cli - currently running into import order related lint errors more often than I'd like.

We are using goland with linters, also you should use make install-linter and then make lint for checking your code.

What are your thoughts on table driven tests?

I think they should be preferred where applicable. I like them, but only for where it makes sense.

[jrkhan]

Hello @sideninja

Wanted to run some work in progress by you:
https://github.com/jrkhan-bot/flow-cli/pull/2/files

It's the general pattern I was hoping to validate:
Before each test (i.e. from TestMain) start the emulator, then each test will run one or more Cobra commands from within the test process itself using cobra.Command.ExecuteC. (This is somewhat similar to how Cobra tests commands: https://github.com/spf13/cobra/blob/master/command_test.go#L45)
So far, just I'm just testing 'json' formatted output, but will also plan to check 'text' and 'inline'.

Here's the result in codecov (we have ever so slightly increased our coverage in the internal package):
https://codecov.io/gh/jrkhan-bot/flow-cli/tree/cc33d05932fb008bace4918b3372de3db554e991

Assuming this looks decent, I will plan to run with it for other commands and output types - but I'd be happy to incorporate any feedback you might have at this point before doing so.

---

Prior to this, I had also tried running commands as separate processes. The test invoked the CLI via an external process per command. However, I didn't find a good way to get coverage metrics that way. I believe it would have involved:

1. Building a binary similar to cmd/flow that also supported -coverprofile
2. Having each command invoked by a test create it's own coverage report
3. Merging all coverage reports through a third party library

[sideninja]

Hi @jrkhan,
thank you for your work on this, however, I would advise you to take a different approach by isolating different parts of the command layer and unit test those parts as much as possible.

Let me give you an example of tests for blocks command.

Testing results
I would take the block result and then construct it in test with mock data and check when calling each of the methods JSON(), String(), Oneliner() that the output is correct. You can even use helpers from go-sdk to build these mock blocks. I would apply the same principle to all results objects.

Testing commands
I would avoid using cobra command as we don't need to test cobra, they have their own tests, but what I like about your approach is that we make sure the flags and all other parameters work (but we should test that again as a separate piece). The way I would test commands is to take the command function and provide arguments and then check the result is matching the expected all that by using mock gateway. Let me again give an example for blocks. We would be testing the get function. Pass different values as arguments and use mock gateway returning the values that make sense for that case. You can check how to construct services with mock gateway here and then how to use mock gateway to provide different responses here

Why I would prefer this way of testing over yours:

• your way depends on starting up the emulator as a separate service, which introduces a lot of uncertainty (network, CI environment, delays on startup, port in use etc etc)
• tests will be much slower and will depend on the outside state of the emulator
• tests are quite complex which creates a problem of tests not working correctly (so you would need tests for tests :D)
• changes will ripple through different parts of your tests
• I could continue but I think you will get the idea

Hopefully, this doesn't discourage you I think you are doing a good job, is just that I should explain the task better.

(p.s. In the future open a PR here on flip fest as a milestone and we can chat under that PR)

[kimcodeashian]

Good day @jrkhan!

Thanks so much for all your hardwork & participation. In order to finalize winners & prepare for prize payout, we'll need the following actions from your end.

Please provide the following information by Nov 17, 2021, (in this GH Issue is fine):

1. Team Information

• Team Members Information - Github Username + Email Contact + Percentage of prize allocation (total should = 100%)
• All mentioned members MUST react to the post with a 👍 which will act as confirmation that the information is correct, or a 👎 to indicate that the information is not correct.
• We will be reaching out via e-mail

🎖IMPORTANT: We will only proceed with prize payouts once all members have confirmed with 👍 on the post.

2. Video Demo (optional)

• Please provide a 5-minute video demo to be featured & showcased in the FLIP Fest Closing Ceremonies
• Link format & Downloadable (eg. Google Drive, Vimeo)
• Content Format (Problem Statement, your work / how you solved it, final outcome)

We will be hosting Closing Ceremonies on November 23rd, 8AM PT where we'll having closing remarks from Dete & will be announcing the winners! I'll share the details here before Nov 17.

[kimcodeashian]

Hey folks,

We've received and reviewed over 82 submissions! What an amazing community on Flow! To commemorate all the hard work done, we have finalized winners and will be announcing them during our Closing Ceremony on Nov 23rd, 8AM PT. Be sure to join us - there may be some attendance prizes & a keynote from our CTO, Dete 😉!

RSVP here so you don't miss out! See you then!