Evaluate test frameworks

[netsniff-ng]

Evaluate test frameworks

[mlafeldt]

Emma, don't waste your time evaluating test frameworks. We "democratically" decided to go with Sharness -- see #18. 🔨

[eroullit]

I am still convinced Ctest covers some ground Shareness does not and vice-versa.
When such "democracy" occurs, people say: ¡No Pasarán! ✊

[mlafeldt]

Yeah, Ctest is better suited for C unit testing. Sharness prevails when it comes to testing command-line interfaces or shell code itself. So we really have two distinct use cases and it's not black and white.

[eroullit]

In the end, looks more like an unanimity.

(Quote from yesterday's IM on the topic)

Daniel: mathias agreed to include his shareness improvements with some basic test bases into netsniiff-ng
Me: yes I saw the ticket and its answer. we will see, but I can already tell you that it is not a silver bullet (e.g. specific function test)
Daniel: okay
Me: but I think ctest and shareness would make a very good combo
Daniel: yes, i think so too
Me: as ctest misses on higher level tests
Daniel: yes, exactly, that's where shareness comes in
Me: in a nutsheel ctest for low-level test and shareness for high-level ones
Daniel: yes, fully ack

[mlafeldt]

No matter what framework we're using, it looks like netsniff-ng always needs root privileges, which is obviously bad for automated testing.

[borkmann]

Maybe there are some hacks for rules.d about which I am not yet aware of. It should definitly be possible to run curvetun, for instance, without root privileges.

[borkmann]

FYI: I just closed issue 12 (http://coccinelle.lip6.fr/) and would like to move the discussion to this ticket, so that we have all 3 frameworks here.

Semantic patches should go into: src/s/
Sharness + Ctest should go into: src/t/

[mlafeldt]

So you even need to be root to let netsniff-ng process a .pcap?

Keep in mind that using Sharness (or any scripts) as root could backfire very easily.

[borkmann]

How exactly do you define "backfire"? Most likely, you need to run those tools (except ifpps, curvetun [with rules.d adaption]) as root. For instance, in netsniff-ng you have RX/TX_RINGs. Only root can allocate them.

[mlafeldt]

It was late and I worried about running all tests via root. But I guess it'd be okay to say sudo netsniff-ng to limit privileges to particular commands. (Need to test how Sharness copes with that.)

And btw: http://www.dict.cc/?s=backfire :)

[borkmann]

Included in PROJECTS file.