We should run each test file in it's own thread.

[lunarmodules]

We should run each test file in it's own thread.

[Tieske]

I'm not so into performance testing, but isn't it dangerous to add a fail/pass verdict on a test based on execution time? that will always be system dependent, so you might be getting different results on different systems.

[DorianGray]

Typically performance tests, like integration tests, run on a nightly basis since they also typically take longer to run than unit tests. This means that typically they are also running on consistent hardware.

This does give us something we can think about though, maybe we have it benchmark a specific routine that it's total execution time is equal to 1 point. We then use points to describe execution time rather than seconds? This should allow us to be consistent across cpu architectures and lua implementations.

[DorianGray]

Maybe also we generate a log of total points, for system performance over time tracking?

[DorianGray]

That is, make total points a part of the output.

[DorianGray]

or points per test, rather

[Tieske]

I would rather think that some report (similar to the coverage output) would be best. Highlighting the code that takes the most time to run, that's the code you would want to optimize. But then again, optimizing testruns is a sub optimization as you should actually optimize production like behaviour

[Tieske]

see http://lua-users.org/wiki/ProfilingLuaCode for some existing profilers that might be integrated

[DorianGray]

I think a separate report is a better idea, I modified the other feature to not specify how it was output but just to expose the data to the output handlers.

There are two use cases here I think.

1. Performance regression testing - target production use cases(basically integration tests that are timed)
2. Unit benchmarking - target small units of code
   The performance regression tests would be run nightly along with the integration tests and the unit benchmarks would run with the unit tests.

Limiting a test to a certain run time is useful more for async tests and not for performance analysis.

I could see myself writing a parser for my performance test output that shoved it all into something like graphite, so I could then see my application performance over time and also break it down all the way to the unit level.

[Tieske]

seems ok to me.

[emberian]

Do you think lanes would be good to use for this? I'm interested in working on this feature.

[Tieske]

iirc @DorianGray had lanes in mind for this feature.

[ajacksified]

@cmr We were looking at lanes, but ran into some problems some months ago running Lanes on Windows and OSX. If you'd like to give it a shot, we're always interested in pull requests.

[eloff]

Just chiming in with my two cents here, I think the most useful for performance tests is to be able to track the results over time, to see what builds contained performance regressions and by how much. Mono does/did a similar thing with nightly performance tests although their links seem to be broken at the moment.

It was just a table with a row per night and a percentage per cell (green for faster, red for negative), the columns being the performance test cases. I'm not saying that should be the format, but I'm offering it as an example.