Scaling this effort up

[syhw]

Scaling this effort up

[tlikhomanenko]

Hi @syhw,

thanks a lot for your huge work on this!

I prefer to have 2. option with a full picture (which still can be helpful to understand what people are doing and how this is realistic) to be prepared :).

However, my suggestion is to have two different tables:

1. for verified results with links to code/repro/proof (like golden test results)
2. with other results which should be still verified somehow. In this case information will be organized in a clear way to cite, to check where we are now, what we still need to reproduce.

Also I prefer to have separate two columns: reproducible and #citations (a lot of citations is not always equivalent to reproducibility).

[lunixbochs]

The most interesting form of "reproducible" for me is if a third party (unaffiliated with the original team) has posted a blog or whitepaper claiming similar results to the original, aka "findings were actually reproduced".

Could have a reproduced column with confidence values such as "the editor repro'd", "link to third party repro", "no repro but we trust the source", and "low confidence"