paperswithcode/releasing-research-code

Tips for releasing research code in Machine Learning (with official NeurIPS 2020 recommendations)

releasing-research-code's Issues

Adding link to `rliable`?

I was wondering if the open-source library rliable, corresponding to our NeurIPS 2021 best paper, for reliably evaluating reporting performance on ML and RL benchmarks, especially when using a handful of runs might be relevant for the README. Adding this as an issue as I wasn't sure though about the best place to add a link to this library in the README.

Add citation recommendation

When finding a paper or code, it is nice to be able to cite it properly, specially if a latex bib style is presented, so citation should be a must.

A regular problem is that usually the citation people tend to provide is to arxiv papers. Don't get me wrong, arxiv definitely is essential, but providing a citation to a conference paper helps assure that the paper passed the criteria to get properly published somewhere.

Software License Recommendation

I've seen some unlicensed research code being published, which makes reuse hard and increases friction. Usually that was not the intent, but a result from a lack of knowledge of the legal implications.

Would it make sense to add a section with a short explanation why adding a software license is important, as well as a recommendation? MIT and APACHE 2.0 seem to be favored and a short explanation of their mechanics could be included.

If that is something that is wanted, I can open a PR.

Will it violate the requirement for anonymous submission if supplying a GitHub repo?

I got a comment of violation for anonymous submission, when I once supplied a GitHub repo along with my submission to another conference. Is it a problem for NeurIPS?

Making the repository a template

Hi,
Since this is a template for releasing code, I think it would be quite useful to mark this repository as a Github template (see here).

As far as I am aware, someone with admin permissions can mark it as a template, such that people can just mirror the repo structure with a single click, rather than copy-pasting or forking the repo.

Results leaderboards

Thank you for these useful tips!

I am happy to contribute the SOTA leaderboards I am maintaining for several ML+systems benchmarking and optimization competitions:

• Collective Knowledge - with 40+ leaderboards

Please keep up this great work!