Ten-year Reproduciblity Challenge: Fluctuation Domains

[rescience]

Ten-year Reproduciblity Challenge: Fluctuation Domains

[rougier]

It's online ! Congratulations! https://rescience.github.io/bibliography/Boettiger_2020.html

[rougier]

You're right, I'll handle the submission from now on. So congratulation, your article has been accepted!
Can you give me the link to your article latest repo?

[rougier]

@ctb Can you edit this reproduction submission for the Ten Years Reproducibility Challenge (you only need one reviewer) ?

[rougier]

@ctb Gentle reminder.

[rougier]

@tpoisot Can you edit this reproduction submission for the Ten Years Reproducibility Challenge (you only need one reviewer) ?

[rougier]

Gentle reminder: @tpoisot Can you edit this reproduction submission for the Ten Years Reproducibility Challenge (you only need one reviewer) ?

[tpoisot]

Absolutely - @ha0ye would you be available to review this?

[ha0ye]

The languages and topic do seem in my wheelhouse. I won't have time until the 2nd week of March (at the earliest), though.

For transparency, I also interact with @cboettig in a variety of places, but we haven't and aren't working on anything together. He did invite me to be on a panel in 2018. (Not sure if any of that matters for COI purposes.)

[rougier]

I think we do not consider this as a COI.

[rougier]

Gentle reminder

[tpoisot]

@ha0ye can you confirm that you will review it?

[ha0ye]

Yup. Likely not this week. I'll be sure to get this done before I leave for fieldwork at the end of the month, though.

[ha0ye]

I'm ready to write up my review, but can't seem to find instructions for the procedure to do so... does it go in this issue?

[rougier]

@ha0ye Yes, exactly (sorry for the delay)

[ha0ye]

Review of 'Reproducing "Fluctuation Domains"'

In reviewing this replication of (Boettiger, Carl, Jonathan Dushoff, and Joshua S. Weitz. 2010. “Fluctuation Domains in Adaptive Evolution.”), I follow the 4-point criteria listed on https://rescience.github.io/edit/

1. The actual replication of the research. The reviewer must evaluate the authors’ claims about a successful replication, applying the standards of the field.

The results from the original paper consist mainly of a numerical simulation (Figure 2 in the original paper). This has been successfully reproduced with the updated code (Figures 4-6 of the replication).

Figure 1 from the original paper has also been reproduced successfully (Figures 1-3 of the replication).

2. Reproducibility of the replication. The reviewers must be able to run the proposed implementation on their computer, and obtain the same results as the authors with the limits of the state of the art of the field.

No issues here. 👍

3. Clarity of the code and the instructions for running it. Uncommented or obfuscated code is as bad as no code at all.

Both the code and the replication are presented in the form of an R package compendium, with the changes for the replication within a branch of the GitHub repo for the code accompanying the original paper, and a rescience folder that contains files associated with the replication paper.

The package functions appear to perform the dual task of doing computations and then plotting the results. This could be better documented (with function names that are more descriptive) and/or refactored (so that computations and plotting are done separately). The latter adds flexibility to save intermediate results and/or subsequent analyses.

Finally, it would be useful to document the replication status and changes to the code within the top-level README. Some basic installation instructions could be added (particularly the note about installing GSL prior to the usual instructions for installing an R package).

4. Clarity and completeness of the accompanying article, in which the authors should clearly state why they think they have replicated the paper (same figures, same graphics, same behavior, etc.) and explain any obstacles they have encountered during the replication work.

The article was straightforward to follow, with detailed discussion of issues around dependency versioning and changing standards. One suggestion I have is to set the chunk options so that the current Figures 1-3 are panels within a single figure, like in the original paper. (Currently, they are separate figures, with duplicated captions.)

These are the chunk settings that I used to re-knit the document:

```{r fig1, fig.width = 5, fig.height = 3, fig.show = "hold", out.extra="", fig.cap="{...}"}

[rougier]

@tpoisot 🛎️

[cboettig]

Thanks @ha0ye, these are great suggestions.

@rougier @tpoisot Just checking in if anything is needed from me here yet?

[rougier]

@cboettig You can edit your submission following @ha0ye suggestions and then @tpoisot will (probably) accept the article. Then there will be some work for the actual publication (mostly editing your metadata file).

[cboettig]

Thanks @rougier. @tpoisot I've made the edits in cboettig/fluctuationDomains#3.

[cboettig]

@rougier @tpoisot ☝️

[rougier]

@tpoisot 🛎️

[rougier]

@tpoisot Now we are very late...

[cboettig]

@rougier Anything we can do here? COVID and all the other crises might be interfering with @tpoisot 's ability to look at this. I think the changes requested and implemented above are pretty straight forward, maybe another independent editor can assess?

[cboettig]

@rougier I've merged the above-mentioned PRs, so everything is now in https://github.com/cboettig/fluctuationDomains/tree/master/rescience, with the pdf at https://github.com/cboettig/fluctuationDomains/blob/master/rescience/article.pdf, and article source here: https://github.com/cboettig/fluctuationDomains/blob/master/rescience/article.Rmd

(I can add the combined .tex file that will be generated from the .Rmd if you want -- for us Rmd folks tex is just an intermediate object between .Rmd source and .pdf output :-) )

[rougier]

I tried compiling from the .Rmd but I had some problems. If you can add the article.tex, that would be great (I need to compile it several times to test the publication process + update metadata)

[rougier]

BTW, I got:

Error in library(fluctuationDomains) :
  there is no package called 'fluctuationDomains'

when inside the rescience directory. I'm not a R user but I imagine I've to tell R I'm using a local package somehow.

[cboettig]

Thanks @rougier !

• tex version is now here: https://github.com/cboettig/fluctuationDomains/blob/master/rescience/article.tex
• I added an install target to the Makefile, which will install fluctuationDomains locally. (you'll just need a copy of libgsl-dev on your machine). Typically we R folks install directly from GitHub by running remotes::install_github("cboettig/fluctuationDomains") in R, which has the added benefit of automatically installing any R package dependencies, but in this case there aren't any :-).

[rougier]

Perfect, it worked. Could you save your github repository at https://www.softwareheritage.org/save-and-reference-research-software/ and give me back the swh id you'll get. Also, just before the conclusion, you have a mising DOI that I think should be a reference to the github repository. It would be better to use the swh id that will in turn points to the github repo. If you update your article.tex, I can merged it in my local fork.

[rougier]

@cboettig ping

[cboettig]

Thanks, queued it on software heritage and forgot to check back next day (queue seems to take a while! though I noticed anyone can queue any public repo, you don't have to be the owner). It's archived now: https://archive.softwareheritage.org/browse/origin/directory/?origin_url=https://github.com/cboettig/fluctuationDomains&timestamp=2020-09-19T00:48:22.704238%2B00:00

Um, which software heritage ID do you recommend I put into the article.tex in place of the DOI? This one? https://archive.softwareheritage.org/browse/revision/e547c62fd9ae8361d601302200bbc90b26299007/?origin_url=https://github.com/cboettig/fluctuationDomains&timestamp=2020-09-19T00:48:22Z or is there something more compact?

[rougier]

Didn't you get a SWH... id at some point? It think the link you get is for a specifc revision(which make sense) but I think there should be an initial SWD id. See for example http://rescience.github.io/bibliography/DiCosmo_2020.html by @rdicosmo

[cboettig]

Thanks @rougier, but even with the DiCosmo example I'm not sure what you are referring to as "the Software Heritage Identifier". I see a DOI there that belongs to the Zenodo record, and I'd be happy to provide that. The link I see that is specific to software heritage is https://archive.softwareheritage.org/swh:1:rev:2db189928c94d62a3b4757b3eec68f0a4d4113f0;origin=https://gitorious.org/ocamlp3l/ocamlp3l_cvs.git/.

I'm a huge fan of software heritage. One thing I think is absolutely brilliant about their approach is that they use content-based identifiers, instead of issuing arbitrary unique strings that are specific to software heritage. You'll not that the sha1 hashes showing up in my link and in the DiCosmo link are literally the hashes of the most recent commit. Software heritage "understands" and can "resolve" these IDs, so my most recent git commit appended with the right prefix swh:1:rev:e547c62fd9ae8361d601302200bbc90b26299007 could be considered my ID, but note that this string doesn't "come from" software heritage, it existed before the Software Heritage archive was created, and before I even pushed that commit to git. Now that the repo is in the SWH archive, SWH can of course resolve that commit hash, or any other commit hash from the repo.

SWH can also resolve individual objects by their SHA-256 sum; e.g. you can literally sha256sum article.pdf in my repo, and then query SWH by that to get just those bits-and-bytes instead of the whole repo. Note this is completely independent of git's hashes.

All this is great and I wish other archives like Zenodo would also allow these content-based queries, but I'm not quite sure what to put in the paper. Readers probably don't know how to resolve a commit hash from software heritage, unless we embed the full link to their API (in which case they probably won't know what the URL signifies, it just looks like a long URL). I think readers expect a DOI and understand what that means; it seems even Prof DiCosmo has provided a Zenodo DOI in addition to the SWH URL. Advice on this? I think you're blazing a trail here.

[rougier]

I think we can use the swh:1:rev:e547c62fd9ae8361d601302200bbc90b26299007 using the address https://archive.softwareheritage.org/swh:1:rev:e547c62fd9ae8361d601302200bbc90b26299007 and the ReScience template should make this cickable. Let me include it in your article and you tell me if it look ok (I'll add the Zenodo DOI as well).

[rougier]

Forgot to ask the Zenodo DOI for the code... Can you request one and give me the DOI?

[rougier]

It's up at Zenodo sandbox server: https://sandbox.zenodo.org/record/676290 (it will be deleted by tomorrow)

[rougier]

@cboettig can you check the PDF for any error? I modified a bit your article.tex to correct layout & fonts.

[cboettig]

@rougier apologies missed my github notifications here. this all sounds good, let me know if you want me to check it over again.

[cboettig]

Also I did request a DOI from zenodo for the rescience-accepted tagged version: https://doi.org/10.5281/zenodo.4067657 if you want to use that.

[cboettig]

Looks like the sandbox version is still there, and it looks good! Thanks and full steam ahead 🚂

[cboettig]

@rougier anything else you need from me?

[rougier]

Nope, I'll publish it right now. Sorry for the looooooooong delay.