Reproduction of Step width enhancement in a pulse-driven Josephson junction

[rescience]

Reproduction of Step width enhancement in a pulse-driven Josephson junction

[rougier]

I've just added the DOI and made a PR on your repo. It's alo published on the ReScienc website with DOI: 10.5281/zenodo.3922195.

You're also on the frontpage ! You can tweet about the publication and cc @ReScienceEds such that we can retweet.

Congratulations again !

[rougier]

Based on @pdebuyl last review I think your paper is now officially accepted ! Congratulations.
For the actual publicaiton, I will need to modify your metadata to adde relevant information and I would need a softxware heritage id. Could you make one (very easy procedure, just go to the software heritage page and "save" your github repository.

[pdebuyl]

@sabinomaggi the editor will take care of the DOI then make another PR with the updated metadata and pdf.

[pdebuyl]

@sabinomaggi the current paper does not state clearly the the visual basic file is "imprisoned" in its binary form and unavailable in source form.

I propose the following:

1. The paper is considered a partial replication.
2. You state that the Visual Basic code is only available in the binary format. You already mention it ("or by opening the Visual Basic project" is not possible for readers of ReScience in the way that we expect it: open-source code that runs on current computers).
3. You include the screenshots as above in the paper (or select 1 or 2 that make the most sense).

When starting the 10-year challenge, it was hard to predict the type of results and your paper is a good illustration of the surprises that occur when resurrecting old programs.

[sabinomaggi]

This submission corresponds to ReScience/ten-years Paper Number #16.

[rougier]

Thanks for your submission. @pdebuyl could you edit this submission (Physics / Fortran) for the ten year reproducibility challenge (only one reviewer needed) ?

[pdebuyl]

Hi @rougier

I can review but must ask first: 1. what is the current timeline? schedule is a mess (my schedule, of course) 2. I'd like to submit something as well, does that matter? (not the same topic of course)

[rougier]

@pdebuyl If you can review, that's all the better. I'll edit it then.

1. No real timeline but end of May would be nice
2. No problem for submitting something.

[pdebuyl]

Hi @sabinomaggi

The reproduction in the article is very interesting, thanks!

Before proceeding to the actual review, I would like to make the licensing checks. The procedure for the challenge (found here ) is a bit different from the usual ReScience one. There are still specific open-source concerns that are important.

1. Was there a license to the original code? Ideally, you should include the "old" code in the git commit history.
2. The new code must be licensed with an open license (see debian's page for instance).
3. Do you have the source for the visual basic programs? All the software used in the replication must be made available.

[sabinomaggi]

Hi @pdebuyl
many thanks to you for the appreciation.

All current code has already been uploaded to this GitHub repository, https://github.com/sabinomaggi/ten-years-challenge-pulsed-drive. Now I see that I missed the last step, the submission to the Software Heritage archive. Is this enough to link everything together? Otherwise, how can I link the paper and the repository containing the code?

As for your specific questions:

1. If I understand well I should upload the "old" fortran code even if it is very similar to the current version. It is not a big problem, can I do that by using a branch?
2. The original code had no license, it was developed only to make the calculations I needed, not to be shared. In the GitHub repository mentioned above I have now licensed everything under CC4, I hope it is enough.
3. Also the original VB sources are already available in the GitHub repository.

[pdebuyl]

Apologies, I looked for a LICENSE file or a notice in the source files and did not see the notice in the README. That being said, creative commons is not suited for software, would you consider one of GPL, LGPL, BSD, MIT (or another established license)?

Regarding the old code, the modifications are rather minimal so I won't insist on that.

[sabinomaggi]

You are absolutely right, I forgot the license stuff, sorry. Now I have added a GPL3 license and changed the README file accordingly. While I was at it, I also added to the repository a directory containing the original Fortran and VB code.
Following the last requirement of the Guidelines, I have just submitted my repository to Software Heritage, and I am waiting for them to process the request.

[pdebuyl]

Thank you for the license update. In the vbprog directory, I can only read stepampl.bas which contains variable declarations. procedur.bas is a binary file that I cannot read, where should I look for the vb code for the analysis?

[sabinomaggi]

Yes you are right, all files of a Visual Basic project were binary files, you can access them only within the Visual Basic IDE.

In particular, the *.frm files contain the layout of the graphic interface (for instance this is loadfile.frm),

while the *bas files contain the actual code, but in the integrated editor you could access only one function (better procedure in basic-speak) at a time (below, you can see the code of the FindParam function, within the file procedure.bas).

Lastly, the file stepampl.mak glued everything together .

I have just checked and it seems there is no possibility to export the project in text form, so at the end the Visual Basic code is pretty useless outside its own environment. Another proof that VB was very immature, even by '90s standards.

[pdebuyl]

Sorry for the delay. I was also wondering the status of the resulting code, as the "updated" code (so, not the 10+ years old code) is supposed to be executable by readers of the ReScience paper. The simplest scenario is one where all code can be executed from source. In case there is a way out, I am reaching out to the editors in chief.

@ReScience/editors : Part of the reproduction involve a binary visual basic program. How does this impact the review process?

[khinsen]

@pdebuyl Badly, of course! However, if I am understanding the situation correctly, it looks like there is nothing anyone can do to fix this. So the only possible action is documenting the problem in the paper. In a sense, it's not very different from lost code.

The remaining philosophical question is if we should classify this as a reproduction failure. Not sure about that. The success/failure dichotomy isn't really appropriate.

[sabinomaggi]

@pdebuyl and @khinsen I understand your perplexities related to the Visual Basic program (the reproduction of the fortran part of the project should have no problem). The reproduction of the Visual Basic program could be easily done by building a virtual machine with MS-DOS, Windows 3.11 and Visual Basic 1.0, as explained briefly in the paper. I have not included a prebuilt VM on GitHub because I expect it would breach some license agreement, even after so many years. Clearly, I could easily share my VM upon request, but I guess it would violate the rules of the challenge.

On the other hand, the role of the Visual Basic application is only to summarize the results of the Fortran numerical simulations, calculating the size of each rf-induced step as a function of the amplitude of the microwave signal. There is nothing really interesting in that code, it only parses each data file and counts the number of occurrences of the same voltage (within a predefined range) for different values of the dc current. When writing the paper I considered all this stuff quite uninteresting, but if you wish I can try to explain it with same detail in the article and maybe add the most important parts of the code within the text.

The last possibility would be to copy the source code of each function from the VB program into a text file to be added to the repository. It is an hard work and it cannot be used anyway (Visual Basic 1.0 does not allow to import/export a source file and in any case the GUI part of the program would be left out), so probably there is not real point in doing that.

[pdebuyl]

@sabinomaggi and @khinsen thank you for the replies.

@sabinomaggi I think that the optimal solution, as Konrad mentions, is to document properly this limitation in the reproduction. It may be the case already, I'll get back to you tomorrow after re-reading the paper.

[khinsen]

Sounds good. Remember that the ultimate point of the challenge is to find out what worked and what didn't, and why. So no matter how small the role of VB in this project is, the fact that it causes trouble with open reviewing is a conclusion that matters.

[sabinomaggi]

No problem for 2 and 3, I will add a a note in the paper following your suggestion and include the screenshots in the Supplementary Material.
I only don't know how to mark the paper as a partial replication (or if I need to do that anyway if I comply with points 2 and 3).

[sabinomaggi]

@pdebuyl and @khinsen I have just pushed an updated version of the paper with a possible solution to the Visual Basic problem, described in a new section, Code availability. It is not perfect but is is better than having no Visual Basic code. There are also some minor changes in section 5, Visual Basic code, that are marked in color in the text, and a new Figure in the Supplementary Material showing one of the screenshots above.

I have also updated the GitHub repo of the paper, https://github.com/sabinomaggi/ten-years-challenge-pulsed-drive, adding the Basic source code and the forms that compose the GUI of the program files, that I managed to extract (somehow) from the original project.

I am not sure if this complies in full with your suggestions, but it surely improves the reproduction (and the paper).

[rougier]

Thanks for the modification and for point 1, I think what you added in the manuscript could be enough. By the way, the type of paper is reproduction (not replication) such that the title should start with [Rp].

[sabinomaggi]

@rougier Ok, thanks. I wait for the reviewer comments to update the paper.

[pdebuyl]

@sabinomaggi sorry for the delay. It is very unusual for this type of review that I cannot execute the full reproduction. Thank you for the modifications, I like that the archive is now, in principle, complete!

@rougier I approve the paper modulo the last change that you requested about [Rp] vs [Re].

[sabinomaggi]

@pdebuyl If you wish I can share with you my DOS/Windows 3.11/Visual Basic virtual machine, maybe after converting it to the Virtual Box format (I am quite sure it is possible). Sharing privately a VM should not be too bad, what I cannot do is to make it available to everyone on a public site.

[pdebuyl]

@sabinomaggi thank you for the proposition. As it will not be ultimately part of the paper, I prefer not to go that route.

[sabinomaggi]

Many thanks to @pdebuyl and @rougier for taking time to review the paper and provide great feedback.
I have just pushed an updated version of the paper with the requested [Rp] marker. The LaTeX code has been cleaned up and I have also added the software heritage ID to the metadata (but I am not 100% sure if it is in the right format).

[rougier]

I've made a PR on your repor and added publication information (but the DOI). Can you check everything is ok?

[sabinomaggi]

Ok, checked and merged. Should I ask for a DOI on Zenodo?

[sabinomaggi]

PR merged and tweeted about the publication. Thanks!

[khinsen]

@rougier Since this paper is now published, can we close this issue?

[rougier]

Yes, done.