Reproduction in Computational Mechanics

[rescience]

Reproduction in Computational Mechanics

[rougier]

It's been published ! Congratulations !

See https://rescience.github.io/read/ and https://zenodo.org/record/3901241.

[rougier]

@labarba Ping !

[pantale]

@rougier You talk about saving the software onto software heritage. This has already been done, here:
https://archive.softwareheritage.org/browse/directory/e45b8b7efb002231f4081c429c4ca33664d0812d/?origin_url=https://github.com/pantale/ReDynELA
I thing the swid is this:
swh:1:dir:e45b8b7efb002231f4081c429c4ca33664d0812d;origin=https://github.com/pantale/ReDynELA;visit=swh:1:snp:4e9f3ab0d0b2a1e806c310a0aba1ff292dd11338;anchor=swh:1:rev:273db43f7a32f956847ce88c2d0e0b7f432a2bb5;path=/
Olivier

[rougier]

@labarba Could you edit this submission ? You only need one reviewer and I think @khinsen is ready to review. I can help you with the publication procedure at the end.

[rougier]

@pantale Thanks for your submission. We'll assign an editor soon.

[rougier]

@labarba Gentle reminder

[labarba]

Hi @rougier — I can be the editor.

@khinsen — You volunteered to review this, I heard?

[khinsen]

@labarba Right. A nice occasion to reactivate dormant FEM experience.

[rougier]

@labarba Thanks.

[khinsen]

@labarba @pantale Here comes my review!

The topic of this reproducton attempt is not just the result of a computation performed 16 years ago, but also its performance characteristics. This makes it a particularly interesting contribution to the reproducibility challenge because performance, being a property of a physical system, lies at the borderline between reproduction and replication. The fact that the author did not re-run unmodified code from 2005, but a combination of later versions and code written specifically for this challenge adds to the borderline status of this work.

Given the ongoing confusion between reproducibility and replicability issues in many domains of science, I would like to encourage the author to make the specific nature of his contribution stand out more clearly.

The paper starts with a description of the historical evolution of the DynELA code, which is interesting because it highlights the impact of technological evolution on research. However, with the discussion of a "classical", "parallal", and "modern" version plus variants, it is not really clear how the code ultimately used in the reproduction differs from the code used in the original paper.

In Figure 2, the left-hand side represents the state before the impact, so I would expect it to be uniformly blue, rather than uniformly red which stands for maximal plastic strain. While the agreement can certainly be qualified as good, the numerical values in Figure 2 are slightly larger than in Figure 9 of the original paper. Can this tendency be explained?

Figure 3 is impossible to understand without a minimal description of the four parallelization strategies being compared. It would be helpful to include the method descriptions from page 368 of the original paper, if necessary with some rewording for avoiding copyright conflicts.

Finally, two minor remarks on the text:

1. Page 1: "the preferred current standard is C++"
   Preferred by who? The FEM community? Certainly not scientific computing in general.

2. Page 2: "classical mathematics does not propose the notion of 4th-order tensor"
   I don't know what "classical" mathematics is, but mathematics certainly does have this notion. The standard term is "tensor of rank 4".

[pantale]

@khinsen @labarba Hi,
I'm currently working on a revised version of the paper, hope to be able to send it by beginning of next week.

Kindly Regards

Olivier Pantalé

[pantale]

@khinsen @labarba
Please find here after, the revised version of the paper proposed for the 10 years reproducibility challenge.

Answers to remarks:
The topic of this reproducton attempt is not just the result of a computation performed 16 years ago, but also its performance characteristics. This makes it a particularly interesting contribution to the reproducibility challenge because performance, being a property of a physical system, lies at the borderline between reproduction and replication. The fact that the author did not re-run unmodified code from 2005, but a combination of later versions and code written specifically for this challenge adds to the borderline status of this work.

Given the ongoing confusion between reproducibility and replicability issues in many domains of science, I would like to encourage the author to make the specific nature of his contribution stand out more clearly.

The paper starts with a description of the historical evolution of the DynELA code, which is interesting because it highlights the impact of technological evolution on research. However, with the discussion of a "classical", "parallal", and "modern" version plus variants, it is not really clear how the code ultimately used in the reproduction differs from the code used in the original paper.
ANSWER: I have modified the document to clarify this.

In Figure 2, the left-hand side represents the state before the impact, so I would expect it to be uniformly blue, rather than uniformly red which stands for maximal plastic strain. While the agreement can certainly be qualified as good, the numerical values in Figure 2 are slightly larger than in Figure 9 of the original paper. Can this tendency be explained?

ANSWER : Agree, the left side is now blue colored. Effectively, values are a little bit different, but after having looked closely at the figure I found a big difference. In fact, the projectile in the new simulation was declared as elastoplastic and not purely elastic, as prposed in the original paper (I have a lot and a lot of different models with various parameters). Then I reran the simulation and the gap reduced as you can see in the updated figure. The value is not exactly the same, but did I rounded the value of the mamimum plastic strain to 0.26 15 years ago, I don't remember anymore.

Figure 3 is impossible to understand without a minimal description of the four parallelization strategies being compared. It would be helpful to include the method descriptions from page 368 of the original paper, if necessary with some rewording for avoiding copyright conflicts.

ANSWER: Yes, agree, I'm going to put some extra information coming from the original publication

Finally, two minor remarks on the text:

Page 1: "the preferred current standard is C++"
Preferred by who? The FEM community? Certainly not scientific computing in general.

ANSWER : Added an extra explanation to this sentence.

Page 2: "classical mathematics does not propose the notion of 4th-order tensor"
I don't know what "classical" mathematics is, but mathematics certainly does have this notion. The standard term is "tensor of rank 4".

ANSWER: In fact I'm talking about classical mathematic libraries where standard implementation concerns vectors and matrices.

article.pdf

All sources files of this publication are available here:
https://github.com/pantale/ReScience-paper2020

[khinsen]

@pantale @labarba The revised article looks good! I particularly appreciate the added explanations about the parallelization methods, which I find clearer than the explanations in the original article.

I had a look at the code and briefly considered a test installation. However, there are no instructions at all. The file CMakeLists.txt suggests that I need to use CMake, but since I haven't touched it for at least 15 years, I wouldn't know where to start. @pantale, could you provide some instructions? Or is this a hopeless task for anyone but you?

[pantale]

@khinsen
Hi,
I updated the README file of the ReDynELA repository to answer your questions. You can have a try on compiling the code. You can contact me directly at [email protected] for more instructions.

Olivier

[khinsen]

@pantale Thanks for the instructions! Unfortunately they don't work for me, see pantale/ReDynELA#1.

[khinsen]

@pantale I forgot a legal detail: could you specify a license for the code? Somewhere in https://github.com/pantale/ReDynELA, ideally by adding a file called LICENSE. My understanding (noting that I am most definitely not a lawyer) is that without a license, nobody may use the code for anything. I'd then be almost in jail already for having tried to compile it ;-)

[pantale]

Yes, I can, but can you help me for this since I never done such thing, and I don't know what license to choose.

[khinsen]

Assuming you have no constraints imposed by your employer, I'd got for one of (1) MIT (2) GPL. The main difference is that an MIT license allows others to improve and re-distribute your code in various ways, including commercial, whereas the GPL requires all derived versions to be GPL-licensed as well.

There's a site providing guidance: https://choosealicense.com/

[pantale]

Thanks a lot.

I have added the MIT license file to the repository.

Olivier

[khinsen]

@labarba I am happy with the current state of the article and the code!

[pantale]

Hi,
@khinsen @labarba What do I have to do now for continuing this process ?

Olivier

[khinsen]

@pantale Putting on my EIC's hat: right now, nothing. It's @labarba as the handling editor who has to decide if the paper is accepted or needs additional review or additional revision.

[pantale]

@khinsen @labarba
Hi,
Just to mention that the associated code has been uploaded to Zenodo:
DOI: 10.5281/zenodo.3739278

Olivier

[rougier]

In fact, we're now using software heritage for saving code.

[pantale]

@rougier Hi,

Do I have to do this myself, or is it in the process of the publication ?
And concerning the proposed paper, what about the future ?

Kindly Regards,

Olivier Pantalé

[rougier]

@labarba Are oyu satisfied with the modification, can we accept the paper ?

@pantale Sorry for the delay. Yes, you have to save the code yourself (very easy to do) and software heritage will give you back an ID. This will need to be included in your metadata but the current template is not up to date (I need to fix it). Then @labarba (or me) will make a PR to your repor to modify your metadata.

[pantale]

@rougier Hi,
The process of saving the code to software heritage is currently done...
I have different ID, depending on directory or revision, which one I have to use, I guess directory, so the following one:
https://archive.softwareheritage.org/swh:1:dir:e45b8b7efb002231f4081c429c4ca33664d0812d;origin=https://github.com/pantale/ReDynELA/

Olivier

[rougier]

Ok, thanks. Now waiting for @labarba aknowledgement.

[pantale]

Ok, thanks. Now waiting for @labarba aknowledgement.

Ok, I'm also waiting for instructions...
Olivier

[labarba]

Where can I share with the author an annotated PDF of their article?

[labarba]

article-3_annotatedLB.pdf

[labarba]

Huh. I guess drag-and-drop just works.

[labarba]

Summary comments coming soon.

[labarba]

Editorial review for ReScience, May 31, 2020

Original paper:
Pantalé, O. (2005). Parallelization of an object-oriented FEM dynamics code: influence of the strategies on the Speedup. Advances in Engineering Software, 36(6), 361-373. https://oatao.univ-toulouse.fr/5378/1/Pantale_5378.pdf

The original paper presents the shared-memory parallelization of the DynaELA code using OpenMP. DynaELA is an explicit finite-element-method solver for elasto-plastic deformation of materials under high-speed impact. The most computationally intensive part of the finite-element solver is the computation of the internal forces on each element of the mesh. Four multi-threading techniques for this part of the solver are discussed in the original paper, each more intricate than the other. They were tested on a benchmark (high-speed impact of a copper rod), reporting speedups and parallel efficiency on up to 8 CPU cores. The tests ran on an 8-way Intel-based Compaq SMP server from 1999 (Intel Pentium III Xeon 700MHz).

The simplest strategy is to add a parallel-for directive on the main loop. On 4 cores, speedup was 2.88 and efficiency 72%, but on 8 cores speedup decreased and efficiency was 29%. The next two parallelization techniques performed similarly, with speedups of 6.45 and 7.88 on 8 cores (efficiencies of 80.7 and 98.5%). The fourth strategy included a load balancing operator (compute time of internal forces varies from element to element), and led to super-linear speedup, presumably due to efficient cache use.

The original paper also presents a typical application run for a dynamic-traction simulation of a cylindrical projectile impacting in a tubular target. The results from DynELA were compared to those using the commercial package Abaqus, and were in agreement. For this case, speedup results with the full DynELA are also shown (5.61x on 8 processors).

The “Reproducibility Challenge” work used an Intel-based Dell server with 24 cores (Intel Xeon E5-2650, 2.20GHz). Naturally, any speedup or parallel-efficiency observations on machines so many generations apart cannot be comparable. But looking at the general findings from a replicability standpoint, can we say that the findings are consistent? The new results confirm that the first strategy for multi-threading quickly reaches a limit of speedup, and the fourth strategy gives the best performance—these findings replicate the original work.

The results of the application demo (dynamic-traction simulation) are shown as plastic strain contours. I left a comment in the PDF about this, too, as follows. The assessment of the results is inadequate. The statement is comparing the maximum value shown in the contourplot legend: 2.6e-2 in the 2005 paper, and 2.58e-2 here. First of all, I don't see a mention of how many processors were used in each case. For two parallel runs, you cannot expect bit-for-bit matching of numerical resutls, especially those arrived at after gather operations. These results are matching! You can look at other diagnostics: is the element with maximum strain at the same location in both cases? What about the total length after deformation?

Congratulations to the author on the success at finding the code (which had not been archived properly), painstakingly merging with a “beta” version with OpenMP, and getting it to compile and run. This is a feat to be commended.

But I found the paper a slog to read, with some incorrect English usage and too verbose. In the end, I could not help myself and fully annotated the PDF with copy editing.

The source code
The GitHub repository has a LICENSE file with MIT License, but the source files have text for GNU. See for example: https://github.com/pantale/ReDynELA/blob/master/Sources/Applications/DynELA_solve/main.C

[labarba]

Note my comment in the PDF about the use of the word "validate." This is a technical word in this field that means comparing the results from a simulation with a physical experiment. The word should not be used with any different meaning.

[labarba]

Please note the annotations on the PDF were done in Acrobat and I'm not sure if any PDF viewer shows them.

[labarba]

An additional comment about the categorization of this paper as a Replication, or Reproduction.

It's tricky because in this case, it is both! The comparison of multi-threading strategies with OpenMP can be called a replication: we have different computing systems (several generations away from each other), and look at whether the experiments with the copper-rod impact test give the same findings. And they do: method 1 saturates and does not scale, and method 4 scales the best with increasing number of processors. (Note: use 'processors' and not 'CPU' in this context.)

The application demo (dynamic-traction simulation) is a reproduction: same code, same computational problem set-up, rerun the simulation and observe the same quantities. The numerical results match, and the reproduction is achieved.

I also want to comment that even if the reproduction was achieved, it is slightly misleading to call the original results 'reproducible.' They were not reproducible to anyone other than the original author, because code and data were not public. And even that was a challenge since the code was lost (but found) in old hard drives! The work is reproducible now, with the code made available with this work.

I think it is worthwhile to clarify this in the paper.

[khinsen]

@labarba Many submissions to the ten-year-challenge end up being at the borderline between reproduction and replicability. That's one of the main conclusions of the challenge, but not one that we had foreseen when writing the author guidelines. So if all contributions to the challenge end up with the "reproduction" label, that's our fault, not the authors'.

However, as you suggest, it's a good idea for authors to elaborate on this in their submissions.

[pantale]

@labarba @khinsen
Thanks a lot for the feedback.
I'm working on the corrections to the article.
Olivier

[labarba]

Many submissions to the ten-year-challenge end up being at the borderline between reproduction and replicability.

Saying "borderline" is confusing. This submission contains BOTH replication (applies to the analysis of multi-threading strategies), and reproduction (applies to the benchmark on dynamic-traction simulation).

I want to add to my points about the dynamic-traction test. The author should remove speculations about whether he "rounded off" 2.58 into 2.6. The results are shown as a contour plot, and the maximum number shown in the legend was output by the plotting software (and unlikely to have been manipulated by hand). From my reading of the paper, I understand that a new or modified post-processing software was used. Thus, it's impossible to know where the difference comes from. One plotting routine may be choosing the maximum value of strain at a node of the element, and the other could be averaging the nodes, for example. Or (as I mentioned), it could simply be accumulation of floating-point errors in the gather operation on these two different multi-threaded runs. In sum, both results (2.58 and 2.6) should be considered equally credible approximations to the true value, and thus consistent.

[pantale]

@labarba @khinsen
All correction are done confirming to the annoted PDF. Only the clarification about reproduction/replication has to be done.
What about the license ? Do you want me to remove allusions to gnu license in source files ?

[labarba]

Re. license --- you have to pick one. Either remove all the text on GNU license from source files and leave the BSD-3 LICENSE file or replace the LICENSE file to match GNU. The difference, as Konrad explained, is BSD is more permissive and GNU is copyleft.

See if this helps:
https://figshare.com/articles/A_short_lecture_on_Open_Licensing/4516892/1

[pantale]

@labarba @khinsen
Hi,
Here is a revised version of the paper.
article.pdf

As far as the changes are concerned, here is a short summary:

• Regarding the difference 2.58, 2.60, I updated the text and added the fact that we're comparing a 32-bit processor and a 64-bit processor on an explicit integration scheme, known to be sensitive to round-error calculations.
• I have added a small paragraph in the concluding section about the fact that I have both replication and replication, depending on the results or performance of parallel computing. I hope this will clarify the position of this paper in the challenge.
• All changes to the text have been taken into account and I thank you for your helpful comments.

I have removed all the text on the GNU license from the source files and made a new commit of the source files to github. The software heritage has also been updated.

Olivier

[labarba]

I'm satisfied with the changes. A few typos remain, including:

p.4, line 6: softwares >> software (an uncountable noun)
p.5, line 8: requested >> required
p.5, line 10: have >> has (subject: "one")
p.5, last paragraph: reproduce>>replicate
p.10, last line: repoducible>>reproducible

[pantale]

@labarba
Hi,
Thank you very much for these last corrections which I have taken into account in the revised version. I also have checked and corrected some other typos in the text.
PDF: article.pdf
GitHub paper source: https://github.com/pantale/ReScience-paper2020

Olivier

[rougier]

@larbarba If satisiied with the corrections ping me if you need help with the publication process

[labarba]

Thanks, @rougier --- after the last small edits, the paper is ready to publish. I'm not up to speed with the mechanics of the acceptance workflow in ReScience. Appreciate if you can take it from here.

[rougier]

Sure. The semi-automated procedure is explained here: https://github.com/ReScience/articles

You'll need to clone this repository and have the article.pdf and 'metadata.yaml at the root directory. The metadata need first to be filled by you or @pantale and the pdf regenerated. Then you'll need to reserve a DOI for the article such that it can be regenerated with proper DOI. Best is to try first on the sandbox Zenodo server because things can go messy.

Last option (ans easier for you) is I publish the paper :)

[rougier]

@labarba Thanks for your work (you can unsubscribe from this thread if you don't want to be flooded by comments).
@pantale I'll take over for the actual publication part which is a bit cumbersome. Can you point me to your repository for the source of the paper such that I can make a PR?

[pantale]

@rougier
Sure, I think that everything you want is there:
https://github.com/pantale/ReScience-paper2020
this repository has been updated to the very last version of the paper

[rougier]

Great. Can you save your repository on software heritage and give me the swid ?

[rougier]

Just sent you a PR. Make sure everything is correct then I will pre-register a DOI and regenerate the paper.

[pantale]

@rougier
I have merged your modifications. Seems that everything is OK.

[rougier]

I made another PR with the final DOI.

[pantale]

@rougier
Commit merged.
Thanks a lot for your work on the final version.

Olivier