Submission: openjournals/joss-reviews#3065
Status: Conditional accept
I was very interested to see this new agent-based modelling package and am glad to have had the opportunity to review it. The quality of the work here seems great and, as my comments show below, I was easily able to get this installed and work with the example models. As it is a package for agent-based simulation there are lots of moving parts and I do not have time to go into detail checking every class and function. However, I am quite happy from my use with the example models that everything is working as claimed. Nice work @JoelForamitti
Below I list comment based on the JOSS review checklist. At this point I have not been able to check off all items - see bold comments below for those still outstanding. I think it should be relatively easy to address my comments (most of which are about documentation) and then I can change my status to accept.
Cheers, JM
General checks
All good
Functionality
Installation
Installation via pip install agentpy
into new conda env worked fine. I have not tried developer installation.
Functionality
Functionality as tested by running all examples models in the model library shows everything works as expected. I have not tried coding up my own model.
Performance
There are no claims to performance that I can see (so this is checked off the JOSS review checklist)
Documentation
Statement of Need
The 'statement of need' checkbox on the JOSS review checklist seems a little out of place (it refers to 'the paper' and seems to duplicate (to all intents and purposes) the checkbox for 'statement of need' in the Software Paper section. So I have checked this off (but see comments below about Comparison)
Installation instructions
Installation instructions are clear.
Example usage
Example usage is very good, via the excellent Models Library. The example models demonstrate the range of current functionality (as shown in the Overview of the docs).
Functionality documentation
This is not yet checked off in the JOSS review checklist. Please see issues below
The API reference seems complete and is useful. One minor comment is that in the contents of the API reference, I think it might be useful to add (Grid) to the title entry for Discrete spaces (to make the entry Discrete spaces (Grid) and even clearer the link to how this is described in the Overview - see comment about environments below).
Throughout the docs, the word separate is frequently misspelled as seperate (also in the Flocking example model) - it's a minor issue but as you edit documentation in future you might fix this.
Most of my comments are for clarification in the Overview:
The Using Environments section is a little confusing as it states, "There are three different types of environments:" then presents four items in the list following. I think this section could clarify better the relationship of Environment to the other types. For example, you might also highlight here that the Wealth Transfer example model does not use any of Network
, Grid
, or Space
(it just assumes an env with no topology). Also, you might state explicitly here that multiple types of environment can operate simultaneously (I infer this is true from the documentation on Agent.neighbours which states "If an agent has the same neighbors in multiple environments, ..."
The Recording data section: it might be useful in this section to emphasise (even more clearly) that this is about recording the model state for output to file and later analysis. I initially mistakenly thought it referred to what information might be accessed at runtime (e.g. I wonder why agents would not be able to access aggregate level information about the state of the system dynamically, but the model examples show this is clearly possible). Once I understood the intention of agentpy is to be used for experiments with multiple runs the importance of this section became clear (but see my comments below on making that intention more prominent). One way to make this clearer might be to refer to the Output and analysis section and make clear how record()
and Model.measure()
are related to DataDict
.
The comparison with Mesa is very useful, both in the paper and in the documentation. I think you should emphasise this more, to make clear why people might want to use agentpy:
- in the paper, highlighting that agentpy has been designed for scientific use with experiments from multiple runs would be useful in the Introduction, rather than leaving it to the final paragraph of the paper
- in the documentation, I suggest you link to the Comparison page from the User Guide page (and emphasise what sorts of modelling purposes you have in mind for agentpy on the User Guide page also)
I wonder if in future you could also provide a comparison to NetLogo. NetLogo is a very useful starting point for people learning agent-based simulation. To attract people from NetLogo to use agentpy a comparison would be useful I think. Building on this, some brief thoughts from someone (myself) coming at this with a background in using NetLogo:
- make clear your Grid and Space environments are not toroidal (which is the default case in NetLogo)
- highlight that experiments are akin to NetLogo's behaviour space
- comment on how patches and patchsets (NetLogo's grid) would be implemented in agentpy
Automated tests
This is not yet checked off in the JOSS review checklist. Please comment
I cannot see any reference currently to automated tests in the documentation. I can see that such tests might be possible given the stochastic nature of ABM. You might recommend users run the Example Models to check their output is generally the same as that in the examples shown on the docs webpages. Please comment
Community guidelines
This is good and I have checked off. However, one addition to the contributions page might be to welcome contributions of (simple) models to the Model Library to expand the examples available?
Software Paper
Summary
This is not yet checked off in the JOSS review checklist.
A summary is provided by the Overview section of the paper and provides a clear description. However, I think you should make clearer in the Introduction that agentpy has been designed for scientific use with experiments from multiple runs. Currently this only becomes clear in the final paragraph of the paper.
Statement of Need
This is not yet checked off in the JOSS review checklist.
I have commented on this in the JOSS review thread:
Although you do include related information elsewhere in the paper, there does not seem to be "a section titled 'Statement of Need' that clearly states what problems the software is designed to solve and who the target audience is". @whedon also discovered this (on 26 Feb). I think the section is required for the journal.
@JoelForamitti has responded and currently we are currently waiting for response to check if this is explicitly needed.
State of the Field
This is not yet checked off in the JOSS review checklist.
The comparison to other packages is present and useful. However, while the Abar et al. (2017) paper is good, it is not open access. Could you either:
- add an open-access equivalent review paper, or
- add links to webpages for the frameworks mentioned (NetLogo, Repast, MASON)
Quality of Writing
This is good.
References
This seems fine