ecojulia/ecosistem.jl

Julia package for ecosystem simulation

ecosistem.jl's Issues

Move to EcoJulia

A few final things to do:

• Set up a project board to keep track of changes
• Sort out ownership in EcoJulia for Claire
• Move the package to EcoJulia

JLD now creates warnings, replace with JLD2

We are using JLD in the code, and it creates warnings because it is behaving improperly with FileIO - I think it's a type piracy problem. I tried to suppress the warnings by being more precise about what we were doing, but it wasn't possible. I think the long term solution is to move to the data pipeline and/or JLD2, but I'm open to other suggestions (and maybe this is fixed in dev?).

Problem with GDAL

This errors:

EcoSISTEM.jl/examples/Africa_run.jl

Line 12 in 8c6b2b8

africa = readfile(file, -25°, 50°, -35°, 40°)

julia> africa = readfile(file, -25°, 50°, -35°, 40°)
ERROR: GDALError (CE_Failure, code 10):
        Pointer 'hDS' is NULL in 'GDALGetRasterBand'.


Stacktrace:
  [1] gdaljl_errorhandler(class::GDAL.CPLErr, errno::Int32, errmsg::Cstring)
    @ GDAL ~/.julia/packages/GDAL/5CClC/src/error.jl:36
  [2] gdalgetrasterband
    @ ~/.julia/packages/GDAL/5CClC/src/gdal_h.jl:927 [inlined]
  [3] getband
    @ ~/.julia/packages/ArchGDAL/CUiBS/src/dataset.jl:606 [inlined]
  [4] (::EcoSISTEM.ClimatePref.var"#17#19"{Vector{Any}})(dataset::ArchGDAL.Dataset)
    @ EcoSISTEM.ClimatePref ~/Documents/git/julia/EcoSISTEM/src/ClimatePref/ReadData.jl:74
  [5] #8
    @ ~/Documents/git/julia/EcoSISTEM/src/ClimatePref/ReadData.jl:22 [inlined]
  [6] read(f::EcoSISTEM.ClimatePref.var"#8#10"{EcoSISTEM.ClimatePref.var"#17#19"{Vector{Any}}}, args::String; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ ArchGDAL ~/.julia/packages/ArchGDAL/CUiBS/src/context.jl:216
  [7] read
    @ ~/.julia/packages/ArchGDAL/CUiBS/src/context.jl:214 [inlined]
  [8] #7
    @ ~/Documents/git/julia/EcoSISTEM/src/ClimatePref/ReadData.jl:21 [inlined]
  [9] environment(f::EcoSISTEM.ClimatePref.var"#7#9"{EcoSISTEM.ClimatePref.var"#17#19"{Vector{Any}}, String}; globalconfig::Vector{Any}, threadconfig::Vector{Any})
    @ ArchGDAL ~/.julia/packages/ArchGDAL/CUiBS/src/context.jl:22
 [10] environment
    @ ~/.julia/packages/ArchGDAL/CUiBS/src/context.jl:16 [inlined]
 [11] read
    @ ~/Documents/git/julia/EcoSISTEM/src/ClimatePref/ReadData.jl:20 [inlined]
 [12] readfile(file::String, xmin::Quantity{Int64, NoDims, Unitful.FreeUnits{(°,), NoDims, nothing}}, xmax::Quantity{Int64, NoDims, Unitful.FreeUnits{(°,), NoDims, nothing}}, ymin::Quantity{Int64, NoDims, Unitful.FreeUnits{(°,), NoDims, nothing}}, ymax::Quantity{Int64, NoDims, Unitful.FreeUnits{(°,), NoDims, nothing}})
    @ EcoSISTEM.ClimatePref ~/Documents/git/julia/EcoSISTEM/src/ClimatePref/ReadData.jl:72
 [13] top-level scope
    @ REPL[11]:1

[feature/enhancement] abstract measurement types

Hi all,

As I start to migrate the list of items in #13 to individual issues, I've found it useful to better define the functionality
for VEL.jl to better plan how the interface between the two will work. This issue mostly relates to my plan to implement
different types of Measurements

As far as I can tell, all this would require from EcoSISTEM to integrate is shifting how storage works storage in simulate_record!(storage::AbstractArray...).

Right now this is filled via,storage[:, :, i] = eco.abundances.matrix

Would it be possible to shift the abundance matrix type in Ecosystem to something that stores an arbitrarily long list of matrices (parameterized by data type) corresponding to different measurements (e.g. abundance and a trait unrelated to the traits being selected on in the EcoSISTEM trait-relationship)?

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Bug when more than 100 MPI processes

The allocation algorithm fails when there are over 100 processes. In fact this is a failure when the number of processes exceeds either sqrt(num_species) or sqrt(num_subcommunities).

Fix MPI coverage measurement

Looking through changes in #66 has made it clear that setting off MPI jobs as separate processes in the testing means their tests do not get picked up by the coverage checking. Not sure how to proceed with this, but some means of accounting for the testing done there would be good - maybe check what, if anything MPI.jl does?

Fix parallelisation on transition code

The code in main parallelises efficiently using MPI.jl, but currently only works on one process (multi-threaded) on dev. This needs fixed.

Updating to Julia v1.6

Julia v1.6-rc1 is now out, so I wonder whether the github workflow will now work with 1.6?

Fix memory allocation on MPI gather_abundance()

Trying to fix some MPI problems in #66 has led to an issue with gather_abundance() doing a lot of allocation while trying to collect the abundance data sensibly onto one node. It seems to be non-trivial to fix, but doesn't get done very often, so this issue just notes it needs fixed when possible. This might involve one or many of:

1. Some kind of custom view into the full abundance array on the root process where the abundances are gathered to
2. Some kind of custom view into the process-specific abundance array on every process
3. Distributed calculation instead of whatever is being done on the root process

I suspect 1 may be the easiest to accomplish, then the array can be copied as is from the other processes, but 3 might be the ideal solution in many cases.

a few feature requests

hi everyone,

as discussed a bit in the zulip I'm working on building a set of tools within EcoJulia that will use Simulation.jl as a base. as i've started to familiarize myself with the code i've noticed a few things i'd like to add

1. abstract measurement type for trajectories (abundance / occupancy)
2. abstract landscape geometries, integration with SpatialGraphs.jl (as far as i can tell you can only use a grid at the moment?)
3. integration with EcologicalNetworks.jl, define a set of consumer-resource relationships from an EN.jl metaweb
4. integration with the new RasterDataSources.jl
5. abstract environmental variables: at the moment, in Biodiversity/Habitats.jl, HabitatsUpdate.jl, and AbioticEnv.jl, etc. have lots of duplicate/nearly-duplicate code,
   e.g.

function TempChange(eco::AbstractEcosystem, hab::ContinuousHab, timestep::Unitful.Time)
  val = hab.change.rate
  v = uconvert(K/unit(timestep), val)
  hab.matrix .+= v * timestep
end
function RainfallChange(eco::AbstractEcosystem, hab::ContinuousHab, timestep::Unitful.Time)
  val = hab.change.rate
  v = uconvert(mm/unit(timestep), val)
  hab.matrix .+= v * timestep
end

which could be condensed if each environmental variable was a type that holds its unit.

happy to help with implementation of any/all

Examples folder untested

Need to add at least one of the examples to the testing suite so that we can check that the examples Project loads properly. See #83

Restructuring everything!

I think we need to do a major shift around of everything now we've merged in the SCRC branch:

• Rename Simulation.jl to EcoSISTEM.jl
• Rename main to dev
• Only allow PR merges to dev if approved by one of the dev team
• Rename biodiversity to main
• Create @boydorr/ecosistem-admins group
• Only allow PRs to main when we are going to make a release (approved by > 1 of the admin team?)
• Add in .zenodo.json file to capture metadata (can be mostly copied from Phylo)
• Add an automatic trigger to Zenodo to capture tagged releases of EcoSISTEM
• Resolve warnings in tests
• Link to contribution agreement from README and/or docs
• Release 0.1 of the package based on main (ex biodiversity)
• Fix license by removing first line and quotes, so GitHub recognises the license correctly
• Update to 0.1.1 with minor fixes identified post 0.1 release
• Permanent fix to Diversity incompatibility
• Update to 0.1.2
• Merge main from 0.1.2 into dev
• Add SCRC authors to the dev .zenodo.json file

Julia nightly fails

Seems to be an issue with Ubuntu 20.04 and loading .so files related to NetCDF:

ERROR: LoadError: InitError: could not load library "/home/runner/.julia/artifacts/a990d3d23ca4ca4c1fcd1e42fc198f1272f7c49b/lib/libnetcdf.so"
libmbedtls.so.13: cannot open shared object file: No such file or directory
Stacktrace:
  [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
    @ Base.Libc.Libdl ./libdl.jl:117
  [2] dlopen(s::String, flags::UInt32)
    @ Base.Libc.Libdl ./libdl.jl:117
  [3] macro expansion
    @ ~/.julia/packages/JLLWrappers/QpMQW/src/products/library_generators.jl:54 [inlined]
  [4] __init__()
    @ NetCDF_jll ~/.julia/packages/NetCDF_jll/2PbeF/src/wrappers/x86_64-linux-gnu.jl:14
  [5] _include_from_serialized(path::String, depmods::Vector{Any})
    @ Base ./loading.jl:812
  [6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, depth::Int64)
    @ Base ./loading.jl:921
  [7] _require_search_from_serialized
    @ ./loading.jl:893 [inlined]
  [8] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1196
  [9] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1089
 [10] macro expansion
    @ ./loading.jl:1069 [inlined]
 [11] macro expansion
    @ ./lock.jl:267 [inlined]
 [12] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1033
 [13] include
    @ ./Base.jl:427 [inlined]
 [14] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
    @ Base ./loading.jl:1423
 [15] top-level scope
    @ stdin:1
during initialization of module NetCDF_jll
in expression starting at /home/runner/.julia/packages/NetCDF/CS38M/src/NetCDF.jl:1
in expression starting at stdin:1