pecanproject / pecan Goto Github PK
View Code? Open in Web Editor NEWThe Predictive Ecosystem Analyzer (PEcAn) is an integrated ecological bioinformatics toolbox.
Home Page: www.pecanproject.org
License: Other
The Predictive Ecosystem Analyzer (PEcAn) is an integrated ecological bioinformatics toolbox.
Home Page: www.pecanproject.org
License: Other
when dev_mode is on, it uses a library (~/R-dev
by default) that is only in the .libPaths()
when dev_mode()
is on, (and not in R_LIBS_USER paths). The build.sh
should not write to this development library by default.
To get around having to recompile all of the packages, I have a script that just loads all of the functions and data into a dev environment:
#!/usr/bin/env Rscript
library(devtools)
dev_mode(on = TRUE)
lapply(list("utils", "db", "settings", ...package list truncated, ... "all"),
function(x) install(x, quick = TRUE, local = TRUE, quiet = TRUE))
It would be quicker if it only loaded updated files / packages, but I have already written the above and can submit, but I wanted to write this up as a place for feedback.
An alternative would be to update build.sh
to install to dev library packages and a -q
to specify --no-install-vignettes
and other speed enhancing features.
What I have been doing is putting my test pecan.xml in the working directory, adding a debugonce
function after require(PEcAn.all)
, e.g.
require(PEcAn.all)
debugonce(write.config.BIOCRO)
and then launching R and using source
to run the file
source('workflow.R')
workflow.R <pecan.xml>
The function plot.trait
and its dependencies were written using the syntax of ggplot2 v 0.90.
Mostly this will require changing use of the opts()
function to theme()
and renaming theme_xx()
functions to element_xx()
. There is a comprehensive guide here: https://github.com/wch/ggplot2/wiki/New-theme-system
This should be very similar to the changes made when I updated the variance decomposition and sensitivity plots (0dde7f0)
To get a list of the warnings thrown by ggplot2 deprecated functions, try the following:
library(PEcAn.priors)
plot.trait("Vcmax", prior = list('norm',1,1), posterior.sample = rnorm(1000, 1.1, 0.5))
help(package = "PEcAn.priors")
?plot.trait
If ED fails to execute the PEcAn code thinks it finished correctly and does not trigger an error, resulting the web interface sitting in an endless loop.
Is there a way to include a parameter in the PEcAn sensitivity analysis, but to 'turn off' the meta-analysis. In other words is there a way to use the parameter prior from the database directly and not automatically query trait data if some exists.... other than removing the trait data itself? Can this be parameter specific (i.e. still perform meta-analysis for other parameters?)
Having a space in the path name breaks the job.sh
set start.date/end.date for sensitivy.analysis default to start/end date run if no start/end date is specified in check.settings.
Eventually will need a function that converts old ALMA files to netCDF CF format. Some legacy ALMA files have already been converted to ED specific format.
This contains tests dependent on model specific code, which should be in the model specific package as well as code that depends on the DB package for read.settings.
Util package should not be dependent on any other PEcAn package.
biocro does not have restrictions on input data and thus should work on all sites.
read.settings writes the corrected settings to a file in current folder as pecan.xml. Often this will overwrite the pecan.xml file that was originally loaded and will result in a complex pecan.xml file.
Should show Long.Name instead of abbreviation
Should enable a way to put pecan-web behind a username/password. We can use the same username/password as stored in bety database. Most of the code already exists in the db app.
Using a server that doesn't support rsa keys, I'd like to be able to connect without entering my password each time PEcAn exchanges information with a server (rsync, ssh, etc). What settings do I need?
When writing the ED2IN file it should use the MET start/end date for NL%METCYC1 and NL%METCYCF. The templates are wrong and have @START_YEAR@ which should be @MET_START_YEAR@ and should be defined in write.config.ED.
first review related code, e.g.
utils/R/distn.stats.R
utils/R/plots.R
utils/R/utils.R
and the modules/priors package.
If outdir is not specified use PEcAn_ as with web.
If I run read.settings
with the current biocro settings file (inst/extdata/settings.biocro.xml
), I get this error:
Error in mysqlExecStatement(conn, statement, ...) :
RS-DBI driver: (could not run statement: Incorrect integer value: '' for column 'model_id' at row 1)
Called from: mysqlExecStatement(conn, statement, ...)
I propose to fix it by setting the default model id = -1 (not all uses of a PEcAn settings file require a model). An alternative would be to use other model tags (like the name or path) to fix this.
Thoughts?
rename file to be biocro.revsion same as for ED. If no template specified in pecan.xml it should try to load the biocro.revision file.
Does PECAN always require 'T' files in order to perform the sensitivity analysis and variance decomposition plots. If I could output the yearly mean flux of NEE and NPP with the 'Y' file can PECAN use that instead?
The way PECAN is set up currently, is it even necessary to output any other files besides the 'T' files in order to perform the sensitivity analysis? It seems it doesn't use the monthly or yearly files, but I haven't looked at this very closely.
I can run the command that this generates at the linux prompt and it runs fine, but it fails when run within R. In this context I'm connecting to geo.bu.edu and I already have an ssh connection open between the VM and the cluster.
All subsequent ssh and rsync commands fail as well.
We were able to get all of this to (almost) work in the previous VM version (all files were copied back and forth but the submission to the queue was failing), but now I can't even get the files to copy.
> system2("ssh", c(settings$run$host$name, "mkdir",
+ "-p", file.path(settings$run$host$rundir, run)),
+ stdout = TRUE)
ssh_askpass: exec(rpostback-askpass): No such file or directory
Permission denied, please try again.
ssh_askpass: exec(rpostback-askpass): No such file or directory
Permission denied, please try again.
ssh_askpass: exec(rpostback-askpass): No such file or directory
Permission denied (password,hostbased).
Warning: running command ''ssh' [email protected] mkdir -p /usr2/faculty/dietze/ED.r82/ED/run/56' had status 255
It would be great if in addition to a standard time-series plot of output we could have PEcAn generate a flux "fingerprint" plot (e.g. figure 4 in https://www.ideals.illinois.edu/bitstream/handle/2142/34295/Hickman_George.pdf ) and/or a "weather radial" plot (http://www.weather-radials.com)
When I try to follow the instructions on the wiki for upgrading bety, the sudo bundle install
line results in the following error:
Installing nokogiri (1.6.0)
Gem::InstallError: nokogiri requires Ruby version >= 1.9.2.
An error occurred while installing nokogiri (1.6.0), and Bundler cannot continue.
Make sure that `gem install nokogiri -v '1.6.0'` succeeds before bundling.
I have tried upgrading ruby, installing rvm, running sudo bundle update
, but none of these fix the issue.
Currently we don't check the result of a model run. If the model fails we should have our script throw an error (exit code != 0). We can either leverage the error code from the model, or check if the output file was generated.
Description: Cannot open connection to database and write Rdata files for get.trait.data() in R modules demo (step 4.6) (writing from Path: home/carya/pecan/output/PEcAn_1/pft/). Below is one of the twenty warning messages:
In file.remove(old.files[which(file.info(list.files(path = settings$pfts[i]$pft$outdir, ... :
cannot remove file '/home/carya/output//PEcAn_1/pft/1//SLA.model.bug', reason 'Permission denied'
Reproduction Steps: Follow R modules demo hand out up to step 4.6. Running get.trait.data() will fail and return multiple error / warning messages.
Steps to Fix: Changed R/W permissions via shell
db.exists checks to see if the database exists. Right now it opens a connection and returns. We should add a simple select statement to test for read access and a simple write to test write permissions (we could rewrite a single field that was fetched for the write test).
Right now utils.R defines some variables for mstmip that are not in standard. These should be added to a local addition to the mstmip file.
the source found at http://www.hdfgroup.org/ftp/HDF5/current/src/ has been updated to hdf5-1.8.11.tag.gz. I had to make this change when installing on my make to get HDF5 libs installed (on my Mac OSX machine). All other instructions for HDF5 still worked fine.
In an effort to try to speed up my sensitivity analysis runs, I have realized I only need a handful of state/flux variables in my T file in order to get the results I need from PECAN. Right now there are roughly 40+ state variables loaded to the T files where I really only need ~5.
I think I can do the following: 1) go into the ED2 source code and turn off those state variables written to the T file I don't need, then 2) edit the model2netcdf.ED2.R script within PECAN to eliminate the conversion of variables I don't need. Does that sound reasonable, or will this cause big problems? Any advice on what files to edit would be appreciated.
Many places in the code assume settings as a global variable, this should be removed redmine-1636
Right now the templates are in both models/ed/inst and web/templates. This is confusing and the files should be merged
I get the following error when I perform the 'start.model.run' step in PECAN
bash: qsub: command not found
Error in system2("ssh", c(settings$run$host$name, qsub, file.path(settings$run$host$rundir, :
error in running command
I think it may be a problem with the way I declared my tag in the xml file:
<qsub>qsub -N @NAME@ -o @STDOUT@ -e @STDERR@ -l nodes=1:ppn=1 -l walltime=23:00:00</qsub>
Do you notice anything I am doing wrong in declaring the qsub tag?
(see redmine #1920)
Who receives it?
When received by email, will the reply via email be recorded on github (e.g. without requiring a luddite to open up a browser)?
That would be a huge benefit over Redmine.
@serbinsh @dlebauer I think I found a problem with the way that 'write.configs.ed.R' converts the leaf_respiration_rate_m2 to dark respiration factor. In order to convert: the code does the following 1) re-scales both the leaf_respiration_rate_m2 and Vcmax from 25C to 15C 2) then dark_resp_factor= leaf_resp@15C/Vcmax@15C.
However, it seems the code rescales Vcmax from 25 C two times, for example starting at line 53.
if('Vcmax' %in% names(trait.samples)) {
vcmax <- trait.samples[['Vcmax']]
trait.samples[['Vcmax']] <- arrhenius.scaling(vcmax, old.temp = 25, new.temp = 15)
}
## Convert leaf_respiration_rate_m2 to dark_resp_factor
if('leaf_respiration_rate_m2' %in% names(trait.samples)) {
leaf_resp = trait.samples[['leaf_respiration_rate_m2']]
vcmax <- trait.samples[['Vcmax']]
## First scale variables to 15 degC
trait.samples[['leaf_respiration_rate_m2']] <-
arrhenius.scaling(leaf_resp, old.temp = 25, new.temp = 15)
vcmax_15 <- arrhenius.scaling(vcmax, old.temp = 25, new.temp = 15)
## Calculate dark_resp_factor -- Will be depreciated when moving from older versions of ED2
trait.samples[['dark_respiration_factor']] < trait.samples[['leaf_respiration_rate_m2']]/
vcmax_15
It seems that Vcmax has already been re-scaled to 15C even before it enters into the dark respiration factor loop, because the code rescales the Vcmax parameter anyway in the previous section of code. I tested this out and it seems to be the case. For example my median posterior value for leaf respiration rate from the meta-analysis is 2.433. If I convert this by hand correctly, assuming Vcmax@25C=79.357, then I get a dark respiration factor of 0.0306. However, if I rescale Vcmax twice, I get a dark respiration factor of 0.0435, which is exactly what the code is giving me.
Does my reasoning make sense or am I way off on this? This might explain why I have been getting 'high' dark respiration factor values from PECAN, that don't work with my simulation. Not sure if the code has recently switched to using Rd0 only, but I am still using dark respiration factor.
When selecting a new host/model it should try to preserve site/model
build.sh will not run without the following on HPC systems:
module load netcdf nco R jags gdal
Some options:
echo "module load netcdf nco R jags gdal" > ~/.bashrc
)if type module > /dev/null; then
module load netcdf/4.3.1.1 nco R/3.0.2 JAGS gdal/1.10.1
fi
Suggestions?
I am not sure if this is a bug, but I can't get dbfile.input.check to work, for example, I would like to find the MsTMIP climate driver data so I use
dbfile.input.check(1118, "2004-01-01 06:00:00", "2004-01-01 06:00:00", "application/x-netcdf", "MsTMIP driver", con)
But I don't get anything returned.
Also, most of the functions in dbfiles.R are not exported. Is this supposed to be the case? Are these functions in progress (I don't see any tests or other uses in pecan directory)?
since R arguments use partial matching, user
will work for both the MySQL and PostgreSQL drivers.
Need to
find . -name "*.xml" | xargs grep username
related to redmine 1860
running job.sh on the BU geo cluster gave the following error
require (PEcAn.ED2)
Loading required package: PEcAn.ED2
Warning message:
In library(package, lib.loc = lib.loc, character.only = TRUE, logical.return = TRUE, :
there is no package called 'PEcAn.ED2'
model2netcdf.ED2('/usr4/ugrad/mgianott/geoDemo//15', 71.3225, -156.626, '2000/01/01', '2001/01/01')
Error: could not find function "model2netcdf.ED2"
Execution halted
cp: cannot stat `/usr4/ugrad/mgianott/ED.r82/ED/run/15/README.txt': No such file or directory
Right now the web interface uses file_id instead of id as well as it does not check to see if a file is an input and not posterior.
This is was not really an issue before but now can result in trouble since posteriors are also stored in dbfiles. This could result in files being returned that are posteriors and not inputs.
Versions of updatedb.sh and addsites.sh available here do not match the database version http://isda.ncsa.illinois.edu/~kooper/EBI/betydump.sql .
The columns have changed. models.revision is now numeric where it is passed a value of 'unk' from the script.
addsites.sh also references nonexistent columns.
Need to document all the special tags in ED2IN templates.
Should be able to launch integration tests from build script.
please update original issue as ideas are added / changed in comments
Rob and I wrote a set of scripts to combine the NCEP 2 degree global data set. These are in pecan/modules/data.atmosphere/inst/scripts/ncep.
We want to do the same with cru-ncep 0.5 degree data, as described in pecan/modules/data.atmosphere/inst/scripts/cruncep/README.md
But the Cru-ncep 0.5 degree x > 100 year data set (~1TB) requires more careful thought.
I noticed that for the variance decomposition plot that the absolute variance is calculated for each parameter and not the partial variance. This is because within line 145 of 'sensitivity.analysis.R' :
partial.variances <- variances #/ sum(variances)
The sum(variances) was commented out leaving behind the total variance for each parameter. This doesn't fundamentally change the interpretation of the variance decomposition plot, but was just curious why this was done Was it to keep the variance units??
Just thought of a small tweak to 08-finished.php that I think will make it a bit more intuitive. Under "Outputs" could you move the box "File" and the button "Show Run Output" to after "Plot run/year/variable" and then rename "Show Run Output" to "Show Run File". This way the selection boxes for Run, Year, and Variable are right in a row and followed by "Plot run/year/variable" and there is less ambiguity in "Show Run Output" (for me show output and plot output are too close to synonyms, but show file makes this more clear)
The "troubleshooting pecan" wiki provides a some details about a very specific PHP issue. This should probably be filed as a resolved issue with question and answer here on GitHub. But a more general description of how to debug could be handy, including use of debugonce
(with links to relevant documentation such as Hadley Wickham's chapter on debugging and exceptions as well as links to our wiki page on testing
On Stampede, I was unable to install ncdf4, udunits2, rjags, MCMCpack, and rgdal.
These will all require local installation of dependencies. Has anyone done this? Is the best approach to build all of the dependencies from source? Are there tricks to identifying and loading the correctly configured modules? I currently have this in my .bashrc
but am still testing it out:
module load gsl hdf5 parallel-netcdf/4.2.1.1 netcdf nco udunits/2.1.24 R_mkl/3.0.1
In general, it seems that it might be useful to share libraries on such a machine rather than have everyone fiddle around. e.g. if I set /home1/02014/dlebauer/lib/R
to read-only, and people put libPaths("~/lib/R/", "/home1/02014/dlebauer/lib/R")
it in their .Rprofile
, they wouldn't have to install any dependencies, but could if they wanted to (e.g. a different version of PEcAn etc).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.