compevol/morph-models

Models for dealing with morphological data

morph-models

Models for dealing with standard discrete morphological data.

Aiming at MK and MKv models from Lewis, 2001 (http://sysbio.oxfordjournals.org/content/50/6/913.short)

A tutorial is available here

morph-models's Issues

Remove constant characters automatically for Mkv model

If an Mkv model is chosen, constant characters should be removed (or ignored) from the matrix printing a Warning with the details about removed characters.

The transition probability matrix in Lewis Mk model is incorrect

double should be used instead of integer for calculation line 108

Lewis MK with all equal frequencies by default

Should the default setting for Lewis MK model be that all frequencies are equal? This is the simplest case of the model and is commonly used for morphological data.

Likelihood incorrectly calculated using Mkv with frequencies

When running the penguins data under Mkv model with frequencies estimated, I get
Likelihood incorrectly calculated: -22865.94213914816 != -22865.76245795682(-0.17968119134093286) Operator: beast.evolution.operators.ScaleOperator

To narrow down the problem, first I only enabled the operators changing the frequencies, it works fine, but once I enable a tree operator, say the treeScaler, I got something like this:
P(posterior) = -22865.76245795682 (was -22865.94213914816) **
P(prior) = -462.11152443659694 (was -462.11152443659694)
P(birthDeath) = -450.6982587466785 (was -450.6982587466785)
P(origin_prior) = -5.075173815233827 (was -5.075173815233827)
P(diversificationRate_prior) = -0.5092379526435218 (was -0.5092379526435218)
P(turnover_prior) = 0.0 (was 0.0)
P(samplingProportion_prior) = 0.0 (was 0.0)
P(MeanRatePrior.c:penguins) = 2.8929944227178996 (was 2.8929944227178996)
P(ucldStdevPrior.c:penguins_morph) = 0.819364145120802 (was 0.819364145120802)
P(ucldStdevPrior.c:penguins_dna) = 0.819364145120802 (was 0.819364145120802)
P(ratePrior.s:morph3) = -1.3068528194400546 (was -1.3068528194400546)
P(ratePrior.s:morph4) = -1.3068528194400546 (was -1.3068528194400546)
P(ratePrior.s:morph5) = -1.3068528194400546 (was -1.3068528194400546)
P(ratePrior.s:morph6) = -1.3068528194400546 (was -1.3068528194400546)
P(gammaShape_morph_prior) = -0.13316535724040832 (was -0.13316535724040832)
P(gammaShape_rag1_prior) = -1.0 (was -1.0)
P(gammaShape_cyt_b_prior) = -1.0 (was -1.0)
P(gammaShape_coi_prior) = -1.0 (was -1.0)
P(gammaShape_12srna_prior) = -1.0 (was -1.0)
P(gammaShape_16srna_prior) = -1.0 (was -1.0)
P(likelihood) = -22403.65093352022 (was -22403.83061471156) **
P(treeLikelihood.morph2) = -825.6129223393757 (was -825.7926035307171) **
P(treeLikelihood.morph3) = -535.803294293985 (was -535.803294293985)
P(treeLikelihood.morph4) = -223.3398850703398 (was -223.3398850703398)
P(treeLikelihood.morph5) = -56.00861874956596 (was -56.00861874956596)
P(treeLikelihood.morph6) = -83.35182917539032 (was -83.35182917539032)
P(treeLikelihood.morph7) = -21.20816564864092 (was -21.20816564864092)
P(treeLikelihood.rag1) = -5564.232312404348 (was -5564.232312404348)
P(treeLikelihood.cyt_b) = -5213.817694448532 (was -5213.817694448532)
P(treeLikelihood.coi) = -2412.530466591211 (was -2412.530466591211)
P(treeLikelihood.12srna) = -3583.1025130164585 (was -3583.1025130164585)
P(treeLikelihood.16srna) = -3884.6432317823724 (was -3884.6432317823724)
At sample 190000
Likelihood incorrectly calculated: -22865.94213914816 != -22865.76245795682(-0.17968119134093286) Operator: beast.evolution.operators.ScaleOperator

Xml to reproduce the error is attached, with a much smaller xml to reproduce the same thing.

xmls.zip

BEAUti does not open fileChooser window for morphological data

When clicking on the plus sign and choosing "add morphological data" nothing happens. If one adds morphological data from the file menu then it works properly.

There is no operators for estimated frequencies.

BEAUti sets Frequencies to be estimated for Lewis MK model by default however there is no operator for Frequencies in the template.

java.lang.NullPointerException at initialization

At the beginning of the run, BEAST2 shows:
java.lang.NullPointerException
at beast.evolution.likelihood.BeagleTreeLikelihood.setUpSubstModel(BeagleTreeLikelihood.java:474)
at beast.evolution.likelihood.BeagleTreeLikelihood.initialize(BeagleTreeLikelihood.java:364)
at beast.evolution.likelihood.BeagleTreeLikelihood.initAndValidate(BeagleTreeLikelihood.java:99)
at beast.core.BEASTInterface.initByName(BEASTInterface.java:105)
at beast.evolution.likelihood.TreeLikelihood.initAndValidate(TreeLikelihood.java:132)
at beast.util.XMLParser.initBEASTObjects(XMLParser.java:377)
at beast.util.XMLParser.parse(XMLParser.java:535)
at beast.util.XMLParser.parseFile(XMLParser.java:302)
at beast.app.BeastMCMC.parseArgs(BeastMCMC.java:311)
at beast.app.BeastMCMC.main(BeastMCMC.java:567)

although the run will continue and will have results.
A toy xml will reproduce the problem:
test.xml.zip

beast xml files do not specify (correct) datatype

I am trying to run beast with morph-traits, using the latest versions from github [beast: commit 892e22a227b37e4c56eedddd2abe03e3885c8a9e, Fri Mar 13 16:13:06 2015 morph-models: commit c11f1d7, Fri Mar 13 16:28:51 2015]. I am running them from Eclipse's debug mode.

Using both a beauti generated beast xml configuration or examples/M3982.xml from this repository, I get the error logs below.

File: M3982.xml seed: 127 threads: 1
Probing: beagle.jar Skip loading file:/usr/home/gereon/lang/beast/beast2/lib/beagle.jar: contains classs beagle.Beagle that is already loaded
Probing: colt.jar Skip loading file:/usr/home/gereon/lang/beast/beast2/lib/colt.jar: contains classs cern.clhep.PhysicalConstants that is already loaded
Probing: commons-math3-3.1.1.jar Skip loading file:/usr/home/gereon/lang/beast/beast2/lib/commons-math3-3.1.1.jar: contains classs org.apache.commons.math3.linear.OpenMapRealVector$OpenMapEntry that is already loaded
Probing: debug-1.0.jar Skip loading file:/usr/home/gereon/lang/beast/beast2/lib/debug-1.0.jar: contains classs org.jdesktop.swinghelper.debug.CheckThreadViolationRepaintManager$1 that is already loaded
Probing: fest-sources.jar URL file:/usr/home/gereon/lang/beast/beast2/lib/fest-sources.jar is already in the CLASSPATH
Probing: fest.jar Skip loading file:/usr/home/gereon/lang/beast/beast2/lib/fest.jar: contains classs net.jcip.annotations.GuardedBy that is already loaded
Probing: jam.jar Skip loading file:/usr/home/gereon/lang/beast/beast2/lib/jam.jar: contains classs jam.console.ConsoleApplication$1 that is already loaded
Probing: junit-4.8.2.jar Skip loading file:/usr/home/gereon/lang/beast/beast2/lib/junit-4.8.2.jar: contains classs junit.extensions.ActiveTestSuite$1 that is already loaded
Probing: BDSSM.addon.jar Loaded URL file:/home/gereon/.beast/2.2/BDSKY/lib/BDSSM.addon.jar
Probing: GEO.addon.jar Loaded URL file:/home/gereon/.beast/2.2/GEO_SPHERE/lib/GEO.addon.jar
Probing: SubstBMA.addon.jar Loaded URL file:/home/gereon/.beast/2.2/SubstBMA/lib/SubstBMA.addon.jar
Probing: colt.jar Skip loading file:/home/gereon/.beast/2.2/SubstBMA/lib/colt.jar: contains classs cern.clhep.PhysicalConstants that is already loaded
Uniform => beast.math.distributions.Uniform
Exponential => beast.math.distributions.Exponential
LogNormal => beast.math.distributions.LogNormalDistributionModel
Normal => beast.math.distributions.Normal
Beta => beast.math.distributions.Beta
Gamma => beast.math.distributions.Gamma
LaplaceDistribution => beast.math.distributions.LaplaceDistribution
prior => beast.math.distributions.Prior
InverseGamma => beast.math.distributions.InverseGamma
OneOnX => beast.math.distributions.OneOnX
Alignment(M3982)
  55 taxa
  16 sites
  16 patterns

Filter 1,2,4,7,8,12,15,16
55 taxa
8 sites
8 patterns
java.lang.RuntimeException: number of stationary frequencies does not match number of states.
    at beast.evolution.substitutionmodel.LewisMK.setFrequencies(LewisMK.java:53)
    at beast.evolution.substitutionmodel.LewisMK.initAndValidate(LewisMK.java:82)
    at beast.util.XMLParser.initPlugins(XMLParser.java:314)
    at beast.util.XMLParser.parse(XMLParser.java:443)
    at beast.util.XMLParser.parseFile(XMLParser.java:244)
    at beast.app.BeastMCMC.parseArgs(BeastMCMC.java:253)
    at beast.app.BeastMCMC.parseArgs(BeastMCMC.java:238)
    at beast.app.BeastMCMC.main(BeastMCMC.java:517)

Error 110 parsing the xml input file

validate and intialize error: number of stationary frequencies does not match number of states.

Error detected about here:
  <beast>
      <run id='mcmc' spec='MCMC'>
          <distribution id='posterior' spec='util.CompoundDistribution'>
              <distribution id='likelihood' spec='util.CompoundDistribution'>
                  <distribution id='morphTreeLikelihood.M39822' spec='TreeLikelihood'>
                      <siteModel id='morphSiteModel.s:M39822' spec='SiteModel'>
                          <substModel id='LewisMK.s:M39822' spec='LewisMK'>

data

File: test.xml seed: 127 threads: 1
Probing: beagle.jar Skip loading file:/usr/home/gereon/lang/beast/beast2/lib/beagle.jar: contains classs beagle.Beagle that is already loaded
Probing: colt.jar Skip loading file:/usr/home/gereon/lang/beast/beast2/lib/colt.jar: contains classs cern.clhep.PhysicalConstants that is already loaded
Probing: commons-math3-3.1.1.jar Skip loading file:/usr/home/gereon/lang/beast/beast2/lib/commons-math3-3.1.1.jar: contains classs org.apache.commons.math3.linear.OpenMapRealVector$OpenMapEntry that is already loaded
Probing: debug-1.0.jar Skip loading file:/usr/home/gereon/lang/beast/beast2/lib/debug-1.0.jar: contains classs org.jdesktop.swinghelper.debug.CheckThreadViolationRepaintManager$1 that is already loaded
Probing: fest-sources.jar URL file:/usr/home/gereon/lang/beast/beast2/lib/fest-sources.jar is already in the CLASSPATH
Probing: fest.jar Skip loading file:/usr/home/gereon/lang/beast/beast2/lib/fest.jar: contains classs net.jcip.annotations.GuardedBy that is already loaded
Probing: jam.jar Skip loading file:/usr/home/gereon/lang/beast/beast2/lib/jam.jar: contains classs jam.console.ConsoleApplication$1 that is already loaded
Probing: junit-4.8.2.jar Skip loading file:/usr/home/gereon/lang/beast/beast2/lib/junit-4.8.2.jar: contains classs junit.extensions.ActiveTestSuite$1 that is already loaded
Probing: BDSSM.addon.jar Loaded URL file:/home/gereon/.beast/2.2/BDSKY/lib/BDSSM.addon.jar
Probing: GEO.addon.jar Loaded URL file:/home/gereon/.beast/2.2/GEO_SPHERE/lib/GEO.addon.jar
Probing: SubstBMA.addon.jar Loaded URL file:/home/gereon/.beast/2.2/SubstBMA/lib/SubstBMA.addon.jar
Probing: colt.jar Skip loading file:/home/gereon/.beast/2.2/SubstBMA/lib/colt.jar: contains classs cern.clhep.PhysicalConstants that is already loaded
Uniform => beast.math.distributions.Uniform
Exponential => beast.math.distributions.Exponential
LogNormal => beast.math.distributions.LogNormalDistributionModel
Normal => beast.math.distributions.Normal
Beta => beast.math.distributions.Beta
Gamma => beast.math.distributions.Gamma
LaplaceDistribution => beast.math.distributions.LaplaceDistribution
prior => beast.math.distributions.Prior
InverseGamma => beast.math.distributions.InverseGamma
OneOnX => beast.math.distributions.OneOnX
java.lang.Exception: Either input 'stateNumber' or 'datatype' needs to be specified
    at beast.core.Input.validate(Input.java:702)
    at beast.core.BEASTInterface.validateInputs(BEASTInterface.java:315)
    at beast.util.XMLParser.createObject(XMLParser.java:697)
    at beast.util.XMLParser.parseInputs(XMLParser.java:830)
    at beast.util.XMLParser.createObject(XMLParser.java:693)
    at beast.util.XMLParser.parseInputs(XMLParser.java:830)
    at beast.util.XMLParser.createObject(XMLParser.java:693)
    at beast.util.XMLParser.parseInputs(XMLParser.java:830)
    at beast.util.XMLParser.createObject(XMLParser.java:693)
    at beast.util.XMLParser.parseInputs(XMLParser.java:830)
    at beast.util.XMLParser.createObject(XMLParser.java:693)
    at beast.util.XMLParser.parseInputs(XMLParser.java:830)
    at beast.util.XMLParser.createObject(XMLParser.java:693)
    at beast.util.XMLParser.parseRunElement(XMLParser.java:561)
    at beast.util.XMLParser.parse(XMLParser.java:442)
    at beast.util.XMLParser.parseFile(XMLParser.java:244)
    at beast.app.BeastMCMC.parseArgs(BeastMCMC.java:253)
    at beast.app.BeastMCMC.parseArgs(BeastMCMC.java:238)
    at beast.app.BeastMCMC.main(BeastMCMC.java:517)

Error 110 parsing the xml input file

validate and intialize error: Either input 'stateNumber' or 'datatype' needs to be specified

Error detected about here:
  <beast>
      <run id='mcmc' spec='MCMC'>
          <distribution id='posterior' spec='util.CompoundDistribution'>
              <distribution id='likelihood' spec='util.CompoundDistribution'>
                  <distribution id='treeLikelihood.anolis' spec='TreeLikelihood'>
                      <siteModel id='SiteModel.s:anolis' spec='SiteModel'>
                          <substModel id='LewisMK.s:anolis' spec='LewisMK'>

Installing morph-models with bModelTest

The combination shows BEAST Model Test as site model when importing morphological data. Perhaps the MM template has a site model name that does not match the default gamma site model?

The morph-models package breaks the Java package naming conventions for BEAST2 packages

BEAST2 packages should not use Java package names beast.* as this increases the likelihood of package name clashes in BEAST2 installations when multiple packages clash with each other or the core. BEAST2 packages should have their own top level package name.

Importing data with 1 state

When importing data where there is a constant site, the importer creates a partition for that site, but the LewisMK model fails at getTransitionProbabilities since it tries to calculate stateCount/(stateCount-1), which with stateCount=1 gives a division by zero error.

Possible solutions:

• warn the user when importing
• let the importer assist in removing these sites
• make LewisMK robust to stateCount=1 cases
• create error message in LewisMK to prevent the error occurring

Beast fails to initialize model when the number of states is different.

When using the ordinal model and ordinal data with a different number of states, beast fails to initialize the model giving -infinite likelihood.

This is an input user error and should be caught during the model verification stage (such as when the sum of frequencies is checked) and communicated with an appropriate error.

During creating a sample test case with short sequences, beast even managed to run (although gave a few "incorrect likelihood was calculated" errors), this should not have happened.

Suggested fix: Additional assert/check that the number of states in the model agrees with the data.

Sample example: XML with randomly generated sequences and, to my best knowledge, vanilla ordinal model specification.