Giter Site home page Giter Site logo

easystats / performance Goto Github PK

View Code? Open in Web Editor NEW
990.0 27.0 87.0 671.61 MB

:muscle: Models' quality and performance metrics (R2, ICC, LOO, AIC, BF, ...)

Home Page: https://easystats.github.io/performance/

License: GNU General Public License v3.0

R 95.78% TeX 4.22%
r2 aic performance models loo r easystats mixed-models statistics hacktoberfest

performance's Introduction

easystats: An R Framework for Easy Statistical Modeling, Visualization, and Reporting

downloads total

What is easystats?

easystats is a collection of R packages, which aims to provide a unifying and consistent framework to tame, discipline, and harness the scary R statistics and their pesky models.

However, there is not (yet) an unique “easystats” way of doing data analysis. Instead, start with one package and, when you’ll face a new challenge, do check if there is an easystats answer for it in other packages. You will slowly uncover how using them together facilitates your life. And, who knows, you might even end up using them all.

Installation

CRAN_Status_Badge insight status badge R-CMD-check

Type Source Command
Release CRAN install.packages("easystats")
Development r-universe install.packages("easystats", repos = "https://easystats.r-universe.dev")
Development GitHub remotes::install_github("easystats/easystats")

Finally, as easystats sometimes depends on some additional packages for specific functions that are not downloaded by default. If you want to benefit from the full easystats experience without any hiccups, simply run the following:

easystats::install_suggested()

Citation

To cite the package, run the following command:

citation("easystats")
To cite easystats in publications use:

  Lüdecke, Patil, Ben-Shachar, Wiernik, Bacher, Thériault, & Makowski
  (2022). easystats: Framework for Easy Statistical Modeling,
  Visualization, and Reporting. CRAN.
  doi:10.32614/CRAN.package.easystats
  <https://doi.org/10.32614/CRAN.package.easystats>

A BibTeX entry for LaTeX users is

  @Article{,
    title = {easystats: Framework for Easy Statistical Modeling, Visualization, and Reporting},
    author = {Daniel Lüdecke and Mattan S. Ben-Shachar and Indrajeet Patil and Brenton M. Wiernik and Etienne Bacher and Rémi Thériault and Dominique Makowski},
    journal = {CRAN},
    doi = {10.32614/CRAN.package.easystats},
    year = {2022},
    note = {R package},
    url = {https://easystats.github.io/easystats/},
  }

If you want to do this only for certain packages in the ecosystem, have a look at this article on how you can do so! https://easystats.github.io/easystats/articles/citation.html

Getting started

Each easystats package has a different scope and purpose. This means your best way to start is to explore and pick the one(s) that you feel might be useful to you. However, as they are built with a “bigger picture” in mind, you will realize that using more of them creates a smooth workflow, as these packages are meant to work together. Ideally, these packages work in unison to cover all aspects of statistical analysis and data visualization.

  • report: 📜 🎉 Automated statistical reporting of objects in R
  • correlation: 🔗 Your all-in-one package to run correlations
  • modelbased: 📈 Estimate effects, group averages and contrasts between groups based on statistical models
  • bayestestR: 👻 Great for beginners or experts of Bayesian statistics
  • effectsize: 🐉 Compute, convert, interpret and work with indices of effect size and standardized parameters
  • see: 🎨 The plotting companion to create beautiful results visualizations
  • parameters: 📊 Obtain a table containing all information about the parameters of your models
  • performance: 💪 Models’ quality and performance metrics (R2, ICC, LOO, AIC, BF, …)
  • insight: 🔮 For developers, a package to help you work with different models and packages
  • datawizard: 🧙 Magic potions to clean and transform your data

Frequently Asked Questions

How is easystats different from the tidyverse?

You’ve probably already heard about the tidyverse, another very popular collection of packages (ggplot, dplyr, tidyr, …) that also makes using R easier. So, should you pick the tidyverse or easystats? Pick both!

Indeed, these two ecosystems have been designed with very different goals in mind. The tidyverse packages are primarily made to create a new R experience, where data manipulation and exploration is intuitive and consistent. On the other hand, easystats focuses more on the final stretch of the analysis: understanding and interpreting your results and reporting them in a manuscript or a report, while following best practices. You can definitely use the easystats functions in a tidyverse workflow!

easystats + tidyverse = ❤️

Can easystats be useful to advanced users and/or developers?

Yes, definitely! easystats is built in terms of modules that are general enough to be used inside other packages. For instance, the insight package is made to easily implement support for post-processing of pretty much all regression model packages under the sun. We use it in all the easystats packages, but it is also used in other non-easystats packages, such as ggstatsplot, modelsummary, ggeffects, and more.

So why not in yours?

Moreover, the easystats packages are very lightweight, with a minimal set of dependencies, which again makes it great if you want to rely on them.

Documentation

Websites

Each easystats package has a dedicated website.

For example, website for parameters is https://easystats.github.io/parameters/.

Blog

In addition to the websites containing documentation for these packages, you can also read posts from easystats blog: https://easystats.github.io/blog/posts/.

Other learning resources

In addition to these websites and blog posts, you can also check out the following presentations and talks to learn more about this ecosystem:

https://easystats.github.io/easystats/articles/resources.html

Dependencies

easystats packages are designed to be lightweight, i.e., they don’t have any third-party hard dependencies, other than base-R packages or other easystats packages! If you develop R packages, this means that you can safely use easystats packages as dependencies in your own packages, without the risk of entering the dependency hell.

library(deepdep)

plot_dependencies("easystats", depth = 2, show_stamp = FALSE)

As we can see, the only exception is the {see} package, which is responsible for plotting and creating figures and relies on {ggplot2}, which does have a substantial number of dependencies.

Usage

Total downloads

Total insight datawizard parameters performance bayestestR effectsize correlation see modelbased report easystats
22,865,116 6,696,001 4,049,281 2,787,191 2,709,620 2,646,921 2,105,929 698,380 577,658 343,099 190,066 60,970

Trend

Contributing

We are happy to receive bug reports, suggestions, questions, and (most of all) contributions to fix problems and add features. Pull Requests for contributions are encouraged.

Here are some simple ways in which you can contribute (in the increasing order of commitment):

  • Read and correct any inconsistencies in the documentation
  • Raise issues about bugs or wanted features
  • Review code
  • Add new functionality

Code of Conduct

Please note that the ‘easystats’ project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

performance's People

Contributors

andthewings avatar bwiernik avatar dominiquemakowski avatar egouldo avatar etiennebacher avatar github-actions[bot] avatar indrajeetpatil avatar jmgirard avatar krassowski avatar mattansb avatar mcanouil avatar pablobernabeu avatar pdwaggoner avatar rekyt avatar rempsyc avatar strengejacke avatar vincentarelbundock avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

performance's Issues

Unable to install from GitHub.

Hi there. I've been unable to install this package from GitHub. Code and errors from the install follows:

expand
> check_compiler()
[1] TRUE
> has_rtools()
[1] TRUE
> devtools::install_github("easystats/performance")
Downloading GitHub repo easystats/performance@master
√  checking for file 'C:\Users\Anthony\AppData\Local\Temp\Rtmp8mqayi\remotes3b2441315bf\easystats-performance-0d7ed43/DESCRIPTION' ... 
-  preparing 'performance': (376ms)
√  checking DESCRIPTION meta-information ... 
-  installing the package to process help pages
         -----------------------------------
-  installing *source* package 'performance' ...
   ** using staged installation
   ** R
   ** byte-compile and prepare package for lazy loading
   Error: object 'area_under_curve' is not exported by 'namespace:bayestestR'
   Execution halted
   ERROR: lazy loading failed for package 'performance'
-  removing 'C:/Users/Anthony/AppData/Local/Temp/RtmpUFphiY/Rinst2b743ec856ec/performance'
         -----------------------------------
   ERROR: package installation failed
Error in (function (command = NULL, args = character(), error_on_status = TRUE,  : 
  System command error

Output from Sys.getenv():

expand
> Sys.getenv()
__COMPAT_LAYER              RunAsAdmin
AGSDESKTOPJAVA              F:\Program Files (x86)\ArcGIS\Desktop10.6\
ALLUSERSPROFILE             C:\ProgramData
APPDATA                     C:\Users\Anthony\AppData\Roaming
BINPREF                     D:/Rtools/mingw_$(WIN)/bin/
CLICOLOR_FORCE              1
CommonProgramFiles          C:\Program Files\Common Files
CommonProgramFiles(x86)     C:\Program Files (x86)\Common Files
CommonProgramW6432          C:\Program Files\Common Files
COMPUTERNAME                UNICRON
ComSpec                     C:\WINDOWS\system32\cmd.exe
developer tools             D:\Working Folder\util\platform-tools
DISPLAY                     :0
DriverData                  C:\Windows\System32\Drivers\DriverData
FP_NO_HOST_CHECK            NO
GDAL_DATA                   D:/R/R-3.6.0/library/sf/gdal
GFORTRAN_STDERR_UNIT        -1
GFORTRAN_STDOUT_UNIT        -1
HOME                        D:/Users/Anthony/Documents
HOMEDRIVE                   C:
HOMEPATH                    \Users\Anthony
LOCALAPPDATA                C:\Users\Anthony\AppData\Local
LOGONSERVER                 \\UNICRON
MSMPI_BIN                   C:\Program Files\Microsoft MPI\Bin\
MSYS2_ENV_CONV_EXCL         R_ARCH
NUMBER_OF_PROCESSORS        8
NVIDIAWHITELISTED           0x01
OneDrive                    C:\Users\Anthony\OneDrive
OS                          Windows_NT
PATH                        D:\R\R-3.6.0\bin\x64;D:\Rtools\bin;C:\Program
                            Files\Microsoft MPI\Bin\;C:\Program Files (x86)\Common
                            Files\Oracle\Java\javapath;C:\ProgramData\Oracle\Java\javapath;C:\Windows\System32;C:\Windows;C:\Windows\System32\wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\Program
                            Files\Calibre2\;C:\Windows\System32;C:\Windows;C:\Windows\System32\wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program
                            Files (x86)\QuickTime\QTSystem\;C:\Program Files
                            (x86)\NVIDIA
                            Corporation\PhysX\Common;C:\Windows\System32;C:\Windows;C:\Windows\System32\wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program
                            Files (x86)\Windows Kits\10\Windows Performance
                            Toolkit\;C:\Windows\System32\OpenSSH\;C:\Program
                            Files\Intel\WiFi\bin\;C:\Program Files\Common
                            Files\Intel\WirelessCommon\;D:\Program Files\Microsoft
                            VS Code\bin;D:\Program Files\PuTTY\;C:\Program
                            Files\NVIDIA Corporation\NVIDIA NvDLISR;D:\Program
                            Files\SASHome\SASFoundation\9.4\ets\sasexe;D:\Program
                            Files\SASHome\Secure\ccme4;D:\Program
                            Files\SASHome\x86\Secure\ccme4;C:\Program Files\MiKTeX
                            2.9\miktex\bin\x64\;D:\Presence;D:\Rtools\mingw_64\bin;C:\Users\Anthony\AppData\Local\Microsoft\WindowsApps;E:\TDM-GCC-64\bin
PATHEXT                     .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE      AMD64
PROCESSOR_IDENTIFIER        Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
PROCESSOR_LEVEL             6
PROCESSOR_REVISION          3c03
ProgramData                 C:\ProgramData
ProgramFiles                C:\Program Files
ProgramFiles(x86)           C:\Program Files (x86)
ProgramW6432                C:\Program Files
PROJ_LIB                    D:/R/R-3.6.0/library/sf/proj
PSModulePath                C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules\
PUBLIC                      C:\Users\Public
QT_D3DCREATE_MULTITHREADED
                            1
R_ARCH                      /x64
R_COMPILED_BY               gcc 4.9.3
R_DOC_DIR                   D:/R/R-3.6.0/doc
R_HOME                      D:/R/R-3.6.0
R_LIBS_USER                 D:/Users/Anthony/Documents/R/win-library/3.6
R_PACKRAT_DEFAULT_LIBPATHS
                            D:/R/R-3.6.0/library
R_PACKRAT_SYSTEM_LIBRARY    D:/R/R-3.6.0/library
R_USER                      D:/Users/Anthony/Documents
RMARKDOWN_MATHJAX_PATH      D:/Program Files/RStudio/resources/mathjax-26
RS_LOCAL_PEER               \\.\pipe\28163-rsession
RS_RPOSTBACK_PATH           D:/Program Files/RStudio/bin/rpostback
RS_SHARED_SECRET            63341846741
RSTUDIO                     1
RSTUDIO_CONSOLE_COLOR       256
RSTUDIO_CONSOLE_WIDTH       80
RSTUDIO_MSYS_SSH            D:/Program Files/RStudio/bin/msys-ssh-1000-18
RSTUDIO_PANDOC              D:/Program Files/RStudio/bin/pandoc
RSTUDIO_SESSION_PORT        28163
RSTUDIO_USER_IDENTITY       Tony Kroeger
RSTUDIO_WINUTILS            D:/Program Files/RStudio/bin/winutils
SHIM_MCCOMPAT               0x810000001
SystemDrive                 C:
SystemRoot                  C:\WINDOWS
TEMP                        C:\Users\Anthony\AppData\Local\Temp
TERM                        xterm-256color
TMP                         C:\Users\Anthony\AppData\Local\Temp
USERDOMAIN                  UNICRON
USERDOMAIN_ROAMINGPROFILE   UNICRON
USERNAME                    Tony Kroeger
USERPROFILE                 C:\Users\Anthony
windir                      C:\WINDOWS

I've been able to install other packages from github previously without issue. TIA!

R2 bayes: default to robust?

I think it would make sense to set robust to TRUE by default, since it's consistent with the remaining defaults used by in parameters, report bayestestR and such?

R2 bayes "fixed" -> "marginal"

Should we rename the current R2_Bayes_fixed to R2_Bayes_marginal so it is more consistent with the frequentist mixed models?

PCA to parameters?

Looking at the functions, I wonder if pca should be moved to parameters. As it is not technically related to performance, but rather used eventually to decrease the number of parameters of the model. Maybe it would be more discoverable there. What do you think?

Naming of functions

@DominiqueMakowski Since we have several functions that compute the model quality or performance, maybe we can find a common prefix for those as well? E.g.

performance_rmse(), performance_epcp(), performance_logloss() etc., or quality_rmse(), quality_epcp(), quality_logloss().

This would be in line with other "bundles" like r2_*(). item_*() and check_*(). What would you say?

Vignettes

I think we could start by adding two vignettes, one "Different R2s" discussing the differences and application of the R2 indices. We already have this information in the docs so it's mainly a reformatting. A second one could be "Comparing models" As I feel that this feature will need a bit of highlighting :)

I'll try making drafts

ICC docs

Minor point, currently in the ICC docs it says:

can be interpreted as \dQuote{the proportion of the variance explained by the grouping structure in the population} \cite{(Hox 2010: 15)}

Here the definition is:

The intraclass correlation ρ can also be interpreted as the expected correlation between two randomly drawn units that are in the same group. (Hox, 2010, p. 15)

Should we also add this second definition (if correct) to the docs?

BF in compare_models

I am not really sure how, but it might be relevant to make compare_models compatible with bayesfactor_models

Workaround is_hurdle

Since insight::model_info() does not yet return $is_hurdle, this needs to be retrieved manually for this update and to be removed after CRAN update of insight.

Performance for ordinal models

Because in both, brms and rstanarm, the R^2 metric is not available for ordinal models (in contrast with elpd, loo or waic), model_performance(ordinal_model ) does not work.

Clean up my mess...

@DominiqueMakowski Due to this unfortunate setup of Github, cloud sync and two PCs for working at packages I tried to commit via webinterface, which completely messed up the file structure (with R files put in the root folder).

I now cloned the repo here, cleaned up everything and commited via GitHub app again (hope that my sync understands that it doesn't have revert the sync from GitHub after syncing from cloud at home...).

Please check the spelling, some of the performance_* functions have lower case, other upper case R.

check_collinearity for interaction terms

Great! it works now!
I know I am at risk of going off-topic but may I ask why the VIF is calculated only for the main effects terms and not for the interaction term? For example:


> library(car)
> library(performance)
> beta1<- 0.5
> beta2<- -2
> pred1<- runif(n=1000,0,1)
> pred2<- runif(n=1000,0,1)
> Int<- 1
> logE.y<- Int + pred1*beta1 + pred2*beta2
> E.y<- exp(logE.y)
> resp<- rpois(n=1000,lambda=E.y)
> dati<- data.frame(resp=resp,pred1=pred1,pred2=pred2)
> 
> glm1<- glm(resp~pred1*pred2,data=dati,poisson)
> vif(glm1)
      pred1       pred2 pred1:pred2 
   2.791190    4.807468    6.259886 
> check_collinearity(glm1)
# Check for Multicollinearity

Low Correlation

 Predictor  VIF Increased SE
     pred1 2.79         1.67
     pred2 4.81         2.19

Originally posted by @LincolnPetersen in #55 (comment)

model_performance for Stan models

Looking at the blog-post, two things came to my mind.

  1. Should we remove the CI from the R2-values? It really bloats the output.
  2. Should we use a lighter theme for the code-chunks? Dark mode was cool for some months when Apple introduced it, but Apple is on the skids and no longer trend setting. I think light themes are better (at least for the code, the sidebar looks good!)

check_assumptions()

Maybe we should think of a new "class" of functions, prefixed with check_*, made for... checking stuff for our models. This could be a way of regrouping them together and making them more discoverable. They would become check_overdispersion(), check_zeroinflation() etc.

Based on recent questions that I had from students, other possible functions (not necesserily implemented in performance) come to my mind, such as check_normality and check_homogeneity (of variances), and other functions that would facilitates the checks to see if it's suited for parametric analyses... which would be very useful for students.

Originally posted by @DominiqueMakowski in #24 (comment)

ranking of models and indices differences

This is an extension of #28

It would indeed be interesting to either have a separate function (e.g., rank_models), or an argument in compare_models (e.g., show_ranks), to "rank" the models, i.e., displaying "the best" at the top, and then the following, which indices could be displayed in terms of difference with the best model.

This presents several challenges:

  • encoding somewhere the directionality of each index (for instance, for R2 the bigger the better, whereas for AIC the smaller the better).
  • finding some sort of unique index of "best" model, for instance, the ratio of higher indices (as these are not always congruent). For instance, model 2 can have 4 out of 6 indices placing it as the best.
  • dealing with edge cases: what to do in the case of equality, etc.
  • some indices can be compared through NH testing, resulting in a p-value. What to do in these cases?

check_outlier

@pdwaggoner @strengejacke
That's a great idea, a awesome implementation here and in see 🎉 .

Few minor things that came through my mind:

  • check_outlier -> check_outliers? 🤔 (plural)
  • Add distance type as a method argument (allowing for other types of distances, e.g. Mahalanobis etc)
  • This function could be extended to dataframes and single vectors, for which it could return the outliers based on a traditional threshold of SD (e.g., 1.96 SD etc)
  • Make the threshold explicit and changeable with a threshold argument (currently is it for Cook's distance (4 / n) or 4, for example). We could provide a "default" mode that would pick a reasonable default depending on the distance method.
  • how should we deal with Bayesian models? since they usually have their own measure (the pareto k) should we wrap around this method?

I can't wait to have vignettes and posts presenting all of these cool functions ☺️

Indices

Possible indices to add:

  • ICC
  • Bayes factor (?)
  • R2 Nakagawa
  • R2 for lme
  • Goodness-of-Fit for Logistic Regression?
  • Hosmer-Lemeshow for Log. Reg.?

Consistent return values for R2-functions

We must think about the behaviour of some r-squared functions...

  1. Do we want r2() to return different values for lm? Currently, DF etc. are also returned.
  2. Do we want named vectors to be returned or unnamed?
  3. Do we want a list to be returned for r2_*()?

I would say "yes" to 2 and 3, but not sure about 1?

Performance for logistic regression

Related to #1

they say (on wikipedia 😀):

"The Hosmer-Lemeshow test is for overall calibration error, not for any particular lack of fit such as quadratic effects. It does not properly take overfitting into account, is arbitrary to choice of bins and method of computing quantiles, and often has power that is too low."

"For these reasons the Hosmer-Lemeshow test is no longer recommended. Hosmer et al have a better one d.f. omnibus test of fit, implemented in the R rms package residuals.lrm function."

Other alternatives have been developed to address the limitations of the Hosmer-Lemeshow test. These include the Osius-Rojek test and the Stukel test, available in the R script AllGOFTests.R: www.chrisbilder.com/categorical/Chapter5/AllGOFTests.R

I don't really have any opinion, never used any of those...

r2-differences

Moved from strengejacke/sjstats#67 over here...

@hauselin due to the re-organization of packages, all "model-performance" related stuff will now be implemented in the performance package.

@DominiqueMakowski What do you think, can we make r2() let accept multiple model objects, and when the user passes multiple models, we can make an "anova"-like output? I.e. the r-squared values for all models, and an extra column indicating the difference(s)?

KL-Divergence-Based R2

Looking for something else, I just came across this package, which seems to be relatively recent, implementing some other R2 metrics, created by the package's author (Zhang, 2017). Leaving it here for future reference.

Updating `model_performance()`

Now that we have mse(), rmse() and rse(), we might include one of these measures in model_performance() as well, at least the RMSE I would say.

Maybe error_rate() also for model_performace()-glm?

r2 fails for intercept only models

performance::r2(insight::download_model("lm_0"))
 Error in stats::pf(out$F, out$DoF, out$DoF_residual, lower.tail = FALSE) : 
  Non-numeric argument to mathematical function 

Should we return NA instead of throwing error?

Install fails with Error "object ‘has_intercept’ is not exported by 'namespace:insight'"

I tried to install performance, but the install is currently failing with the following error:

Downloading GitHub repo easystats/performance@master
✔  checking for file ‘/private/var/folders/[...]/easystats-performance-122d3d0/DESCRIPTION’ ...
─  preparing ‘performance’:
✔  checking DESCRIPTION meta-information ...
─  installing the package to process help pages
         -----------------------------------
─  installing *source* package ‘performance’ ...
   ** using staged installation
   ** R
   ** byte-compile and prepare package for lazy loading
   Error: object ‘has_intercept’ is not exported by 'namespace:insight'
   Execution halted
   ERROR: lazy loading failed for package ‘performance’
─  removing ‘/private/var/folders/[...]/performance’

Possible to some bug got into the process recently?
(FYI, I am on macOS 10.14.4, running R 3.6.0)

compare_models disfunction

I looked at the other issues and I think that this has not been raised. The function "compare_performance" shuffles the names and factors (AIC and so forth) around, so that the model that is first named in the function always stands in front of the lowest AIC.

Example:
mpd_1 <- lm(log(super_data$naturalized_proc+1)~super_data$mpd)
mpd_2 <- lm(log(super_data$naturalized_proc+1)~super_data$MPD.obs)
mpd_3 <- lm(log(super_data$naturalized_proc+1)~super_data$MPD.ses)

compare_performance(mpd_1,mpd_2,mpd_3)

output
name class AIC BIC R2 R2_adjusted RMSE
1 mpd_1 lm 2114.482 2127.295 0.15494980 0.1533463 1.775275
2 mpd_2 lm 2122.197 2135.010 0.14253483 0.1409078 1.788268
3 mpd_3 lm 2156.999 2169.812 0.08422591 0.0824882 1.848070

compare_performance(mpd_2,mpd_1,mpd_3)

output
name class AIC BIC R2 R2_adjusted RMSE
1 mpd_2 lm 2114.482 2127.295 0.15494980 0.1533463 1.775275
2 mpd_1 lm 2122.197 2135.010 0.14253483 0.1409078 1.788268
3 mpd_3 lm 2156.999 2169.812 0.08422591 0.0824882 1.848070

Add tests for release

Add tests for

  • check_convergence
  • check_overdispersion
  • check_singularity
  • check_zeroinflation
  • cronbachs_alpha
  • icc
  • item_difficulty
  • item_intercor
  • item_reliability
  • item_split_half

Revise readme for release

Since we have no vignettes yet, I would like to revise the readme a bit, starting with a short "about" paragraph, then maybe sections on

  • r2 / icc
  • check-functions
  • item-functions
  • model_performance()

Further performance metrics to add

This would definitely fit.

Other things that come to my mind when I think of the scope:

  • Other indices used in the structural equation field (this bunch of guys). As they are computed by default by lavaan, it would mostly consist of extractors consistent with the easyverse.
  • Convenience methods for PCAs / Factor Analysis, returning the % of variance explained.
  • I think that in general, and in the future, the easyverse (that is a thing now :) methods could be useful in the machine learning world, where people struggle with different models/packages. Providing a unifying syntax for extracting, interpreting and understanding their models could be quite appreciated. Although this will probably wait for the help of a future contributor, expert in this kind of things. But providing the tools to bridge the regression world with the ML world could be quite cool.

Originally posted by @DominiqueMakowski in #14 (comment)

equivalence-test for frequentist models

When reading this article, that encourages to think of CI as compatible intervals, and that there's always a difference (even from NULL) etc., I though about implementing a frequentist-equivalent to the Bayesian equivalence_test() we have in bayestestR.

  • rope_range() would be identical, so we can call bayestestR::rope_range()
  • rope() will differ, because we have no posterior and no HDI, but just the CI. But we don't really need this function, just internally we need to calculate how much of the CI is inside the rope-range. We could probably just check the proportion of overlap of rope in CI.
  • equivalence_test() will get .lm, .lmer, .glm generics etc., that actually all do the same thing: 1) determine rope-range, 2) compute the xx%-CI, 3) check overlap of CI within rope-range.

This will of cource not completely overcome the dichotomia-problem of significant vs. non-significant, but it shows which predictors might be more under discussion, because their status is "undecided".

Marginal vs. conditional R2 for Bayesian models

It would be a great feature to have something for that. Options are:

  • Expand R2_nakagawa to work with Bayesian models
  • Adapt R2 MLM to work with Bayesian models
  • Create a function based on rstanarm::bayes_R2

For the last option, this thread could be relevant:

bayes_R2(model, re.form = NULL) vs bayes_R2(model, re.form = NA) is fine, but it shouldn’t be used to “test” a null hypothesis that the group-specific intercept and coefficients are irrelevant.

Complete docs

  • empty section \examples in check_overdispersion()
  • empty section \examples in check_zeroinflation()
  • empty section \value in rmse()

check_collinearity

OK, thanks, I have managed to install the last version of performance by first installing (via github) bayestestR. Now the function check_collinearity() is available. However, I get an error message:


> check_collinearity(glmCompois)
Error in .subset2(x, i, exact = exact) : 
  attempt to select less than one element in get1index

Originally posted by @LincolnPetersen in glmmTMB/glmmTMB#473 (comment)

Package Maintainer

We have not defined a maintainer for this package yet...
I think we both have made substantial contributions to this package. Regarding the other packages, where I would in most cases see you as "main-maintainer", I can take maintainance over for this package, if you like. So we can share some "workload".

r2 lmer error for singular fit

it seems that r2_nakagawa fails when singular fit. Better would be to warn and return NA? Unfortunately, don't have any reproducible example at the moment.

ICC

Currently, icc() works for the packages lme4, * glmmTMB* and rstanarm. brms calculates the ICC in a different way, based on a variance-decomposition. For non-Gaussian models, this approach is also recommended by Stan team members. For non-Bayesian models (lme4/glmmTMB), the ICC for non-Gaussian models is based on some approximations of the variances (which is not perfect, but apparently the current state of the art).

Questions:

  1. Do we want rstanarm models behave like lme4 or brms? The latter would mean that non-Gaussian models fitted with rstanarm would return a more "Bayesian correct" ICC.

  2. For brms, we can't use the variance-calculation we use for the r2_nagakawa(). We could only calculate a comparable ICC for brms models when they are Gaussian and only have one random intercept. Do we want to cover this "edge case" by implementing extra code particularly for these types of models, or do we always want the variance decomposition for brms?

Let me give a comparison, only for the same brms-model

model4 <- brms::brm(Petal.Length ~ Petal.Width + (1 | Species), data = iris)

# Variance decomposition
performance::icc(model4)
#> $ICC_decomposed
#> [1] 0.7477364
#> 
#> $CI
#>      2.5%     97.5% 
#> 0.8543512 0.6156847 

# ICC only for linear models, with only random intercept
sjstats::icc(model4)
#> ## Species
#>           ICC: 0.94  HDI 89%: [0.86  1.00]

If you would use the current icc() for a mixed model from rstanarm, you get the ICC based on Nagakawa, as it would be returned for lme4 etc. as well:

performance::icc(model2)
#> $ICC_adjusted
#> [1] 0.8786005
#> 
#> $ICC_conditional
#> [1] 0.5603696

So, method 1) (Variance Decomposition) gives an ICC of .75, method 2) (simple ICC for linear models with random intercept only) gives ans ICC of .94 while method 3) (Nagakawa ICC) returns .88 and .56.

We could use method 1) or 3) for brms linear models with random intercept only, where method 1) is below (or between those two ICC values) and method 3) is above the values of method 2)...

  1. What values should be returned? We have the CI for brms-models, but only one "ICC" (which is not exactly an ICC, but something comparable, the variance decomposition in a Bayesian way...)

  2. How do we call what icc() returns for brms models? Still ICC, and explain this in the docs of icc()?

CRAN submission 0.2.0

Anything to do? I think we have a nice set of new functions and features, I would say, if at all, we could add a vignette (#47).

Any bayestestR must be submitted before.

Add additional information as attributes

For:

  • ICC
  • R2

(closing issues #12 and #10 in favor of this one)

We need:

  • a helper function, like r2_details() or icc_details() or similar
  • a class-attribute for all r2_*() and icc() return values, so we know which details we can expect

Rename functions

overdispersion() -> check_overdispersion()
zerocount() -> check_zeroinflation()

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.