nanxstats/stackgbm

🌳 Stacked Gradient Boosting Machines

stackgbm

stackgbm offers a minimalist, research-oriented implementation of model stacking (Wolpert, 1992) for gradient boosted tree models built by xgboost (Chen and Guestrin, 2016), lightgbm (Ke et al., 2017), and catboost (Prokhorenkova et al., 2018).

Installation

The easiest way to get stackgbm is to install from CRAN:

install.packages("stackgbm")

Alternatively, to use a new feature or get a bug fix, you can install the development version of stackgbm from GitHub:

# install.packages("remotes")
remotes::install_github("nanxstats/stackgbm")

To install all potential dependencies, check out the instructions from manage dependencies.

Model

stackgbm implements a classic two-layer stacking model: the first layer generates "features" produced by gradient boosting trees. The second layer is a logistic regression that uses these features as inputs.

Related projects

For a more comprehensive and flexible implementation of model stacking, see stacks in tidymodels, mlr3pipelines in mlr3, and StackingClassifier in scikit-learn.

Code of Conduct

Please note that the stackgbm project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

stackgbm's Issues

Dependencies

As catboost and lightgbm are not CRAN packages, you should probably move them to Suggests dependencies and add some requireNamespace (cf. http://r-pkgs.had.co.nz/description.html).

Add code of conduct

Include code of conduct.

Add GitHub Actions workflow for pkgdown

Need to customize the r-lib/actions pkgdown workflow (to install catboost) so the vignette and code examples can render with results. This would save a lot of manual running of pkgdown and reduce repo size.

Revisit license

Perhaps MIT would be a more permissive choice.

lightgbm uses MIT; xgboost and catboost both use Apache 2.0.

Add code examples for all exported functions

As title suggests

Release stackgbm 0.1.0

First release

• Proof read Title: and Description: and ensure they are informative
• Check that all exported functions have @returns and @examples
• Check that Authors@R: includes a copyright holder (role 'cph')
• Review extrachecks
• usethis::use_cran_comments() (optional)

Prepare for release

• Check current CRAN check results
• Check licensing of included files
• Review pkgdown reference index for, e.g., missing topics
• Bump version
• Update cran-comments.md (optional)
• Update NEWS.md
• Review pkgdown website
• urlchecker::url_check()
• Check with local machine
• Check with GitHub Actions
• Check with win-builder

Submit to CRAN

• Draft GitHub release
• Submit to CRAN via web form
• Approve emails

Wait for CRAN

• Accepted 🎉
• Blog post

Improve example in vignette

The current example in the vignette gives better results on macOS but not on Linux.

Could use a better example that shows some consistency.

Pass CV parameter objects to `stackgbm()` directly

Make the stackgbm() less tedious by passing params objects directly.

May need to make them S3 objects and parse them.

Thanks @svm-zhang for the suggestion.

Add contributing guidelines

Add contributing guidelines.

Future-based parallel over parameter grid

• My goal is to maximize CPU utilization as only increasing n_threads doesn't seem to make that happen.
• The idea is to refactor the for-loop with foreach + %dofuture% loops to parallel over the parameter grid.
• Remove the seed argument to let users set.seed() outside.
• Let users configure the future parallel backend outside (or not, to use sequential).
• Use progressr (%dofuture% compatible) to display progress bar.
• Would probably recommend to set n_threads to 1 when using this parallel.
• A vignette to explain these would be nice.

Optimize default parameter grid

Previous grid:

• n_iterations = c(10, 50, 100, 200, 500, 1000)
• max_depth = c(2, 3, 4, 5)
• learning_rate = c(0.001, 0.01, 0.02, 0.05, 0.1)

Optimized grid:

• n_iterations = c(100, 200, 500, 1000)
• max_depth = c(3, 5, 7, 9)
• learning_rate = c(0.01, 0.05, 0.1, 0.2)

These values follow the common ranges of the three parameters, and provide a balance between model complexity, risk of overfitting, and training time.

Add GitHub Actions workflows

We should add GitHub Actions workflows for R CMD check.

Rethink the parallel parameter

For example, consider having this parameter as a global option instead of a function argument.

• To save repeated typing - set it once and for all.
• This is mostly a side effect not controlled by R but the underlying boosting libraries.

Need to do some research on best practices and decide.