⚖Open Source Toolkit for Quantitative Risk Assessment
Home Page: https://evaluator.tidyrisk.org
License: Other
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
When using the following command required the following packages:
Command: explore_scenarios(input_directory, results_directory)
DT
statip
flexdashboard
SUGGESTION: add these packages along with the others for evaluator.
The README is currently written for the old initial evaluator release. Need to rewrite to bring this current with the usage vignette directions, which are much more comprehensive.
Reports are awkward to customize styles and fonts. Investigate CSS & font overrides to improve the user experience.
Promoting this to an issue rather than a comment buried in a closed issue (#26)
I am trying to model events with a TEF of (min = 0.0, mode = 0.1, max = 0.5) i.e. expected is once in 10 years and max is once in 2 years.
I'm finding that sample_tef() returns all zeros as the function converts the rpert distribution into a rounded integer. As all values in this pert distribution are below 0.5 they will be rounded to 0.
max(as.integer(round(mc2d::rpert(100000, 0, 0.1, 0.5)))) >= 1
Any thoughts or suggestions on how to overcome this problem or whether I have misunderstood the application of sample_tef()
Regards Ross
Originally posted by @rah in #26 (comment)
Getting errors with run_simulations(quantitative_scenarios)
Using steps outlined and packaged data. No changes to any table or spreadsheet.
Running Mac OSX 10.13.3 as admin user
Rv3.4.1
Rstudio 1.0.153
Next steps: Duplicating procedure on multiple OSs to see if I can duplicate error.
PS: Evaluator is inspirational.
Final Update: I managed to not duplicate these errors in Windows 10 VM. The difference was not the OS, but just in terms of how I read the instructions. I kept interpreting references to methods in the paragraph dialogue as steps I should take immediately. I realized going through the second time that these were condensed references to the main code for that section and that I should follow along in the code portion only. Moving on to analyzing results of my first simulation.
[Deprecated] There are multiple errors:
"xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
Error: .onLoad failed in loadNamespace() for 'tcltk', details:
call: dyn.load(file, DLLpath = DLLpath, ...)
error: unable to load shared object '/Library/Frameworks/R.framework/Versions/3.4/Resources/library/tcltk/libs/tcltk.so':
dlopen(/Library/Frameworks/R.framework/Versions/3.4/Resources/library/tcltk/libs/tcltk.so, 10): Library not loaded: /opt/X11/lib/libX11.6.dylib
Referenced from: /Library/Frameworks/R.framework/Versions/3.4/Resources/library/tcltk/libs/tcltk.so
Reason: image not found
In addition: Warning message:
running command ''/usr/bin/otool' -L '/Library/Frameworks/R.framework/Resources/library/tcltk/libs//tcltk.so'' had status 1 "
Hello,
We're going to release vctrs 0.2.0 soon (perhaps on Monday). This update has a lot of internal changes and is no longer compatible with your package. Once it is on CRAN, evaluator will have to be updated for compatibility. Please see https://github.com/r-lib/vctrs/blob/master/NEWS.md for a list of breaking changes.
Thanks!
Hi team,
I would like to thank you for your development to make easier Quantitative Cyber Risk Analysis.
After installing Evaluator on a Debian VM and configured R environment. I am able to run the initial evaluator simulation from the templates provided and get the results on the different files, after that
When launching explore_scenarios command, i got the following errors on the individual risk scenarios view through web browser where:
1 - in the left pane under, Threat Profile, Controls and Loss Magnitude the following error : "Error: Column 1 must be named."
2 - in the right pane, under Loss Table,
in the Minimum colomn, Inf value for all scenarios
in the Maximum colomn, -Inf value for all scenarios
in the Mean and Mode colomns, NA values for all scenarios
Could you please, let me know what can i do to correct these errors?
NB: You can see bellow the screenshots of my browser and the results of R CLI
Kind regards
M
Refactor away from a hard-coded dependency on mc2d:bpert to a more functional specification where a sampling function can be passed in along with its optional dependencies (think purrr style). This opens up normal, log, and all sorts of interesting possibilities on modeling.
Hi there,
when I simplify the example provided and just keep two domains, the output reports are not generated anymore.
survey.xlsx
Are there some parameters I have to change in addition to the spreadsheet?
Notice that I just kept everything as is except for the 2 entries.
Cheers.
Progress bars were lost when the furrr dependency was dropped. This should be returned for at least multi-scenario simulations.
I have verified this to be the case on the docker image, as well as new-build.
Any major modifications to the template survey.xlsx or new tabs/domains in the spreadsheet generates this error when trying to generate a report:
generate_report("/evaluator/inputs", "/evaluator/results", "~/evaluator/risk_report.html")
...
|........................ | 37%
label: make_scenario_table (with options)
List of 1
$ echo: logi FALSE
|......................... | 38%
inline R code fragments
|......................... | 39%
label: scenarioA_details (with options)
List of 1
$ echo: logi FALSE
Quitting from lines 308-309 (analyze_risk.Rmd)
Error in summarise_impl(.data, dots) :
Column Value at Risk must be length 1 (a summary value), not 0
Hi there,
I am just running the demo as described in vignette.
I am getting two main issues as per screenshot: icon is missing and some column error.
OS: Win10
RStudio: Version 1.4.1106
I believe the first two lines are related to the logo icon, something about the file path?
Console errors:
Warning in file.create(to[okay]) :
cannot create file 'C:\Users\xxx\AppData\Local\Temp\Rtmp2hssV5\file4df065136f50\explore_scenarios_files/D:/Users/xxx/Documents/R/win-library/4.1/evaluator/rmd/img/evaluator_hex_48px.png', reason 'Invalid argument'
Warning in file.create(to[okay]) :
cannot create file 'C:\Users\xxx\AppData\Local\Temp\Rtmp2hssV5\file4df065136f50\explore_scenarios_files/D:/Users/xxx/Documents/R/win-library/4.1/evaluator/rmd/styles/html-styles.css', reason 'Invalid argument'
Warning in file.create(to[okay]) :
cannot create file 'C:\Users\xxx\AppData\Local\Temp\Rtmp2hssV5\file4df065136f50\explore_scenarios_files/D:/Users/xxx/Documents/R/win-library/4.1/evaluator/rmd/img/evaluator_hex_48px.png', reason 'Invalid argument'
Warning: Error in : Column 1 must be named.
Use .name_repair to specify repair.
144: <Anonymous>
Warning: Error in : Column 1 must be named.
Use .name_repair to specify repair.
141: <Anonymous>
Warning: Error in : Column 1 must be named.
Use .name_repair to specify repair.
149: <Anonymous>
Warning in min(scenario_data$loss_events) :
no non-missing arguments to min; returning Inf
Warning in min(scenario_data$sle_min) :
no non-missing arguments to min; returning Inf
Warning in min(scenario_data$ale) :
no non-missing arguments to min; returning Inf
Warning in mean.default(scenario_data$loss_events, na.rm = TRUE) :
argument is not numeric or logical: returning NA
Warning in mean.default(scenario_data$sle_mean, na.rm = TRUE) :
argument is not numeric or logical: returning NA
Warning in mean.default(scenario_data$ale, na.rm = TRUE) :
argument is not numeric or logical: returning NA
Warning in max(scenario_data$loss_events) :
no non-missing arguments to max; returning -Inf
Warning in max(scenario_data$sle_max) :
no non-missing arguments to max; returning -Inf
Warning in max(scenario_data$ale) :
no non-missing arguments to max; returning -Inf
@davidski, I was able to use scenario explorer, but I did have trouble generating reports. I'm sure I'm doing something stupid.
I. Commands and Errors
A. With Studio Viewer:
">" generate_report(input_directory, results_directory) %>% rstudioapi::viewer()
Error in rmarkdown::render(system.file("rmd", "analyze_risk.Rmd", package = "evaluator"), :
argument "output_file" is missing, with no default
B. With Attempting to Generate a Word Report
">" generate_report(input_directory, results_directory, output_format = "word_document")
Error in rmarkdown::render(system.file("rmd", "analyze_risk.Rmd", package = "evaluator"), :
formal argument "output_format" matched by multiple actual arguments
Evaluator currently expects to have TEF, TS, DIFF, and LM data. This hits several different levels of the FAIR taxonomy and doesn't lend itself to doing an analysis at different levels of granularity/summary. This is a tracking issue for a fairly large body of work to decompose the simulation functions, allowing a roll up from various points, ideally with either a series of distribution inputs or sampled values.
Encountered the following error when performing the "run_simulations()":
Error: 'by_row' is not an exported object from 'namespace:purrr'
It seems like data-frame based mappers (dmap(), dmap_at(), dmap_if(), invoke_rows(), slice_rows(), map_rows(), by_slice(), by_row(), and unslice()) have been moved to a new package, purrrlyr. Refer to the link below for more information:
https://github.com/tidyverse/purrr/blob/aea1ce446f3e06b28b2e867a10efbaa29a7498cb/NEWS.md
I am trying to launch Scenario Explorer and I am getting the following error in my browser:
"Error: pandoc document conversion failed with error 1"
Alicia explains very well why this happens when transitioning from an html document: ropensci/iheatmapr#4
This did not work: Alternative fix involving removing of Pandoc: rstudio/rmarkdown#1184 [devtools::install_github('rstudio/rmarkdown')]
Alternative fix involved installing a markdown package found here: https://stackoverflow.com/questions/47323389/pandoc-document-conversion-failed-with-error-1-after-update-to-r-version-3-4-2
The various markdown driven functions (generate_report and friends) do not have tests. Unsure how to easily create tests for these via testthat and mockery. Test coverage will have some limits on upper bounds until this is addressed (I would rather not exclude those functions from codecov reporting).
Typo at line 44 in encode.R where "scenarios" is mis-typed as "senarios"
# fetch LM params
scenarios <- left_join(senarios, mappings[mappings$type == "lm",],
by = c("lm" = "label")) %>%
rename_("lm_l" = "l", "lm_ml" = "ml", "lm_h" = "h", "lm_conf" = "conf") %>%
select_('-c(lm, type)')
This is causing an error when validating scenarios.
Current CRAN release does not work with current tidyverse. Cut a release with current master or wait for more complete refactoring?
Warning message:
package ‘evaluator’ is not available for this version of R
the markdown reports cannot find the files in the styles subdirectory when generating report
Part of the simulation cleanup removed the convert_quant_to_qual function in favor of a simple left-join. The Scenario Explorer app currently calls that function. Rework to use a simple left join.
In following the directions to generate a report using Rstudio viewer, I was prompted to install a number of packages that were not a part of the original evaluator package. This was trivial, but consider including these if appropriate:
psych
pander
ggalt
The tidyrisk pkgdown sites should have a prominent "part of tidyrisk" link in the header, similar to the tidyverse sites. Explore how the tidysite template accomplishes this and adapt the technique (note that tidysite is (C)rstudio with no open license).
I think that there is an incorrect variable called within summarize.R at lines 65-69, which means that the scenario_summary.rda file is not created
scenario_summary <- summarize_scenarios(simulation_results)
save(scenario_summary, file = file.path(results_dir, "scenario_summary.rda"))
domain_summary <- summarize_domains(simulation_results, domains)
save(domain_summary, file = file.path(results_dir, "domain_summary.rda"))
results_dir should be results_directory ?
I've been getting progressive issues when trying to run generate_report() as it will claim it cannot find rda files - the path seems to change between user Dir and R/win-library. Trying to confirm if this is my issue.
Cheers
Trying to get to grips with evaluator using the templates which works reasonably well using the bootstrap script apart from the fact that I'm getting the following errors:
Warning: Error in : Column 1 must be named.
140:
Warning: Error in : Column 1 must be named.
140:
Warning: Error in : Column 1 must be named.
140:
Warning in min(scenario_data$loss_events) :
no non-missing arguments to min; returning Inf
Warning in min(scenario_data$sle_min) :
no non-missing arguments to min; returning Inf
Warning in min(scenario_data$ale) :
no non-missing arguments to min; returning Inf
Warning in mean.default(scenario_data$loss_events, na.rm = TRUE) :
argument is not numeric or logical: returning NA
Warning in mean.default(scenario_data$sle_mean, na.rm = TRUE) :
argument is not numeric or logical: returning NA
Warning in mean.default(scenario_data$ale, na.rm = TRUE) :
argument is not numeric or logical: returning NA
Warning in max(scenario_data$loss_events) :
no non-missing arguments to max; returning -Inf
Warning in max(scenario_data$sle_max) :
no non-missing arguments to max; returning -Inf
Warning in max(scenario_data$ale) :
no non-missing arguments to max; returning -Inf
The image shows where it manifests itself
"mutate_each()" function is now deprecated.
Some of the tests are using tempdir(check = TRUE). The check parameter was added in R 3.5 and will cause builds to fail on older versions of R. Review and determine if we should take a dependency on R 3.5 or if we should remove this parameter and continue to support R >= 3.3.
Hi David,
First, let me start by thanking you for this amazing package. It has made what I am working on much much much easier and more organized. I have a couple questions:
First, in regard too TEF I am currently using the code below:
TEFestimate <- list(func = "mc2d::rpert", min = input$tefl,
mode = input$tefml, max = input$tefh,
shape = input$tefconf)
TCestimate <- list(func = "mc2d::rpert", min = input$tcapl,
mode = input$tcapml, max = input$tcaph,
shape = input$tcapconf)
DIFFestimate <- list(list(func = "mc2d::rpert", min = input$csl,
mode = input$csml, max = input$csh,
shape = input$csconf))
LMestimate <- list(func = "mc2d::rpert", min = input$lml,
mode = input$lmml, max = input$lmh,
shape = input$lmconf)
single_scen <- tidyrisk_scenario(
tef_params = TEFestimate,
tc_params = TCestimate,
diff_params = DIFFestimate,
lm_params = LMestimate)
values$simulation_result <- run_simulation(single_scen, iterations = values$iterations)
If I want to derive TEF from Contact frequency and propability of action, how can I do that?
Also if I want to include primary in secondary losses in this scenario how can I do that?
Again thank you so much for this!
Tibble 3.0 (and perhaps some of the more recent rlang releases) introduces breaking changes and evaluator fails. Working branch of tibble_update is in progress to work through the issues. https://github.com/tidyverse/tibble/blob/master/NEWS.md for more info.
Tooling is focused on getting a qualitative scenario into quantitative parameters. Ensure that more mature orgs have a straightforward path. Includes not just the importing, but also the reporting (which is also focused on quant scenarios).
Hi there,
I am fairly new to R and so please forgive my ignorance.
I launch the docker this way:
docker run -p 8789:8787 -e PASSWORD=secret -v $(pwd)/data:/home/rstudio tidyrisk/evaluator:latest
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] add: executing...
Nothing additional to add
[cont-init.d] add: exited 0.
[cont-init.d] userconf: executing...
[cont-init.d] userconf: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
This is all good then I go to the webpage: http://[myip]:8789/
I login all good and then I get this:
Do I have to run other components?
Is there a dockercompose file that I can use to launch the entire stack?
Cheers.
I have adjusted the code (openfair.R) "sample_tef" samples to allow for samples to be of type "double" and adjusted the L,M,H column type to col_double:
From - samples = as.integer(round(purrr::invoke(func, params))),
To - samples = as.numeric(round(purrr::invoke(func, params), 2)),
I have also adjusted the "mappings" to:
mappings <- readr::read_csv(file.path(inputs_dir, "qualitative_mappings.csv"),
col_types = readr::cols(
type = readr::col_character(),
label = readr::col_character(),
l = readr::col_double(),
ml = readr::col_double(),
h = readr::col_double()))
The simulation runs and I do see the TEF values, however, none of the other values populate (seems as if because it is not an integer of 1 or greater that the simulation assumes the TEF is a value of 0).
The encoding seems to work fine and the "Quantitative Scenarios" contains all of the expected variables.
I am unable to determine what I have missed.
Qualitative_mappings.csv
type,label,l,ml,h,conf
tef,frequent,.2,.3,.5,1
tef,occasional,.1,.2,40,1
tef,rare,.05,.1,.2,1
tc,high,6.5e+01,8.5e+01,9.5e+01,1
tc,medium,3.5e+01,5.0e+01,8.0e+01,1
tc,low,1.0e+01,2.0e+01,4.5e+01,1
diff,5 - Optimized,65,85,95,1
diff,4 - Managed,65,85,90,1
diff,3 - Defined,3.5e+01,5.0e+01,8.0e+01,1
diff,2 - Repeatable,20,45,60,1
diff,1 - Initial,10,20,45,1
lm,high,1860300,3780150,5700000,1
lm,medium,20600,1860300,3700000,1
lm,low,2060,20600,2100000,1
Prepare for release:
devtools::check()devtools::check_win_devel()rhub::check_for_cran()revdepcheck::revdep_check(num_workers = 4)Submit to CRAN:
usethis::use_version('minor')cran-comments.mddevtools::submit_cran()Wait for CRAN...
usethis::use_github_release()usethis::use_dev_version()Build an options framework with initial support for the scales::dollar prefix (EUR, GBP, etc.).
Please briefly describe your problem and what output you expect.
type mismatch, using docker image.
Brief description of the problem
Following the instructions here I am getting a type mismatch:
> validate_scenarios(qualitative_scenarios, capabilities, domains, mappings)
Error: Can't join on 'controls' x 'capability_id' because of incompatible types (character / numeric)I don't really know R but it seems that the helper function either isn't casting the controls to a chr, but I can't figure out how to fix this.
I was trying to follow this guide. However, the run_simulations function throws this exception "Error: All scenarios must be tidyrisk_scenario object".
Below, is the snippet from the code I ran:
library(collector)
library(evaluator)
create_templates(".")
domains <- readr::read_csv("./inputs/domains.csv")
import_spreadsheet("./inputs/survey.xlsx", domains, output_dir="./inputs")
qualitative_scenarios <- readr::read_csv("./inputs/qualitative_scenarios.csv")
mappings <- readr::read_csv("./inputs/qualitative_mappings.csv")
capabilities <- readr::read_csv("./inputs/capabilities.csv")
validate_scenarios(qualitative_scenarios, capabilities, domains, mappings)
quantitative_scenarios <- encode_scenarios(qualitative_scenarios, capabilities, mappings)
simulation_results <- run_simulations(quantitative_scenarios, simulation_count = 1000)I'm sitting down this morning to figure out how to incorporate the numbers from Cyentia's IRIS 2020 report into my org's Evaluator workflow; any strong opinions about how best to do that?
At first glance, the Loss Magnitude values seem pretty straightforward since they're at the same level of FAIR, so I'm mostly looking at how they might interact with TEF in qualitative_mappings.csv.
Some ideas I'm throwing around:
analyze_risk.Rmd reportI get this error message running the vignette code:
`simulation_results <- run_simulations(quantitative_scenarios,
iterations = 100L)
simulation_results_error.zip
`
Investigate some returning some basic sensitivity analysis in the details section for at least DIFF estimation (providing guidance to the user as to which capabilities, if improved, would provide most impact).
Hi
I've discovered what I think is a problem when I try to run generate_report(). When processing analyze_risk.Rmd the params set for input_directory and results_directory are set as:
params:
input_directory: "/Users/dseversk/rstats/evaluator/vignettes/data"
results_directory: "/Users/dseversk/rstats/evaluator/vignettes/results"
focus_scenario_ids: !r c(51, 12)
This is then used at lines 64,-65
input_directory <- params$input_directory
results_directory <- params$results_directory
when loading data at line 69, I get errors as the directories are incorrectly set.
Looking at https://github.com/davidski/evaluator/blob/master/vignettes/usage.Rmd should it read instead
params:
input_directory <- "~/evaluator/data"
results_directory <- "~/evaluator/results"
focus_scenario_ids: !r c(51, 12)
Apologies in advance if this is incorrect, I am completely new to R.
Cheers!
Kevin
I am receiving the errors below when generating a report using this code:
generate_report(input_directory, results_directory) %>% rstudio::viewer()
I previously defined results_directory as "simulation_results.rda" per the instructions.
The heart of the error below seems to be that it is looking for a file named "scenario_summary.rda", which I never created.
Next steps: I'm going to create a copy of the file I have with this new name in the directory.
"Quitting from lines 68-78 (analyze_risk.Rmd)
Error in readChar(con, 5L, useBytes = TRUE) : cannot open the connection
In addition: Warning message:
In readChar(con, 5L, useBytes = TRUE)
cannot open compressed file 'C:/Users/username/Documents/evaluator/results/scenario_summary.rda', probable reason 'No such file or directory'
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.