solararbiter/solararbiter.github.io

Contains the Solar Forecast Arbiter static website

solararbiter.github.io's Issues

fill in metrics with basic info

probably start with a list of metrics and some references. can take from PMP and stakeholder materials. Order by deterministic, probabilistic, and cost.

let's not do too much until we've thought through it more carefully.

use cases contents has funky numbering

\1. Evaluate forecasts

\9. Analyze forecasts

\3. Perform forecast evaluation in a standard manner

\6. Administer the framework

adjust data model for regular observations

and allow missing values. Also add recommendations for resampling

brier score, quantile score descriptions

I was testing Quantile Skill Score in the reports and came across some results that initially confused me. Table below shows metrics for 4 pairs of forecasts

Forecast	BS	QS	BSS	QSS
Table Mountain Boulder CO Day Ahead GEFS ghi Prob(f <= x) = 0.0%	0.46	46.3	-0.113	0.653
Table Mountain Boulder CO Day Ahead GEFS ghi Prob(f <= x) = 40.0%	0.282	39.6	-0.161	0.688
Table Mountain Boulder CO Day Ahead GEFS ghi Prob(f <= x) = 50.0%	0.25	44.7	0.00e+00	0.643
Table Mountain Boulder CO Day Ahead GEFS ghi Prob(f <= x) = 100.0%	0.264	93.4	0.55	0.201
Table Mountain Boulder CO Hour Ahead Prob Persistence ghi Prob(f <= x) = 0.0%	0.415	129	nan	nan
Table Mountain Boulder CO Hour Ahead Prob Persistence ghi Prob(f <= x) = 40.0%	0.243	122	nan	nan
Table Mountain Boulder CO Hour Ahead Prob Persistence ghi Prob(f <= x) = 50.0%	0.25	121	nan	nan
Table Mountain Boulder CO Hour Ahead Prob Persistence ghi Prob(f <= x) = 100.0%	0.585	112	nan	nan

I came away from this wanting a few things in the brier score documentation

1. a statement about how the Arbiter computes o for non-event forecasts (is f <= x?).
2. some discussion about brier score vs. quantile score for quantile forecasts (like all of our reference forecasts as of today).

fix readme

update stakeholder page with links to emails

Mailchimp will let us link to a web page version of the emails that we send out. We could add those links to the stakeholder page as they come out. Maybe useful for summarizing public stakeholder engagement? Maybe just busy work? Here's what we have so far:

Use cases
Use cases revised

consider adding support for disqus comments

Only an idea.

Pros:

• centralizes information exchange on the website
• we don't have to manage forms or feedback through email (possibly in premade documents)

Cons:

• unstructured
• we don't control public data
• anonymous responses are not possible

Trials documentation improvements

Distilled from my notes and @dplarson's notes on the second test trial.

• Further clarify the provided python script is meant as a simple example to help users get started and is not recommended for a real trial.
• Add warning that a password reset link is single use and cannot be reissued without breaking anonymity.
• Email with forecast links are nice. A simple copy/pasteable list of uuids would also be nice.
• Send email notifications when a trial is about to begin and when it has ended.
• Include an example of a probabilistic forecast in the documentation.
• Clarify wording around the forecast submission window.

improve user supplied benchmarks

Discuss:

• Multiple user-supplied benchmarks
• Metadata for running benchmarks (WRF namelists, maybe scripts, other information for reasonable level of reproducibility)
• Reiterate that The framework-generated benchmarks will be GFS/NAM/HRRR forecasts with straight-forward post-processing. We anticipate that the framework-generated benchmarks will complement the benchmarks supplied by the TA2 teams.
• WRF 4.0 vs. WRF Solar 1.2 vs. WRF Solar 4.0
• Model configuration may depend on requirements for ramps and variability, so we should set targets sooner than later
• and maybe other things from communication with Larry

clearly document 'create' restrictions

mainly that when creating an object, the organization is set to the creating user's organization. Thus sharing of create permissions to user's outside of the role/permission org is pointless.

improve SETO funding reference

Probably need the program number.

Make benchmark forecasting page

karin et al climate zones

Consider supporting these climate zones presented at the 2019 PVSC. They're designed as zones for degredation analysis, but I think they'd work for forecasts too. 2019_pvsc_submission.pdf

python package supporting the paper: https://github.com/toddkarin/pvcz

clean up, prettify framework architecture schematic

The one at the bottom of https://solarforecastarbiter.org/documentation/framework/

home page should provide a more concrete description of the project

Users shouldn't need to dig to get a concrete idea of what we're doing. Maybe steal text/graphics from the SETO review poster and recent talks.

Add PICP and PINAW metrics

Somehow two probabilistic forecast metrics were mistakenly not added to the metrics page during previous PRs:

• prediction interval coverage probability (PICP)
• prediction interval normalized average width (PINAW)

These two should be added to the metrics page, with a separate issue/PR open on the core code repo for implementing the functions themselves.

document quality flag and data validation results

The only documentation we currently have for quality flags and the data validation toolkit is the core api. We need dashboard documentation that

1. explains the graphs on the dashboard's observation pages
2. explains how the data is validated when uploaded and on a recurring basis
3. explains the quality flags in the downloads

add more references to metrics

following example introduced in #108. Start with the references section of the April survey doc.

add link to implementation for each metric

Could link to...

the core function docstring: https://solarforecastarbiter-core.readthedocs.io/en/latest/generated/solarforecastarbiter.metrics.deterministic.normalized_mean_absolute.html#solarforecastarbiter.metrics.deterministic.normalized_mean_absolute

or to the RTD source: https://solarforecastarbiter-core.readthedocs.io/en/latest/_modules/solarforecastarbiter/metrics/deterministic.html#normalized_mean_absolute

I'm not aware of any way to cleanly link to the source on github based on function names.

site coordinate precision comic

Maybe include this xkcd in the documentation for how to specify site location, especially in an section that describes how to specify an anonymous plant location.

blog post on satellite data

We received a handful of questions about if we'd use satellite data to validate forecasts. We should record the response in a blog post.

add links to climate zone files in /assets

add links to AMS 2019 slides and poster

Maybe put it on the homepage?

https://ams.confex.com/ams/2019Annual/meetingapp.cgi/Paper/354714

review, reference, incorporate IEA Task 36 recommendations

https://www.ieawindforecasting.dk/

In particular, we can use the Task 36 work products to provide more guidance on evaluation scenario design.

add user stories or use cases

update data policies page

include links to Data Use Agreement, other docs

Update climate zone map

Update the climate zone map with more refined outlines of the climate zones, preferably in the form of a GIS-type file (e.g. a shapefile) that can be re-used in the future.

• refine the climate zones
  • save in a re-usable GIS-type format (e.g. a shapefile)
• update the climate zone plot
  • replace the current figure file with the revised one (PNG or JPG format)

Update permissions reference table

Add Site data type and Update permissions to the permission reference table found at https://solarforecastarbiter.org/documentation/dashboard/administration/#permissions-reference-table

Correct typo in CRPS definition

Correct the CRPS definition on the metrics page which should use (F_i - O_i)^2, not |F_i - O_i|^2.

Broken link on Data Model page

Due to a line break, the [Definitions](../definitions/#probforecastdef) link under the Data Models page (Metadata >> Forecasts >> Probabilistic forecasts: https://solarforecastarbiter.org/datamodel/#probabilistic-forecasts) shows up as plaintext instead of as an HTML link.

Remove the leeward islands from region 9

They're mostly uninhabited and we can greatly reduce the area of the models we have to save

rename data rights to data policies

I think the "Data Rights" page should be renamed "Data Policies". I'll do this by end of week unless I hear objections.

improve documentation of reports

The report documentation should include

• Considerations around specifying probabilistic reports
• What to expect in a deterministic report, an event report, and a probabilistic report
• The list under "following these steps" should bold the action sections.
• Clarify the process around sites vs aggregates

I also want to consider moving the report subsection from the "working with data" section into its own top level section. Maybe add some examples.

(I couldn't remember the status of the event reports, so I tried to look it up in the documentation.)

improve data sharing documentation

https://solarforecastarbiter.org/documentation/dashboard/administration/ is a fine reference. Especially for data sharing, we need more documentation that builds understanding and provides concrete examples.

I've thought highly of the approach described by https://documentation.divio.com/ since I saw their PyCon talk a few years ago, but I haven't actually tried to implement it. So here are some ideas for new documentation in that framework that are specific to data sharing:

How-to:

• How to allow users within your organization to upload data
• How to share a site and an observation
• How to share a forecast
• How to share a report (metrics-only)
• How to share a report (including time series)

Tutorials:

• How to share a report, including time series, but in more detail?

Explanation:

• Why is our permissions system structured the way it is
• How does the system work (api/sql land)
• Strategies for how forecast users and vendors can set up permissions and roles. For example, vendors might choose to set up roles for each client rather than for each site. Users might choose to use "applies to all".

If we like the approach we can blow up and reassemble the entire documentation, but that's more work than I want to commit to right now.

update data model with correct columns

to mirror api/database. For example, sites no longer have network or well known text but do have extra parameters. Should also update the number of new aggregate parameters.

improve event metric description

A stakeholder was confused about our event metrics and thought that they were only applicable to solar power ramps. We could improve the writing a little and add a second example such as clear/cloudy.

update dashboard documentation for real users

The getting started section describes how to use test log in information. Not sure if we want to keep the test info, but we definitely need to say how to use real log in.

Perhaps also move the data access control help page's account creation and establishing or joining an organization sections into the primary dashboard help page.

Dashboard interaction Page

• Make Dashboard Interaction Page.
• Add link to Data Model Introduction.

reference data map needs to be checked against the database

For example, the reference data map shows a SRML site in Richland but the database does not include it because the data is too old (1996).

Perhaps the thing to do is recreate the google map using a metadata dump from the database. Or perhaps just do SolarArbiter/solarforecastarbiter-dashboard#286

update home page

Should explain what this page is (static information about the project, help pages) and link to the dashboard. Remove references to early development and June 2019.

Brier Score decomposition: typo

On the metrics page, the Brier Score decomposition is (incorrectly) listed asBS = REL + RES + UNC, when it should actually be BS = REL - RES + UNC (i.e. subtract RES, not add RES).

This was a typo from when I wrote up the Word report on metrics in Markdown.

improve metrics

catch all issue for things I want to change on the metrics page.

new reference: https://www.wpc.ncep.noaa.gov/html/hpcverif.shtml worthwhile because it's standard NCEP product.

number of forecasts that failed to be submitted by the submission deadline

Add notes on cost metrics

Add content on cost metrics (i.e. the financial costs of mis-forecasting) to the website, based on the document shared by Aidan. Probably add the content to the end of the Metrics page (unless a more obvious place is identified). The content should provide general guidance and mirror the notation used in other parts of the website (e.g. the error metric definitions).

add license

MIT?

Copyright Solar Forecast Arbiter Team?

Replace metrics document with HTML

Currently, the metrics page links to a document that summarizes the metrics. Instead, it would be better to list out the metrics on the metrics page itself (e.g. as a bullet point list).

The main question with this approach is how to represent mathematical symbols (e.g. the square root in the RMSE definition). There are libraries for rendering LaTeX in an HTML page (e.g. MathJax), which may be better than trying to use an ASCII-representation (e.g. RMSE = sqrt( 1/n * sum( (F_i - O_i)^2 ) )).

Document exatly what each permission does

Not sure if this belongs here, but we should document what permissions allow. For example, having the write_values permission allows one to write values to specified id but also allows one to read the interval_length of the parent object and to get the last value uploaded before a certain time.

add reference data map

We should add a link or even embed the @lboeman's nice reference data map. It's not clear to me where it should go. We might need a new Reference Data section. Thoughts?

comment form

Add a comment form to allow people to submit comments anonymously.

This will require something like google's recaptcha system.

allow users to sign up for a mailing list

Implementation depends on what kind of system we use for a mailing list. Might only need a link to another page. That's a separate discussion.

And no signup feature unless they can also unsubscribe.

add nmae, nmbe to metrics, clarify normalization

Dashboard currently allows for the following report options:

MAPE
NMAE
NMBE
NRMSE

and links to the relevant sections of https://solarforecastarbiter.org/metrics . But the links for NMAE and NMBE are broken since they don't exist on the page.

We should also clarify the normalization rules as discussed in SolarArbiter/solarforecastarbiter-core#269

@dplarson can you take this on?