apel/grafana-dashboards

Grafana dashboards for the IRIS, SCD Cloud, RAL Tier1, and SeaDataCloud accounting portals.

grafana-dashboards

Grafana dashboards for the IRIS, SCD Cloud, RAL Tier1, and SeaDataCloud accounting portals.

Directory Structure

• /dashboards - Dashboard JSON files that go into the directory specified in the provisioning files (/var/lib/grafana/dashboards/ in this case).
• /docs - Admin documentation.
• /img - Images used in the user guide panels which should be put in /usr/share/grafana/public/img so they can be accessed by the dashboard.
• /provisioning - Files that go into /etc/grafana/provisioning to define where Grafana can locate dashboard JSON files and what datasources it has access to. Note that the datasources file is a skeleton that would need the url, user and password added.
• /schemas - Contains SQL files used to set up and initialise additional databases used by the dashbaords.
• /scripts - Contains a utility script and config to apply a fixed colour mapping to all series within a dashboard JSON.

grafana-dashboards's Issues

Create IRIS-specific database schema with views on existing schemas

Having the data available with proper datetime fields should make working with Grafana simpler.

Publishing user guide

Including which fields are required and relevant info for data submission fields.

Projection table query doesn't take into account new year

When collecting the 'previous' value column, the current SQL query just minuses 1 from the month. This won't work in January. Fix using MySQL date_sub function.

Interval of data updates will lead to slightly inaccurate projection values

APEL tables are only updated once a day which causes misleading projections during certain periods of the day.

Fictional example:
Today's date: 08/11
Value so far this month for grid: 100
Projected value at the end of the month: 100/8 * 30 = 375
Let's say that the tables are updated at 2pm every day.
Tomorrow morning the projection graph will be recalculated as: 100/9 * 30 = 333.33
Arguably there is no reason for this dip as we do not know what the data will show for tomorrow (we should stick to today's value)

Possible Solution: use the latest update time in the table as today's date in the calculation.

Including allocation history for each activity

Somehow displaying the allocations for each VO over time, whether that be combined in a graph or in another format.

This will involve creating a new view in the iris_shared schema, likely containing a columns for VO, date, and allocation value.

Broken image rendering

Image rendering (when sharing a dashboard) has moved to a plugin now, but it doesn't seem to be working.

https://grafana.com/grafana/plugins/grafana-image-renderer

Combine cloud and grid filters

Maybe add a separate filter for type of system?
At least add a ‘none’ option for each type of accounting so that the one you don't want can be filtered out.

Add extrapolation chart for current month

Let’s Encrypt tidy up

Auto renewal cron of some sort needs setting up.

https://certbot.eff.org/docs/using.html#managing-certificates

Duplicated sites & VOs where both grid and cloud used

We need to figure out a more elegant way of handling these cases as it is confusing to users.

Also covers situations of different capitalisation for different types of resource.

Connect to IRIS IAM OAuth

Scrollbar on pie-chart table

Solution: increase height of the pie-chart panels

Add projection to main chart

As the main chart gets most of the eyeball time, it might be good to add an additional "estimated additional usage" data point to the latest month on top of the bar of the actual usage, in a suitably greyed-out colour, to show where we might be at the end of the month.

This probably shouldn't be added before we've fixed the issues with series that don't have data for intermediate months. Interpolation removed from later versions of Grafana so this shouldn't be a blocker.

Add allocation to Resource Manager projection table

Philip wants to be able to view allocations within the monthly projection table for each activity VO

Priority 4

Some email domains not able to log in with IAM

Someone with a ukaea.uk email address couldn't log in using IAM, being given the "Login Failed Required email domain not fulfilled" error.

This shouldn't be a problem as that domain is listed in allowed_domains. Does Grafana have a limit on the number of domains it checks from that setting, or does it require domains with a certain number of elements?

Provider project overall graph

Philip wants to see an overall graph of capacity and usage for the three provider projects (SCD, GridPP, DiRAC) over the last 12 months.

Priority 1

Resource manager chart y-axis limit too low

Resource manager charts have a fixed limit that the bars are now exceeding, so either the limit needs raising (c.f. future capacity), or should somehow be set dynamically (as the two charts needs to match).

Add a chart showing balance of over/under usage

It would be useful to have some view on what the balance of over/under usage is compared to the capacity of the IRIS infrastructure for the currently selected time period. (So need to calculate total usage vs. capacity for each month and sum up.)

The idea is to show the benefits of the flexibility in the infrastructure in that it shares resources with larger infrastructures, so some periods we use more, other times we use less, but overall we generally (hopefully) use our resources.

Related idea was maybe showing how IRIS resources are used when we're not using them, but we don't want to focus on our non-usage of our resources.

Double counting in Resource Manager chart?

Adding up both columns seems to suggest that some sites/VOs might be being included in both columns. This behaviour might be ok, but we should see if it's acceptable and if it is, then clarify it on the chart. Otherwise, we'll have to fix it.

Set Grafana role from IAM

Should be possible to set Grafana role (Admin, Editor, Viewer) from IRIS IAM groups:

https://grafana.com/docs/grafana/latest/auth/generic-oauth/#jmespath-examples

This can then be used to have general group (Viewer) that can view top level dashboards, and another group that can see privileged views (Editor), with Admin reserved for APEL team.

Create VO project filter table

Create table to specify what "project" a VO is part of for filtering purposes.

Probably two columns of VO + project. Only need to list IRIS ones at the moment, but good to be able to extend in future.

Add more detail to resource centre (+ activity?) view

Difficulty of validating data was raised at F2F meeting so if we can show the source of the data (SubmitHost?) then that would help.

(New monitoring system can also be extended to IRIS to help https://github.com/apel/monitoring)

Quattor-ise deployment

Dashboard user guide

It would be good to have a user guide for the dashboard, especially to point out the features that people might not notice/find by themselves (I'm thinking screenshots with arrows/boxes).

Improvements to Cloud and T1 views

• Filter down to just showing RAL data - ral-cloud for SCD Cloud views, and RAL-LCG2 for Tier1 views
• Remove IRIS VO filtering from Cloud T1 views
• Remove any unnecessary panels - e.g. panels and dashboards that show by-site views as each only shows one site
• Point T1 views at APEL main repo

Display capacity/allocation line for all time ranges

At the moment, if you select a time range that doesn't include at least two capacity or allocation points, no line will show. It would be nice if the capacity and allocation lines could display whatever the time range selected, but this is not a high priority issue.

Could be solved by adding a point for each month, but doing that in the database is messy and doing it in the dashboard is fiddly.

Add capacity table to iris specific schema and show in Grafana

Grafana supports overriding the visuals for regex selected series so we can add a table to store the capacity history (with date) and then add it to the main Grafana chart and show it as a line graph.

Graph tooltip displays incorrect values when missing data

When using the graph hover tooltip on the main top graph some of the values are incorrect because they are showing the nearest number rather than 0. This happens because there is no data for that month with that VO. Grafana still expects a database entry for them (either with null or 0 value) with which to populate the graph. Because of the way the data comes in, month-VO combinations with no data to show have no database entries.

Possible solution would be to insert dummy record summaries for month-VO combinations with no data.

SCD Cloud + RAL Tier1 dashboards

Depends on #40.

Once #40 is implemented, we can add SCD Cloud and RAL Tier1 dashboards (probably in separate folders, which will hopefully simplify permissions).

Check that all allocations added

Know that CCP4 hasn't been added. There are a few other that don't have any usage - what to do about them? as we don't know what name they'll actually be using.

Get cert for HTTPS

Explain Capacity and Allocation in user guide

Combined VO list still includes 'None' option

Add links back to IRIS website, STFC, other?

Table of Month,Site,Activity,Usage

PJ raised this a little while ago, but it's a lower priority than the recent requests. He'd like to be able to see/extract a table of usage by activity by site by month.

Priority 5.

Possibly map virgo to LIGO

LIGO are apparently doing their work under the virgo VO.

Mapping it would be a solution, but not the best, in case they do correct how they submit work and virgo start doing their own work.

Improve clarity of units

Add a fixed colour mapping for each VO/site

It would be good if the colours for each VO and site were consistent across all the panels.

Remove unnecessary time range options

Any time option less than 30 days doesn't make a lot of sense, so the list could be tidied up. Looks like the options are in the JSON.

Tidy up bottom charts

These charts haven't been touched for a while and don't include the cloud data (this may be why the colours don't match between charts). They should also be updated to use bars (rather than a line).

See if possible to log logins

May need to log all HTTP access and Logstash it:

https://grafana.com/docs/grafana/latest/installation/configuration/#router-logging

Filter data by selected date range

There seem to be some issues with the display of data in the Dashboard, with the y-axes being overly compressed. This seems to be due to the way Grafana is handling the data for EUCLID, dirac and gaia-dev. If these series are hidden, then the charts scale correctly.

Missing zero values cause display artifacts

Grafana has the annoying feature that it tries to interpolate values in stacked charts if there's no data for a particular key for a particular date. To work around this we could add zero values somehow - perhaps as a default zero entry for each site and VO combination for each month. There seemingly is an option in the Grafana chart settings about how these are handled, but it doesn't work.

Filter by provider project

Philip wants to be able to filter sites using provider project (SCD, GridPP, DiRAC) tags, this will update to show provider project capacity usage at each site within that provider

Priority 2

Ensure all panels are using the new combined view and project filter

Follow on from #78. Ensure all panels are using the new combined view VCombinedSummaries and project filter.

Links on pie charts not working

They just seem to take you back to the main view.

(If it's not possible to get them working, then we could just deprecate them.)

Add data source filter

Follow on from #78. Add data source filter variable (cloud/grid) which can be used to select on the SourceSchema column in VCombinedSummaries.

Cumulative usage by activity VO

Philip wants to have access to a table that contains the cumulative usage and cumulative allocation amount for each activity VO over a chosen time period

Priority 3

Create Python script to assign colours to panels

Add GRACC style bin/bucket size (1M, 1Q, 1Y, auto?)

Top left of https://gracc.opensciencegrid.org/dashboard/db/site-summary?orgId=1 there's a "Bin" option that groups the data into larger aggregations than 1 day.