teamdatatonic/looker-gcp-control

A Looker block for analysing Google Cloud Platform Billing and BigQuery data access logs

looker-gcp-control's Issues

not 100% sure here but total credits measure needs sql_distinct_key set

Otherwise it uses the pk which means since its unnesting over this key it throws key clash error.
I set it to sql distinct key to creditid and it looks right to me!?

Support the newer BigQueryAuditMetadata format

Google introduced a new format for audit logs: the BigQueryAuditMetadata as opposed to the older AuditData format which is supported by the model of this block.

Google recommends to use the new format.
Please note that the table names used by the log sink when exporting the new logs and the old logs are the same (but the schema is different!), so if you want to keep both you will need to use separate datasets. :-(

Can someone explain how to use the provided looker model files?

Docs don't explain how to actually use the model files.

I already have a billing dashboard setup so I just want to get the BQ utilization stuff.

I see three different files that appear to be for this, how do I use them with looker?

• bigquery_audit.dashboard.lookml
• bigquery_data_access.view.lkml
• gcp_bigquery_logs.model.lkml

Need GKE model view of looker

Hi Team,

Just wanted to know if there is any GKE model view of looker ML. We are exploring Looker as an alternative that would connect to our GKE metering usage BQ dataset, so that we can build dashboards on the top of it. We have already explored GCP Billing part. Let me know if there is any datatonic template for GKE as well.

Thanks.

New Billing Table Avaliable

https://cloud.google.com/billing/docs/how-to/export-data-bigquery-tables#data-schema

cloud_pricing_export is a new billing table that can land with the log sync. This deals with discounts applied (not credits) to your GCP billing, which has been a request from clients.
Help required to add this to the existing model, should anyone in the community wish to do this.

Data structure mismatch

I tried setting this up but the current Stackdriver export doesn't seem to match what is here. I had to update my database naming convention to match:

x.y.cloudaudit_googleapis_com_activity_* instead of:
x.y.cloudaudit_googleapis_com_data_access_*

Which is easy enough. Once updated though, I'm getting errors on the structures contained in the _payload.servicedata_v1_bigquery nest.

Is this code potentially out of date with the current export format?

bigquery cost 10% out

credit to gordonball for highlighting

GB usually refers to 10^9, with GiB meaning 2^30, but in Google’s docs they say: “Storage usage is calculated in gigabytes (GB), where 1 GB is 2^30 bytes. This unit of measurement is also known as a gibibyte (GiB)” which means according to Google, GB = GiB = 2^30

This means the cost is $5 per 240 bytes and not $5 per 1012 bytes

Fix iminent. Note this will change back dated costings too.