dynatrace-oss/dynatrace-gcp-monitor

Dynatrace integration for Google Cloud Platform monitoring

Dynatrace integration for Google Cloud Platform monitoring

This is the home of dynatrace-gcp-monitor which provides the mechanism to pull all Google Cloud metrics and Cloud logs into Dynatrace.

This integration consists of K8s container and few auxiliary components. This setup will be running in your GCP project and will be pushing data to Dynatrace. We provide bash script that will deploy all necessary elements.

To deploy this integration, see the docs on how to run it on Google Kubernetes Engine.

(legacy info) Integration renamed from GCP Function to GCP Monitor, starting from version 1.1.0

The whole project was renamed, so all internal references and files are now named dynatrace-gcp-monitor. This includes created resources names and docker images. Previous versions will still be available (but not updated), under the name dynatrace-gcp-function.

(legacy info) Cloud Function deployment is deprecated

Up until version 1.1.8, there was an option to deploy GCP integration as a Cloud Function (instead of a K8s container). It is now deprecated and has no support. If you are using this kind of deployment, please refer to this migration guide.

(legacy info) Migrating to 1.0.x from previous 0.1.x installations

If you already have a previous version of dynatrace-gcp-monitor deployed, please refer to this migration guide before installing the latest version.

Pricing

• Ingested metrics will consume DDUs. For more details GCP service monitoring consumption
• Ingested logs will consume DDUs. For more details Log monitoring consumption

Support

Before you create a ticket check troubleshooting guides specific to your deployment.
If you didn't find a solution please contact Dynatrace support.

Additional resources

• Architecture overview of Kubernetes deployment
• Monitoring multiple projects
• Expand monitoring in a Kubernetes container
• Self-monitoring in Google Cloud for metrics
• Self-monitoring for logs
• Dynatrace Azure Log Forwarder
• Dynatrace AWS log forwarder

Contributing

See CONTRIBUTING for details on submitting changes.

License

dynatrace-gcp-monitor is under Apache 2.0 license. See LICENSE for details.

dynatrace-gcp-monitor's Issues

Setup script failing to create self-monitoring dashboards

Heyo, folks.

We recently tried installing this function to our GCP environment only for it to fail to create the self-monitoring dashboards within GCP. This is the error we're seeing:

- create self monitoring dashboard
cat: dashboards/dynatrace-gcp-function_self_monitoring.json: No such file or directory
ERROR: (gcloud.monitoring.dashboards.list) Term operand expected [displayName: *HERE*].
ERROR: (gcloud.monitoring.dashboards.create) argument --config-from-file: Unable to read file [dashboards/dynatrace-gcp-function_self_monitoring.json]: [Errno 2] No such file or directory: 'dashboards/dynatrace-gcp-function_self_monitoring.json'
Usage: gcloud monitoring dashboards create (--config=CONFIG | --config-from-file=CONFIG_FROM_FILE) [optional flags]
  optional flags may be  --config | --config-from-file | --help
For detailed information on this command and its flags, run:
  gcloud monitoring dashboards create --help
- cleaning up
~
- removing archive [dynatrace-gcp-function.zip]
- removing temporary directory [dynatrace-gcp-function.zip]

It looks like the issue is on lines 276 & 280 of setup.sh. It's trying to pull the files from the dashboard folder but it's nowhere near that folder.

If this is only happening because we're attempting to install it incorrectly, then I believe the docs need to be updated to reflect this because we were simply following these docs and ran into this issue.

Failed to load configuration file

getting following error messages for all yaml files in config folder
Failed to load configuration file: '/Users/lidsc82/Workspaces/cloudFunction/local-dyn-gcp-function-venv/source/src/config/cloud_function.yml'. Error details: unsupported type for timedelta seconds component: str

this is happening for all GCP services

metrics are not continuously polled/ingested for every minute

Hi, team,

I am deploying dynatrace-gcp-monitor as Kubernetes pod and polling metrics from a GCP monitoring scoping projects. In the scoping project, there are about 10 projects monitored. dynatrace-gcp-monitor polls metrics from the scoping project together with all monitored 10 projects.

Initially, I set the polling interval as 3 minutes, in the running log of dynatrace-gcp-monitor, I find there are errors like:
[ERROR][MAIN_LOOP] Single polling timed out and was stopped, timeout: 300s

Based on the code, I find the timeout minutes is interval + 2min, then I increase polling interval to 6 mins, there are also some errors like:

[ERROR][MAIN_LOOP] Single polling timed out and was stopped, timeout: 480s

Once the metrics are ingested into Dynatrace, I find the gathered metrics are not continuously shown in the graph and there are sometimes 2 minutes or 3 minutes gap with no metrics. Looks like once polling is time out, the polled metrics are dropped and will not be processed and ingested into Dynatrace backend.

I have thought maybe I added too many monitored project into the scoping project. But based on doc https://www.dynatrace.com/support/help/setup-and-configuration/setup-on-cloud-platforms/google-cloud-platform/gcp-integrations/gcp-guide/monitor-multiple-projects, "This approach should work up to roughly 375 moderately sized projects."

Could you please advise what configuration should be checked to make sure we can get the metrics are continuously polled and ingested for every minutes?

Thanks.

Dashboards are not imported when using k8s deployment

When I use k8s deployment to run the integration then dashboards are not being installed.

Creating missing metric descriptor

Even after giving the project.editor role, getting permission error....

• commented few lines in run_docker.py
  site = web.TCPSite(runner, '0.0.0.0', 8080) loop.run_until_complete(site.start())

Manually executed on pod - python run_docker.py

Starting execution for project(s): ['xxx']
Selected services: k8s_container,k8s_node,gce_instance,k8s_cluster
Trying to use default service account
Successfully obtained access token
Starting processing...
Finished fetching data in 1.666172742843628
Ingest response: {'linesOk': 486, 'linesInvalid': 0, 'error': None}
Finished uploading metric ingest lines to Dynatrace in 0.6894481182098389 s
Fetched and pushed GCP data in 2.3575189113616943 s
Processed 486 lines
Pushing self monitoring time series to GCP Monitor...
Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/connectivity'
Failed to create descriptor for 'custom.googleapis.com/dynatrace/connectivity' due to '{'error': {'code': 403, 'message': 'Permission monitoring.metricDescriptors.create denied (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}'
Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/ingest_lines'
Failed to create descriptor for 'custom.googleapis.com/dynatrace/ingest_lines' due to '{'error': {'code': 403, 'message': 'Permission monitoring.metricDescriptors.create denied (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}'
Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/request_count'
Failed to create descriptor for 'custom.googleapis.com/dynatrace/request_count' due to '{'error': {'code': 403, 'message': 'Permission monitoring.metricDescriptors.create denied (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}'
Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/phase_execution_time'
Failed to create descriptor for 'custom.googleapis.com/dynatrace/phase_execution_time' due to '{'error': {'code': 403, 'message': 'Permission monitoring.metricDescriptors.create denied (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}'
Failed to push self monitoring time series, error is: 403 => {'error': {'code': 403, 'message': 'Permission monitoring.timeSeries.create denied (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}
Execution took 4.200105667114258

WHEN NOT COMMENTED LINES in run_docker.py, we get below error

Monitoring disabled. Check your project(s) settings.
Traceback (most recent call last):
File "run_docker.py", line 99, in
loop.run_until_complete(site.start())
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/aiohttp/web_runner.py", line 100, in start
self._server = await loop.create_server( # type: ignore
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1463, in create_server
raise OSError(err.errno, 'error while attempting '
OSError: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8080): address already in use
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f8264864610>

Config folder is missing in source code

Code used to have a folder called Config, which contains list of Yaml files for every GCP service. in the latest version, this config folder is missing, but when we are running the code locally.. yaml files listed in config folder are referrenced.

Two risks found with aqua scanner scanning

With aqua/scanner:2022.4.217 to scan image dynatrace/dynatrace-gcp-monitor:release-1.1.1, there are two risks with actions needed.

Could you please take a look to see if can fix them in next release?

Appreciate!

YQ 4+ required but script doesn't proceed

Folks,

First of all the instructions here: https://www.dynatrace.com/support/help/shortlink/deploy-gcp downloads version 3.x.x of yq

Secondly, the script doesn't allow the installation without version 4+

Then after installing yq version 4+, the script still fails... Take a look here:

Self Monitoring project

Am i correct in assume that the selfMonitoring abilities of the gcp-function are specifically tied to a project name 'dynatrace-gcp-extension'?

It seems that the create metrics call will always default to the context.project_owner_id that is pass thru the MetricsContext from, initially, HandleEvent in main.py that default it to dynatrace-gcp-extension.

As far as I can see. this project is also being used to get the dynatrace api keys, which is odd to me because we don't have this project.

How do I direct this function to make the metrics in the GCP_PROJECT env variable specified project?

How to view custom metrics in GCP after selfMonitoringEnabled is set to true?

https://github.com/dynatrace-oss/dynatrace-gcp-monitor/blob/master/k8s/helm-chart/dynatrace-gcp-monitor/values.yaml

Send custom metrics to GCP to diagnose quickly if your dynatrace-gcp-monitor processes and sends metrics/logs to Dynatrace properly.

Allowed values: "true"/"yes", "false"/"no"

selfMonitoringEnabled: "false"

How do we view custom metrics in GCP after we set selfMonitoringEnabled to true?

RFE: dt.security_context attribute

Is your feature request related to a problem? Please describe.
The attribute dt.security_context is important for data segmentation, analysis, and permission mapping within the Dynatrace platform. Currently, the dynatrace-gcp-monitor does not set or define this attribute on the metrics and logs sent from this integration. As a result, by default, the dt.security_context attribute cannot be used. As a workaround, processing rules in the Dynatrace environment can be configured to parse data and apply a value to dt.security_context. However, this requires elevated access to the Dynatrace environment and hinders self-service deployment and adoption of Dynatrace for GCP.

Describe the solution you'd like
As a first step, setting the dt.security_context to a static value during the helm chart deployment should be possible by specifying the desired value in the values.yaml. This should apply the desired dt.security_context attribute to all logs and metric dimensions sent from the dynatrace-gcp-monitor to Dynatrace.

A better solution, would be to allow specifying an existing attribute as a dynamic value passed as the dt.security_context. For example, in the values.yaml, having a setting called securityContext with a value like fromAttribute: gcp.project.id would set the value of dt.security_context as the same value of gcp.project.id.

securityContext:
fromAttribute: 'gcp.project.id' # set the value equal to that of gcp.project.id if it exists
default: 'my default context' # set the value to 'my default context' if the value of 'gcp.project.id' does not exist

Describe alternatives you've considered
See above

Additional context
https://docs.dynatrace.com/docs/observe-and-explore/logs/lma-security-context

Failing tasks when monitoring all services

I used the Function (v0.0.23) deployment with all services and all metrics. Once run I started getting errors in logs. I would expect the lines to have Error severity for better filtering.

Please take a look:

insertId,labels.execution_id,logName,receiveTimestamp,resource.labels.function_name,resource.labels.project_id,resource.labels.region,resource.type,severity,textPayload,timestamp,trace
000000-03466100-899b-4226-a146-1a43c30aa2bc,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,2021-02-22 12:59:16.359508 [dynatrace-gcp-extension] : Finished fetching data in 8.522547960281372,2021-02-22T12:59:16.359Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-03466100-899b-4226-a146-1a43c30aa2bc,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:15.691604 [dynatrace-gcp-extension] : Failed to finish task for [spanner.googleapis.com/instance/backup/used_bytes], reason is Exception {'error': {'code': 500, 'message': 'Internal error encountered. Please retry after a few seconds. If internal errors persist, contact support at https://cloud.google.com/support/docs.', 'status': 'INTERNAL'}}",2021-02-22T12:59:15.691Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-f117c1db-c0d1-41d6-89bd-7871fad381b1,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:15.083628 [dynatrace-gcp-extension] : Failed to finish task for [serviceruntime.googleapis.com/quota/limit], reason is Exception {'error': {'code': 400, 'message': 'Field aggregation.alignmentPeriod had an invalid value of ""0"": The alignment period must be greater than or equal to 10s.', 'status': 'INVALID_ARGUMENT'}}",2021-02-22T12:59:15.083Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-f117c1db-c0d1-41d6-89bd-7871fad381b1,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:14.660401 [dynatrace-gcp-extension] : Failed to finish task for [knative.dev/internal/eventing/trigger/event_processing_latencies], reason is Exception {'error': {'code': 403, 'message': 'User [e-mail] does not have permission to see metric knative.dev/internal/eventing/trigger/event_processing_latencies.', 'status': 'PERMISSION_DENIED'}}",2021-02-22T12:59:14.660Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-7937c9c6-bd48-463b-9f72-1b4568ebc0d1,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:13.691595 [dynatrace-gcp-extension] : Failed to finish task for [knative.dev/internal/eventing/trigger/event_dispatch_latencies], reason is Exception {'error': {'code': 403, 'message': 'User [e-mail] does not have permission to see metric knative.dev/internal/eventing/trigger/event_dispatch_latencies.', 'status': 'PERMISSION_DENIED'}}",2021-02-22T12:59:13.691Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-7937c9c6-bd48-463b-9f72-1b4568ebc0d1,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:13.675049 [dynatrace-gcp-extension] : Failed to finish task for [knative.dev/internal/eventing/trigger/event_count], reason is Exception {'error': {'code': 403, 'message': 'User [e-mail] does not have permission to see metric knative.dev/internal/eventing/trigger/event_count.', 'status': 'PERMISSION_DENIED'}}",2021-02-22T12:59:13.675Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-715301c5-0bbb-4804-972b-90e204ed52ae,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:13.045363 [dynatrace-gcp-extension] : Failed to finish task for [knative.dev/internal/serving/revision/request_latencies], reason is Exception {'error': {'code': 403, 'message': 'User [e-mail] does not have permission to see metric knative.dev/internal/serving/revision/request_latencies.', 'status': 'PERMISSION_DENIED'}}",2021-02-22T12:59:13.045Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-715301c5-0bbb-4804-972b-90e204ed52ae,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:13.043511 [dynatrace-gcp-extension] : Failed to finish task for [knative.dev/internal/serving/revision/request_count], reason is Exception {'error': {'code': 403, 'message': 'User [e-mail] does not have permission to see metric knative.dev/internal/serving/revision/request_count.', 'status': 'PERMISSION_DENIED'}}",2021-02-22T12:59:13.043Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-715301c5-0bbb-4804-972b-90e204ed52ae,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:13.041706 [dynatrace-gcp-extension] : Failed to finish task for [knative.dev/internal/serving/activator/request_latencies], reason is Exception {'error': {'code': 403, 'message': 'User [e-mail] does not have permission to see metric knative.dev/internal/serving/activator/request_latencies.', 'status': 'PERMISSION_DENIED'}}",2021-02-22T12:59:13.041Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-715301c5-0bbb-4804-972b-90e204ed52ae,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:13.040909 [dynatrace-gcp-extension] : Failed to finish task for [knative.dev/internal/serving/activator/request_count], reason is Exception {'error': {'code': 403, 'message': 'User [e-mail] does not have permission to see metric knative.dev/internal/serving/activator/request_count.', 'status': 'PERMISSION_DENIED'}}",2021-02-22T12:59:13.040Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-715301c5-0bbb-4804-972b-90e204ed52ae,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:24.258411090Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:13.038194 [dynatrace-gcp-extension] : Failed to finish task for [knative.dev/internal/eventing/broker/event_count], reason is Exception {'error': {'code': 403, 'message': 'User [e-mail] does not have permission to see metric knative.dev/internal/eventing/broker/event_count.', 'status': 'PERMISSION_DENIED'}}",2021-02-22T12:59:13.038Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-40cb4b5e-f754-46b6-b50b-0ed1b8025069,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:14.390275496Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:12.656580 [dynatrace-gcp-extension] : Failed to finish task for [kubernetes.io/container/memory/page_fault_count], reason is Exception {'error': {'code': 400, 'message': 'Field aggregation.perSeriesAligner had an invalid value of ""ALIGN_SUM"": The aligner cannot be applied to metrics with kind CUMULATIVE and value type INT64.', 'status': 'INVALID_ARGUMENT'}}",2021-02-22T12:59:12.656Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-ef4753e0-65d0-4b9e-a074-57cc4ab8d3e1,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:14.390275496Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:10.754083 [dynatrace-gcp-extension] : Failed to finish task for [firebasedatabase.googleapis.com/storage/disabled_for_overages], reason is Exception {'error': {'code': 400, 'message': 'Field aggregation.alignmentPeriod had an invalid value of ""0"": The alignment period must be greater than or equal to 10s.', 'status': 'INVALID_ARGUMENT'}}",2021-02-22T12:59:10.754Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-ef4753e0-65d0-4b9e-a074-57cc4ab8d3e1,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:14.390275496Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:10.700913 [dynatrace-gcp-extension] : Failed to finish task for [firebasedatabase.googleapis.com/storage/total_bytes], reason is Exception {'error': {'code': 400, 'message': 'Field aggregation.alignmentPeriod had an invalid value of ""0"": The alignment period must be greater than or equal to 10s.', 'status': 'INVALID_ARGUMENT'}}",2021-02-22T12:59:10.701Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-cd235d69-db4a-4d64-a882-c54e761f93e4,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:14.390275496Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:10.655435 [dynatrace-gcp-extension] : Failed to finish task for [serviceruntime.googleapis.com/quota/limit], reason is Exception {'error': {'code': 400, 'message': 'Field aggregation.alignmentPeriod had an invalid value of ""0"": The alignment period must be greater than or equal to 10s.', 'status': 'INVALID_ARGUMENT'}}",2021-02-22T12:59:10.655Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-483cbd4d-065e-4dd2-9afe-2899b505527b,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:14.390275496Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,2021-02-22 12:59:07.837048 [dynatrace-gcp-extension] : Starting processing...,2021-02-22T12:59:07.837Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-483cbd4d-065e-4dd2-9afe-2899b505527b,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:14.390275496Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,2021-02-22 12:59:07.836718  : Access to following projects: dynatrace-gcp-extension,2021-02-22T12:59:07.836Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-483cbd4d-065e-4dd2-9afe-2899b505527b,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:14.390275496Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,2021-02-22 12:59:07.485475  : Successfully obtained access token,2021-02-22T12:59:07.485Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000001-483cbd4d-065e-4dd2-9afe-2899b505527b,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:14.390275496Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,2021-02-22 12:59:07.462862  : Trying to use default service account,2021-02-22T12:59:07.462Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38
000000-cc188f3e-7061-4623-b5ad-ae7784ddf6e6,cv8qfyg0i3fz,projects/dynatrace-gcp-extension/logs/cloudfunctions.googleapis.com%2Fcloud-functions,2021-02-22T12:59:14.390275496Z,dynatrace-gcp-function,dynatrace-gcp-extension,us-central1,cloud_function,INFO,"2021-02-22 12:59:07.462572  : Selected feature sets: api/default,apigee.googleapis.com/Environment/default,apigee.googleapis.com/Proxy/default,apigee.googleapis.com/ProxyV2/default,assistant_action_project/default,autoscaler/default,bigquery_biengine_model/default,bigquery_dataset/default,bigquery_project/default,bigtable_cluster/default,bigtable_table/default,cloud_composer_environment/default,cloud_dataproc_cluster/default,cloud_dlp_project/default,Cloud Function/default,cloud_run_revision/default,cloud_tasks_queue/default,cloudiot_device_registry/default,cloudml_job/default,cloudml_model_version/default,cloudsql_database/default,cloudtrace.googleapis.com/CloudtraceProject/default,cloudvolumesgcp-api.netapp.com/NetAppCloudVolumeSO/default,consumed_api/default,consumer_quota/default,dataflow_job/default,datastore_request/default,dns_query/default,filestore_instance/default,firebase_domain/default,firebase_namespace/default,firestore_instance/default,gae_app/default,gae_app_uptime_check/default,gae_instance/default,gce_instance/default,gce_instance/agent,gce_instance/appenginee,gce_instance/firewallinsights,gce_instance/istio,gce_instance/uptime_check,gce_instance_vm_flow/,gce_router/default,gce_zone_network_health/default,gcs_bucket/default,gke_container/default,https_lb_rule/default,iam_service_account/default,instance_group/default,interconnect/default,interconnect_attachment/default,internal_http_lb_rule/default,internal_tcp_lb_rule/default,internal_udp_lb_rule/default,istio_canonical_service/default,k8s_cluster/default,k8s_container/default,k8s_container/agent,k8s_container/apigee,k8s_container/istio,k8s_container/nginx,k8s_node/default,k8s_pod/default,knative_broker/default,knative_revision/default,knative_trigger/default,logging_sink/default,microsoft_ad_domain/default,nat_gateway/default,netapp_cloud_volume/default,network_security_policy/default,producer_quota/default,pubsub_snapshot/default,pubsub_subscription/default,pubsub_topic/default,pubsublite_subscription_partition/default,pubsublite_topic_partition/default,recaptchaenterprise.googleapis.com/Key/default,redis_instance/default,spanner_instance/default,tcp_lb_rule/default,tcp_ssl_proxy_rule/default,tpu_worker/default,transfer_service_agent/default,udp_lb_rule/default,uptime_url/default,vpc_access_connector/default,vpn_gatewayv/",2021-02-22T12:59:07.462Z,projects/dynatrace-gcp-extension/traces/102dbadbc67135eedee3c2f7d5fc6a38

Failed to push self monitoring time series

I followed the Kubernetes quick start in the readme on a simple cluster, when I check the container status I get a 403 permissions denied error. Any idea what this might be? When I follow the bash instructions and deploy a cloud function it works fine and I'm seeing metrics in Dynatrace, but I have some constraints and have to go via the kubernetes route, I used the same Dynatrace API Token in both cases. Many thanks in advance

kubectl -n dynatrace logs -l app=dynatrace-gcp-function

2020-11-02 17:06:34.795092 [c5410aad] : Finished uploading metric ingest lines to Dynatrace in 1.1920928955078125e-06 s
2020-11-02 17:06:34.795097 [c5410aad] : Pushing self monitoring time series to GCP Monitor...
2020-11-02 17:06:34.889516 [c5410aad] : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/connectivity'
2020-11-02 17:06:34.988816 [c5410aad] : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/ingest_lines'
2020-11-02 17:06:35.086464 [c5410aad] : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/request_count'
2020-11-02 17:06:35.186966 [c5410aad] : Creating missing metric descriptor for 'custom.googleapis.com/dynatrace/phase_execution_time'
2020-11-02 17:06:35.285764 [c5410aad] : Failed to push self monitoring time series, error is: 403 => {'error': {'code': 403, 'message': 'Permission monitoring.timeSeries.create denie
d (or the resource may not exist).', 'status': 'PERMISSION_DENIED'}}
2020-11-02 17:06:35.286386 [c5410aad] : Execution took 2.5885870456695557

Instrument dynatrace-gcp-monitor with opentelemetry

We would like to monitor dynatrace-gcp-monitor performance with OneAgent. Could we instrument dynatrace-gcp-monitor with opentelemetry?

Vulnerabilities to be addressed

With aqua/scanner:2022.4.217 to scan image dynatrace/dynatrace-gcp-monitor:release-1.1.0, there are two high issues found. Can those issues be fixed please?

Can not create dashboard

I'm getting the following error when script creates dashboards in my tenant:

- Create ["Google Cloud APIs"] dashboard from file [dashboards/api.json]
{"error":{"code":400,"message":"Could not map JSON at 'tiles[2].queries[0].filterBy' near line 53 column 25"}}
- Create ["Google Cloud Function"] dashboard from file [dashboards/cloud_function.json]
{"error":{"code":400,"message":"Could not map JSON at 'tiles[1].queries[0].filterBy' near line 43 column 25"}}
- Create ["Google Cloud SQL"] dashboard from file [dashboards/cloudsql_database.json]
{"error":{"code":400,"message":"Could not map JSON at 'tiles[0].queries[0].filterBy' near line 31 column 25"}}
- Create ["Google Cloud Datastore"] dashboard from file [dashboards/datastore.json]
{"error":{"code":400,"message":"Could not map JSON at 'tiles[0].queries[0].filterBy' near line 31 column 25"}}
- Create ["Google Cloud Filestore"] dashboard from file [dashboards/filestore_instance.json]
{"error":{"code":400,"message":"Could not map JSON at 'tiles[3].queries[0].filterBy' near line 73 column 25"}}
- Create ["Google Cloud Storage"] dashboard from file [dashboards/gcs_bucket.json]
{"error":{"code":400,"message":"Could not map JSON at 'tiles[1].queries[0].filterBy' near line 43 column 25"}}
- Create ["Google Cloud Load Balancing"] dashboard from file [dashboards/lb.json]
{"id":"5d7c0de3-1344-4216-a755-313a3bc30892","name":"Google Cloud Load Balancing"}
- Create ["Google Cloud Pub/Sub"] dashboard from file [dashboards/pubsub.json]
{"error":{"code":400,"message":"Could not map JSON at 'tiles[1].queries[0].filterBy' near line 43 column 25"}}

Installation documentation misleading/incorrect

Overview

Heyo, folks! As I mentioned in the other issue I created (#96), we recently tried installing the dynatrace-gcp-function to our GCP environment and we ran into an issue when providing the setup.sh script with our tenant URL.

Details

According to the official documentation it states the following for the prerequisites:

Then, when the script is running, it asks for the following values. For the tenant URL specifically, it mentions to refer to the prerequisites:

Suspected Problem

The problem with that is the script doesn't allow for any of the recommended values to be true. The regex on line 189 of setup.sh expects a URL to end with a slash and doesn't accept /api anywhere at the end. Unless I'm reading the regex incorrectly (which is entirely possible, in which case I apologize for making a fool of myself 😅), I'm reading it allows for:

• Up to 256 characters to define the DNS.
• Followed by an optional field for /e/<36-digit-id>/
• End string

Wrap-Up

So, TL;DR: I believe the regular expression on line 189 of the setup.sh script is either incorrect, or the prerequisites section of the install documentation needs to be updated to reflect the proper format for the tenant URI.

GCP Integration creates infinite dashboards

https://github.com/dynatrace-oss/dynatrace-gcp-function/blob/1150562f5095471b126896fc31d1300c1f95302c/src/lib/self_monitoring.py#L309-L311

Our GCP service account did not have the monitoring.dashboards.list permissions, because this is not listed anywhere in the dosc nor is it being created by the helm deployment package provided.

Please also not that we had self_monitoring off, and we only notice dthe dashboards when we started peeking at audit logs.

If the result HTTP 403, like it was with us, this will just keep on creating dashboards until the quota is hit.

{
  "error": {
    "code": 403,
    "message": "The caller does not have permission",
    "status": "PERMISSION_DENIED"
  }
}

Removing it requires some annoying hop jumping

$ gcloud monitoring dashboards list > way_too_many_dashboards.yaml
$ docker run --rm -v "${PWD}":/workdir mikefarah/yq -N e 'select(.displayName == "dyna*") | .name' way_too_many_dashboards.yaml | xargs -n 1 gcloud monitoring dashboards delete --quiet

Wrong token scopes

Documentation https://github.com/dynatrace-oss/dynatrace-gcp-function states that token with

• API v2 Ingest metrics
• API v1 Read configuration

is required while Write configuration scope is required to write dashboards.

See error:

- Create ["Google Cloud APIs"] dashboard from file [dashboards/api.json] {"error":{"code":403,"message":"Token is missing required scope. Use one of: WriteConfig (Write configuration)"}}

run_docker.py failing when using proxy config

Hey @Dawid-Kaszubski, Thanks for getting this in so quickly. I'm running into some issues running the code with the proxy settings enabled.

The environment variables exist

HTTPS_PROXY="http://<REDACTED>"
USE_PROXY="DT_ONLY"

However the run_docker.py script fails in due to an exception raised in the fast_check lib.

2021-03-05 09:26:57.480281 : Unable to get Dynatrace token metadata. Error details:

exception block:

async def get_dynatrace_token_metadata(dt_session: ClientSession, context: LoggingContext, dynatrace_url: str, dynatrace_api_key: str, timeout: Optional[int] = 2) -> dict:
    try:
        response = await dt_session.post(
            url=f"{dynatrace_url.rstrip('/')}/api/v1/tokens/lookup",
            headers={
                "Authorization": f"Api-Token {dynatrace_api_key}",
                "Content-Type": "application/json; charset=utf-8"
            },
            json={
                "token": dynatrace_api_key
            },
            timeout=timeout)
        if response.status != 200:
            context.log(f'Unable to get Dynatrace token metadata: {response.status}, url: {response.url}, reason: {response.reason}')
            return {}

        return await response.json()
    except Exception as e:
        context.log(f'Unable to get Dynatrace token metadata. Error details: {e}')
        return {}

If i execute the local_test.py script the gcp function respects the environment variables and is able to connect to both the Dynatrace tenant and google apis successfully. Do you have any suggestions to help me debug this futher?

Many thanks
Jack

Metric unit mappings

Hey, I am trying to map metric units configured in the GCP service yaml files to the metric unit format in Dynatrace so that i can create custom alerts in the UI or via the API. Would you be able to help with the mappings containing "????" or point me in the right direction?

{
"%" : "Percentage",
"10^2.%" : "Percentage", // multiplies the value by 100 before ingestion
"1" : "Count",
"ms" : "MilliSecond",
"s" : "Second",
"By" : "Bytes",
"GiBy" : "GigaByte",
"" : "UNSPECIFIED",
"{CPU}" : "????",
"{packets}" : "????",
"d" : "????",
"s" : "Second",
"us" : "MicroSecond",
"1/s" : "PerSecond",
"{connection}" : "????",
"{request}" : "????",
"s{CPU}" : "????",
"By/s" : "BytePerSecond",
"GBy.s" : "???? GigaBytes per second?",
"GiBy.s" : "???? GigaBytes per second?",
"{inode}" : "????",
"{dBm}" : "????",
"ns" : "NanoSecond",
"s{idle}" : "????",
"s{uptime}" : "????",
"{port}" : "????",
"{packet}" : "????",
"{connection}" : "????"
}

Many thanks
Jack

Request feature to support self-signed cert for cert validation

In file https://github.com/dynatrace-oss/dynatrace-gcp-monitor/blob/master/k8s/helm-chart/dynatrace-gcp-monitor/values.yaml, there are following lines:

requireValidCertificate: if true function requires valid SSL certificates when communicating with Dynatrace cluster

may be used to bypass SSL certificates errors when traffic is proxied through Active Gate with self-signed certificate.

Allowed values: "true"/"yes", "false"/"no"

requireValidCertificate: "true"

Could please add feature to support self-signed certificate as following? Thanks.

1. set requireValidCertificate as"true".
2. add a new variable like "selfSignedCert" which accept PEM format cert, or variable like "selfSignedCA" which accept PEM format CA cert(so multiple certs signed by this CA can be validated).
3. Once dyntrace-gcp-monitor pod is running and talk to Dynatrace ActiveGate which has cert as "selfSignedCert" or has cert signed by "selfSignedCA", it can accept the self-signed cert.

skipped services (no extractors) for apigee

Hello,
can you please explain to me the concept of extractor services? because in the logs of my cluster under GCP, I encounter the following logs:

- skipped services (no extractors): [apigee_googleapis_com_Environment, apigee_googleapis_com_ProxyV2, apigee_googleapis_com_Proxy]

I would like to be able to activate these metrics

best regards,
Badr

HTTPS_PROXY and HTTP_PROXY environment variables are not defined in the deployment or config map

The proxy environment variables are not set by the deployment YAML and likewise there are no parameters to set them in the config map.

https://github.com/dynatrace-oss/dynatrace-gcp-function/blob/master/k8s/dynatrace-gcp-function.yaml

Dataflow dimension breaching the metric ingestion API dimension character limit

I came across some dimensions that are breaching a character limit. I believe this error is returned by the metric ingestion API as an invalid line with the message on the exceeded length. Speaking with the project owner, they require the dimension and can’t truncate it on their end. Can we get this char limit increased to say 500 and as a fall back truncate the dimension if it is over that?

INVALID LINE: 'cloud.gcp.dataflow_googleapis_com.job.estimated_byte_count,job_id=" REDACTED ",pcollection="SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Schema View/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)/Combine.GroupedValues/Partial.out0",job_name="REDACTED",region="europe- REDACTED",project_id="REDACTED" gauge,20755 1616588614000', reason: 'dimension value length exceeded: 401 > 250'

Least permission needed instead of complete role.

HI Team,

To enable sending metrics from CloudSQL service, below role is mentioned to be added on service account which has some additional permissions which may not needed for sending metrics. Same applies to other mentioned roles for rest of the services having additional permissions.

Can we know what exact permissions are need for sending metrics rather than having the complete Role?
There are some security concerns on using the full Role.

roles/cloudsql.viewer

• cloudsql.*.export
• cloudsql.*.get
• cloudsql.*.list
• cloudsql.instances.listServerCa

Adding additional Google Projects to cloud Function

I have set up a project with Dynatrace Cloud Function and it is working, I have added 2 additional projects successfully, but after adding those 2 I am following the directions as I did before but an not seesin gthe new project data, when I look in the function log I do not see the new project mentioned.

Here are the gcloud project bind ID commands

gcloud projects add-iam-policy-binding New-Project-Id --member="serviceAccount:dynatrace-gcp-function-pre-sa@Cloud-Function_project-35c8980e.iam.gserviceaccount.com" --role=roles/monitoring.editor
gcloud projects add-iam-policy-binding New-Project-Id --member="serviceAccount:dynatrace-gcp-function-pre-sa@Cloud-Function_project-35c8980e.iam.gserviceaccount.com" --role=roles/monitoring.viewer
gcloud projects add-iam-policy-binding New-Project-Id --member="serviceAccount:dynatrace-gcp-function-pre-sa@Cloud-Function_project-35c8980e.iam.gserviceaccount.com" --role=roles/compute.viewer
gcloud projects add-iam-policy-binding New-Project-Id --member="serviceAccount:dynatrace-gcp-function-pre-sa@Cloud-Function_project-35c8980e.iam.gserviceaccount.com" --role=roles/cloudsql.viewer
gcloud projects add-iam-policy-binding New-Project-Id --member="serviceAccount:dynatrace-gcp-function-pre-sa@Cloud-Function_project-35c8980e.iam.gserviceaccount.com" --role=roles/cloudfunctions.viewer
gcloud projects add-iam-policy-binding New-Project-Id --member="serviceAccount:dynatrace-gcp-function-pre-sa@Cloud-Function_project-35c8980e.iam.gserviceaccount.com" --role=roles/file.viewer
gcloud projects add-iam-policy-binding New-Project-Id --member="serviceAccount:dynatrace-gcp-function-pre-sa@Cloud-Function_project-35c8980e.iam.gserviceaccount.com" --role=roles/pubsub.viewer
gcloud projects add-iam-policy-binding New-Project-Id --member="serviceAccount:dynatrace-gcp-function-pre-sa@Cloud-Function_project-35c8980e.iam.gserviceaccount.com" --role=roles/editor

where New-Project-Id is the project being added and Cloud-Function-project is the project running the cloud function app

Any ideas as to why it is failing?

Script ran but message is not descriptive

I had the script and I see the function deployed. However the script ended like this with no useful message. I hope these can be improved.

App Engine not created when running install.

Hey folks... when running the install script, the question about the creation of the App Engine was never prompted, as state at the docs:

Please note dynatrace-gcp-function uses Cloud Scheduler that requires App Engine to be created. If you don't have App Engine enabled yet, installer script will prompt you to Create it and select region, where it will run. Reference: Cloud Scheduler documentation

Same is displayed at internal docs.: https://www.dynatrace.com/support/help/setup-and-configuration/setup-on-cloud-platforms/google-cloud-platform/set-up-integration-gcp/deploy-as-gcp-function/

At the end of the install script shows this error and no schedule is run or AE installed.

- schedule the runs
ERROR: (gcloud.scheduler.jobs.list) Please use the location flag to manually specify a location.
ERROR: (gcloud.scheduler.jobs.create.pubsub)Please use the location flag to manually specify a location.

Error: - deployment failed, please examen error message and run again."

If run manually, information actually reach the SaaS environment. But not via schedule.

Is this expected? or docs need to be updated?

Thanks!

Can not provide custom URL as a tenant URL

See: https://github.com/dynatrace-oss/dynatrace-gcp-function/blob/master/scripts/setup.sh

while ! [[ "${DYNATRACE_URL}" =~ ^https:\/\/[a-z0-9-]{8}\.(live|sprint|dev)\.(dynatrace|dynatracelabs)\.com\/$ ]]; do read -p "Enter Dynatrace tenant URI: " DYNATRACE_URL done

script supports only SaaS tenant URLs. It does not support custom Cluster AG URL for managed.

Proxy support

Does the kubernetes deployment respect proxy environment variables ?

We tried deploying the code adding the environment variables for https_proxy and no_proxy as we have an air-gapped install and all traffic should go via our proxy including the traffic for the xxxx..live.dynatrace.com but the app does not seem to use the variables.