appmetrics/elasticsearch

App Metrics Extensions for Elasticsearch reporting

App Metrics

What is App Metrics?

App Metrics is an open-source and cross-platform .NET library used to record metrics within an application. App Metrics can run on .NET Core or on the full .NET framework. App Metrics abstracts away the underlaying repository of your Metrics for example InfluxDB, Graphite, Prometheus etc, by sampling and aggregating in memory and providing extensibility points to flush metrics to a repository at a specified interval.

App Metrics provides various metric types to measure things such as the rate of requests, counting the number of user logins over time, measure the time taken to execute a database query, measure the amount of free memory and so on. Metrics types supported are Gauges, Counters, Meters, Histograms and Timers and Application Performance Indexes Apdex.

App.Metrics includes an Exponentially Forward Decaying, Sliding Window and Algorithm R reservoir implementations. For more details on reservoir sampling see the docs.

Documentation

• Getting Started
• ASP.NET Core 2.0
• Reporting
• Sample Applications & Grafana Dashboards

Latest Builds, Packages & Repo Stats

Branch	Azure Devops
dev
main

---

Visualization

Dashboards can be imported from Grafana

Grafana Web Monitoring

Grafana OAuth2 Client Web Monitoring

How to build

Azure Devops builds are triggered on commits and PRs to the dev branch

• Install the latest .NET Core 2.x SDK
• Run build.ps1 or build.sh in the root of the repository

How to run benchmarks

App.Metrics includes benchmarking using BenchmarkDotNet.

Two benchmark projects exist targeting App.Metrics.Core and App.Metrics.Concurrency

	cd .\src\Core\benchmarks\App.Metrics.Benchmarks.Runner
	dotnet run -c "Release" --framework netcoreapp3.1

	cd .\src\Concurrency\benchmarks\App.Metrics.Concurrency.Benchmarks.Runner
	dotnet run -c "Release" --framework netcoreapp3.1

You'll then be prompted to choose a benchmark to run which will output a markdown file with the result in directory.

You can find the benchmark results here and here.

Contributing

See the contribution guidlines for details.

Acknowledgements

• ASP.NET Core
• Grafana
• Fluent Assertions
• xUnit.net
• Hugo
• Netlify

Thanks for providing free open source licensing

• NDepend
• Jetbrains

License

This library is release under Apache 2.0 License ( see LICENSE ) Copyright (c) 2016 Allan Hardy

See LICENSE

---

App Metrics is based on the Metrics.NET library, and at the moment uses the same reservoir sampling code from the original library which is a port of the Java Dropwizard Metrics library.

Metrics.NET Licensed under these terms: "Metrics.NET is release under Apache 2.0 License Copyright (c) 2014 Iulian Margarintescu" see LICENSE

Dropwizard Metrics Licensed under these terms*: "Copyright (c) 2010-2013 Coda Hale, Yammer.com Published under Apache Software License 2.0, see LICENSE"

elasticsearch's Issues

Remove dependency on Microsoft DI and Configuration

Use IMetricsBuilder from latest App Metrics core

There is a bug in grafana's elasticsearch template

Error Rate there is always data.
search query missing space characters.

old:
name:application.httprequests__error_rateAND tags.env:$environment AND tags.app:$application AND tags.server:$server
new:
name:application.httprequests__error_rate AND tags.env:$environment AND tags.app:$application AND tags.server:$server

ES Authentication Scheme not set correctly

See #12

Breaking change in ElasticSearch 6

Upgraded to ElasticSearch 6, now seeing the following in the ElasticSearch logs when reporting more than one type of metric:

Rejecting mapping update to [metrics] as the final mapping would have more than 1 type: [meter, counter].

See https://www.elastic.co/guide/en/elasticsearch/reference/6.x/removal-of-types.html

App Metrics 2.0 package updates

Move formatting to separate Formatter package

Same as InfluxDB - AppMetrics/InfluxDB#8

System.MissingMethodException on call to .Report.ToElasticSearch()

Hi - I'm trying to use version 2.0.0-alpha in my project, and am stumbling at the first hurdle with an exception being thrown from the call to .Report.ToElasticSearch() on startup

I'm using the overload which takes the string url and elasticsearchIndex arguments

System.MissingMethodException
HResult=0x80131513
Message=Method not found: 'Void App.Metrics.GeneratedMetricNameMapping..ctor(System.Collections.Generic.IDictionary2<App.Metrics.HistogramValueDataKeys,System.String>, System.Collections.Generic.IDictionary2<App.Metrics.MeterValueDataKeys,System.String>, System.Collections.Generic.IDictionary2<App.Metrics.ApdexValueDataKeys,System.String>, System.Collections.Generic.IDictionary2<App.Metrics.CounterValueDataKeys,System.String>)'.
Source=App.Metrics.Formatters.Elasticsearch
StackTrace:
at App.Metrics.Formatters.Elasticsearch.MetricsElasticsearchDocumentFormattingOptions..ctor()
at App.Metrics.Formatters.Elasticsearch.MetricsElasticsearchOutputFormatter..ctor(String elasticsearchIndex)
at App.Metrics.Reporting.Elasticsearch.ElasticsearchReporter..ctor(MetricsReportingElasticsearchOptions options, IElasticsearchClient elasticsearchClient)
at App.Metrics.MetricsElasticsearchReporterBuilder.ToElasticsearch(IMetricsReportingBuilder metricReporterProviderBuilder, String url, String elasticsearchIndex)
..........

is this currently a known issue? Thanks

Wrong mapping fields names for webtracking with ElasticSearch

In the doc (https://www.app-metrics.io/reporting/reporters/elasticsearch/) it is recommended to create an index and to send the mapping :

Add an index and the document mappings to Elasticsearch using the default mappings provided in the github repository

But when i send this command to ES :

PUT appmetricssandbox/_mapping/timer?update_all_types
{
  "properties": {
    "tags.env": { 
      "type":     "text",
      "fielddata": true
    },
    "tags.exception": { 
      "type":     "text",
      "fielddata": true
    },
    "tags.app": { 
      "type":     "text",
      "fielddata": true
    },
    "tags.route": { 
      "type":     "text",
      "fielddata": true
    },
    "tags.server": { 
      "type":     "text",
      "fielddata": true
    },
    "tags.item": { 
      "type":     "text",
      "fielddata": true
    },
    "tags.client_id": { 
      "type":     "text",
      "fielddata": true
    },
    "tags.http_status_code": { 
      "type":     "text",
      "fielddata": true
    },
    "tags.health_check_name": { 
      "type":     "text",
      "fielddata": true
    }
  }
}

I get this error for ES :

{
"error": {
"root_cause": [
{
"type": "mapper_parsing_exception",
"reason": "Field name [tags.env] cannot contain '.'"
}
],
"type": "mapper_parsing_exception",
"reason": "Field name [tags.env] cannot contain '.'"
},
"status": 400
}

Grafana dashboard doesn't work

I use elastic search 5.x + grafana 5.x + asp net core 2.
Index and mappings are the same like in this file https://github.com/AppMetrics/Elasticsearch/blob/dev/visualization/App.Metrics.Sandbox-Elasticsearch-IndexAndMappingSetup.json but I had to add one more property
timestamp": { "type": "date", "format": "epoch_millis" }
in each type mappings because I was not able to create data source in grafana without some date property (need to specify this in "Time field name" input).

Part of Program.cs

public static IWebHost BuildWebHost(string[] args) =>
            WebHost.CreateDefaultBuilder(args)
                .ConfigureMetricsWithDefaults(builder => 
                {
                    builder.Report.ToElasticsearch(options =>
                    {
                        options.Elasticsearch.Index = "appmetricssandbox";
                        options.Elasticsearch.BaseUri = new Uri("http://127.0.0.1:9200");
                        options.Elasticsearch.AuthorizationSchema = ElasticSearchAuthorizationSchemes.Anonymous;
                        options.HttpPolicy.BackoffPeriod = TimeSpan.FromSeconds(30);
                        options.HttpPolicy.FailuresBeforeBackoff = 5;
                        options.HttpPolicy.Timeout = TimeSpan.FromSeconds(10);
                        options.MetricsOutputFormatter = new MetricsJsonOutputFormatter();
                        options.FlushInterval = TimeSpan.FromSeconds(20);
                        options.Filter = new MetricsFilter().WhereType(MetricType.Timer);
                    });
                })
                .UseMetrics()
                .UseStartup<Startup>()
                .Build();

And in configure services method I have

public void ConfigureServices(IServiceCollection services)
{
         services.AddMvc().AddMetrics();
}

Error in grafana

{
    "root_cause": [
        {
            "type": "query_shard_exception",
            "reason": "Failed to parse query [name:application.httprequests__active AND tags.env: AND tags.app: AND tags.server:()]",
            "index_uuid": "Nt8ntKjkS-qM8VrbhHrxDw",
            "index": "appmetricssandbox"
        }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
        {
            "shard": 0,
            "index": "appmetricssandbox",
            "node": "3tw4J0WDQGKTYKvrzUbtgQ",
            "reason": {
                "type": "query_shard_exception",
                "reason": "Failed to parse query [name:application.httprequests__active AND tags.env: AND tags.app: AND tags.server:()]",
                "index_uuid": "Nt8ntKjkS-qM8VrbhHrxDw",
                "index": "appmetricssandbox",
                "caused_by": {
                    "type": "parse_exception",
                    "reason": "Cannot parse 'name:application.httprequests__active AND tags.env: AND tags.app: AND tags.server:()': Encountered \"  \"AND \"\" at line 1, column 52.\r\nWas expecting one of:\r\n     ...\r\n    \"(\" ...\r\n    \"*\" ...\r\n     ...\r\n     ...\r\n     ...\r\n     ...\r\n     ...\r\n    \"[\" ...\r\n    \"{\" ...\r\n     ...\r\n    ",
                    "caused_by": {
                        "type": "parse_exception",
                        "reason": "Encountered \"  \"AND \"\" at line 1, column 52.\r\nWas expecting one of:\r\n     ...\r\n    \"(\" ...\r\n    \"*\" ...\r\n     ...\r\n     ...\r\n     ...\r\n     ...\r\n     ...\r\n    \"[\" ...\r\n    \"{\" ...\r\n     ...\r\n    "
                    }
                }
            }
        }
    ]
}

Dashboard was added using ID from this page https://grafana.com/dashboards/2140 .
In my asp net core project I use App.Metrics.AspNetCore.Mvc and App.Metrics.Reporting.Elasticsearch.

Spent some time with research and didn't find any useful solution. What I need to change to start it work ?
Thanks

Include exception tracking in web dashboard

See AppMetrics/AppMetrics#35

Add sourcelink support

Support https://github.com/ctaggart/SourceLink which enables the source code repo to be the source server. Requires portable PDBs so will have to sacrifice OpenCover support for now.

Add server to Grafana dashboard and allowing filtering all metrics

State of project

Hi,

We would like to use AppMetrics for our metric needs, we use ELK stack for monitoring.
Seeing that this repository was not updated in a while, wondering is there any plans to update this reporter to latest AppMetrics version?

Thank you!

Project dependency updates

• Test projects package updates
• Benchmark projects package updates
• Library projects package updates

Has this project been abandoned?

Hi just wantet to ask whether this project is still active or not? If not, it would be nice if it were mentioned in the description of the repo or the repo was to be archived!

Posting to AWS elastic search

Hello,

do you support AWS elastic search somehow or should I create my own implementation?

_bulk api call not compatible with ES 6.2

I can't say if this is a bug because I don't really know the intent of the code. It seems that _bulk is being used in order to send multiple events to ES.

One thing I noticed is that the endpoint used is not _bulk for the configured index but at the ES app root. The index endpoint is preferred: /{index}/_bulk or {index}/{type}/_bulk instead of /_bulk

\r\n is being used in the payload which is not allowed except for delimiting records.

It could be that a recent ES version changed the format but the docs don't specify a required version. The payload seems all wrong for the bulk API.

Move to AppMetrics GitHub org

Workaround for - ES6 Rejecting mapping update - the final mapping would have more than 1 type

Hi,

I saw your workaround for ES6 release, which doesn't support more than 1 type for mapping more about it here.
I don't think that this pull request solves the problem completely.
If so, and I am mistaken, please provide new documentation for Grafana/Elasticsearch Web Application Setup.

Thanks,
Jana

Rename packages to keep consistency with App Metrics 2.0 package naming

• App.Metrics.Extensions.Reporting.ElasticSearch to App.Metrics.Reporting.Elasticsearch
• App.Metrics.Formatting.ElasticSearch to App.Metrics.Formatters.Elasticsearch
• App.Metrics.Formatters.ElasticSearch to App.Metrics.AspNetCore.Formatters.Elasticsearch

Allow loading reporter settings from config

Dependency on Microsoft DI and Configuration has been removed by default, still require a way to load settings from config