signoz/opentelemetry-collector-contrib

---

Getting Started   •   Getting Involved   •   SigNoz Slack Community

---

This is a fork of OpenTelemetry Collector repository.

Fork Versioning Description

Let's take the recent release as example: 0.43.0-0.1.

• 0.43.0 is the version of open-telemetry/opentelemetry-collector-contrib that SigNoz is synced.
• 0.1 is the fork version, where major version is bumped at each major feature releases and breaking changes, while minor version is bumped for the patch releases.

OpenTelemetry Collector Contrib

This is a repository for OpenTelemetry Collector contributions that are not part of the core distribution of the Collector. Typically, these contributions are vendor specific receivers/exporters and/or components that are only useful to a relatively small number of users.

Please note that this repository and its releases are a superset of the core repository.

Contributing

See CONTRIBUTING.md.

opentelemetry-collector-contrib's Issues

clickhousemetricsexporter panic on inital install of SigNoz

Describe the bug

This crash periodically occurs on a new installation of SigNoz 0.8 when there is no data

2022-05-16T06:25:21.858Z	info	service/telemetry.go:95	Setting up own telemetry...
2022-05-16T06:25:21.859Z	info	service/telemetry.go:115	Serving Prometheus metrics	{"address": ":8888", "level": "basic", "service.instance.id": "91608d37-eea4-4139-8dd7-c3258df6798c", "service.version": "latest"}
2022-05-16T06:25:21.859Z	info	service/collector.go:229	Starting otelcontribcol...	{"Version": "latest", "NumCPU": 2}
2022-05-16T06:25:21.859Z	info	service/collector.go:124	Everything is ready. Begin running and processing data.
panic: runtime error: index out of range [-1]

goroutine 105 [running]:
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousemetricsexporter.addSingleHistogramDataPoint({0xc00054c000}, {0x8}, {0x46df53}, {0x0, 0x0}, 0x5, 0xc000430320)
	/src/exporter/clickhousemetricsexporter/helper.go:397 +0x8bd
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousemetricsexporter.(*PrwExporter).PushMetrics(0xc00054c000, {0x4fe1728, 0xc0006ac360}, {0x4fe1760})
	/src/exporter/clickhousemetricsexporter/exporter.go:168 +0xad7
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsRequest).export(0x4fe1760, {0x4fe1728, 0xc0006ac360})
	/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/metrics.go:67 +0x34
go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send(0xc0009ee248, {0x504d4d0, 0xc00098c120})
	/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/common.go:232 +0x96
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send(0xc000a5e000, {0x504d4d0, 0xc00098c120})
	/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:176 +0x5eb
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send(0xc00011c930, {0x504d4d0, 0xc00098c120})
	/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/metrics.go:134 +0x88
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1({0x43215a0, 0xc00098c120})
	/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry_inmemory.go:105 +0x5c
go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume(0xc000cabfa8, {0x43215a0, 0xc00098c120})
	/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/internal/bounded_memory_queue.go:99 +0x2c
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2()
	/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/internal/bounded_memory_queue.go:78 +0xd6
created by go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers
	/go/pkg/mod/go.opentelemetry.io/[email protected]/exporter/exporterhelper/internal/bounded_memory_queue.go:68 +0xa5

Failing line sugests pt.BucketCounts() is 0 when there is no data:

cumulativeCount += pt.BucketCounts()[len(pt.BucketCounts())-1]

Steps to reproduce

helm --namespace platform install signoz signoz/signoz --set clickhouseOperator.storage=100Gi --set frontend.image.tag="0.8.0" --set queryService.image.tag="0.8.0"

do nothing else

What version did you use?
Version: 0.8.0

What config did you use?
Config: (e.g. the yaml config file)

otel-collector-metrics-config.yaml:
----
exporters:
  clickhousemetricswrite:
    endpoint: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=${CLICKHOUSE_DATABASE}&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}
extensions:
  health_check: {}
  zpages: {}
processors:
  batch:
    send_batch_size: 1000
    timeout: 10s
receivers:
  otlp:
    protocols:
      grpc: null
      http: null
  prometheus:
    config:
      scrape_configs:
      - job_name: otel-collector
        scrape_interval: 30s
        static_configs:
        - targets:
          - signoz-otel-collector:8889
service:
  extensions:
  - health_check
  - zpages
  pipelines:
    metrics:
      exporters:
      - clickhousemetricswrite
      processors:
      - batch
      receivers:
      - otlp
      - prometheus

Environment
OS: "Ubuntu 18.04"
Azure Kubernetes

Migration down failing for clickhousetracesexporter

Schema migration for JSON Object type

We will need to do data migrations for tables. The time_series table needs the migration for labels. The clickhouse-go doesn't support the JSON Object type yet. We should be able to achieve the same with queries but there is an unexpected with JSON type https://bit.ly/3aem6wG.

For example.

ALTER TABLE time_series ADD COLUMN metric_name LowCardinality(String) CODEC(ZSTD(1))


ALTER TABLE time_series UPDATE metric_name=JSONExtractString(labels, '__name__') WHERE true; // this one succeeds

ALTER TABLE signoz_metrics_v2.time_series_temp MODIFY COLUMN labels JSON CODEC(ZSTD(5)); // this one fails

Or we can create a new table and migrate data

CREATE TABLE signoz_metrics_v2.time_series_json_zstd
(
   `date` Date CODEC(DoubleDelta, LZ4),
   `metric_name` LowCardinality(String) CODEC(ZSTD(1)),
   `fingerprint` UInt64 CODEC(DoubleDelta, LZ4),
   `labels` Object('json') CODEC(ZSTD(5))
)
ENGINE = ReplacingMergeTree
PARTITION BY date
ORDER BY (fingerprint, metric_name)


INSERT INTO signoz_metrics_v2.time_series_json_zstd SELECT
    date,
    JSONExtractString(labels, '__name__') AS metric_name,
    fingerprint,
    labels
FROM time_series

DROP TABLE signoz_metrics_v2.time_series

// cc @ankitnayan

database not update?

opentelemetry-collector-contrib/exporter/clickhouseexporter/clickhouse_factory.go

Line 119 in 90625b1

clickhouseUrl = fmt.Sprintf("clickhouse://%s:%s@%s/%s?x-multi-statement=true", username[0], password[0], host, database)

When I deploy the test environment, use an external database and don't have permission to access the default default database, like this:

use external Clickhouse start failed

Describe the bug

[root@wwww signoz]# kubectl -n signoz get pod signoz-otel-collector-6cdf8f5698-kf9ms  -o yaml
apiVersion: v1
kind: Pod
......
spec:
  containers:
  - command:
    - /otelcontribcol
    - --config=/conf/otel-collector-config.yaml
    env:
    - name: CLICKHOUSE_HOST
      value: 172.20.42.189
    - name: CLICKHOUSE_PORT
      value: "9013"
    - name: CLICKHOUSE_HTTP_PORT
      value: "8123"
    - name: CLICKHOUSE_CLUSTER
      value: cluster
    - name: CLICKHOUSE_DATABASE
      value: mydb
    - name: CLICKHOUSE_TRACE_DATABASE
      value: mydb
    - name: CLICKHOUSE_USER
      value: xxxxxx
    - name: CLICKHOUSE_PASSWORD
      value: xxxxxxxxx
    - name: CLICKHOUSE_SECURE
      value: "false"
    - name: CLICKHOUSE_VERIFY
      value: "false"
......

[root@wwww signoz]# kubectl -n signoz logs -f signoz-otel-collector-6cdf8f5698-kf9ms
022-07-20T11:05:38.088Z        error   clickhousetracesexporter/writer.go:101  Could not write a batch of spans        {"kind": "exporter", "name": "clickhousetraces", "error": "code: 81, message: Database signoz_traces doesn't exist"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter
        /src/exporter/clickhousetracesexporter/writer.go:101
2022-07-20T11:05:48.097Z        error   clickhousetracesexporter/writer.go:101  Could not write a batch of spans        {"kind": "exporter", "name": "clickhousetraces", "error": "code: 81, message: Database signoz_traces doesn't exist"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter
        /src/exporter/clickhousetracesexporter/writer.go:101
2022-07-20T11:05:58.098Z        error   clickhousetracesexporter/writer.go:101  Could not write a batch of spans        {"kind": "exporter", "name": "clickhousetraces", "error": "code: 81, message: Database signoz_traces doesn't exist"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhousetracesexporter.(*SpanWriter).backgroundWriter
        /src/exporter/clickhousetracesexporter/writer.go:101
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x10636f2]

What did you expect to see?
use signoz helm chart deploy in kubernetes

What did you see instead?
A clear and concise description of what you saw instead.

What version did you use?
Version: v0.10.0, 36283d3

panic: runtime error: index out of range

Stack trace

panic: runtime error: index out of range [17] with length 17 [recovered]
	panic: runtime error: index out of range [17] with length 17

goroutine 6470 [running]:
go.opentelemetry.io/otel/sdk/trace.(*recordingSpan).End(0x14000cf1e00, {0x0, 0x0, 0x0})
	/Users/makeavish/go/pkg/mod/go.opentelemetry.io/otel/[email protected]/trace/span.go:346 +0x8cc
panic({0x1070031a0, 0x14004ae6a68})
	/usr/local/go/src/runtime/panic.go:1052 +0x2b0
github.com/open-telemetry/opentelemetry-collector-contrib/processor/signozspanmetricsprocessor.(*processorImp).updateLatencyMetrics(0x1400071ae00, {0x14004b90cc0, 0x3e}, 0x42b0c6f7a0b5c0fa, 0x11)
	/Users/makeavish/signoz/opentelemetry-collector-contrib/processor/signozspanmetricsprocessor/processor.go:542 +0x264
github.com/open-telemetry/opentelemetry-collector-contrib/processor/signozspanmetricsprocessor.(*processorImp).aggregateMetricsForSpan(0x1400071ae00, {0x140045a7ea0, 0x10}, {0x14004abf930}, {0x140008febe0})
	/Users/makeavish/signoz/opentelemetry-collector-contrib/processor/signozspanmetricsprocessor/processor.go:509 +0x244
github.com/open-telemetry/opentelemetry-collector-contrib/processor/signozspanmetricsprocessor.(*processorImp).aggregateMetricsForServiceSpans(0x1400071ae00, {0x140008febe0}, {0x140045a7ea0, 0x10})
	/Users/makeavish/signoz/opentelemetry-collector-contrib/processor/signozspanmetricsprocessor/processor.go:490 +0x128
github.com/open-telemetry/opentelemetry-collector-contrib/processor/signozspanmetricsprocessor.(*processorImp).aggregateMetrics(0x1400071ae00, {0x140047a9cf8})
	/Users/makeavish/signoz/opentelemetry-collector-contrib/processor/signozspanmetricsprocessor/processor.go:479 +0x130
github.com/open-telemetry/opentelemetry-collector-contrib/processor/signozspanmetricsprocessor.(*processorImp).ConsumeTraces(0x1400071ae00, {0x10741cd58, 0x14003939380}, {0x140047a9cf8})
	/Users/makeavish/signoz/opentelemetry-collector-contrib/processor/signozspanmetricsprocessor/process
ogo.opentelemetry.io/collector/model/otlpgrpc.rawTracesServer.Export({{0x1073a3520, 0x1400000c978}}, {0x10741cd58, 0x14003939260}, 0x140047a9c50)
	/Users/makeavish/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/otlpgrpc/traces.go:166 +0x64
go.opentelemetry.io/collector/model/internal/data/protogen/collector/trace/v1._TraceService_Export_Handler.func1({0x10741cd58, 0x14003939260}, {0x10713a720, 0x140047a9c50})
	/Users/makeavish/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/data/protogen/collector/trace/v1/trace_service.pb.go:217 +0xbc
go.opentelemetry.io/collector/config/configgrpc.enhanceWithClientInformation.func1({0x10741cd58, 0x14003939170}, {0x10713a720, 0x140047a9c50}, 0x14004a87f60, 0x140047a9c68)
	/Users/makeavish/go/pkg/mod/go.opentelemetry.io/[email protected]/config/configgrpc/configgrpc.go:382 +0x8c
google.golang.org/grpc.chainUnaryInterceptors.func1.1({0x10741cd58, 0x14003939170}, {0x10713a720, 0x140047a9c50})
	/Users/makeavish/go/pkg/mod/google.golang.org/[email protected]/server.go:1116 +0xdc
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x10741cd58, 0x14003939170}, {0x10713a720, 0x140047a9c50}, 0x14004a87f60, 0x14004af0600)
	/Users/makeavish/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:325 +0x5f4
google.golang.org/grpc.chainUnaryInterceptors.func1.1({0x10741cd58, 0x14003938ea0}, {0x10713a720, 0x140047a9c50})
	/Users/makeavish/go/pkg/mod/google.golang.org/[email protected]/server.go:1119 +0x1cc
google.golang.org/grpc.chainUnaryInterceptors.func1({0x10741cd58, 0x14003938ea0}, {0x10713a720, 0x140047a9c50}, 0x14004a87f60, 0x140047a9c68)
	/Users/makeavish/go/pkg/mod/google.golang.org/[email protected]/server.go:1121 +0x1d0
go.opentelemetry.io/collector/model/internal/data/protogen/collector/trace/v1._TraceService_Export_Handler({0x106a6ff60, 0x1400011f870}, {0x10741cd58, 0x14003938ea0}, 0x140006bf8c0, 0x14000cc8d80)
	/Users/makeavish/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/data/protogen/collector/trace/v1/trace_service.pb.go:219 +0x1fc
google.golang.org/grpc.(*Server).processUnaryRPC(0x14000b1b500, {0x107474bc0, 0x14004a6eea0}, 0x14003ce0b40, 0x1400067d3b0, 0x109aa7b70, 0x0)
	/Users/makeavish/go/pkg/mod/google.golang.org/[email protected]/server.go:1282 +0x1254
google.golang.org/grpc.(*Server).handleStream(0x14000b1b500, {0x107474bc0, 0x14004a6eea0}, 0x14003ce0b40, 0x0)
	/Users/makeavish/go/pkg/mod/google.golang.org/[email protected]/server.go:1616 +0x80c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0x140079e02a0, 0x14000b1b500, {0x107474bc0, 0x14004a6eea0}, 0x14003ce0b40)
	/Users/makeavish/go/pkg/mod/google.golang.org/[email protected]/server.go:921 +0xa4
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/Users/makeavish/go/pkg/mod/google.golang.org/[email protected]/server.go:919 +0x260
Socket connection to remote was closed
Process exiting with code: 0

@makeavish any chance you were sending some data which has some really big duration? Please share the max duration value in you load data.

Table creation and schema migration for metrics tables

Table creation is currently part of code https://github.com/SigNoz/opentelemetry-collector-contrib/blob/develop/exporter/clickhousemetricsexporter/clickhouse.go#L95. We should pull it out into sql files. This will help us maintain the schema migration in similar fashion to trace tables.

Check impact of keeping cache size to 1M or 100M in signozspanmetricsprocessor

It is not possible to determine the cache size beforehand as users keep on adding services to signoz. We should have an idea of the impact of keeping a high number for dimensions_cache_size like 1M or 100M. Or should we change the size dynamically based on some metric/number?

Add otellogsexporter to otel collector with the finalized schema

• Add the schema changes to otellogsexporter.

Collector crashes due to custom timezone configured on Clickhouse server

Describe the bug
When running a clickhouse cluster with the America/New_York timezone configured the collector crashes with this error message:

Error: cannot build exporters: error creating clickhousetraces exporter: error connecting to primary db: could not load time location: unknown time zone America/New_York
2022/07/20 18:14:42 collector server run finished with error: cannot build exporters: error creating clickhousetraces exporter: error connecting to primary db: could not load time location: unknown time zone America/New_York

It seems the otel-collector has no time zone data included in its binary or within the docker image.

Steps to reproduce

1. Configure a clickhouse server with the America/New_York timezone or any other timezone.
2. Run the collector against the clickhouse cluster and it should exit in error (crash).

What did you expect to see?
The collector should not crash and support whatever timezone is set on the server.

What did you see instead?
Indefinite crashing of the collector; unusable

What version did you use?
Version: signoz/otelcontribcol:0.45.1-1.1

Extended above image with this dockerfile:

FROM signoz/otelcontribcol:0.45.1-1.1
COPY otel-collector-config.yaml /etc/otel/config.yaml

What config did you use?
Config:

receivers:
  otlp/spanmetrics:
    protocols:
      grpc:
        endpoint: localhost:12345
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    send_batch_size: 10000
    send_batch_max_size: 11000
    timeout: 10s
  signozspanmetrics/prometheus:
    metrics_exporter: prometheus
    latency_histogram_buckets: [100us, 1ms, 2ms, 6ms, 10ms, 50ms, 100ms, 250ms, 500ms, 1000ms, 1400ms, 2000ms, 5s, 10s, 20s, 40s, 60s ]
    dimensions_cache_size: 10000

exporters:
  clickhousetraces:
    datasource: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=signoz_traces&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PWD}
  clickhousemetricswrite:
    endpoint: tcp://${CLICKHOUSE_HOST}:${CLICKHOUSE_PORT}/?database=signoz_metrics&username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PWD}
    resource_to_telemetry_conversion:
      enabled: true
  prometheus:
    endpoint: 0.0.0.0:8889

extensions:
  health_check:
    endpoint: 127.0.0.1:13133
service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [signozspanmetrics/prometheus, batch]
      exporters: [clickhousetraces]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhousemetricswrite]
    metrics/spanmetrics:
      receivers: [otlp/spanmetrics]
      exporters: [prometheus]

Environment
OS: Ran in it in Docker on my Mac

Additional context
I fixed the issue using this dockerfile which downloaded TZ data and baked it into the image (inspired by this SO post):

FROM alpine:3.15.5 as build
ADD https://github.com/golang/go/raw/master/lib/time/zoneinfo.zip /zoneinfo.zip
RUN chmod 777 /zoneinfo.zip

FROM signoz/otelcontribcol:0.45.1-1.1
ENV ZONEINFO /zoneinfo.zip
COPY --from=build /zoneinfo.zip /zoneinfo.zip
COPY otel-collector-config.yaml /etc/otel/config.yaml

Collector booted up without exception after this fix.

With Go 1.15 you can also embed this into the binary instead of installing into the docker image: https://go.dev/doc/go1.15#time/tzdata

Either approach will work I believe (I only tested the dockerfile approach).

Slack Signoz thread: https://signoz-community.slack.com/archives/C01HWQ1R0BC/p1658340994426549

Remove deprecate columns (clickhouse trace exporter)

With this PR columns like HTTP status code, gRPC status code and gRPC Method are deprecated and has to removed.

Collector container crashes with status code 137

Main issue in observed in one of deployment env. Here https://github.com/SigNoz/opentelemetry-collector-contrib/blob/develop/exporter/clickhousemetricsexporter/clickhouse.go#L166-L186 we are loading the data from time_series table into memory. It unmarshalls the whole json encoded data into memory and that operation is known to be memory heavy. The container crashed instantly after it brought up as there is not enough memory.