mjpitz/cognative

cognative is an opinionated, minimalistic approach to business intelligence 🧠 and operations 🚨.

cognative

cognative is an opinionated, minimalistic approach to business intelligence 🧠 and operations 🚨.

Goals

1. Reduce the complexity and number of systems operations staff is required to have expertise on.
2. Maximize the breadth of functionality they are able to support across business intelligence and operations.
3. Simplify the experience for developers.
4. Enable companies to own their own data.
5. Be flexible enough to support a variety of deployment models.

Background

I started cognative as a recent reflection on the operational constraints faced by smaller organizations and how to make best use of their skills and expertise. Requiring operations staff to be experts on a large number of systems is not sustainable and will only lead to burn out. And yet, an increasing number of complex systems need to be run to support business intelligence and operations.

The project name is a clever play on words. It takes the acronym for the tech stack (COG) and joins it with "native". It came out of a discussion I had with Tim Banks about the pitfalls of "traditional" operations tech stacks and how it's often short-sighted once you start to consider the broader needs of business intelligence. COG stands for Clickhouse, OpenTelemetry, and Grafana.

• Clickhouse is our data warehouse layer. It stores logs, metrics, traces, as well as other business related data sets. Clickhouse also comes with tons of integrations which makes it easy to source data from a variety of locations.
• OpenTelemetry provides instrumentation for our ecosystem. Information is collected from a variety of sources using vendor neutral solutions. The community also supports a large number of languages, making it easy to add to many systems today.
• Grafana provides our data exploration, visualization, and alerting layer. It can easily integrate with on-call solutions like PagerDuty, OpsGenie, BetterStack, and many more.

License

While no code currently exists in this repository, it seemed prudent to include a software license early on.

Copyright (C) 2024 The cognative authors
SPDX-License-Identifier: AGPL-3.0-only

cognative's Issues

Port helm chart out of mya's repository once public

Once this repository is public, we should look at porting out the standalone Helm chart from Mya's charts repository (https://github.com/mjpitz/mjpitz). I initially put it there so I could leverage my existing signing infrastructure after hitting issues with chart-releaser running on private repositories. I'd also like to iron out issues with the bitnami chart so our chart repository contains more than one deployment option.

Also, once the chart is under the scope of this repository, we will be able to add additional capabilities such as automatically including the dashboards and alerts generated from this project.

Example references

It would be really good to enumerate a handful of example references that we want to illustrate in the project. For example, I figured having some code samples for emitting metrics from applications would be useful (we can probably point people at the opentelemtry.io docs for this). Another example was how to write data enrichment pipelines to be able transform and enrich data within the ecosystem...

Scope of work

• enumerate a list of useful code references that we want to be able to include in the repo

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update helm release grafana to v7.3.11
• chore(deps): update helm release clickhouse to v6

Detected dependencies

docker-compose

docker-compose.yaml

• clickhouse/clickhouse-server 24.4
• otel/opentelemetry-collector-contrib 0.100.0
• grafana/grafana-oss 10.4.2

github-actions

.github/workflows/code-style.yaml

• actions/checkout v4
• actions/setup-node v4

helmv3

charts/cognative-bitnami/Chart.yaml

• clickhouse 5.3.3
• opentelemetry-collector 0.90.1
• opentelemetry-collector 0.90.1
• opentelemetry-collector 0.90.1
• grafana 7.3.10

npm

package.json

• prettier ^3.2.5

---

• Check this box to trigger a request for Renovate to run again on this repository

Replicate kubernetes-monitoring/kubernetes-mixin

The monitoring-mixin project under the Kubernetes space is a great tool for those looking to monitor and alert on events happening within the cluster. While this project is great, the queries that it uses are tied to the kube-prometheus-stack deployment.

The other thing I'm not a huge fan of with this project is jsonnet. Don't get me wrong, I thought it was really interesting at first, but every time I sit down to use it, I need to re-learn it again. Very few people out there have worked with it, making it harder to find contributors.

Scope of work (each should probably be their own task...)

• research jsonnet alternatives
• prototype using jsonnet alternative
• triage dashboards from monitoring-mixin project
  • TODO: enumerate dashboards
• triage alerts from monitoring-mixin project
  • TODO: enumerate dashboards
• research playbooks as a possible inclusion into this project

Document and compare to alternatives

Used to be in the README, but I want to break the content into an actionable ticket...

• ELK - ElasticSearch, LogStash, Kibana
  • The ELK stack is a long-standing solution for logs and metrics.
  • LogStash has a well-established history of being deployed as an ETL pipeline.
• LGTM/P - Loki, Grafana, Tempo, Mimir / Prometheus
  • The full Grafana stack requires a lot of operational experience. It effectively requires learning three new "databases"
    for data that is largely the same. Loki is effectively a database for logs. Tempo, a database for traces. And finally,
    Mimir / Prometheus, a database for metrics. Each of these systems have their own resource usage and scaling requirements.
  • In addition, this is a partial solution as it does not cover the business intelligence side of the world. An additional
    database can be added to support your business analytics, but doing so will only add to the complexity.
• XOG - ?, OpenTelemetry, Grafana
  • Because OpenTelemetry is so flexible, why not pick one of the many other databases?
  • For a starter or simplified deployment, this is a great option. Leveraging an existing database technology may simplify
    complexity today, it will pose some interesting technical challenges later on. Importing data from an existing database
    technology into a solution like Clickhouse will be relatively easy.