openmeterio/openmeter

Cloud Metering for AI, Billing and FinOps. Collect and aggregate millions of usage events in real-time.

OpenMeter

OpenMeter is a Real-Time and Scalable Usage Metering for AI, Usage-Based Billing, Infrastructure, and IoT use-cases.

Learn more about OpenMeter at https://openmeter.io.

Quickstart

Check out the quickstart guide for a 5-minute overview and demo of OpenMeter.

Links

• Examples
• Demo Video
• Stripe UBP Demo
• Decisions

Community

To engage with our community, you can use the following resources:

• Discord - Get support or discuss the project.
• Contributing to OpenMeter - Start here if you want to contribute.
• Code of Conduct - Our community guidelines.
• Adopters - Companies already using OpenMeter.
• Blog - Stay up-to-date.

Examples

See our examples to learn about common OpenMeter use-cases.

• Metering OpenAI Chat GPT Usage
• Metering Kubernetes Pod Execution Time
• Usage Based Billing with Stripe

API

OpenMeter exposes a REST API for integrations.

Client SDKs

Currently, we offer the following Client SDKs:

• Node.js
• Python
• Go
• Web

In cases where no specific SDK is available for your preferred programming language, you can utilize the OpenAPI definition. Please raise a GitHub issue to request SDK support in other languages.

Development

For an optimal developer experience, it is recommended to install Nix and direnv.

Installing Nix and direnv

Note: These are instructions that SHOULD work in most cases. Consult the links above for the official instructions for your OS.

Install Nix:

sh <(curl -L https://nixos.org/nix/install) --daemon

Consult the installation instructions to install direnv using your package manager.

On MacOS:

brew install direnv

Install from binary builds:

curl -sfL https://direnv.net/install.sh | bash

The last step is to configure your shell to use direnv. For example for bash, add the following lines at the end of your ~/.bashrc:

eval "\$(direnv hook bash)"

Then restart the shell.

For other shells, see https://direnv.net/docs/hook.html.

MacOS specific instructions

Nix may stop working after a MacOS upgrade. If it does, follow these instructions.

---

Run the dependencies:

make up

Run OpenMeter:

make run

Run tests:

make test

Run linters:

make lint

Roadmap

Visit our website at https://openmeter.io for our public roadmap.

License

The project is licensed under the Apache 2.0 License.

openmeter's Issues

Fix logger error strings

slog does not support %w format string

Avoid group by duplicates between Kafka msg key and value for meter

Currently, we copy window and group by properties from Kafka message key to value with AS_VALUE to be able to sink records with desired primary key in Kafka Connect sink connector:

    'pk.mode'                                 = 'record_key',
    'transforms'                              = 'RenameField,ValueToKey,tsWindowStart,tsWindowEnd',
    'transforms.RenameField.type'             = 'org.apache.kafka.connect.transforms.ReplaceField$Value',
    'transforms.RenameField.renames'          = 'WINDOWSTART_TS:WINDOWSTART,WINDOWEND_TS:WINDOWEND',
    'transforms.ValueToKey.type'              = 'org.apache.kafka.connect.transforms.ValueToKey',
    'transforms.ValueToKey.fields'            = 'WINDOWSTART,WINDOWEND',

The challenge is that WINDOWSTART and WINDOWEND is not available key so can't be used as primary key without copying it to values. But then with the built-in transformers we can't combine keys with values to a primary key so we need to copy others keys to values too. Leading to duplicating data between Kafka keys and values.

Find a way to eliminate the need for duplicating properties between kafka key and value. Likely we need to create a custom transform to create a sink primary key containing timewindowstart, timewindowend, subject, and all group by(s).

Apply meter best practices to examples

Apply meter config best practices to examples

Do not build docker container for docs and example changes

We could be more selective with container builds and skip them for docs and example changes.
Benefits:

• faster builds
• less noise in container registry

Use npm library in examples

Refactor Node examples to use @openmeter/sdk: #64

Integrate OTel with Sink Worker

Report metrics in Sink Worker around:

• events processed by namespace, status (success | invalid)
• batch sizes by namespce
• deadletter count by namespace, reason (malformed | meter_missing | invalid_value | invalid_groupby)

Support batch event ingestion

Problem

Currently, we only support event ingestion one at a time.

Solution

• The cloudevents specification allows batched HTTP content mode.
• Update the OpenAPI document to include Content-Type: application/cloudevents-batch+json with a body of an array of events.
• Use the JSON decoder for the content type: https://github.com/openmeterio/openmeter/blob/main/internal/server/router/router.go#L24
• Update the request handler to unmarshal the body based on the content-type: https://github.com/openmeterio/openmeter/blob/main/internal/server/router/router.go#L53
• Publish the events one by one (transaction handling would be nice, can be implemented later)

Improve Kafka Config around disabling IPv6 for OSX

For Kafka config today we disable IPv6 for localhost due to Kafka brokers on OSX return the IPv6 addresses first, causing connection error logs.

On the long term make this configurable and more robust by checking.
For example removing localhost and using the loopback IP address would also be okay.

// This is needed when using localhost brokers on OSX,
// since the OSX resolver will return the IPv6 addresses first.
if strings.Contains(c.Broker, "localhost") || strings.Contains(c.Broker, "127.0.0.1") {
   config["broker.address.family"] = "v4"
}

Create and configure DQL (Dead Letter Queue) topic programmatically

Create the DQL topic via Kafka Admin API to make retention configurable.

Related to: #225

Keep API version in sync with releases

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Problem Description

The API version isn't updated automatically: https://github.com/openmeterio/openmeter/blob/main/api/openapi.yaml#L4

Proposed Solution

Add a CI workflow or pre-commit hook to update the version automatically.

Alternatives Considered

No response

Additional Information

No response

Deprecate average aggregation

Current implementation of avg is confusing and easy to get incorrect values.

Currently avg only makes sense on window size. Over windows avg is incorrect.

Detect OpenAPI breaking changes

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Problem Description

Make sure the OpenAPI won't have breaking changes after the 1.0 stable release.

Proposed Solution

Add to CI: https://github.com/Tufin/oasdiff

Alternatives Considered

No response

Additional Information

No response

Fix FOSSA runs on Dependabot PRs

Return RFC7807 type errors

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Problem Description

Currently we return a custom error format to clients. While that's not necessarily a problem, some standardisation would be nice.

Proposed Solution

Return errors formatted according to RFC7807.

A library I've used in the past: https://github.com/moogar0880/problems

Alternatives Considered

Given the size of the library, we could just implement our own problem types.

Additional Information

No response

Time-based quotas + webhook events when quota is reached

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Problem Description

A lot of projects use quota-based billing instead of usage-based billing (eg. pay $X for Y credits/month). Currently the only way to implement this with OpenMeter is to make a GET request after every event to check the meter total. That's not very scalable.

Proposed Solution

This is how I imagine it would work:

1. Each meter can have optional associated quotas with timeframes of Daily/Weekly/Monthly/Yearly/No timeframe. The timeframe determines when a quota resets.
2. A webhook event is triggered when a quota is reached within the given timeframe. This can be used to inform another application that a quota has been reached.
3. Allowing multiple quotas to be associated with each meter would let us trigger webhooks events for things warnings too.

Smaller and indie projects often prefer quotas because of the simpler billing model. I think adding this functionality would open up OpenMeter to a lot more developers.

PS. Another option could be returning the current meter's usage in the HTTP response when every event is posted. But I'm assuming that would require a database lookup on every event, which could get expensive.

Alternatives Considered

A simple Redis-based counter that is checked before each outgoing request is made.

Additional Information

No response

Support GPT Assistant API ?

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Problem Description

GPT Assistant API is really hard to track.

Proposed Solution

Would it be possible for openmeter to support it ? I guess solving this issue, would benefit a lot of people and bring a lot of traction to the project as Assistant API has a lot of usage and no one knows how to bill users for it.

Alternatives Considered

No response

Additional Information

No response

Refactor Kafka Connect configuration

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Problem Description

#229 refactors Kafka Connect, but I believe there is more room for improvement.

The configuration feels a bit "disorganized" at the moment.

I'd also note that we probably do not want to use this feature in production, but want to manually configure Kafka Connect (I may be wrong about that).

Proposed Solution

I can imagine two potential routes, both involve supporting multiple connectors.

The advantage of supporting multiple connectors is that we can experiment with different configurations without breaking existing ones.

Option one is to allow passing raw connector configuration:

sink:
  kafkaConnect:
    enabled: true
    connectors:
      - name: clickhouse
        config:
          connector.class: "com.clickhouse.kafka.connect.ClickHouseSinkConnector"
          database: "openmeter"

          # ...

The downside of this solution is that we have no way to validate the configuration.

An alternative solution would be adding config schema for supported connectors (currently only clickhouse anyway):

sink:
  kafkaConnect:
    enabled: true
    connectors:
      - name: clickhouse
        type: clickhouse
        config:
          database: "openmeter"

          # ...

Although option two is a bit more work, I probably prefer that for the moment because it makes misconfiguration harder.

Alternatives Considered

No response

Additional Information

No response

Sweep: Support batch event ingestion

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Problem Description

Currently, we only support event ingestion one at a time.

Proposed Solution

The cloudevents specification allows batched HTTP content mode.
Update the OpenAPI document to include Content-Type: application/cloudevents-batch+json with a body of an array of events.
Use the JSON decoder for the content type: https://github.com/openmeterio/openmeter/blob/main/internal/server/router/router.go#L24
Update the request handler to unmarshal the body based on the content-type: https://github.com/openmeterio/openmeter/blob/main/internal/server/router/router.go#L53
Publish the events one by one (transaction handling would be nice, can be implemented later)

Alternatives Considered

No response

Additional Information

No response

Python client

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Problem Description

A lot of AI code is written in Python, a nice client would be ideal!

Proposed Solution

Likely generate it off the openapi spec

Alternatives Considered

Might be nice to directly work it into something like FastAPI

Additional Information

No response

Customizable prefix for Kafka topics

Currently, we use a hardcoded prefix om_ to prefix Kafka topics. For example:

• om_events
• om_detected_events
• om_meter_m1

Requirements:

• Support prefixes via header as OM-Prefix.
• Default to om_ prefix.
• Add API to create events and detected_events topics: POST /api/ingests
• By default create om_events and om_detected_events topics.
• Optionally configure ingestion topics via Kafka instead of ksql streams (in this case move partition and scheme a config to Kafka create)

Write some basic tests to ensure configuration is properly loaded

Introduce interface for deadlettering

So we can add another deadletter reporting capability in the future other than the default Kafka dead lettering behavior.

Create new subjects endpoint

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Problem Description

The UI needs to figure out subjects from the returned data

Proposed Solution

Create a new /subjects endpoint for a meter that returns all available subjects.

Alternatives Considered

No response

Additional Information

No response

Implement subject mapping

Implement subject mapping endpoints to assign display name and metadata to subjects

#534

Benthos schedule input does not exhaust non-batch inputs

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.
• I am not looking for support or already pursued the available support channels without success.

OpenMeter Version

v1.0.0-beta.53

Expected Behavior

When wrapping a non-batch input in schedule input, only one document is being read from the input instead of reading all.

Actual Behavior

All documents should be read

Steps To Reproduce

input:
  schedule:
    interval: "10s"
    input:
      sql_select:
        driver: postgres
        dsn: postgres://postgres:postgres@service_a:5432/service_a?sslmode=disable
        table: usage
        columns:
          - id
          - realm_id
          - subject
          - time
        where: time >= ?

        # Interval is 10 seconds, but we go back 15 seconds to make sure no events are missed.
        # Deduplication should take care of duplicate events
        # Tweak this based on your needs (ie. your priority of duplicated event ingestion vs potentially missed events)
        args_mapping: 'root = [ (now().ts_unix() - 15).ts_format(format: "2006-01-02 15:04:05", tz: "UTC") ]'

Additional Information

Workaround: use generate input with sql_select processor and unarchive processor.

Also see benthosdev/benthos#2314

Create new groupBys endpoint

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Problem Description

We may want to filter by groupBy fields in the future.

Proposed Solution

Create a new /groupBys field for meters that returns all values for all groupBy fields

Alternatives Considered

Call it /groupBy-values

Additional Information

No response

New docker compose breaks quickstart

quickstart [main] docker compose version  
Docker Compose version v2.24.6-desktop.1
quickstart [main] docker compose up -d    
service "kafka" can't be used with `extends` as it declare `depends_on`

Related: docker/compose#11544

Run pre-commit hooks in CI

RFC: split StreamingConnector into ingestion and aggregation component

Currently ingestion and aggregation are tightly coupled together. I believe they should be two separate domains, even though the components of the underlying technology are tightly coupled as well.

By separating these modules it becomes easier to reason about the system as each component (ingestion, aggregation) will be responsible for their part only. It also becomes easier to cover the system with tests.

Add CI check for generated code

Ensure that every code is regenerated.

Add redis for dedupe support to Helm chart

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Problem Description

The Helm chart currently requires an external redis for dedupe to work.

Proposed Solution

Add a default redis installation so it's easy to test deduplication.

Alternatives Considered

No response

Additional Information

No response

Move log configuration under telemetry section

Create new meter query endpoint

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Problem Description

The current /values endpoint has a few shortcomings that makes it less useful (at least in the UI):

• Grouping results by default window size when no windowSize parameter is present
• groupBy parameter does not accept multiple values (it does accept comma separated values)

Proposed Solution

Implement a new /query endpoint:

• Do not group results by default window size when no windowSize is present
• Accept array values for groupBy and subject
• Return from and to parameters (and make both of them optional)

The new endpoint will be able to return values aggregated for a given (open ended) interval:

• without any window size, subject or groupBy grouping
• groupped by window size (but not subject or groupBy fields)
• groupped by window size and subject (but not groupBy fields)
• groupped by window size subject and any/all group by fields

The /query endpoint will deprecate the /values endpoint once we have adopted it.

Alternatives Considered

No response

Additional Information

What's the default window size for if we are not going to use it?

Add license header to all files

• Add licensei to the tooling and CI to verify headers (example: https://github.com/bank-vaults/vault-secrets-webhook)
• Add license headers to Go files (addlicense can make it easier for the first time)

Copyright holder is Tailfin Cloud Inc..

Move configuration to its own package and split into smaller files (`config.go` is long as it is and not really readable)

Refactor configuration

Second round of configuration refactor before tagging a stable version:

• #207
• #257
  • Currently ksqldb is the default. There should probably be no default.
• #208
• #205
• Move meters out to a separate file (?)
• Return multiple errors from validation functions
• #230

ClickHouse migrations

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Problem Description

Use migrations for ClickHouse: events table.

Proposed Solution

https://atlasgo.io/guides/clickhouse
https://clickhouse.com/blog/manage-your-clickHouse-schema-as-code-using-atlas

Alternatives Considered

No response

Additional Information

Integrate into Helm chart and docker-compose.
https://atlasgo.io/guides/deploying/image
https://atlasgo.io/guides/testing/docker-compose
https://atlasgo.io/guides/deploying/helm

Add OTEL tracing

Benchmark ClickHouse queries

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Problem Description

The meters are currently implemented as materialized views on the events table. This has a drawback that meters have to be created first for the events to be processed:

Materialized views in ClickHouse are implemented more like insert triggers. If there’s some aggregation in the view query, it’s applied only to the batch of freshly inserted data. Any changes to existing data of source table (like update, delete, drop partition, etc.) does not change the materialized view.
We do not recommend using POPULATE, since data inserted in the table during the view creation will not be inserted in it.

Proposed Solution

Benchmark:

1. reading from the events table directly on query (optional, as we already know it's significantly slower)
2. use projections instead of materialized views

End goal

Back with data that projections can be used or not as an alternative to materialized views.

Additional Information

Seeder: https://github.com/openmeterio/openmeter/blob/main/etc/seed/seed.yaml (make seed)
Grafana K6: https://k6.io/ (load tester)

Provide docs and Helm chart for Kubernetes deployment

Filter events by data properties

Use case, I have two meters, one tracks all durations, one tracks only successfull executions.

{
   ...
   data: {
      duration: 123,
      result: 'success'
   }
}

OpenMeter .NET 8 SDK

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Problem Description

Greetings, since a few days ago .NET 8 was released, which will be long-term supported, it would be great if you made an open meter SDK for .NET

Proposed Solution

Create OpenMeter .NET 8 SDK

Alternatives Considered

No response

Additional Information

No response

Make Kafka optional for dev use cases

In dev environments, we could make Kafka and sink-worker optional to reduce resource footprint.

We can either use an in-memory queue or ClickHouse batch inserts.

Add tracing

Investigate query timeouts due to ClickHouse connection pool limit

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.
• I am not looking for support or already pursued the available support channels without success.

OpenMeter Version

1.0.0-beta

Expected Behavior

We don't exhaust connection pool.the

Actual Behavior

Query timeouts.

Steps To Reproduce

No response

Additional Information

• Is it related to ClickHouse/clickhouse-go#100?
• Maybe query timeout config missing?

Instrument Kafka

Plug-n-play deploy a kubernetes pod usage metering

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Problem Description

I have a cluster and i want to know how much resources a namespace uses via metering the pods in it

Proposed Solution

There is a demo already for it in examples but it is not prod ready, it is a common use-case that can be useful for many users of openmeter

Alternatives Considered

No response

Additional Information

No response

FinOps Open Cost & Usage Specification Support

Preflight Checklist

• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Problem Description

At the moment we use CloudEvent (https://cloudevents.io/) to define OpenMeter ingesting API specification.

{
  "specversion": "1.0",
  "type": "api-calls",
  "id": "00001",
  "time": "2023-01-01T00:00:00.001Z",
  "source": "service-0",
  "subject": "customer-1",
  "data": {
    "duration": "12",
    "path": "/hello"
  }
}

The result of any metering system is costs&usage reports. One of the important use-cases of report usage is data exporting.

There are two goals of data exports:

1. Integration (accounting/payment systems). A scenario in which reports are exported from OpenMeter to target systems such as Stripe, Paypal, OEBS, NetSuite, etc. In this case, each system has its own data format, and standardization is impossible.

2. FinOps Observability (datalakes, dwh). In this scenario, data is typically uploaded to customer's S3 in a specific format at a specific frequency. Then transferred by customers into company-wide DWH/Data Lake. This practice is presented, for example, in all top cloud providers such as GCP, AWS, Azure and some SaaS services.

So, we use CloudEvent CNCF specification for OpenMeter as input at Ingesting API. Probably we should consider FOCUS (FinOps Open Cost & Usage Specification) as OpenMeter output at Reports Exporting API.

Proposed Solution

1. Research CNCF proposed specification - https://focus.finops.org/
2. Add this format as an option in planned S3 exports API if that makes sense

Alternatives Considered

No response

Additional Information

No response

Process Messages in DQL (Dead Letter Queue)

When Kafka Connect ClickHouse gets a malformed message, it sends the entire batch to DQL.
It means the DQL topic can have bad and good messages mixed. Ideally, OpenMeter would have a secondary consumer for DQL, which runs validation against those messages and tries to re-process the ones that it finds the corresponding meter and matches the validator.

Related to: #225

Instrument redis

• Register health check
• Add tracing and metrics