greptimeteam/promql-parser

PromQL Rust parser

PromQL Lexer and Parser

The goal of this project is to build a PromQL lexer and parser capable of parsing PromQL that conforms with Prometheus Query.

Example

To parse a simple instant vector selector expression:

use promql_parser::parser;

let promql = r#"
    http_requests_total{
        environment=~"staging|testing|development",
        method!="GET"
    } offset 5m
"#;

match parser::parse(promql) {
    Ok(expr) => {
        println!("Prettify:\n\n{}", expr.prettify());
        println!("AST:\n{expr:?}");
    }
    Err(info) => println!("Err: {info:?}"),
}

or you can directly run examples under this repo:

cargo run --example parser

This outputs:

Prettify:
http_requests_total{environment=~"staging|testing|development",method!="GET"} offset 5m

AST:
VectorSelector(VectorSelector { name: Some("http_requests_total"), matchers: Matchers { matchers: [Matcher { op: Re(staging|testing|development), name: "environment", value: "staging|testing|development" }, Matcher { op: NotEqual, name: "method", value: "GET" }] }, offset: Some(Pos(300s)), at: None })

PromQL compliance

This crate declares compatible with prometheus v2.45.0, which is released at 2023-06-23. Any revision on PromQL after this commit is not guaranteed.

Community Extensions

There are a number of community projects that extend promql-parser or provide integrations with other systems.

Language Bindings

• py-promql-parser Python binding of this crate.

Known Uses

Here are some of the projects known to use promql-parser:

• GreptimeDB Open Source & Cloud Native Distributed Time Series Database
• OpenObserve High performance, petabyte scale Elasticsearch/Splunk/Datadog alternative for (logs, metrics, traces, RUM, Error tracking, Session replay)

If your project is using promql-parser, feel free to make a PR to add it to this list.

Contributing

Contributions are highly encouraged!

Pull requests that add support for or fix a bug in a feature in the PromQL will likely be accepted after review.

Licensing

All code in this repository is licensed under the Apache License 2.0.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be licensed as above, without any additional terms or conditions.

promql-parser's Issues

walk_expr does not traverse function arguments

For example this query:

(sum by (namespace) (max_over_time(pg_stat_activity_count{}[1h])))

the walk looks like

walking... Paren(ParenExpr { expr: Aggregate(AggregateExpr { op: TokenType(53), expr: Call(Call { func: Function { name: "max_over_time", arg_types: [Matrix], variadic: false, return_type: Vector }, args: FunctionArgs { args: [MatrixSelector(MatrixSelector { vector_selector: VectorSelector { name: Some("pg_stat_activity_count"), matchers: Matchers { matchers: {Matcher { op: Equal, name: "__name__", value: "pg_stat_activity_count" }} }, offset: None, at: None }, range: 3600s })] } }), param: None, modifier: Some(Include({"namespace"})) }) })
walking... Aggregate(AggregateExpr { op: TokenType(53), expr: Call(Call { func: Function { name: "max_over_time", arg_types: [Matrix], variadic: false, return_type: Vector }, args: FunctionArgs { args: [MatrixSelector(MatrixSelector { vector_selector: VectorSelector { name: Some("pg_stat_activity_count"), matchers: Matchers { matchers: {Matcher { op: Equal, name: "__name__", value: "pg_stat_activity_count" }} }, offset: None, at: None }, range: 3600s })] } }), param: None, modifier: Some(Include({"namespace"})) })
walking... Call(Call { func: Function { name: "max_over_time", arg_types: [Matrix], variadic: false, return_type: Vector }, args: FunctionArgs { args: [MatrixSelector(MatrixSelector { vector_selector: VectorSelector { name: Some("pg_stat_activity_count"), matchers: Matchers { matchers: {Matcher { op: Equal, name: "__name__", value: "pg_stat_activity_count" }} }, offset: None, at: None }, range: 3600s })] } })

I believe it misses function arguments in Call. I think it also misses vector selector of a matrix selector.

feat: Support Filtering by multiple or filters

@yuanbohan Hi Yuanbo, I want to support or filter but even if I add something in promql.y I get the error "invalid promql query", I am new to yacc and I have used antlr before (anltr is relatively It's relatively simple, just modify the grammar definition, but yacc doesn't seem to work), so I hope you can give me some suggestions.

【victoriametrics or expression】

https://docs.victoriametrics.com/keyconcepts/#filtering-by-multiple-or-filters

【promql.y】

label_match_list -> Result<Matchers, String>:
                label_match_list COMMA label_matcher { Ok($1?.append($3?)) }
        |       label_match_list LOR label_matcher { Ok($1?.append($3?)) }  // support or filter
        |       label_matcher { Ok(Matchers::empty().append($1?)) }
;

【error】
expression：
a{label1="1" or label2="2"}

 Err("invalid promql query")

Lex: context with &str and lifetime, instead of String and clone

Reference

• calc_manual_lex
• grmtools parsing idioms

Cannot parse `time() > 1`

I got an error when trying to parse PromQL time() > 1:

Failure during query parsing, query: time() >= 1, source: comparisons between scalars must use BOOL modifier

This expression is on the compliance list, we should add support for it.

Multiple `__name__` matchers not permitted

A PromQL query can contain multiple __name__ matchers, e.g.

{__name__=~"node_cpu_.+", __name__=~".+total"}

Here's a demo of such a query

However with this lib it fails with metric name must not be set twice: '.+total' or 'node_cpu_.+'

Many thanks for this high quality effort btw!

Merge AggModifier and VectorMatchModifier

pub enum AggModifier {
    By(Labels),
    Without(Labels),
}

pub enum VectorMatchModifier {
    On(Labels),
    Ignoring(Labels),
}

AggModifier and VectorMatchModifier expresses a similar concept, but are different structs. This may increase the burden for downstream consumers to handle them both. I propose to merge them into one struct, like

pub enum LabelModifier {
  Include(Labels),
  Exclude(Labels),
}

Lexer: keep the lines in lex

LRNonStreamingLexer needs to keep the lines information.

Support pretty print of Expr

I.e., be able to convert a parsed Expr back to PromQL literal. This is useful for debug and implement format interface in Prometheus

Add Expr Visitor trait

Add ExprVisitor like this in DataFusion's PlanVisitor:

pub trait PlanVisitor {
    type Error;

    fn pre_visit(&mut self, plan: &LogicalPlan) -> Result<bool, Self::Error>;

    fn post_visit(&mut self, _plan: &LogicalPlan) -> Result<bool, Self::Error> {
        Ok(true)
    }
}

It's a util trait convenient for traversing an Expr. Like in this scenario https://github.com/GreptimeTeam/greptimedb/blob/6833b405d95239e4bfd2b000106f5b1ddc94bfac/src/servers/src/promql.rs#L373-L398

check regex compatibility

Error: better error mechanism is needed

Use String in terms of easy coding for now. Chumsky can be a good choice. This should include at least:

• Parser
• Lexer

Add Extension Expr type for customizing the parsed AST

Proposal

This ticket proposes to add a new variant to Expr enum, which allows others to customize and insert extra nodes into the Expr. Like

pub trait ExtensionExpr {
    fn as_any(&self) -> &dyn Any;

    fn name(&self) -> &str;
}

pub enum Expr {
    // omitted
    VectorSelector(VectorSelector),
    MatrixSelector(MatrixSelector),
    Call(Call),
    Ext(Box<dyn ExtensionExpr>),
}

Rationale

With this functionality, we can add other nodes to the AST like EXPLAIN in GreptimeTeam/greptimedb#986, or rewrite the parsed AST to suit the planner (i.e., preprocess before the planning phase).

Parser: rules conflict

UnaryExpr, BinaryExpr, signed_number conflicts because of the precedence of +/-

use nom

@yuanbohan May I ask if we have considered using nom to rewrite the parsing of promql and sql?

support native histograms

• prometheus/prometheus#11170
• prometheus/prometheus#11218

Failing to parse valid regex

The following regex fails to parse in rust, but works with the official go parser:

uri=~"/v[1-9]/.*/{gid}/{uid}"

Reproduce

use promql_parser::parser;

fn main() {
    let promql = r#"rate(http_server_requests_seconds_bucket{method="GET",status=~"2..",uri=~"/v[1-9]/.*/{gid}/{uid}",le="0.25"}[1h])"#;

    match parser::parse(promql) {
        Ok(expr) => {
            println!("Prettify:\n\n{}", expr.prettify());
            println!("AST:\n{expr:?}");
        }
        Err(info) => println!("Err: {info:?}"),
    }
}

# `Err: "illegal regex for /v[1-9]/.*/{gid}/{uid}"`

The same query works with in go:

package main

import (
	"github.com/prometheus/prometheus/promql/parser"
)

func main() {
	_, err := parser.ParseExpr(`rate(http_server_requests_seconds_bucket{method="GET",status=~"2..",uri=~"/v[1-9]/.*/{gid}/{uid}",le="0.25"}[1h])`)

	if err != nil {
		panic(err)
	}
}