camunda-community-hub/zeeqs

GraphQL API for Zeebe data

zeeqs's Issues

Filter incidents by state

• filter incidents by their state (created|resolved)

I can subscribe for updates of workflows

Using the GraphQL API, I can subscribe for updates of workflows:

• a new workflow is deployed
• a new workflow instance is created
• a workflow instance is completed/terminated

GraphQL's subscriptions should be suitable for this.

Suggestion chang enums to @Enumerated(EnumType.STRING)

Using @Enumerated(EnumType.STRING) on enum fields leads to more meaningful data in the database.

It is much easier to debug something in the database when you can read "ACTIVATED" instead of 0

Would you consider changing to strings ?

I could implement a PR that does the change, and provides a SQL script to migrate the schema and data.

ZeeQS with Hazelcast and Postgresql : Missing relation in Postgresql

Hello everyone !

I am trying to run ZeeQS with Hazelcast and Postgresql in Kubernetes.
I first tried to run this Dockerfile :

version: "2"

networks:
  zeebe_network:
    driver: bridge

services:
  zeebe:
    container_name: zeebe_broker
    image: camunda/zeebe:1.2.3
    environment:
      - ZEEBE_LOG_LEVEL=debug
    ports:
      - "26500:26500"
      - "9600:9600"
      - "5701:5701"
    volumes:
      - ../target/exporter/zeebe-hazelcast-exporter.jar:/usr/local/zeebe/exporters/zeebe-hazelcast-exporter.jar
      - ./application.yaml:/usr/local/zeebe/config/application.yaml
    networks:
      - zeebe_network
      
  zeeqs:
    container_name: zeeqs
    image: ghcr.io/camunda-community-hub/zeeqs:2.1.0
    environment:
      - zeebe.client.worker.hazelcast.connection=zeebe:5701
      - spring.datasource.url=jdbc:postgresql://db:5432/postgres
      - spring.datasource.username=postgres
      - spring.datasource.password=zeebe
      - spring.datasource.driverClassName=org.postgresql.Driver
      - spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.PostgreSQLDialect
    volumes:
      - ./lib/postgresql-42.2.12.jar:/app/libs/postgresql-42.2.12.jar
    ports:
      - "9000:9000"
    depends_on:
      - db
    networks:
      - zeebe_network

  db:
    image: postgres:12.2
    restart: always
    environment:
      POSTGRES_PASSWORD: zeebe
    volumes:
      - database-data:/var/lib/postgresql/data/
    networks:
      - zeebe_network

volumes:
  database-data:

I have this error from Postgresql :

db_1     | 2021-11-29 18:08:01.484 UTC [33] ERROR:  relation "hazelcast_config" does not exist at character 82
db_1     | 2021-11-29 18:08:01.484 UTC [33] STATEMENT:  select hazelcastc0_.id as id1_3_0_, hazelcastc0_.sequence as sequence2_3_0_ from hazelcast_config hazelcastc0_ where hazelcastc0_.id=$1

And from ZeeQS :

zeeqs    | 2021-11-29 18:08:01.499  WARN 1 --- [           main] o.h.engine.jdbc.spi.SqlExceptionHelper   : SQL Error: 0, SQLState: 42P01
zeeqs    | 2021-11-29 18:08:01.500 ERROR 1 --- [           main] o.h.engine.jdbc.spi.SqlExceptionHelper   : ERROR: relation "hazelcast_config" does not exist
zeeqs    |   Position: 82
zeeqs    | 2021-11-29 18:08:01.510  INFO 1 --- [           main] o.h.e.internal.DefaultLoadEventListener  : HHH000327: Error performing load command
zeeqs    | 
zeeqs    | org.hibernate.exception.SQLGrammarException: could not extract ResultSet
...
zeeqs    | Caused by: org.postgresql.util.PSQLException: ERROR: relation "hazelcast_config" does not exist

There is a missing relation in Postgresql

Querying completed workflow instances by datetime range

Support querying completed workflow instances by datetime range. e.g. For processes and variables on a particular day.

{
  processInstances (stateIn: [COMPLETED], minStartTime: "2021-11-02T00:00:00.000Z", maxEndTime: "2021-11-02T23:59:59.999Z") {
  	nodes {
          state
          startTime
          endTime
          variables {
          name
          value
      }
    }
  }
}

Variable Updates, Element Instance Transitions and Message Correlations are not persisted

If ZeeQS imports records from a Zeebe cluster with multiple partitions then it can happen that variable updates, element instance transitions, and message correlations are not persisted.

The problem is caused by the importer. It uses the record position as ID for the entities. But the positions are not unique across multiple partitions.

Related issue: camunda-community-hub/zeebe-simple-monitor#278

Import error records from Hazelcast and expose them via GraphQL

In case, that an unexpected error happens (e.g. during the workflow processing) then an error record is written. This can contain useful information for root cause (e.g. if an instance is was added to the blacklist).

AT:

• import error records from Hazelcast
• expose query for errors in GraphQL

Return only timers of root timer start events for process

You can get the timers of a process. Currently, it returns all timers that belong to the process, including timers from timer start events and intermediate timer events. It is expected that it returns only the timers from the root timer start events but not from intermediate events. The behavior should be aligned with message subscriptions of a process.

Docker image for 2.0.0 not released to Docker Hub

https://hub.docker.com/r/camunda/zeeqs/tags?page=1&ordering=last_updated

Could you please provide it so that we can update the Helm charts?

Querying workflow instances by variables

Support querying workflow instances by specific variables names & values.

ERROR: value too long for type character varying(255)

Hi Team,

I have used the Zeeqs application to connect to PostgreSQL Db using the below configurations.

Zeebe Setup([Version 0.23.2]One broker One Standalone Gateway) with Zeebe-hazelcast-exporter(0.9.0) and Zeeqs(image- zeeqs:latest) deployed on kubernetes cluster using helm charts in same namespace

The application was working fine (as expected) before it threw the following Error , after which there are NO Traces/Logs

2020-07-23 11:20:04.426 ERROR 1 --- [pool-1-thread-1] o.h.engine.jdbc.spi.SqlExceptionHelper : ERROR: value too long for type character varying(255)

Hazelcast exporter receives the data and pushes to H2 database, as we see the changes in Simple Monitor. But the same is not reflected through Zeeqs. It does not capture inputs into Database.
The Error thrown is not very specific as to which Table and Field.

On trying to replicate, we noticed :

• Connections to Jdbc closed abruptly without proper logs
• On every restart of Zeeqs, the old data from Hazelcast was updated
• The startup Logs of Zeeqs always had this :

2020-08-10T12:40:38.448886336Z 2020-08-10 12:40:38.448 WARN 1 --- [pool-1-thread-1] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 0, SQLState: 22001

2020-08-10T12:40:38.448927509Z 2020-08-10 12:40:38.448 ERROR 1 --- [pool-1-thread-1] o.h.engine.jdbc.spi.SqlExceptionHelper : ERROR: value too long for type character varying(255)

Importing data from Elasticsearch

• import the data that is provided by the Elasticsearch exporter

Build ZeeQS for multiple architectures

Hello!
My colleague has a problem when he is running spec tests on his M1 Mac. Sometimes ZeeQS container hangs out on the Spring Data JPA's repositories scanning phase.

So, I build the multiplatform ZeeQS image using the jib in the ZeeQS repo: https://github.com/users/aivinog1/packages/container/package/zeeqs. We have tested it and it works fine (no one hanging so far). So, I would like to add this support to the ZeeQS.

Hazelcast JDBC Driver

Hey!

I think we can simplify the first-look example by adding the Hazelcast JDBC Driver and switch to this by default in examples (since we obviously have Hazelcast in our environment because we import data from it). WDYT? I can propose a PR to see how it looks.

WorkflowInstance.variables query returns "Internal Server Error(s) while executing query"

ZeeQS 1.0.0-alpha5 docker image.
PostgreSQL 13.1

I want to get the variables of a completed workflow instance :

query {
          workflowInstance (key: 2251799813916461) {
    				state
            variables {
              name
              value
            }
          }
        }

But I get :

{
  "errors": [
    {
      "message": "Internal Server Error(s) while executing query"
    }
  ],
  "data": {
    "workflowInstance": {
      "state": "COMPLETED",
      "variables": null
    }
  }
}

Searching in the database, I do not find my variables values, there is only numbers in the value column of the variables table...

I can fetch decision evaluations

Is your feature request related to a problem? Please describe.
I want to see the evaluations of a decision. The evaluation contains all data of the evaluated decision, for example, the matched rules of a decision table.

Describe the solution you'd like

• I can fetch all evaluations of a decision
• A decision evaluation has the following properties
  • connection to the related decision
  • decision output
  • connection to evaluated inputs
    • input id
    • input name
    • input value
  • connection to matched rules
    • rule id
    • rule index
    • connection to evaluated outputs
      • output id
      • output name
      • output value
  • connection to the related process instance
  • connection to the related element instance
  • evaluation failure message
  • failed decision id

Describe alternatives you've considered
None.

Additional context
Requires camunda-community-hub/zeebe-hazelcast-exporter#215
Requires #285

Does Zeeqs really guarantee the order of events as they occurred?

I have a Zeebe broker that streams data to a PostgreSQL DB through Zeeqs/Hazelcast.

It seems that events are not always recorded as they actually happened; in other words, the order of events does not seem to be always guaranteed. Here is a screenshot of my PostgreSQL:

The chat-message event has happened after all other events/variables. Also, between every two chat-message there must always be a userInputText and a watsonResponse; yet, this is not the case as you can see in the chunk shown in the screen. This is not the only case I spotted; there are others…

Could you please let me know if Zeeqs guarantees the order of events as they actually occurred in reality?

Thank you,

Is GraphiQL disabled when running on docker-compose?

Thank you for this awesome extension to Camunda! I am currently using the in-memory profile from the docker-compose.yml however, GraphiQL seems to be disabled as I receive a 400 when executing a GET request. Is there an environment variable or application config option to enable GraphiQL?

Logs

zeeqs          | 2022-09-05 09:53:01.290  INFO 1 --- [nio-9000-exec-2] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
zeeqs          | 2022-09-05 09:53:01.291  INFO 1 --- [nio-9000-exec-2] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
zeeqs          | 2022-09-05 09:53:01.292  INFO 1 --- [nio-9000-exec-2] o.s.web.servlet.DispatcherServlet        : Completed initialization in 1 ms
zeeqs          | 2022-09-05 09:53:24.249  INFO 1 --- [nio-9000-exec-3] g.k.servlet.HttpRequestHandlerImpl       : Bad request: cannot handle http request
zeeqs          |
zeeqs          | graphql.GraphQLException: Query parameter not found in GET request
zeeqs          |        at graphql.kickstart.servlet.GraphQLGetInvocationInputParser.getGraphQLInvocationInput(GraphQLGetInvocationInputParser.java:36) ~[graphql-java-servlet-10.1.0.jar:na]
zeeqs          |        at graphql.kickstart.servlet.HttpRequestHandlerImpl.handle(HttpRequestHandlerImpl.java:38) ~[graphql-java-servlet-10.1.0.jar:na]
zeeqs          |        at graphql.kickstart.servlet.AbstractGraphQLHttpServlet.doRequest(AbstractGraphQLHttpServlet.java:82) ~[graphql-java-servlet-10.1.0.jar:na]
zeeqs          |        at graphql.kickstart.servlet.AbstractGraphQLHttpServlet.doGet(AbstractGraphQLHttpServlet.java:69) ~[graphql-java-servlet-10.1.0.jar:na]
zeeqs          |        at javax.servlet.http.HttpServlet.service(HttpServlet.java:645) ~[javax.servlet-api-4.0.1.jar:4.0.1]
zeeqs          |        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750) ~[javax.servlet-api-4.0.1.jar:4.0.1]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91) ~[spring-web-5.3.19.jar:5.3.19]
zeeqs          |        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.19.jar:5.3.19]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.19.jar:5.3.19]
zeeqs          |        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.19.jar:5.3.19]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.3.19.jar:5.3.19]
zeeqs          |        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.19.jar:5.3.19]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) ~[spring-boot-actuator-2.6.7.jar:2.6.7]
zeeqs          |        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.19.jar:5.3.19]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.3.19.jar:5.3.19]
zeeqs          |        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.19.jar:5.3.19]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
zeeqs          |
zeeqs          | 2022-09-05 09:53:24.250 ERROR 1 --- [nio-9000-exec-3] g.k.servlet.AbstractGraphQLHttpServlet   : Error executing GraphQL request!
zeeqs          |
zeeqs          | graphql.GraphQLException: Query parameter not found in GET request
zeeqs          |        at graphql.kickstart.servlet.GraphQLGetInvocationInputParser.getGraphQLInvocationInput(GraphQLGetInvocationInputParser.java:36) ~[graphql-java-servlet-10.1.0.jar:na]
zeeqs          |        at graphql.kickstart.servlet.HttpRequestHandlerImpl.handle(HttpRequestHandlerImpl.java:38) ~[graphql-java-servlet-10.1.0.jar:na]
zeeqs          |        at graphql.kickstart.servlet.AbstractGraphQLHttpServlet.doRequest(AbstractGraphQLHttpServlet.java:82) ~[graphql-java-servlet-10.1.0.jar:na]
zeeqs          |        at graphql.kickstart.servlet.AbstractGraphQLHttpServlet.doGet(AbstractGraphQLHttpServlet.java:69) ~[graphql-java-servlet-10.1.0.jar:na]
zeeqs          |        at javax.servlet.http.HttpServlet.service(HttpServlet.java:645) ~[javax.servlet-api-4.0.1.jar:4.0.1]
zeeqs          |        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750) ~[javax.servlet-api-4.0.1.jar:4.0.1]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91) ~[spring-web-5.3.19.jar:5.3.19]
zeeqs          |        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.19.jar:5.3.19]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.19.jar:5.3.19]
zeeqs          |        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.19.jar:5.3.19]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.3.19.jar:5.3.19]
zeeqs          |        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.19.jar:5.3.19]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) ~[spring-boot-actuator-2.6.7.jar:2.6.7]
zeeqs          |        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.19.jar:5.3.19]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.3.19.jar:5.3.19]
zeeqs          |        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.19.jar:5.3.19]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.62.jar:9.0.62]
zeeqs          |        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
zeeqs          |

Querying the endpoint with a POST works perfectly by the way 😊

Improve GraphQL tests with Spring

Description

We could improve the tests for GraphQL by using the Spring test harness. See the docs for details.

Data persistence issue

Hi,

Zeeqs can query current insert data only, When i'm restart zeeqs service it can't query exist(old active) data in cluster. But will create new instance and query using zeeqs, it ability to query and show it.
Therefore

• Zeeqs have to connect with elasticsearch server or zeebe cluster and it ability get old active data of cluster

Why does zeeqs need hazelcast exporter?

Description

I have a few questions on zeeqs architecture

1. As I understand the way zeeqs functions, it reads the workflow events from hazelcast and pushes them to postgres db for persistence. I am curious on why is it designed this way? Can zeebe not export the events directly to a postgres db and zeeqs read from postgres db?
2. What is the recommended production configuration for hazelcast? Should it be running in embedded mode or standalone mode? We observed that when hazelcast runs in standalone mode if it restarts, zeebe also needs to be restarted to connect with hazelcast.

Thanks.

Add `bpmn:documentation` for BpmnElement

At the moment I am pulling the whole process XML to get element Documentation.
How difficult would be to add bpmn:documentation into BpmnElement?

Thanks

Support different databases

• document how to use a different database (e.g. Postgres, MySQL, etc.) to store the aggregated data in

Element Instance contains meta-information about the workflow

An element instance contains a list of meta-information from the workflow

• message name of message event
• condition expression of sequence flows on an exclusive gateway
• job-type of service task
• error code of error event
• timer definition from timer event
• called process id of call activity

See also https://github.com/camunda-community-hub/zeebe-simple-monitor/blob/master/src/main/java/io/zeebe/monitor/rest/ProcessesViewController.java#L245

Filter jobs

• filter jobs by their state
• filter jobs by their type

JobRecord - record.Metadata.Key is set to -1 when intent is CREATE

Hey,

I think that when JobRecord intent is CREATE the value of record.Metadata.Key is set to -1 (at least that's the case with broker 0.22.2, not sure if it's a bug or as expected)

I think that this will cause JPA to create a record with -1 as ID.

Maybe you should filter jobs when key = -1,

note: this is true for other entities such incidents & messages

Thanks,

Asaf.

how to avoid loosing data in production

Hey,

Few things related to production:

1. How to avoid errors when current Hazelcast sequence < sequence stored in config (may occur if Hazelcast is restarted)
2. How to avoid losing data when Zeeqs is down for a while (Hazelcast with persistence ? migrate to ES?)

Providing a REST API

• provide a REST API to access the data

Filter workflow instances by state

• filter workflow instances by their state

workflowInstance(states:[ACTIVATED]) {
  ...  
} 

// or 
workflowInstance(stateIn:[COMPLETED,TERMINATED]) {
  ...  
} 

Use common pagination parameters

The GraphQL queries support pagination of the result. Currently, it uses the parameters limit: 10 and page: 0. But the parameter limit is not so common and it is not intuitive. It defines the items per page.

• replace the pagination parameter limit with perPage

Providing an initial GraphQL API

• create an initial schema that includes workflows, workflow instances, variables, jobs and incident
• provide the resolvers for the API

Support namespaces/domains/multi-tenancy

In order to support different namespaces/domains (i.e. multi-tenancy), we need a way to store the data based on the namespace and be able to query by the namespace This can be useful to collect data from multiple environments (e.g. dev/stage/prod) by a single consumer, especially, for building a testing tool and running multiple test cases.

Each data entity (e.g. processes, process instances, jobs, etc.) contains a new field for the namespace.

The importer sets the namespace when reading exported records.

The GraphQL queries contain an optional field to filter by the namespace.

Related to/depends on camunda-community-hub/zeebe-hazelcast-exporter#132.

Importing process records fail with PostgreSQL

On starting zeeqs with postgres db (docker-compose file) the log file is filled up with errors -
Caused by: org.postgresql.util.PSQLException: ERROR: invalid byte sequence for encoding "UTF8": 0x00
(complete log file is shared here).

On submitting the process definition to zeebe we sometimes get null values in the api response instead of the process instance information. The process definition retured is also null. Linked are the screenshots of the graphql requests made through postman client.

On a slack discussion here, the issue was reproduced when running zeeqs with postgres.

Evaluate other GraphQL framework

Look into https://github.com/Netflix/dgs-framework and evaluate if it fits better for this project.

I can query decisions and decision requirements

Is your feature request related to a problem? Please describe.
I want to fetch the deployed decisions and related decision requirements. Similarly, to the deployed processes.

Describe the solution you'd like

• I can query all decisions
• I can query all decision requirements (DRG)
• A decision has the following properties
  • key
  • decision id
  • version
  • decision name
  • connection to the related decision requirements
• A decision requirements (DRG) has the following properties
  • key
  • decision requirements id
  • decision requirements name
  • version
  • namespace
  • XML
  • connection to all containing decisions

Describe alternatives you've considered
None.

Additional context
Requires camunda-community-hub/zeebe-hazelcast-exporter#214

Add helm charts to the repo

The helm charts have been removed from https://github.com/camunda-community-hub/camunda-cloud-helm in order to further provide helm charts they need to be hosted somewhere else, e.g. directly in this repository.

Attaching the sources here so they can easily be added

zeeqs-charts.zip

Should we expose (common) zeeqs queries as a cli?

ZeeQS exposes its data as a GraphQL API. This is great for applications that want to query Zeebe's data programmatically. I believe it might also be useful to offer this data through a command-line interface.

For example:

zeeqs workflows

Could output generally interesting data about the currently deployed workflows, e.g. similar to the GraphQL query:

{
  workflows {
    totalCount
    nodes {
      key
      bpmnProcessId
      version
      workflowInstances {
        totalCount
      }
    }
  }
}

Query Response

{
  "data": {
    "workflows": {
      "totalCount": 3,
      "nodes": [
        {
          "key": "2251799813685249",
          "bpmnProcessId": "demo-process",
          "version": 1,
          "workflowInstances": {
            "totalCount": 3
          }
        },
        {
          "key": "2251799813685251",
          "bpmnProcessId": "demo-2",
          "version": 1,
          "workflowInstances": {
            "totalCount": 2
          }
        },
        {
          "key": "2251799813685269",
          "bpmnProcessId": "demo-3",
          "version": 1,
          "workflowInstances": {
            "totalCount": 1
          }
        }
      ]
    }
  }
}

A potential use-case I see is to quickly query for Zeebe data during local development.

I've created a draft PR to show what this could look like in #40. I would like to validate whether this is something the community would like to use, before I continue to develop this idea further.

Do you see potential in a tool like this? Please comment or 👍 / 👎 on this issue.

A timer of a timer start event links to the created process instance

When a timer of a timer start event is triggered then it should link to the process instance. Currently, the process instance is always null, even if the timer triggered and created a new process instance.

Optimize GraphQL with batching

Description

We could improve the performance of GraphQL by using batch loading. See the docs for defails.

Suggestion: add Time.creationTime

Hey,

Since timers are overridden, it's currently impossible to know the amount of time took for a timer to complete its work,

Adding timer. creationTime in createTimer can help.

Dockerfile?

I really need a Dockerfile to be able to add the Postgres jar file and in general build a new image. But for some reason there is no such thing in the repo.

Can you please assist?

I can subscribe for updates of a workflow instance

Using the GraphQL API, I can subscribe for updates of a specific workflow instance (e.g. new element instance, a new variable, a new incident, etc.).

GraphQL's subscriptions should be suitable for this.

I can access user task's metadata

Description

I can access the following static properties of a user task.

• the assignee
• the candidate groups

Extend the GraphQL API to expose these properties as element metadata.

Importing data from Hazelcast

• import the data that is provided by the Hazelcast exporter (https://github.com/zeebe-io/zeebe-hazelcast-exporter)

Building an initial data model

• initial data model includes: workflows, workflow instances, variables, jobs and incidents
• proving JPA entities and CRUD repositories

Elasticsearch exporter

Hey there, I noticed in https://github.com/camunda-community-hub/zeeqs/blob/master/docs/ZeeQS.png there's reference to being able to use an elasticsearch exporter as input to ZeeQS. Is this currently supported at all? How much of an undertaking, in your estimation, would it be support ES?

I can query user tasks

In Zeebe, user tasks are a special kind of task. They are similar to other tasks but have a few special properties. And they should be completed by users, in form of a user task application (e.g. Tasklist).

We should handle user tasks as first-class citizens.

• I can fetch all user tasks of a process instance
• I can query all user tasks
• A user task has the following properties
  • key
  • state (created or completed)
  • timestamp
  • startTime
  • endTime
  • connection to the processInstance
  • connection to the elementInstance
  • assignee (i.e. the value of the custom job header)
  • candidate groups (i.e. the value of the custom job header)
  • connection to the form, based in the value of the custom job header
  • connection to variables (i.e. all visible variables from this element instance; local variables and from upper scopes; required to render the form)
• the forms of the user tasks are embedded in the process
  • import the forms during the import of a process

Support pagination for Queries

• support a uniform way of pagination for the queries (e.g. workflows, workflow instances, jobs, etc.)

See the GraphQL best practices: https://graphql.org/learn/pagination/

perPage parameter not working, should be limit?

The query

{
  workflowInstances(page: 0, perPage: 20) {
    totalCount
    nodes {
      key
    }
  }
}

yield:

{
  "errors": [
    {
      "message": "Validation error of type UnknownArgument: Unknown field argument perPage @ 'workflowInstances'",
      "locations": [
        {
          "line": 2,
          "column": 30
        }
      ],
      "extensions": {
        "classification": "ValidationError"
      }
    }
  ],
  "data": null
}

Changing "perPage" parameter into "limit" does yield to correct result. It might be good to change to "Pagination Section" in the README.md to include a working ##example.

Entities using record.position as primary key

First of all: great community project.

While evaluating a solution that replaces the Hazelcast exporter/importer with a Kafka based solution, I found the following entities using the position as the primary key:

• ElementInstanceStateTransition
• MessageCorrelation
• VariableUpdate

As far as I know, the position is unique per partition, which would mean that the key is not unique when multiple partitions are used. Shouldn't this be a compound key then ?
For example for VariableUpdate, position+variableKey should be always unique.