killbill/killbill-commons

Kill Bill reusable Java components

killbill-commons

Kill Bill reusable Java components:

• automaton: framework to build state machines
• clock: clock library
• concurrent: extensions to java.util.concurrent.Executors
• config-magic: fork of config-magic
• embeddeddb: library to embed databases
• jdbi: fork of jDBI
• locker: locking library
• metrics: annotation-based metrics
• queue: provides persistent bus events and notifications
• skeleton: framework to build web services
• utils: provides utility methods for core Java(tm) classes (lang, IO, collection) and Cache
• xmlloader: library to load, parse and validate XML files

Kill Bill compatibility

Commons version	Kill Bill version
0.15.y	0.16.z
0.20.y	0.18.z
0.21.y	0.19.z
0.23.y	0.20.z
0.23.y	0.22.z
0.24.y	0.22.z
0.26.y	0.24.z

We've upgraded numerous dependencies in 0.24.x (required for Java 11 support).

Usage

Add the relevant submodule(s) to a project:

<dependency>
    <groupId>org.kill-bill.commons</groupId>
    <artifactId>killbill-automaton</artifactId>
    <version>... release version ...</version>
</dependency>
<dependency>
    <groupId>org.kill-bill.commons</groupId>
    <artifactId>killbill-clock</artifactId>
    <version>... release version ...</version>
</dependency>
<dependency>
    <groupId>org.kill-bill.commons</groupId>
    <artifactId>killbill-concurrent</artifactId>
    <version>... release version ...</version>
</dependency>
<dependency>
    <groupId>org.kill-bill.commons</groupId>
    <artifactId>killbill-config-magic</artifactId>
    <version>... release version ...</version>
</dependency>
<dependency>
    <groupId>org.kill-bill.commons</groupId>
    <artifactId>killbill-embeddeddb-common</artifactId>
    <version>... release version ...</version>
</dependency>
<dependency>
    <groupId>org.kill-bill.commons</groupId>
    <artifactId>killbill-embeddeddb-h2</artifactId>
    <version>... release version ...</version>
</dependency>
<dependency>
    <groupId>org.kill-bill.commons</groupId>
    <artifactId>killbill-embeddeddb-mysql</artifactId>
    <version>... release version ...</version>
</dependency>
<dependency>
    <groupId>org.kill-bill.commons</groupId>
    <artifactId>killbill-embeddeddb-postgresql</artifactId>
    <version>... release version ...</version>
</dependency>
<dependency>
    <groupId>org.kill-bill.commons</groupId>
    <artifactId>killbill-jdbi</artifactId>
    <version>... release version ...</version>
</dependency>
<dependency>
    <groupId>org.kill-bill.commons</groupId>
    <artifactId>killbill-locker</artifactId>
    <version>... release version ...</version>
</dependency>
<dependency>
    <groupId>org.kill-bill.commons</groupId>
    <artifactId>killbill-metrics</artifactId>
    <version>... release version ...</version>
</dependency>
<dependency>
    <groupId>org.kill-bill.commons</groupId>
    <artifactId>killbill-queue</artifactId>
    <version>... release version ...</version>
</dependency>
<dependency>
    <groupId>org.kill-bill.commons</groupId>
    <artifactId>killbill-skeleton</artifactId>
    <version>... release version ...</version>
</dependency>
<dependency>
<groupId>org.kill-bill.commons</groupId>
<artifactId>killbill-utils</artifactId>
<version>... release version ...</version>
</dependency>

About

Kill Bill is the leading Open-Source Subscription Billing & Payments Platform. For more information about the project, go to https://killbill.io/.

killbill-commons's Issues

Replace cglib with Byte Buddy for Java 16 compatibility

• Fork and re-implement proxies in config-magic
• Re-implement proxies in jdbi

Failover of IP seems to cause problems in the bus events connection pool

We noticed during failover testing that if the IP of the database changes, failover takes much longer than it does when it remains the same.

Amazon RDS does failover based on DNS with a very short TTL. We can configured the JVM to prevent DNS caching, and set "org.killbill.dao.connectionTimeout" and "org.killbill.dao.idleConnectionTestPeriod" to 1s. We tried experimenting with various connection pool settings as well.

After a failover it takes about 5 minutes before we stop seeing errors and requests go through again. They appear to come from bus event threads that continue to trickle in for a few minutes:

2015-02-06 16:13:21,563 [bus_ext_events-th] WARN o.k.queue.DefaultQueueLifecycle - Bus: Thread bus_ext_events-th [39] got an exception, catching and moving on...
org.skife.jdbi.v2.exceptions.UnableToObtainConnectionException: java.sql.SQLException: Timeout of 10000ms encountered waiting for connection.

We tried using haproxy to proxy the database connections and provide a static IP to kill bill. When we do this, we can get failover to happen in just a few seconds.

A fix is not a priority for us since deploying haproxy solves the issue entirely, but we were just wondering if there were settings we were missing to make it work without it.

Reaper should not reap entries when there is only one node

In scenario where there is only one node, reaping late notifications with the same Creator name should lead to no-op. This is an edge case, but can happen on testing environment.

queue: update EventBusThatThrowsException to support Guava 16.0

It looks like handlersByTypeLock has been renamed.

NotificationSqlDao should provide search on both keys

NotificationSqlDao should provide getReadyQueueEntriesForSearchKeyS, not only because it is required for Kill Bill (to search by account_record_id and tenant_record_id), but also because the only index present today is a composite on both keys.

Duplicate entry exception when moving entry to notifications_history

at org.killbill.queue.DBBackedQueue.moveEntryToHistoryFromTransaction(DBBackedQueue.java:370)

Bus and notification queue uniformization

The code between the two is very similar these days: can we unify the libraries further?

Additionally, could retries be built-in? We have several implementations:

• Built-in (simple version, retry right away then give up)
• Custom:
  • JAX-RS push notifications,
  • Payment
  • Retries for entitlement, subscription, invoice and overdue bus and notifications queues (for plugins)

queue: notification queue can become stuck if some handlers are not found

When a node claims some entries, but cannot find the NotificationQueueHandler for them, the entries are left claimed in the database:

https://github.com/killbill/killbill-commons/blob/master/queue/src/main/java/com/ning/billing/notificationq/NotificationQueueDispatcher.java#L184

This will result in another node eventually stealing them from the first node. If the handler is never present, the notification queue can become stuck (by always claiming these entries which cannot be processed).

queues: safer streaming APIs

See discussion at killbill/killbill#862.

Refactor DB lockers

Code is duplicated between the PostgreSQLGlobalLocker and the MySqlGlobalLocker, we should refactor both classes.

DefaultClock doesn't respect user timezone

See https://github.com/killbill/killbill-commons/blob/master/clock/src/main/java/com/ning/billing/clock/DefaultClock.java#L46.

Re Implements MultiMap

During Guava refactoring, we need replacement of Guava Multimap. Forking/copy every implementation is too much at that time because it contains a lot of import to other Guava classes. At initial work, what we really need is simple Map with ability to store multiple values. Thus we have MultiValueMap.

The main problem with MultiValueMap is that it uses List as value, thus we cant make use of other collection types as a value. So far, class(es) that need non-List as a value are:

• org.killbill.billing.lifecycle.DefaultLifeCycle#handlersByLevel : Originally, use SortedSet as value. As currently no replacement, we need to use Map<LifecycleLevel, SortedSet<LifecycleHandler<? extends KillbillService>>> handlersByLevel;

• RequestOptions in killbill-client-java : Not mandatory (like DefaultLifeCycle above), but more to usability problem. Currently, MultiValueMap is useless if user/client code of killbill-client-java want to use MultiValueMap.

  (as part of killbill-client-java no-guava refactoring, I'm planning to use Map<String, Collection<String>> to queryParams and queryParamsForFollow. This make MultiValueMap useless because it based on Map<String, List<String>>, so client code still need to deal with if-key-exist-add-else-put)

We can refactor MultiValueMap, but it is scattered in other modules. I think the real cost is not in refactor vs re-write, but in apply it.

STICKY_EVENTS optimization for the cloud

To make #29 work with the STICKY_EVENTS configuration (bus only), we also need to review the behavior of finding orphan entries.

Instead of simply logging:

killbill-commons/queue/src/main/java/org/killbill/queue/DBBackedQueue.java

Line 346 in 624a819

log.warn(DB_QUEUE_LOG_ID + "Detected unprocessed bus event {}, may need to restart server...", previousLowestOrphanEntry);

we now need to re-insert such orphan ids in the in-memory inflightEvents queue.

Looking at the code written for STICKY_POLLING, there is an easier approach to this:

• Keep the orphan code as-is
• In the reapEntries method, instead of simply calling transactional.insertEntries, do something similar to insertEntryFromTransaction, i.e. write the entries to disk and add them to the inflight queue in bulk on success. Because of the LAST_INSERT_ID logic, we wouldn't be able to insert them in bulk in that case though

For testing:

• Additional tests to write in TestDBBackedQueue
• Manual testing: you could have one Kill Bill server running in STICKY_EVENTS and insert a dummy bus event entry for another fictive node (other creator name). That event should eventually be picked up by the server

Ability to mock clock movement for a single or desired billing record.

Ability to mock clock movement for a single record.
We are trying to test several billing scenarios in our preproduction system. We are trying to parallelize our tests in order to save time. Our constraint is that the clock is global to all the billing records. Would this be a valid request?

STICKY_POLLING optimizations for the cloud

Current behavior for STICKY_POLLING configuration, for both the bus and notification queue, is to only look at entries in the database that match the Creator name of the current node. The issue is when a node goes away: currently, a manual update of the database entries is required (see https://github.com/killbill/killbill/wiki/Kill-Bill-Bus-and-Notification-Queue-Configuration).

Instead, we would now have a new Reaper thread that periodically monitors such entries and re-dispatch them automatically. To detect such entries, we need to look for bus events with an old created date or notification events with an effective date too far in the past (actual thresholds configurable, 5' by default). Such detection is already done today in the queries (see second part of the where clause in

killbill-commons/queue/src/main/resources/org/killbill/bus/dao/PersistentBusSqlDao.sql.stg

Line 14 in 624a819

and (processing_owner IS NULL OR processing_available_date \<= :now)

and

killbill-commons/queue/src/main/resources/org/killbill/notificationq/dao/NotificationSqlDao.sql.stg

Line 21 in 624a819

and (processing_owner IS NULL OR processing_available_date \<= :now)

), we would need to introduce similar queries but focusing only on such old entries. These entries would then be updated to reflect the current Creator name (maybe re-dispatch 10 at a time, configurable).

Additional tests to write in TestDBBackedQueue.

Materialize processing time for bus/notifications

We can estimate the processing time of a notification since we reset the processing date as shown here, but this only works if/when bus/notifications are not late.

Instead, we could add a new column to materialize the time it took the handler to process the entry.

Queue stuck entries with single node system

In situations where some notification entries are left IN_PROCESSING state, we typically have a reaper mechanism to re-dispatch such entries. However in a single node scenario, we cannot re-dispatch to another node and currently the code just prints a warning. Note that such situations could arise as a result of a shutdown that timed out or a node that was killed abruptly.

Such entries will not be picked up since the query for ready entries only looks at AVAILABLE state.

CreatorName should use Kill Bill config source properties

Currently, it looks at System Properties directly.

can find ColonStatementLexer

It can't find ColonStatementLexer

if i import jdbi , the argument incompatible

org.jdbi
jdbi
2.78

Investigate using a third party system to solve distributed synchronization in KB

Production KB will always involve several instances, and today distributed synchronization is achieved through the use of a distributed lock from the SQL engine. This approach is great because it allows to deploy with no dependencies, but fundamentally databases are not the best tool to implement reliable distributed synchronization logic.

We would like to investigate the use of some existing OSS distributed synchronization system. Probably a good place to start would be zookeeper or potentially consul.

The work would be to

1. Create new implementation of the GlobalLocker, and extend the configuration to make it one possible option.
2. Create a new docker compose to start a KB stack with zookeeper
3. Run default tests to verify behavior is as expected
4. Extend tests to create conditions where we hit multiple requests for same accounts across different nodes.

can not configure pool's org.killbill.dao.minIdle connections

sample KB configuration :

JAVA_OPTS="${JAVA_OPTS} -Dorg.killbill.dao.maxActive=80"
JAVA_OPTS="${JAVA_OPTS} -Dorg.killbill.dao.minIdle=30"

KB's relevant logs and exception on startup :

2015-01-12 11:30:13,274 [localhost-startStop-1] INFO  o.s.c.ConfigurationObjectFactory - Assigning default value [true] for [org.killbill.dao.cachePrepStmts] on [org.killbill.commons.jdbi.guice.DaoConfig#isPrep$
2015-01-12 11:30:13,275 [localhost-startStop-1] INFO  o.s.c.ConfigurationObjectFactory - Assigning default value [true] for [org.killbill.dao.useServerPrepStmts] on [org.killbill.commons.jdbi.guice.DaoConfig#is$
2015-01-12 11:30:13,276 [localhost-startStop-1] INFO  o.s.c.ConfigurationObjectFactory - Assigning null default value for [org.killbill.dao.dataSourceClassName] on [org.killbill.commons.jdbi.guice.DaoConfig#get$
2015-01-12 11:30:13,277 [localhost-startStop-1] INFO  o.s.c.ConfigurationObjectFactory - Assigning null default value for [org.killbill.dao.driverClassName] on [org.killbill.commons.jdbi.guice.DaoConfig#getDriv$
2015-01-12 11:30:13,277 [localhost-startStop-1] INFO  o.s.c.ConfigurationObjectFactory - Assigning default value [4.0] for [org.killbill.dao.mysqlServerVersion] on [org.killbill.commons.jdbi.guice.DaoConfig#get$
2015-01-12 11:30:13,277 [localhost-startStop-1] INFO  o.s.c.ConfigurationObjectFactory - Assigning default value [HIKARICP] for [org.killbill.dao.poolingType] on [org.killbill.commons.jdbi.guice.DaoConfig#getCo$
2015-01-12 11:30:13,278 [localhost-startStop-1] INFO  o.s.c.ConfigurationObjectFactory - Assigning default value [0m] for [org.killbill.dao.maxConnectionAge] on [org.killbill.commons.jdbi.guice.DaoConfig#getMax$
2015-01-12 11:30:13,278 [localhost-startStop-1] INFO  o.s.c.ConfigurationObjectFactory - Assigning default value [5m] for [org.killbill.dao.idleConnectionTestPeriod] on [org.killbill.commons.jdbi.guice.DaoConfi$
2015-01-12 11:30:13,279 [localhost-startStop-1] INFO  o.s.c.ConfigurationObjectFactory - Assigning default value [500] for [org.killbill.dao.prepStmtCacheSize] on [org.killbill.commons.jdbi.guice.DaoConfig#getP$
2015-01-12 11:30:13,279 [localhost-startStop-1] INFO  o.s.c.ConfigurationObjectFactory - Assigning default value [2048] for [org.killbill.dao.prepStmtCacheSqlLimit] on [org.killbill.commons.jdbi.guice.DaoConfig$
2015-01-12 11:30:13,280 [localhost-startStop-1] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [jdbc:mysql://killbill.cjafxh9qc2le.us-west-2.rds.amazonaws.com:3306/killbill] for [org.killbill.dao.url]$
2015-01-12 11:30:13,280 [localhost-startStop-1] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [killbill] for [org.killbill.dao.password] on [org.killbill.commons.jdbi.guice.DaoConfig#getPassword()]
2015-01-12 11:30:13,281 [localhost-startStop-1] INFO  o.s.c.ConfigurationObjectFactory - Assigning default value [DEBUG] for [org.killbill.dao.logLevel] on [org.killbill.commons.jdbi.guice.DaoConfig#getLogLevel$
2015-01-12 11:30:13,289 [localhost-startStop-1] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [killbill] for [org.killbill.dao.user] on [org.killbill.commons.jdbi.guice.DaoConfig#getUsername()]
2015-01-12 11:30:13,290 [localhost-startStop-1] INFO  o.s.c.ConfigurationObjectFactory - Assigning default value [10s] for [org.killbill.dao.connectionTimeout] on [org.killbill.commons.jdbi.guice.DaoConfig#getC$
2015-01-12 11:30:13,290 [localhost-startStop-1] INFO  o.s.c.ConfigurationObjectFactory - Assigning value [80] for [org.killbill.dao.maxActive] on [org.killbill.commons.jdbi.guice.DaoConfig#getMaxActive()]
2015-01-12 11:30:14,225 [localhost-startStop-1] INFO  com.google.inject.Guice - An exception was caught and reported. Message: java.lang.IllegalArgumentException: maxPoolSize cannot be negative or greater than $
java.lang.IllegalArgumentException: maxPoolSize cannot be negative or greater than maximumPoolSize
        at com.zaxxer.hikari.HikariConfig.setMinimumIdle(HikariConfig.java:552) ~[HikariCP-java6-2.0.1.jar:na]
        at org.killbill.commons.jdbi.guice.DataSourceProvider.getHikariCPDataSource(DataSourceProvider.java:101) ~[killbill-jdbi-0.2.28.jar:na]
        at org.killbill.commons.jdbi.guice.DataSourceProvider.getDataSource(DataSourceProvider.java:88) ~[killbill-jdbi-0.2.28.jar:na]
        at org.killbill.commons.jdbi.guice.DataSourceProvider.get(DataSourceProvider.java:72) ~[killbill-jdbi-0.2.28.jar:na]
        at org.killbill.billing.server.modules.KillbillPlatformModule.configureDao(KillbillPlatformModule.java:97) ~[killbill-platform-server-0.1.1-classes.jar:na]
        at org.killbill.billing.server.modules.KillbillServerModule.configureDao(KillbillServerModule.java:99) ~[KillbillServerModule.class:na]
        at org.killbill.billing.server.modules.KillbillPlatformModule.configure(KillbillPlatformModule.java:75) ~[killbill-platform-server-0.1.1-classes.jar:na]
        at org.killbill.billing.server.modules.KillbillServerModule.configure(KillbillServerModule.java:88) ~[KillbillServerModule.class:na]
        at com.google.inject.AbstractModule.configure(AbstractModule.java:59) ~[guice-3.0.jar:na]
        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:223) ~[guice-3.0.jar:na]
        at com.google.inject.spi.Elements.getElements(Elements.java:101) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:133) ~[guice-3.0.jar:na]
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:103) [guice-3.0.jar:na]
        at com.google.inject.Guice.createInjector(Guice.java:95) [guice-3.0.jar:na]
        at org.killbill.commons.skeleton.listeners.GuiceServletContextListener.getInjector(GuiceServletContextListener.java:92) [killbill-skeleton-0.2.28.jar:na]
        at com.google.inject.servlet.GuiceServletContextListener.contextInitialized(GuiceServletContextListener.java:45) [guice-servlet-3.0.jar:na]
        at org.killbill.commons.skeleton.listeners.GuiceServletContextListener.contextInitialized(GuiceServletContextListener.java:54) [killbill-skeleton-0.2.28.jar:na]
        at org.killbill.billing.server.listeners.KillbillPlatformGuiceListener.initializeGuice(KillbillPlatformGuiceListener.java:137) [killbill-platform-server-0.1.1-classes.jar:na]
        at org.killbill.billing.server.listeners.KillbillPlatformGuiceListener.contextInitialized(KillbillPlatformGuiceListener.java:89) [killbill-platform-server-0.1.1-classes.jar:na]

hopefully swapping these lines will fix the issue ... if not it's a Hikari bug

Reaper should be enabled for polling mode

Because of this code change, we now expect the reaper to be started to be able to re-dispatch stuck entries.

Investigate an alternate mechanism for the bus/notification events

We have build our own internal bus event mechanism in Kill Bill to provide some very strong semantics, and in particular ensure that if any state has been changed, then an associated bus event will be sent. The implementation relies on two pieces:

• The guava bus event framework
• The persistence is implemented on top of our SQL engine -- mysql, postgresql, ...

We also support multiple configurations, some of them much more performant than others, mostly to address those 2 use cases:

• On Prem data center where instances are stable -- and same instance gets restarted after if fails
• Cloud deployment where instances may come and go -- and different instances are being rolled during deployment, expansions, ...

We would like to add a configuration to rely on some third part bus event systems. A good place to start would be activeMQ, but looking around for options might be good.

The work would be:

1. Implement the Queue interface to work with third part backend
2. Design/change current way of sending busEvents -- today this is mostly called from transactions, probably we need some other scheme, such as using a success transaction handler to post such events.
3. ... Down the line provide a docker-compose with activeMQ to bring up the stack.

The work would be similar for notificationQ, but maybe starting with bus events would be a good place.

single node consumer can not comsume message

we has single node consumer, when this node is down or deploying,
then producer post message , message can be move to history table ,but message handler not invoker，

I check sourcecode, found no subscribers can found ,

so I register first ,then start queue, but found "Attempting to register handler " + handlerInstance + " in a non initialized bus"

any help?

org.killbill.queue.org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException

Caused by: org.killbill.queue.org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'getReadyEntries' at line 1 [statement:"getReadyEntries", located:"getReadyEntries", rewritten:"getReadyEntries", arguments:{ positional:{}, named:{max:100,now:Fri Sep 04 15:26:39 CST 2020}, finder:[]}]
at org.killbill.queue.org.skife.jdbi.v2.SQLStatement.internalExecute(SQLStatement.java:1340)
at org.killbill.queue.org.skife.jdbi.v2.Query.fold(Query.java:177)
at org.killbill.queue.org.skife.jdbi.v2.Query.list(Query.java:86)
at

org.killbill.queue.org.skife.jdbi.v2.sqlobject.ResultReturnThing$IterableReturningThing.result(ResultReturnThing.java:259)
at org.killbill.queue.org.skife.jdbi.v2.sqlobject.ResultReturnThing.map(ResultReturnThing.java:52)
at org.killbill.queue.org.skife.jdbi.v2.sqlobject.QueryHandler.invoke(QueryHandler.java:49)
at org.killbill.queue.org.skife.jdbi.v2.sqlobject.SqlObject.invoke(SqlObject.java:184)
at org.killbill.queue.org.skife.jdbi.v2.sqlobject.SqlObject$2.intercept(SqlObject.java:97)
at org.killbill.queue.org.skife.jdbi.v2.sqlobject.CloseInternalDoNotUseThisClass$$EnhancerByCGLIB$$3ecc764e.getReadyEntries()
at org.killbill.queue.DBBackedQueueWithPolling.fetchReadyEntries(DBBackedQueueWithPolling.java:122)
at org.killbill.queue.DBBackedQueueWithPolling.access$000(DBBackedQueueWithPolling.java:46)
at org.killbill.queue.DBBackedQueueWithPolling$1.inTransaction(DBBackedQueueWithPolling.java:81)
at org.killbill.queue.DBBackedQueueWithPolling$1.inTransaction(DBBackedQueueWithPolling.java:76)
at org.killbill.queue.DBBackedQueue$9.inTransaction(DBBackedQueue.java:354)
at org.killbill.queue.org.skife.jdbi.v2.tweak.transactions.LocalTransactionHandler.inTransaction(LocalTransactionHandler.java:188)
at org.killbill.queue.org.skife.jdbi.v2.BasicHandle.inTransaction(BasicHandle.java:332)
at org.killbill.queue.org.skife.jdbi.v2.DBI$5.withHandle(DBI.java:332)
at org.killbill.queue.org.skife.jdbi.v2.DBI.withHandle(DBI.java:302)
... 14 common frames omitted

Implicit service name limitation

The service name needs to be at least 3 letters long - could we remove this limitation?

See

killbill-commons/queue/src/main/java/org/killbill/notificationq/NotificationQueueDispatcher.java

Line 204 in d8dc83d

final String metricName = new StringBuilder(parts[0].substring(0, 3))

Repeating data read

  @SqlQuery
    List<T> getReadyEntries(@Bind("now") Date now,
                            @Bind("max") int max,
                            // This is somewhat a hack, should really be a @Bind parameter but we also use it
                            // for StringTemplate to modify the query based whether value is null or not.
                            @Nullable @Define("owner") String owner,
                            @Define("tableName") final String tableName);

I found there are multiple thread run getReadyEntries at the same time in the code. Will different thread repeated data right?

Clarify config for NotificationQ and BusEvent

Today they both use similar properties exposed in the base class PersistentQueueConfig, but some properties are not available for both, and some mean different things (e.g getQueueCapacity which refers to a different queue)

Cleanup and visibility enhancement

This task is to track some enhancements we could do on 0.24.x release

New Queue API

Change PersistentBus to get rid of Transmogrifier:

public void postFromTransaction(BusEvent event, final Transmogrifier transmogrifier) throws EventBusException;

becomes

public void postFromTransaction(BusEvent event, final Connection connection) throws EventBusException;

where connection is a generic java.sql.Connection (autoCommit is false).

Changes in killbill-commons:

• Fix implementation in DefaultPersistentBus
• Create a DataSource to return that Connection
• Create a DBI from this DataSource
• Call onDemand

Changes in killbill:

• Implement new TransactionHandler (KillBillTransactionHandler)
• Have KillBillTransactionHandler delegate to RestartTransactionRunner (configurable)
• Intercept the transaction and cache the Handle in a thread local
• EntitySqlDaoTransactionalJdbiWrapper can now get the TransactionHandler from the DBI and pass the Connection in EntitySqlDaoTransactionWrapper#inTransaction (signature change)

As part of this work, we should

• check with upstream if they can expose the Handle/Connection in the transaction directly
• upgrade JDBI to the latest version

TestJerseyBaseServerModule is failing

Missing userToken from RetryableService json

The originalEvent from RetryableService is missing the userToken. Example:

{
  "originalEvent": "{\"busEvent\":{\"extBusEvent\":{\"objectId\":null,\"objectType\":\"INVOICE\",\"eventType\":\"INVOICE_NOTIFICATION\",\"accountId\":\"ba135655-1de0-4847-abcf-a4f0343fc733\",\"tenantId\":\"d05188d5-a9ee-4f07-908c-6174c437973a\",\"metaData\":null},\"extBusEventClass\":\"org.killbill.billing.beatrix.extbus.DefaultBusExternalEvent\",\"userToken\":null,\"searchKey1\":4184783402330769479,\"searchKey2\":5787557447408176903},\"busEventClass\":\"org.killbill.billing.osgi.KillbillEventRetriableBusHandler$OSGIBusEvent\"}",
  "originalEventClass": "org.killbill.queue.retry.SubscriberNotificationEvent",
  "originalEffectiveDate": "2018-03-26T04:53:18.000Z",
  "retryNb": 5

Address Spotbugs errors

1. Enforce the Spotbugs checks by enabling the property check.fail-spotbugs
2. Remove the various check.spotbugs-exclude-filter-file properties and associated spotbugs-exclude.xml files
3. Remove the @SuppressFBWarnings annotations
4. Address the errors.

NotificationQueue should rely on CachedThreadPool executor

The class NotificationQueueDispatcher should not manage the NotificationRunner itself and let the CachedThreadPool do it. The code for DefaultQueueLifecycle could be greatly simplified since there is only one thread.

Reaper over-reporting late entries

We shouldn't call addAll in the for loop:

killbill-commons/queue/src/main/java/org/killbill/queue/DBBackedQueue.java

Line 298 in a152b9f

lateEntries.addAll(entriesLeftBehind);

Review batchClaimEntries suspicious log

Seen in the field:

DBBackedQueue-notif-analytics_notifications: batchClaimEntries only claimed partial entries 100/98

Why can this still happen?

Improve profiling framework for DB transactions

DAO and DAO_DETAILS only record ResultSet execution time, which is good for checking slow queries but this doesn't give us the full DB picture. In particular, it doesn't measure the impact of the following calls:

• GET/SET TRANSACTION ISOLATION LEVEL
• SET AUTOCOMMIT
• EXECUTE STATEMENT
• DEALLOCATE PREPARE STATEMENT
• COMMIT

We don't have to go too low level (if needed, we can always enable the jdbc.audit logger), but it would be useful to at least measure the time taken to run full transactions.

TestDOTBuilder is failing

From Pierre:
" it's a bug indeed in the test. The ordering of key/value pairs is not deterministic, i.e. node_16 [color=grey style=filled label=INIT]; becomes node_16 [style=filled color=grey label=INIT]; on your machine. Funny we never ran into it, maybe you have a different Java version?"

Java version: Java(TM) SE Runtime Environment (build 1.8.0_05-b13)

GM16799:surefire-reports kpostlewaite$ cat testng-results.xml

node_1 [label="Op|S"]; node_0 -> node_2 [label="Op|F"]; node_2 -> node_2 [label="Op|F"]; } subgraph cluster_1 { label="Transaction"; subgraph cluster_2 { label="Authorize"; node_3 [style=filled color=grey label=INIT]; node_4 [style=filled color=grey label=SUCCESS]; node_5 [style=filled color=grey label=FAILED]; node_6 [label=PENDING]; node_3 -> node_4 [label="Op|S"]; node_3 -> node_5 [label="Op|F"]; node_3 -> node_6 [label="Op|P"]; node_6 -> node_4 [label="Op|S"]; node_6 -> node_5 [label="Op|F"]; } subgraph cluster_3 { label="Capture"; node_7 [style=filled color=grey label=INIT]; node_8 [style=filled color=grey label=SUCCESS]; node_9 [style=filled color=grey label=FAILED]; node_7 -> node_8 [label="Op|S"]; node_7 -> node_9 [label="Op|F"]; } subgraph cluster_4 { label="Purchase"; node_10 [style=filled color=grey label=INIT]; node_11 [style=filled color=grey label=SUCCESS]; node_12 [style=filled color=grey label=FAILED]; node_10 -> node_11 [label="Op|S"]; node_10 -> node_12 [label="Op|F"]; } subgraph cluster_5 { label="Void"; node_13 [style=filled color=grey label=INIT]; node_14 [style=filled color=grey label=SUCCESS]; node_15 [style=filled color=grey label=FAILED]; node_13 -> node_14 [label="Op|S"]; node_13 -> node_15 [label="Op|F"]; } subgraph cluster_6 { label="Refund"; node_16 [style=filled color=grey label=INIT]; node_17 [style=filled color=grey label=SUCCESS]; node_18 [style=filled color=grey label=FAILED]; node_16 -> node_17 [label="Op|S"]; node_16 -> node_18 [label="Op|F"]; } node_4 -> node_7 [style=dotted label=CAPTURE_AMOUNT_CHECK]; node_4 -> node_13 [style=dotted]; node_8 -> node_13 [style=dotted]; node_8 -> node_16 [style=dotted label=REFUND_AMOUNT_CHECK]; node_11 -> node_16 [style=dotted label=REFUND_AMOUNT_CHECK]; } subgraph cluster_7 { label="DirectPayment"; node_19 [style=filled color=grey label=INIT]; node_20 [style=filled color=grey label=OPEN]; node_21 [style=filled color=grey label=CLOSED]; node_19 -> node_20; node_20 -> node_21; node_19 -> node_3 [style=dotted color=blue]; node_19 -> node_10 [style=dotted color=blue]; node_20 -> node_4 [style=dotted color=green]; node_20 -> node_8 [style=dotted color=green]; node_21 -> node_17 [style=dotted color=green]; node_21 -> node_14 [style=dotted color=green]; } } ] but found [digraph Payment { splines=false; subgraph cluster_0 { label="Retry"; node_0 [color=grey style=filled label=INIT]; node_1 [color=grey style=filled label=SUCCESS]; node_2 [color=grey style=filled label=FAILED]; node_0 -> node_1 [label="Op|S"]; node_0 -> node_2 [label="Op|F"]; node_2 -> node_2 [label="Op|F"]; } subgraph cluster_1 { label="Transaction"; subgraph cluster_2 { label="Authorize"; node_3 [color=grey style=filled label=INIT]; node_4 [color=grey style=filled label=SUCCESS]; node_5 [color=grey style=filled label=FAILED]; node_6 [label=PENDING]; node_3 -> node_4 [label="Op|S"]; node_3 -> node_5 [label="Op|F"]; node_3 -> node_6 [label="Op|P"]; node_6 -> node_4 [label="Op|S"]; node_6 -> node_5 [label="Op|F"]; } subgraph cluster_3 { label="Capture"; node_7 [color=grey style=filled label=INIT]; node_8 [color=grey style=filled label=SUCCESS]; node_9 [color=grey style=filled label=FAILED]; node_7 -> node_8 [label="Op|S"]; node_7 -> node_9 [label="Op|F"]; } subgraph cluster_4 { label="Purchase"; node_10 [color=grey style=filled label=INIT]; node_11 [color=grey style=filled label=SUCCESS]; node_12 [color=grey style=filled label=FAILED]; node_10 -> node_11 [label="Op|S"]; node_10 -> node_12 [label="Op|F"]; } subgraph cluster_5 { label="Void"; node_13 [color=grey style=filled label=INIT]; node_14 [color=grey style=filled label=SUCCESS]; node_15 [color=grey style=filled label=FAILED]; node_13 -> node_14 [label="Op|S"]; node_13 -> node_15 [label="Op|F"]; } subgraph cluster_6 { label="Refund"; node_16 [color=grey style=filled label=INIT]; node_17 [color=grey style=filled label=SUCCESS]; node_18 [color=grey style=filled label=FAILED]; node_16 -> node_17 [label="Op|S"]; node_16 -> node_18 [label="Op|F"]; } node_4 -> node_7 [style=dotted label=CAPTURE_AMOUNT_CHECK]; node_4 -> node_13 [style=dotted]; node_8 -> node_13 [style=dotted]; node_8 -> node_16 [style=dotted label=REFUND_AMOUNT_CHECK]; node_11 -> node_16 [style=dotted label=REFUND_AMOUNT_CHECK]; } subgraph cluster_7 { label="DirectPayment"; node_19 [color=grey style=filled label=INIT]; node_20 [color=grey style=filled label=OPEN]; node_21 [color=grey style=filled label=CLOSED]; node_19 -> node_20; node_20 -> node_21; node_19 -> node_3 [style=dotted color=blue]; node_19 -> node_10 [style=dotted color=blue]; node_20 -> node_4 [style=dotted color=green]; node_20 -> node_8 [style=dotted color=green]; node_21 -> node_17 [style=dotted color=green]; node_21 -> node_14 [style=dotted color=green]; } } ]]]> node_1 [label="Op|S"]; node_0 -> node_2 [label="Op|F"]; node_2 -> node_2 [label="Op|F"]; } subgraph cluster_1 { label="Transaction"; subgraph cluster_2 { label="Authorize"; node_3 [style=filled color=grey label=INIT]; node_4 [style=filled color=grey label=SUCCESS]; node_5 [style=filled color=grey label=FAILED]; node_6 [label=PENDING]; node_3 -> node_4 [label="Op|S"]; node_3 -> node_5 [label="Op|F"]; node_3 -> node_6 [label="Op|P"]; node_6 -> node_4 [label="Op|S"]; node_6 -> node_5 [label="Op|F"]; } subgraph cluster_3 { label="Capture"; node_7 [style=filled color=grey label=INIT]; node_8 [style=filled color=grey label=SUCCESS]; node_9 [style=filled color=grey label=FAILED]; node_7 -> node_8 [label="Op|S"]; node_7 -> node_9 [label="Op|F"]; } subgraph cluster_4 { label="Purchase"; node_10 [style=filled color=grey label=INIT]; node_11 [style=filled color=grey label=SUCCESS]; node_12 [style=filled color=grey label=FAILED]; node_10 -> node_11 [label="Op|S"]; node_10 -> node_12 [label="Op|F"]; } subgraph cluster_5 { label="Void"; node_13 [style=filled color=grey label=INIT]; node_14 [style=filled color=grey label=SUCCESS]; node_15 [style=filled color=grey label=FAILED]; node_13 -> node_14 [label="Op|S"]; node_13 -> node_15 [label="Op|F"]; } subgraph cluster_6 { label="Refund"; node_16 [style=filled color=grey label=INIT]; node_17 [style=filled color=grey label=SUCCESS]; node_18 [style=filled color=grey label=FAILED]; node_16 -> node_17 [label="Op|S"]; node_16 -> node_18 [label="Op|F"]; } node_4 -> node_7 [style=dotted label=CAPTURE_AMOUNT_CHECK]; node_4 -> node_13 [style=dotted]; node_8 -> node_13 [style=dotted]; node_8 -> node_16 [style=dotted label=REFUND_AMOUNT_CHECK]; node_11 -> node_16 [style=dotted label=REFUND_AMOUNT_CHECK]; } subgraph cluster_7 { label="DirectPayment"; node_19 [style=filled color=grey label=INIT]; node_20 [style=filled color=grey label=OPEN]; node_21 [style=filled color=grey label=CLOSED]; node_19 -> node_20; node_20 -> node_21; node_19 -> node_3 [style=dotted color=blue]; node_19 -> node_10 [style=dotted color=blue]; node_20 -> node_4 [style=dotted color=green]; node_20 -> node_8 [style=dotted color=green]; node_21 -> node_17 [style=dotted color=green]; node_21 -> node_14 [style=dotted color=green]; } } ] but found [digraph Payment { splines=false; subgraph cluster_0 { label="Retry"; node_0 [color=grey style=filled label=INIT]; node_1 [color=grey style=filled label=SUCCESS]; node_2 [color=grey style=filled label=FAILED]; node_0 -> node_1 [label="Op|S"]; node_0 -> node_2 [label="Op|F"]; node_2 -> node_2 [label="Op|F"]; } subgraph cluster_1 { label="Transaction"; subgraph cluster_2 { label="Authorize"; node_3 [color=grey style=filled label=INIT]; node_4 [color=grey style=filled label=SUCCESS]; node_5 [color=grey style=filled label=FAILED]; node_6 [label=PENDING]; node_3 -> node_4 [label="Op|S"]; node_3 -> node_5 [label="Op|F"]; node_3 -> node_6 [label="Op|P"]; node_6 -> node_4 [label="Op|S"]; node_6 -> node_5 [label="Op|F"]; } subgraph cluster_3 { label="Capture"; node_7 [color=grey style=filled label=INIT]; node_8 [color=grey style=filled label=SUCCESS]; node_9 [color=grey style=filled label=FAILED]; node_7 -> node_8 [label="Op|S"]; node_7 -> node_9 [label="Op|F"]; } subgraph cluster_4 { label="Purchase"; node_10 [color=grey style=filled label=INIT]; node_11 [color=grey style=filled label=SUCCESS]; node_12 [color=grey style=filled label=FAILED]; node_10 -> node_11 [label="Op|S"]; node_10 -> node_12 [label="Op|F"]; } subgraph cluster_5 { label="Void"; node_13 [color=grey style=filled label=INIT]; node_14 [color=grey style=filled label=SUCCESS]; node_15 [color=grey style=filled label=FAILED]; node_13 -> node_14 [label="Op|S"]; node_13 -> node_15 [label="Op|F"]; } subgraph cluster_6 { label="Refund"; node_16 [color=grey style=filled label=INIT]; node_17 [color=grey style=filled label=SUCCESS]; node_18 [color=grey style=filled label=FAILED]; node_16 -> node_17 [label="Op|S"]; node_16 -> node_18 [label="Op|F"]; } node_4 -> node_7 [style=dotted label=CAPTURE_AMOUNT_CHECK]; node_4 -> node_13 [style=dotted]; node_8 -> node_13 [style=dotted]; node_8 -> node_16 [style=dotted label=REFUND_AMOUNT_CHECK]; node_11 -> node_16 [style=dotted label=REFUND_AMOUNT_CHECK]; } subgraph cluster_7 { label="DirectPayment"; node_19 [color=grey style=filled label=INIT]; node_20 [color=grey style=filled label=OPEN]; node_21 [color=grey style=filled label=CLOSED]; node_19 -> node_20; node_20 -> node_21; node_19 -> node_3 [style=dotted color=blue]; node_19 -> node_10 [style=dotted color=blue]; node_20 -> node_4 [style=dotted color=green]; node_20 -> node_8 [style=dotted color=green]; node_21 -> node_17 [style=dotted color=green]; node_21 -> node_14 [style=dotted color=green]; } } ] at org.testng.Assert.fail(Assert.java:94) at org.testng.Assert.failNotEquals(Assert.java:494) at org.testng.Assert.assertEquals(Assert.java:123) at org.testng.Assert.assertEquals(Assert.java:176) at org.testng.Assert.assertEquals(Assert.java:186) at org.killbill.automaton.dot.TestDOTBuilder.testGenerator(TestDOTBuilder.java:115) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84) at org.testng.internal.Invoker.invokeMethod(Invoker.java:714) at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901) at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231) at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111) at org.testng.TestRunner.privateRun(TestRunner.java:767) at org.testng.TestRunner.run(TestRunner.java:617) at org.testng.SuiteRunner.runTest(SuiteRunner.java:348) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:343) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:305) at org.testng.SuiteRunner.run(SuiteRunner.java:254) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224) at org.testng.TestNG.runSuitesLocally(TestNG.java:1149) at org.testng.TestNG.run(TestNG.java:1057) at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:77) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeMulti(TestNGDirectoryTestSuite.java:189) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:105) at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:117) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:158) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95) ]]> GM16799:surefire-reports kpostlewaite$

Queue lifecycle thread may exit upon Runtime exception

Our queue lifecycle thread catches Throwable and exit upon RuntimeException (i.e CallbackFailedException , but the Executor restart thread with no task so nothing gets dispatched.

A normal stack trace would be:

java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.killbill.queue.DefaultQueueLifecycle$DispatcherRunnable.sleepSporadically(DefaultQueueLifecycle.java:285)
        at org.killbill.queue.DefaultQueueLifecycle$DispatcherRunnable.run(DefaultQueueLifecycle.java:253)
        at org.killbill.commons.concurrent.WrappedRunnable.run(WrappedRunnable.java:48)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Instead, we see:

java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000005c9872b00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)