killbill/killbill-integration-tests

Kill Bill integration tests

Kill Bill - Open-Source Subscription Billing & Payments Platform

Kill Bill has been the leading Open-Source Subscription Billing & Payments Platform over the past 10 years. The platform exists to help you scale your billing and payment infrastructure and to grow your business. Out of the box, you have access to real-time analytics and financial reports. And because you are in control of your business and client data, you avoid any vendor lock-in.

Kill Bill can fulfill the billing and payment needs of nearly any online business. Reliable and robust, it has proven itself over time with even the largest SaaS and e-commerce organizations.

Open-Source + SaaS billing = OpenSaaS billing

Kill Bill offers a lot out of the box:

• You can test various subscription billing models to see which is most profitable for your business.
• It is easy to run trials and accommodate even the briefest promotional discounts.
• Your business doesn't need to rely on the uptime or processing speed of a third-party SaaS provider.
• Kill Bill is not an all-in-one solution. Instead, it is highly modularized, enabling you to disable functionality you don't need or replace functionality with one of your existing systems.
• On premises or in the cloud, Kill Bill will scale as your subscription business grows. You can also start with Kill Bill in one business area and slowly migrate it to other areas.
• Kill Bill provides an unmatched framework for extensibility.

With Kill Bill, you get a SaaS-like subscription management and billing solution out of the box, but with the flexibility and peace of mind of running an open-source stack.

Get your own subscription billing platform

• You can get much more information on killbill.io.
• All of our technical documentation is available on docs.killbill.io (sources in the killbill/killbill-docs repository).
• Eager to get started? Deploy your own instance of Kill Bill on AWS using our one-click deployer.
• Head over to cloud.killbill.io for free hosted sandboxes, developer tools, and more.

Open-Source and governance

Kill Bill is free to use and completely open-source under the Apache Licence 2.0 license. 🎉

However, we do require financial backing to sustain the effort to maintain and enhance the project. Companies, individual users, and contributors can join their peers in supporting the work through GitHub Sponsors. 🍻

Get stats on Kill Bill contributors

Looking for statistics? This project does not use a monolithic repository, but is instead split across many components. To gather accurate counts on contributors, stars, and forks, take a look across our entire GitHub organization. 📈

About Kill Bill

Martin Westhead, Pierre-Alexandre Meyer, and Stéphane Brossier founded the project independently in 2010. The Billing Project, LLC owns the Kill Bill codebase and trademarks. Professional services, sponsorships, and commercial support packages are available upon request.

killbill-integration-tests's Issues

Regression suite for major Kill Bill releases

Releasing a new major Kill Bill version (e.g. 0.18 -> 0.20) always takes a lot of time, because of the testing involved. We basically have to go through all plugins and verify API compatibility.

Building upon #10, we can now automate some of it.

The goal would be to create a new regression suite. For each plugin known by kpm (see kpm info implementation), the test would:

• download the right version of the plugin (we would typically run these tests against a local Kill Bill 0.19 instance)
• start the plugin
• verify the plugin is started (running)
• stop and uninstall the plugin

For now, since 0.19 plugins aren't ready yet, we can just run this suite against a local 0.18.x instance (it would be a good test to verify that indeed all of the plugins are up-to-date).

Running tests - KillBillClient::API::Unauthorized

Sorry, I'm complete novice to Ruby and can't figure out where to put api_secret & api_key in test_base.rb.
I've launched KillBill from repo:

./bin/start-server -s -d -p 8181 > /tmp/server.out 2>&1 < /dev/null &

Here is what I've got on every test (rake test:core)

Error: test_create_subscription_with_tz_plus_9_with_requested_date_today(KillBillIntegrationTests::TestWithDatesAndTimezones)
KillBillClient::API::Unauthorized:
/home/roma/.rvm/gems/jruby-1.7.24/gems/killbill-client-0.29.0/lib/killbill_client/api/net_http_adapter.rb:194:in request' /home/roma/.rvm/gems/jruby-1.7.24/gems/killbill-client-0.29.0/lib/killbill_client/api/api.rb:49:inpost'
/home/roma/.rvm/gems/jruby-1.7.24/gems/killbill-client-0.29.0/lib/killbill_client/models/resource.rb:45:in post' /home/roma/.rvm/gems/jruby-1.7.24/gems/killbill-client-0.29.0/lib/killbill_client/models/tenant.rb:99:increate'
/home/roma/Projects/Github/KillBill/killbill-integration-tests/killbill-integration-tests/mixin-utils/helper.rb:78:in setup_create_tenant' /home/roma/Projects/Github/KillBill/killbill-integration-tests/killbill-integration-tests/test_base.rb:42:insetup_base'
/home/roma/Projects/Github/KillBill/killbill-integration-tests/killbill-integration-tests/core/test_with_dates_timezones.rb:62:in test_scenario_fixed_price_with_future_invoice' /home/roma/Projects/Github/KillBill/killbill-integration-tests/killbill-integration-tests/core/test_with_dates_timezones.rb:56:intest_scenario_fixed_price'
/home/roma/Projects/Github/KillBill/killbill-integration-tests/killbill-integration-tests/core/test_with_dates_timezones.rb:43:in `test_create_subscription_with_tz_plus_9_with_requested_date_today'
40:
41: def test_create_subscription_with_tz_plus_9_with_requested_date_today
42: # Requested date of '2013-08-03' will take today's date (and all conversion in account timezone will lead to '2013-08-03' and NOT '2013-08-02')
=> 43: test_scenario_fixed_price("2013-08-02T18:00:00.000Z", 'Asia/Tokyo', '2013-08-03', '2013-08-03', '2013-08-03')
44: end
45:
46: def test_create_subscription_with_tz_plus_9_with_requested_date_in_slight_past

Fully automate tests with plugins

test_payment_with_control.rb already automates download of the payment test plugin before running the suite.

Enhancements:

• Don't download/start if the plugin is already installed/started - this should help with testing dev versions
• Adapt other tests to do the same (e.g. tax tests)

Overdue test fails with docker containers

If I create an Ubuntu 16.04 server running the Killbill containers alongside a local MariaDB instance (as described in the instructions I've sent through), then running the entire core test suite results in a failure in test_overdue. rb

$ RUBYOPT=W0 rake test:all
/usr/bin/ruby2.3 -w -I"lib:killbill-integration-tests" -I"/var/lib/gems/2.3.0/gems/rake-11.3.0/lib" "/var/lib/gems/2.3.0/gems/rake-11.3.0/lib/rake/rake_test_loader.rb" "killbill-integration-tests/core/test_catalog.rb" "killbill-integration-tests/core/test_entitlement_add_on.rb" "killbill-integration-tests/core/test_entitlement_cancellation.rb" "killbill-integration-tests/core/test_entitlement_change_plan_test.rb" "killbill-integration-tests/core/test_entitlement_create.rb" "killbill-integration-tests/core/test_invoice.rb" "killbill-integration-tests/core/test_invoice_payment.rb" "killbill-integration-tests/core/test_invoice_template.rb" "killbill-integration-tests/core/test_migration_scenario.rb" "killbill-integration-tests/core/test_overdue.rb" "killbill-integration-tests/core/test_pause_resume.rb" "killbill-integration-tests/core/test_payment.rb" "killbill-integration-tests/core/test_price_override.rb" "killbill-integration-tests/core/test_recurring_in_arrear.rb" "killbill-integration-tests/core/test_refund_credit.rb" "killbill-integration-tests/core/test_tag.rb" "killbill-integration-tests/core/test_tenant.rb" "killbill-integration-tests/core/test_transfer.rb" "killbill-integration-tests/core/test_with_dates_timezones.rb" 
Loaded suite /var/lib/gems/2.3.0/gems/rake-11.3.0/lib/rake/rake_test_loader
Started
...............................................................F
===============================================================================
Failure:
test_add_overdue_state(KillBillIntegrationTests::TestOverdue)
/home/ubuntu/killbill-integration-tests/killbill-integration-tests/core/test_overdue.rb:197:in `check_overdue_stage'
/home/ubuntu/killbill-integration-tests/killbill-integration-tests/core/test_overdue.rb:192:in `add_days_and_check_overdue_stage'
/home/ubuntu/killbill-integration-tests/killbill-integration-tests/core/test_overdue.rb:169:in `go_through_all_overdue_stages'
/home/ubuntu/killbill-integration-tests/killbill-integration-tests/core/test_overdue.rb:31:in `test_add_overdue_state'
     28:     end
     29: 
     30:     def test_add_overdue_state
  => 31:       bp = go_through_all_overdue_stages(@account, 'OD3', '2013-08-01')
     32:       check_entitlement_with_events(bp,
     33:                                     '2013-08-01',
     34:                                     [{:type                   => 'START_ENTITLEMENT',
Failed to retrieve overdue status associated with account 80f6adad-c47b-4b21-9985-40398bb728ba
<"OD1"> expected but was
<"__KILLBILL__CLEAR__OVERDUE_STATE__">

diff:
?                    O   D1         
? __KILLBILL__CLEAR__ VER UE_STATE__
===============================================================================
...............................

Finished in 1017.960196075 seconds.

95 tests, 4930 assertions, 1 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
98.9474% passed

0.09 tests/s, 4.84 assertions/s
rake aborted!
Command failed with status (1): [ruby -w -I"lib:killbill-integration-tests" -I"/var/lib/gems/2.3.0/gems/rake-11.3.0/lib" "/var/lib/gems/2.3.0/gems/rake-11.3.0/lib/rake/rake_test_loader.rb" "killbill-integration-tests/core/test_catalog.rb" "killbill-integration-tests/core/test_entitlement_add_on.rb" "killbill-integration-tests/core/test_entitlement_cancellation.rb" "killbill-integration-tests/core/test_entitlement_change_plan_test.rb" "killbill-integration-tests/core/test_entitlement_create.rb" "killbill-integration-tests/core/test_invoice.rb" "killbill-integration-tests/core/test_invoice_payment.rb" "killbill-integration-tests/core/test_invoice_template.rb" "killbill-integration-tests/core/test_migration_scenario.rb" "killbill-integration-tests/core/test_overdue.rb" "killbill-integration-tests/core/test_pause_resume.rb" "killbill-integration-tests/core/test_payment.rb" "killbill-integration-tests/core/test_price_override.rb" "killbill-integration-tests/core/test_recurring_in_arrear.rb" "killbill-integration-tests/core/test_refund_credit.rb" "killbill-integration-tests/core/test_tag.rb" "killbill-integration-tests/core/test_tenant.rb" "killbill-integration-tests/core/test_transfer.rb" "killbill-integration-tests/core/test_with_dates_timezones.rb" ]
/var/lib/gems/2.3.0/gems/rake-11.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => test:all
(See full trace by running task with --trace)

If the test suite is run individually or with an external database (which slows the tests down) then it works just fine.

Occasionally you see messages like wait_for_expected_clause : timed out for account 404ae871-09e6-4d15-aacb-18c12ac9a5af after 5 popping up in the trace which suggests that this may be related to load on the system.

Investigate random NPE in test_entitlement_change_skip_phase

See https://circleci.com/gh/killbill/killbill/738:

Error: test_change_multiple_times_over_initial_trial_period_1(KillBillIntegrationTests::TestEntitlementChangeSkipPhase)
  KillBillClient::API::InternalServerError: {"className":"org.skife.jdbi.v2.exceptions.CallbackFailedException","code":null,"message":"java.lang.NullPointerException","causeClassName":"java.lang.NullPointerException","causeMessage":null,"stackTrace":[]}
/var/lib/jruby/lib/ruby/gems/shared/bundler/gems/killbill-client-ruby-c0071f72fc10/lib/killbill_client/api/net_http_adapter.rb:217:in `request'
/var/lib/jruby/lib/ruby/gems/shared/bundler/gems/killbill-client-ruby-c0071f72fc10/lib/killbill_client/api/api.rb:55:in `put'
/var/lib/jruby/lib/ruby/gems/shared/bundler/gems/killbill-client-ruby-c0071f72fc10/lib/killbill_client/models/resource.rb:53:in `put'
/home/killbill/killbill-integration-tests/killbill-integration-tests/core/test_entitlement_change_skip_phase.rb:34:in `change_plan_with_target_phase'
/home/killbill/killbill-integration-tests/killbill-integration-tests/core/test_entitlement_change_skip_phase.rb:137:in `test_change_multiple_times_over_initial_trial_period_1'
     134:       # Change plan to Gold : NO TRIAL
     135:       requested_date = nil
     136:       billing_policy = "IMMEDIATE"
  => 137:       bp = bp.change_plan_with_target_phase({:productName => 'Gold', :billingPeriod => 'MONTHLY', :priceList => 'DEFAULT'}, @user, nil, nil, requested_date, billing_policy, :EVERGREEN, false, @options)
     138:       check_entitlement(bp, 'Gold', 'BASE', 'MONTHLY', 'DEFAULT', KB_INIT_DATE, nil)
     139:       wait_for_expected_clause(4, @account, @options, &@proc_account_invoices_nb)
     140: 
===============================================================================
E
===============================================================================
Error: test_change_multiple_times_over_initial_trial_period_2(KillBillIntegrationTests::TestEntitlementChangeSkipPhase)
  KillBillClient::API::InternalServerError: {"className":"org.skife.jdbi.v2.exceptions.CallbackFailedException","code":null,"message":"java.lang.NullPointerException","causeClassName":"java.lang.NullPointerException","causeMessage":null,"stackTrace":[]}
/var/lib/jruby/lib/ruby/gems/shared/bundler/gems/killbill-client-ruby-c0071f72fc10/lib/killbill_client/api/net_http_adapter.rb:217:in `request'
/var/lib/jruby/lib/ruby/gems/shared/bundler/gems/killbill-client-ruby-c0071f72fc10/lib/killbill_client/api/api.rb:55:in `put'
/var/lib/jruby/lib/ruby/gems/shared/bundler/gems/killbill-client-ruby-c0071f72fc10/lib/killbill_client/models/resource.rb:53:in `put'
/home/killbill/killbill-integration-tests/killbill-integration-tests/core/test_entitlement_change_skip_phase.rb:34:in `change_plan_with_target_phase'
/home/killbill/killbill-integration-tests/killbill-integration-tests/core/test_entitlement_change_skip_phase.rb:223:in `test_change_multiple_times_over_initial_trial_period_2'
     220:       # Change plan to Gold : NO TRIAL
     221:       requested_date = nil
     222:       billing_policy = "IMMEDIATE"
  => 223:       bp = bp.change_plan_with_target_phase({:productName => 'Gold', :billingPeriod => 'MONTHLY', :priceList => 'DEFAULT'}, @user, nil, nil, requested_date, billing_policy, :EVERGREEN, false, @options)
     224:       check_entitlement(bp, 'Gold', 'BASE', 'MONTHLY', 'DEFAULT', KB_INIT_DATE, nil)
     225:       wait_for_expected_clause(4, @account, @options, &@proc_account_invoices_nb)
     226: 
===============================================================================
.........................................................................

Finished in 995.112 seconds.

132 tests, 7379 assertions, 0 failures, 2 errors, 0 pendings, 0 omissions, 0 notifications
98.4848% passed

Enable tests in main repo CI workflow

After switching off CircleCI, integrations tests aren't running anymore: https://github.com/killbill/killbill/blob/master/.github/workflows/ci.yml

It shouldn't be too hard to enable them, we do something similar for Avatax for instance (see https://github.com/killbill/killbill-avatax-plugin/blob/36b6d58cd610fde9bf8fff66303a084de5a4a16e/.github/workflows/ci.yml#L58), the main difference would be that Kill Bill needs to be started using the start-server script instead of using a published Docker image.

Re-enable disabled tests in CI

• The email notifications plugin seems misconfigured
• The payment test plugin pulls the legacy Ruby one, instead of the Java one

Fix core/test_entitlement_add_on.rb#test_with_ao test

See killbill/killbill#106.

Intermittent failure of test_entitlement_create against docker image

I'm continuing to have problems with the integration tests failing against the 0.16.9 docker container.

I create a docker environment on Ubuntu Xenial with docker-compose using:

killbill:
  image: killbill/killbill:0.16.9
  restart: always
  ports:
    - "8080:8080"
  links:
    - db
  environment:
    - KILLBILL_CONFIG_DAO_URL=jdbc:mysql://db:3306/killbill
    - KILLBILL_CONFIG_DAO_USER=root
    - KILLBILL_CONFIG_DAO_PASSWORD=killbill
    - KILLBILL_PLUGIN_STRIPE=1
kaui:
  image: killbill/kaui:0.7.0
  restart: always
  ports:
    - "9090:8080"
  links:
    - db
    - killbill
  environment:
    - KAUI_CONFIG_DAO_URL=jdbc:mysql://db:3306/kaui
    - KAUI_CONFIG_DAO_USER=root
    - KAUI_CONFIG_DAO_PASSWORD=killbill
    - KAUI_URL=http://killbill:8080
db:
  image: killbill/mariadb:0.16
  restart: always
  expose:
    - "3306"
  environment:
    - MYSQL_ROOT_PASSWORD=killbill

If I then run the integration tests locally on the same server using:

RUBYOPT=W0 rake test:core TEST=killbill-integration-tests/core/test_entitlement_create.rb

then it works as expected:

/usr/bin/ruby2.3 -w -I"lib:killbill-integration-tests" -I"/var/lib/gems/2.3.0/gems/rake-11.3.0/lib" "/var/lib/gems/2.3.0/gems/rake-11.3.0/lib/rake/rake_test_loader.rb" "killbill-integration-tests/core/test_entitlement_create.rb" 
Loaded suite /var/lib/gems/2.3.0/gems/rake-11.3.0/lib/rake/rake_test_loader
Started
...

Finished in 50.474516508 seconds.

3 tests, 160 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed

However if I run the same test on a separate server (having altered test_base.rb to point at the killbill instance), then I get the following error:

/usr/bin/ruby2.3 -w -I"lib:killbill-integration-tests" -I"/var/lib/gems/2.3.0/gems/rake-11.3.0/lib" "/var/lib/gems/2.3.0/gems/rake-11.3.0/lib/rake/rake_test_loader.rb" "killbill-integration-tests/core/test_entitlement_create.rb" 
Loaded suite /var/lib/gems/2.3.0/gems/rake-11.3.0/lib/rake/rake_test_loader
Started
.F
===============================================================================
Failure:
test_create_ao_subscription_aligned_skip_trial(KillBillIntegrationTests::TestEntitlementCreateTest)
/home/ubuntu/killbill-integration-tests/killbill-integration-tests/mixin-checkers/invoice_checker.rb:19:in `check_invoice_item'
/home/ubuntu/killbill-integration-tests/killbill-integration-tests/core/test_entitlement_create.rb:152:in `test_create_ao_subscription_aligned_skip_trial'
     149:       sort_invoices!(all_invoices)
     150:       assert_equal(5, all_invoices.size)
     151:       fifth_invoice = all_invoices[4]
  => 152:       check_invoice_item(fifth_invoice.items[0], fifth_invoice.invoice_id, 37.95, 'USD', 'RECURRING', 'remotecontroladvanced-monthly', 'remotecontroladvanced-monthly-evergreen', '2013-09-30', '2013-10-31')
     153: 
     154:     end
     155: 
invoice_item bbd2c7ea-68e7-4712-b3ff-448d5ede8c49
<37.95> expected but was
<500.0>

diff:
? 37 .95
? 500 0 
===============================================================================


Finished in 45.035286933 seconds.

3 tests, 153 assertions, 1 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
66.6667% passed

0.07 tests/s, 3.40 assertions/s
rake aborted!
Command failed with status (1): [ruby -w -I"lib:killbill-integration-tests" -I"/var/lib/gems/2.3.0/gems/rake-11.3.0/lib" "/var/lib/gems/2.3.0/gems/rake-11.3.0/lib/rake/rake_test_loader.rb" "killbill-integration-tests/core/test_entitlement_create.rb" ]

In the success case the logs have the Invoice dispatcher logs this way around

^[[36mkillbill_1 | ^[[0m2016-10-12T16:54:28,941+0000 lvl='INFO', log='InvoiceDispatcher', th='notifications-th', xff='', rId='', aRId='2', tRId='2', Generated invoiceId='265be1b0-2624-40ca-a8d2-453692f894b5', numberOfItems='2', accountId='f1c63a8c-1a18-47a6-9f80-e6815e66091c', targetDate='2013-09-30':
^[[36mkillbill_1 | ^[[0m         item = RECURRING{startDate=2013-09-30, endDate=2013-10-31, amount=37.95, rate=37.95, subscriptionId=613514bf-7c21-42d6-8b69-2fff8fb1b050, linkedItemId=null}
^[[36mkillbill_1 | ^[[0m         item = RECURRING{startDate=2013-09-30, endDate=2013-10-31, amount=500.00, rate=500.00, subscriptionId=fa6316b0-24ab-471e-be2c-8afd31940305, linkedItemId=null}

and there is a Next billing date log for the top Subscription Id

^[[36mkillbill_1 | ^[[0m2016-10-12T16:54:29,008+0000 lvl='INFO', log='DefaultNextBillingDatePoster', th='notifications-th', xff='', rId='', aRId='2', tRId='2', Queuing next billing date notification at 2013-10-31T06:00:03.000Z for subscriptionId 613514bf-7c21-42d6-8b69-2fff8fb1b050

in the failure case the invoice dispatcher logs are this way around

^[[33mkillbill_1  |^[[0m 2016-10-07T15:29:09,852+0000 lvl='INFO', log='InvoiceDispatcher', th='notifications-th', xff='', rId='', aRId='2', tRId='2', Generated invoiceId='3abdd764-e8c2-49a2-90c1-10f6379d56ea', numberOfItems='2', accountId='042fca47-e970-49f7-ac52-5277b455126a', targetDate='2013-09-30':
^[[33mkillbill_1  |^[[0m         item = RECURRING{startDate=2013-09-30, endDate=2013-10-31, amount=500.00, rate=500.00, subscriptionId=55085f05-fdaf-4e93-ad40-21de154f7514, linkedItemId=null}
^[[33mkillbill_1  |^[[0m         item = RECURRING{startDate=2013-09-30, endDate=2013-10-31, amount=37.95, rate=37.95, subscriptionId=5ebae650-a383-460b-8775-8b123977a599, linkedItemId=null}
^[[33mkillbill_1  |^[[0m

and the NextBillingDate log entry is for the first subscription ID in the pair.

^[[33mkillbill_1  |^[[0m 2016-10-07T15:29:09,950+0000 lvl='INFO', log='DefaultNextBillingDatePoster', th='notifications-th', xff='', rId='', aRId='2', tRId='2', Queuing next billing date notification at 2013-10-31T06:00:03.000Z for subscriptionId 55085f05-fdaf-4e93-ad40-21de154f7514

That seems to be the only difference in the log traces.

Thoughts?

README to describe how to run those tests

• Describe which catalog, properties, ... to use
• Which command to run