killbill/killbill-integration-tests

Kill Bill integration tests

killbill-integration-tests's Issues

README to describe how to run those tests

• Describe which catalog, properties, ... to use
• Which command to run

Re-enable disabled tests in CI

• The email notifications plugin seems misconfigured
• The payment test plugin pulls the legacy Ruby one, instead of the Java one

Overdue test fails with docker containers

If I create an Ubuntu 16.04 server running the Killbill containers alongside a local MariaDB instance (as described in the instructions I've sent through), then running the entire core test suite results in a failure in test_overdue. rb

$ RUBYOPT=W0 rake test:all
/usr/bin/ruby2.3 -w -I"lib:killbill-integration-tests" -I"/var/lib/gems/2.3.0/gems/rake-11.3.0/lib" "/var/lib/gems/2.3.0/gems/rake-11.3.0/lib/rake/rake_test_loader.rb" "killbill-integration-tests/core/test_catalog.rb" "killbill-integration-tests/core/test_entitlement_add_on.rb" "killbill-integration-tests/core/test_entitlement_cancellation.rb" "killbill-integration-tests/core/test_entitlement_change_plan_test.rb" "killbill-integration-tests/core/test_entitlement_create.rb" "killbill-integration-tests/core/test_invoice.rb" "killbill-integration-tests/core/test_invoice_payment.rb" "killbill-integration-tests/core/test_invoice_template.rb" "killbill-integration-tests/core/test_migration_scenario.rb" "killbill-integration-tests/core/test_overdue.rb" "killbill-integration-tests/core/test_pause_resume.rb" "killbill-integration-tests/core/test_payment.rb" "killbill-integration-tests/core/test_price_override.rb" "killbill-integration-tests/core/test_recurring_in_arrear.rb" "killbill-integration-tests/core/test_refund_credit.rb" "killbill-integration-tests/core/test_tag.rb" "killbill-integration-tests/core/test_tenant.rb" "killbill-integration-tests/core/test_transfer.rb" "killbill-integration-tests/core/test_with_dates_timezones.rb" 
Loaded suite /var/lib/gems/2.3.0/gems/rake-11.3.0/lib/rake/rake_test_loader
Started
...............................................................F
===============================================================================
Failure:
test_add_overdue_state(KillBillIntegrationTests::TestOverdue)
/home/ubuntu/killbill-integration-tests/killbill-integration-tests/core/test_overdue.rb:197:in `check_overdue_stage'
/home/ubuntu/killbill-integration-tests/killbill-integration-tests/core/test_overdue.rb:192:in `add_days_and_check_overdue_stage'
/home/ubuntu/killbill-integration-tests/killbill-integration-tests/core/test_overdue.rb:169:in `go_through_all_overdue_stages'
/home/ubuntu/killbill-integration-tests/killbill-integration-tests/core/test_overdue.rb:31:in `test_add_overdue_state'
     28:     end
     29: 
     30:     def test_add_overdue_state
  => 31:       bp = go_through_all_overdue_stages(@account, 'OD3', '2013-08-01')
     32:       check_entitlement_with_events(bp,
     33:                                     '2013-08-01',
     34:                                     [{:type                   => 'START_ENTITLEMENT',
Failed to retrieve overdue status associated with account 80f6adad-c47b-4b21-9985-40398bb728ba
<"OD1"> expected but was
<"__KILLBILL__CLEAR__OVERDUE_STATE__">

diff:
?                    O   D1         
? __KILLBILL__CLEAR__ VER UE_STATE__
===============================================================================
...............................

Finished in 1017.960196075 seconds.

95 tests, 4930 assertions, 1 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
98.9474% passed

0.09 tests/s, 4.84 assertions/s
rake aborted!
Command failed with status (1): [ruby -w -I"lib:killbill-integration-tests" -I"/var/lib/gems/2.3.0/gems/rake-11.3.0/lib" "/var/lib/gems/2.3.0/gems/rake-11.3.0/lib/rake/rake_test_loader.rb" "killbill-integration-tests/core/test_catalog.rb" "killbill-integration-tests/core/test_entitlement_add_on.rb" "killbill-integration-tests/core/test_entitlement_cancellation.rb" "killbill-integration-tests/core/test_entitlement_change_plan_test.rb" "killbill-integration-tests/core/test_entitlement_create.rb" "killbill-integration-tests/core/test_invoice.rb" "killbill-integration-tests/core/test_invoice_payment.rb" "killbill-integration-tests/core/test_invoice_template.rb" "killbill-integration-tests/core/test_migration_scenario.rb" "killbill-integration-tests/core/test_overdue.rb" "killbill-integration-tests/core/test_pause_resume.rb" "killbill-integration-tests/core/test_payment.rb" "killbill-integration-tests/core/test_price_override.rb" "killbill-integration-tests/core/test_recurring_in_arrear.rb" "killbill-integration-tests/core/test_refund_credit.rb" "killbill-integration-tests/core/test_tag.rb" "killbill-integration-tests/core/test_tenant.rb" "killbill-integration-tests/core/test_transfer.rb" "killbill-integration-tests/core/test_with_dates_timezones.rb" ]
/var/lib/gems/2.3.0/gems/rake-11.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => test:all
(See full trace by running task with --trace)

If the test suite is run individually or with an external database (which slows the tests down) then it works just fine.

Occasionally you see messages like wait_for_expected_clause : timed out for account 404ae871-09e6-4d15-aacb-18c12ac9a5af after 5 popping up in the trace which suggests that this may be related to load on the system.

Investigate random NPE in test_entitlement_change_skip_phase

See https://circleci.com/gh/killbill/killbill/738:

Error: test_change_multiple_times_over_initial_trial_period_1(KillBillIntegrationTests::TestEntitlementChangeSkipPhase)
  KillBillClient::API::InternalServerError: {"className":"org.skife.jdbi.v2.exceptions.CallbackFailedException","code":null,"message":"java.lang.NullPointerException","causeClassName":"java.lang.NullPointerException","causeMessage":null,"stackTrace":[]}
/var/lib/jruby/lib/ruby/gems/shared/bundler/gems/killbill-client-ruby-c0071f72fc10/lib/killbill_client/api/net_http_adapter.rb:217:in `request'
/var/lib/jruby/lib/ruby/gems/shared/bundler/gems/killbill-client-ruby-c0071f72fc10/lib/killbill_client/api/api.rb:55:in `put'
/var/lib/jruby/lib/ruby/gems/shared/bundler/gems/killbill-client-ruby-c0071f72fc10/lib/killbill_client/models/resource.rb:53:in `put'
/home/killbill/killbill-integration-tests/killbill-integration-tests/core/test_entitlement_change_skip_phase.rb:34:in `change_plan_with_target_phase'
/home/killbill/killbill-integration-tests/killbill-integration-tests/core/test_entitlement_change_skip_phase.rb:137:in `test_change_multiple_times_over_initial_trial_period_1'
     134:       # Change plan to Gold : NO TRIAL
     135:       requested_date = nil
     136:       billing_policy = "IMMEDIATE"
  => 137:       bp = bp.change_plan_with_target_phase({:productName => 'Gold', :billingPeriod => 'MONTHLY', :priceList => 'DEFAULT'}, @user, nil, nil, requested_date, billing_policy, :EVERGREEN, false, @options)
     138:       check_entitlement(bp, 'Gold', 'BASE', 'MONTHLY', 'DEFAULT', KB_INIT_DATE, nil)
     139:       wait_for_expected_clause(4, @account, @options, &@proc_account_invoices_nb)
     140: 
===============================================================================
E
===============================================================================
Error: test_change_multiple_times_over_initial_trial_period_2(KillBillIntegrationTests::TestEntitlementChangeSkipPhase)
  KillBillClient::API::InternalServerError: {"className":"org.skife.jdbi.v2.exceptions.CallbackFailedException","code":null,"message":"java.lang.NullPointerException","causeClassName":"java.lang.NullPointerException","causeMessage":null,"stackTrace":[]}
/var/lib/jruby/lib/ruby/gems/shared/bundler/gems/killbill-client-ruby-c0071f72fc10/lib/killbill_client/api/net_http_adapter.rb:217:in `request'
/var/lib/jruby/lib/ruby/gems/shared/bundler/gems/killbill-client-ruby-c0071f72fc10/lib/killbill_client/api/api.rb:55:in `put'
/var/lib/jruby/lib/ruby/gems/shared/bundler/gems/killbill-client-ruby-c0071f72fc10/lib/killbill_client/models/resource.rb:53:in `put'
/home/killbill/killbill-integration-tests/killbill-integration-tests/core/test_entitlement_change_skip_phase.rb:34:in `change_plan_with_target_phase'
/home/killbill/killbill-integration-tests/killbill-integration-tests/core/test_entitlement_change_skip_phase.rb:223:in `test_change_multiple_times_over_initial_trial_period_2'
     220:       # Change plan to Gold : NO TRIAL
     221:       requested_date = nil
     222:       billing_policy = "IMMEDIATE"
  => 223:       bp = bp.change_plan_with_target_phase({:productName => 'Gold', :billingPeriod => 'MONTHLY', :priceList => 'DEFAULT'}, @user, nil, nil, requested_date, billing_policy, :EVERGREEN, false, @options)
     224:       check_entitlement(bp, 'Gold', 'BASE', 'MONTHLY', 'DEFAULT', KB_INIT_DATE, nil)
     225:       wait_for_expected_clause(4, @account, @options, &@proc_account_invoices_nb)
     226: 
===============================================================================
.........................................................................

Finished in 995.112 seconds.

132 tests, 7379 assertions, 0 failures, 2 errors, 0 pendings, 0 omissions, 0 notifications
98.4848% passed

Regression suite for major Kill Bill releases

Releasing a new major Kill Bill version (e.g. 0.18 -> 0.20) always takes a lot of time, because of the testing involved. We basically have to go through all plugins and verify API compatibility.

Building upon #10, we can now automate some of it.

The goal would be to create a new regression suite. For each plugin known by kpm (see kpm info implementation), the test would:

• download the right version of the plugin (we would typically run these tests against a local Kill Bill 0.19 instance)
• start the plugin
• verify the plugin is started (running)
• stop and uninstall the plugin

For now, since 0.19 plugins aren't ready yet, we can just run this suite against a local 0.18.x instance (it would be a good test to verify that indeed all of the plugins are up-to-date).

Intermittent failure of test_entitlement_create against docker image

I'm continuing to have problems with the integration tests failing against the 0.16.9 docker container.

I create a docker environment on Ubuntu Xenial with docker-compose using:

killbill:
  image: killbill/killbill:0.16.9
  restart: always
  ports:
    - "8080:8080"
  links:
    - db
  environment:
    - KILLBILL_CONFIG_DAO_URL=jdbc:mysql://db:3306/killbill
    - KILLBILL_CONFIG_DAO_USER=root
    - KILLBILL_CONFIG_DAO_PASSWORD=killbill
    - KILLBILL_PLUGIN_STRIPE=1
kaui:
  image: killbill/kaui:0.7.0
  restart: always
  ports:
    - "9090:8080"
  links:
    - db
    - killbill
  environment:
    - KAUI_CONFIG_DAO_URL=jdbc:mysql://db:3306/kaui
    - KAUI_CONFIG_DAO_USER=root
    - KAUI_CONFIG_DAO_PASSWORD=killbill
    - KAUI_URL=http://killbill:8080
db:
  image: killbill/mariadb:0.16
  restart: always
  expose:
    - "3306"
  environment:
    - MYSQL_ROOT_PASSWORD=killbill

If I then run the integration tests locally on the same server using:

RUBYOPT=W0 rake test:core TEST=killbill-integration-tests/core/test_entitlement_create.rb

then it works as expected:

/usr/bin/ruby2.3 -w -I"lib:killbill-integration-tests" -I"/var/lib/gems/2.3.0/gems/rake-11.3.0/lib" "/var/lib/gems/2.3.0/gems/rake-11.3.0/lib/rake/rake_test_loader.rb" "killbill-integration-tests/core/test_entitlement_create.rb" 
Loaded suite /var/lib/gems/2.3.0/gems/rake-11.3.0/lib/rake/rake_test_loader
Started
...

Finished in 50.474516508 seconds.

3 tests, 160 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed

However if I run the same test on a separate server (having altered test_base.rb to point at the killbill instance), then I get the following error:

/usr/bin/ruby2.3 -w -I"lib:killbill-integration-tests" -I"/var/lib/gems/2.3.0/gems/rake-11.3.0/lib" "/var/lib/gems/2.3.0/gems/rake-11.3.0/lib/rake/rake_test_loader.rb" "killbill-integration-tests/core/test_entitlement_create.rb" 
Loaded suite /var/lib/gems/2.3.0/gems/rake-11.3.0/lib/rake/rake_test_loader
Started
.F
===============================================================================
Failure:
test_create_ao_subscription_aligned_skip_trial(KillBillIntegrationTests::TestEntitlementCreateTest)
/home/ubuntu/killbill-integration-tests/killbill-integration-tests/mixin-checkers/invoice_checker.rb:19:in `check_invoice_item'
/home/ubuntu/killbill-integration-tests/killbill-integration-tests/core/test_entitlement_create.rb:152:in `test_create_ao_subscription_aligned_skip_trial'
     149:       sort_invoices!(all_invoices)
     150:       assert_equal(5, all_invoices.size)
     151:       fifth_invoice = all_invoices[4]
  => 152:       check_invoice_item(fifth_invoice.items[0], fifth_invoice.invoice_id, 37.95, 'USD', 'RECURRING', 'remotecontroladvanced-monthly', 'remotecontroladvanced-monthly-evergreen', '2013-09-30', '2013-10-31')
     153: 
     154:     end
     155: 
invoice_item bbd2c7ea-68e7-4712-b3ff-448d5ede8c49
<37.95> expected but was
<500.0>

diff:
? 37 .95
? 500 0 
===============================================================================


Finished in 45.035286933 seconds.

3 tests, 153 assertions, 1 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
66.6667% passed

0.07 tests/s, 3.40 assertions/s
rake aborted!
Command failed with status (1): [ruby -w -I"lib:killbill-integration-tests" -I"/var/lib/gems/2.3.0/gems/rake-11.3.0/lib" "/var/lib/gems/2.3.0/gems/rake-11.3.0/lib/rake/rake_test_loader.rb" "killbill-integration-tests/core/test_entitlement_create.rb" ]

In the success case the logs have the Invoice dispatcher logs this way around

^[[36mkillbill_1 | ^[[0m2016-10-12T16:54:28,941+0000 lvl='INFO', log='InvoiceDispatcher', th='notifications-th', xff='', rId='', aRId='2', tRId='2', Generated invoiceId='265be1b0-2624-40ca-a8d2-453692f894b5', numberOfItems='2', accountId='f1c63a8c-1a18-47a6-9f80-e6815e66091c', targetDate='2013-09-30':
^[[36mkillbill_1 | ^[[0m         item = RECURRING{startDate=2013-09-30, endDate=2013-10-31, amount=37.95, rate=37.95, subscriptionId=613514bf-7c21-42d6-8b69-2fff8fb1b050, linkedItemId=null}
^[[36mkillbill_1 | ^[[0m         item = RECURRING{startDate=2013-09-30, endDate=2013-10-31, amount=500.00, rate=500.00, subscriptionId=fa6316b0-24ab-471e-be2c-8afd31940305, linkedItemId=null}

and there is a Next billing date log for the top Subscription Id

^[[36mkillbill_1 | ^[[0m2016-10-12T16:54:29,008+0000 lvl='INFO', log='DefaultNextBillingDatePoster', th='notifications-th', xff='', rId='', aRId='2', tRId='2', Queuing next billing date notification at 2013-10-31T06:00:03.000Z for subscriptionId 613514bf-7c21-42d6-8b69-2fff8fb1b050

in the failure case the invoice dispatcher logs are this way around

^[[33mkillbill_1  |^[[0m 2016-10-07T15:29:09,852+0000 lvl='INFO', log='InvoiceDispatcher', th='notifications-th', xff='', rId='', aRId='2', tRId='2', Generated invoiceId='3abdd764-e8c2-49a2-90c1-10f6379d56ea', numberOfItems='2', accountId='042fca47-e970-49f7-ac52-5277b455126a', targetDate='2013-09-30':
^[[33mkillbill_1  |^[[0m         item = RECURRING{startDate=2013-09-30, endDate=2013-10-31, amount=500.00, rate=500.00, subscriptionId=55085f05-fdaf-4e93-ad40-21de154f7514, linkedItemId=null}
^[[33mkillbill_1  |^[[0m         item = RECURRING{startDate=2013-09-30, endDate=2013-10-31, amount=37.95, rate=37.95, subscriptionId=5ebae650-a383-460b-8775-8b123977a599, linkedItemId=null}
^[[33mkillbill_1  |^[[0m

and the NextBillingDate log entry is for the first subscription ID in the pair.

^[[33mkillbill_1  |^[[0m 2016-10-07T15:29:09,950+0000 lvl='INFO', log='DefaultNextBillingDatePoster', th='notifications-th', xff='', rId='', aRId='2', tRId='2', Queuing next billing date notification at 2013-10-31T06:00:03.000Z for subscriptionId 55085f05-fdaf-4e93-ad40-21de154f7514

That seems to be the only difference in the log traces.

Thoughts?

Running tests - KillBillClient::API::Unauthorized

Sorry, I'm complete novice to Ruby and can't figure out where to put api_secret & api_key in test_base.rb.
I've launched KillBill from repo:

./bin/start-server -s -d -p 8181 > /tmp/server.out 2>&1 < /dev/null &

Here is what I've got on every test (rake test:core)

Error: test_create_subscription_with_tz_plus_9_with_requested_date_today(KillBillIntegrationTests::TestWithDatesAndTimezones)
KillBillClient::API::Unauthorized:
/home/roma/.rvm/gems/jruby-1.7.24/gems/killbill-client-0.29.0/lib/killbill_client/api/net_http_adapter.rb:194:in request' /home/roma/.rvm/gems/jruby-1.7.24/gems/killbill-client-0.29.0/lib/killbill_client/api/api.rb:49:inpost'
/home/roma/.rvm/gems/jruby-1.7.24/gems/killbill-client-0.29.0/lib/killbill_client/models/resource.rb:45:in post' /home/roma/.rvm/gems/jruby-1.7.24/gems/killbill-client-0.29.0/lib/killbill_client/models/tenant.rb:99:increate'
/home/roma/Projects/Github/KillBill/killbill-integration-tests/killbill-integration-tests/mixin-utils/helper.rb:78:in setup_create_tenant' /home/roma/Projects/Github/KillBill/killbill-integration-tests/killbill-integration-tests/test_base.rb:42:insetup_base'
/home/roma/Projects/Github/KillBill/killbill-integration-tests/killbill-integration-tests/core/test_with_dates_timezones.rb:62:in test_scenario_fixed_price_with_future_invoice' /home/roma/Projects/Github/KillBill/killbill-integration-tests/killbill-integration-tests/core/test_with_dates_timezones.rb:56:intest_scenario_fixed_price'
/home/roma/Projects/Github/KillBill/killbill-integration-tests/killbill-integration-tests/core/test_with_dates_timezones.rb:43:in `test_create_subscription_with_tz_plus_9_with_requested_date_today'
40:
41: def test_create_subscription_with_tz_plus_9_with_requested_date_today
42: # Requested date of '2013-08-03' will take today's date (and all conversion in account timezone will lead to '2013-08-03' and NOT '2013-08-02')
=> 43: test_scenario_fixed_price("2013-08-02T18:00:00.000Z", 'Asia/Tokyo', '2013-08-03', '2013-08-03', '2013-08-03')
44: end
45:
46: def test_create_subscription_with_tz_plus_9_with_requested_date_in_slight_past

Enable tests in main repo CI workflow

After switching off CircleCI, integrations tests aren't running anymore: https://github.com/killbill/killbill/blob/master/.github/workflows/ci.yml

It shouldn't be too hard to enable them, we do something similar for Avatax for instance (see https://github.com/killbill/killbill-avatax-plugin/blob/36b6d58cd610fde9bf8fff66303a084de5a4a16e/.github/workflows/ci.yml#L58), the main difference would be that Kill Bill needs to be started using the start-server script instead of using a published Docker image.

Fully automate tests with plugins

test_payment_with_control.rb already automates download of the payment test plugin before running the suite.

Enhancements:

• Don't download/start if the plugin is already installed/started - this should help with testing dev versions
• Adapt other tests to do the same (e.g. tax tests)

Fix core/test_entitlement_add_on.rb#test_with_ao test

See killbill/killbill#106.