basho/giddyup

Visual scorecard for riak_test.

giddyup

Visual scorecard for riak_test which updates the Giddyup web page. It runs as ubuntu on giddyup.basho.com (see IPs) and lives in /opt/giddyup. It can be accessed remotely via the Cloud1 VPN by sshing into tools.az1.cloud1. Data is stored in PostgreSQL on postgres2.bos1 and in Amazon S3.

Configuration

Here are some configuration parameters set in the ubuntu environment:

Port = env_or_default("PORT", "5000"),
IP = env_or_default("IP", "0.0.0.0"),
DB = env_or_default("DATABASE_URL", "postgres://localhost/giddyup_dev"),
AuthUser = env_or_default("AUTH_USER", "user"),
AuthPass = env_or_default("AUTH_PASSWORD", "pass"),
S3_AKID = os:getenv("S3_AKID"),
S3_BUCKET = env_or_default("S3_BUCKET", "basho-giddyup-dev"),
S3_SECRET = os:getenv("S3_SECRET"),
S3_HOST = env_or_default("S3_HOST", "s3.amazonaws.com"),
% next 2 used primarily by the coverge report script
Riak_ebins = generate_ebin_paths(env_or_default("RIAK_LIB_PATH", "")),
% this is a different config from the IP/Port above so that if the
% script that uses this config is running some other service that
% happens to have the same name, it doesn't conflict. Also, hostnames
% script that uses this config is running some other service that
% happens to have the same name, it doesn't conflict. Also, hostnames
% are nicer than ip's when talking to a webservice.
GiddyupUrl = env_or_default("GIDDYUP_URL", "http://localhost:5000"),

giddyup's Issues

Once merged, add support for tests/replication2_pg.er

basho/riak_test#244

• "bitcask", "eleveldb", and "memory" backends
• any platform
• since Riak 1.2
• Riak EE only

500 errors cause loading failures/freeze

Sometimes when loading a scorecard matrix or the results, a 500 error will occur on the xhr (on Heroku/Puma this is usually because of too much concurrency). This basically causes any forward progress to be halted because the RESTAdapter does not handle error conditions. We should recover from that and retry the request, perhaps limiting how many requests are in flight at one time, or backing off.

Add replication2_upgrade

replication2_upgrade is the BNW version of replication_upgrade. It should run for testing upgrades from any release after to 1.3.0 (and maybe even the 1.2.7 release we did).

Please add verify_crdt_capability

and if not there already, verifiy_counter_capability too

Remove repl_reduced from 2.0

Change minimum version for the repl_reduced test to be 2.1, not 2.0 and remove it from the 2.0 board. cc: @lordnull @seancribbs

Test flakiness tracker (wacky idea)

Hiya. This wacky idea comes via @kuenishi who'd suggested looking at the doc http://www.chromium.org/developers/how-tos/webkit-gardening for ideas about how a source code "gardener" can be successful.

Part of that doc mentions the "Flakiness Dashboard"

• Dashboard: http://test-results.appspot.com/dashboards/flakiness_dashboard.html#useWebKitCanary=true
• Dashboard docs: http://dev.chromium.org/developers/testing/flakiness-dashboard

Where is repl_rt_ack?

When was this added? Should it be here? I can't find the test in riak_test?

Add overload test

Uses memory backend (set manually by the test, but should probably be in scorecard to reduce confusion)

add cuttlefish_configuration test for 2.0+

Invalid test name

typo in test_multiplie_sink_pg:

06:00:30.578 [warning] replication2_pg:test_multiplie_sink_pg failed: {undef,[{replication2_pg,test_multiplie_sink_pg,[],[]}]}
06:00:30.578 [error] Error in process <0.4484.30> on node '[email protected]' with exit value: {undef,[{replication2_pg,test_multiplie_sink_pg,[],[]}]}

http://giddyup.basho.com/#/projects/riak_ee/scorecards/36/36-848-replication2_pg:test_multiplie_sink_pg-fedora-17-64/14979/artifacts/88121

Add RT heartbeat test for 1.4.0

See: basho/riak_test#311

Please add the new 2i tests to giddyup

The tests are:-

verify_2i_stream
verify_2i_limit
verify_2i_returnterms
verify_cs_bucket

Many thanks!

Please add the two DVV tests to Giddyup

tests/verify_dvv_repl.erl
tests/sibling_explosion.erl

Many thanks.

Repl Reduced

Please add (if not added already) repl_reduced to giddy-up. It is in the tests directory of riak_test.

Add handoff_ttl test for next MAJOR release (not 1.4)

Only applies to the memory backend.

Add Yokozuna Tests to Giddyup

The Yokozuna riak tests live in the yokozuna repo in riak_test. The test modules are:

• aae_test.erl
• yokozuna_essential.erl
• yz_errors.erl
• yz_fallback.erl
• yz_flag_transitions.erl
• yz_index_admin.erl
• yz_languages.erl
• yz_mapreduce.erl
• yz_monitor_solr.erl
• yz_pb.erl
• yz_rs_migration.erl
• yz_rt.erl
• yz_schema_admin.erl
• yz_siblings.erl
• yz_wm_extract_test.erl

A section must be added to the ~/.riak_test.config for Yokozuna testing:

{yz_verify, [
    %% The name of the project/product, used when fetching the test
    %% suite and reporting.
    {rt_project, "yokozuna"},
    {rt_deps, ["/tmp/yz-verify/riak_yz/deps"]},
    %% Paths to the locations of various versions of the project. This
    %% is only valid for the `rtdev' harness.
    {rtdev_path, [
                  %% This is the root of the built `rtdev' repository,
                  %% used for manipulating the repo with git. All
                  %% versions should be inside this directory.
                  {root, "/tmp/yz-verify/rt"},

                  %% The path to the `current' version, which is used
                  %% exclusively except during upgrade tests.
                  {current, "/tmp/yz-verify/rt/riak_yz"}
                 ]},
    {load_intercepts, false}
]}

The entire test suite can be run via a driver script, tools/verify.sh. Current usage is:

 Usage:
  ./verify.sh [OPTIONS] <previous riak dir> <scratch dir>

 Options:
  --yz-source: Where to clone Yokozuna from.  May be a local or
               remote git repo.
  --yz-branch: Use a specific branch of Yokozuna.
  --riak-source: Where to clone Riak from.
  --riak-branch: Usage a specific branch of Riak.
  --test: Run only a specific test.

 Example:
  ./verify.sh ~/riak-builds/1.4.1 /tmp/yz-verify | tee verify.out
  ./verify.sh --yz-source ~/home/user/yokozuna ~/riak-builds/1.4.1 /tmp/yz-verify | tee verify.out

• This version is currently in the yokozuna repo on the develop branch
• Should work on any storage backend
• Should work on any platform with Riak and JVM
• Not supporting update from previous version except for Riak Search

Tracked in basho/yokozuna#132

Add support for repl cascading writes

see: basho/riak_test#261

Pleases add verify_handoff_mixed

The verify_handoff_mixed test from basho/riak_test#400 should be added to all 2.0 (and future) runs.

It operates somewhat like loaded_upgrade, in that it also uses either a previous or a legacy version node, according to the upgrade_version metadata. It would be great to have the test run twice, once with upgrade_version=previous and once =legacy.

I'll warn that the test often takes four minutes to run successfully, and will take over five minutes to fail. It has to wait for handoff to finish, and that can take a while. Raising handoff_concurrency to 8 did not reduce the success time.

Remove replication2_upgrade -2 test from 1.4.x

Legacy test of replication2_upgrade will never pass and it isn't intended to.

Adding Riak CS testing

The CS riak tests live in the riak_cs repo in riak_test/tests. The test modules are:

• cs296_regression_test.erl
• cs347_regression_test.erl
• cs436_regression_test.erl
• cs512_regression_test.erl
• external_client_tests.erl
• list_objects_test.erl
• mp_upload_test.erl
• object_get_conditional_test.erl
• object_get_test.erl
• repl_test.erl
• stats_test.erl
• too_large_entity_test.erl

All of the tests rely on some common code that is in the rtcs module in riak_test/src. The repl test also uses the repl_helpers module in riak_test/src.

The only ee specific test is the repl_test, but all of the tests can be run with an ee build. An ee build for riak_cs can be generated by setting RIAK_CS_EE_DEPS in the environment prior to building.

A normal command-line execution of all of the tests looks like this:

./riak_test -c rtdev -d ~/basho/riak_test_builds/riak_cs/riak_test/ebin/ -d ~/basho/riak_test_builds/riak_cs/ebin/ -v

Some people report not needing the second -d to load the riak_cs top-level beams into the code path, but I use it just in case.

Add replication 1.4 SSL

Test file: tests/replication2_ssl.erl

(merged) PR basho/riak_test#254
requires (merged) PR: basho/riak_core#291

• any backend
• any platform
• for BNW repl in 1.4
• Riak EE only

"open log in new window": New window, check. Log, not so much

Click the link, it doesn't work as advertised.

Add pipe_verify_* tests

Please add the following tests to the matrix: pipe_verify_basics, pipe_verify_examples, pipe_verify_exceptions, pipe_verify_handoff, pipe_verify_handoff_blocking, pipe_verify_restart_input_forwarding, pipe_verify_sink_types.

None of these tests care which backend is in use, because they do not touch Riak KV.

The pipe_verify_handoff_blocking test will fail on Riak 1.3 and earlier. Other tests should be able to run in mixed-version environments, but should not be expected to survive rolling upgrades.

Add SSL tests to proxy_get

The proxy_get SSL tests needed to be added to Giddyup:

-t replication2_pg:test_basic_pg_mode_repl13_ssl,
-t replication2_pg:test_basic_pg_mode_mixed_ssl,
-t replication2_pg:test_12_pg_mode_repl12_ssl,
-t replication2_pg:test_12_pg_mode_repl_mixed_ssl,
-t replication2_pg:test_mixed_pg_ssl,
-t replication2_pg:test_multiple_sink_pg_ssl,
-t replication2_pg:test_bidirectional_pg_ssl,
-t replication2_pg:test_pg_proxy_ssl

The non-SSL tests should already be in Giddyup:

-t replication2_pg:test_basic_pg_mode_repl13,
-t replication2_pg:test_basic_pg_mode_mixed,
-t replication2_pg:test_12_pg_mode_repl12,
-t replication2_pg:test_12_pg_mode_repl_mixed,
-t replication2_pg:test_mixed_pg,
-t replication2_pg:test_multiple_sink_pg,
-t replication2_pg:test_bidirectional_pg,
-t replication2_pg:test_pg_proxy,

New test pb_cipher_suites

Any backend, any version >= 2.0, no special config.

Add verify_api_timeouts test

• No specific backend
• 1.4+

add verify_dynamic_ring

backend: bolth
1.4 min vsn.

commit: basho/riak_test@df75db3

Add `verify_riak_object_reformat` test

• Backend doesn't matter
• Upgrade version (downgrade actually): previous, legacy
• 1.4 minimum version

/cc @jrwest

yz_solr_start_timeout

Please add yz_solr_start_timeout to the test suite for 2.0.0pre3. It's another in the riak_test directory of yokozuna.

Add popup or popout for log view

Scrolling the log output of a test run can be frustrating when only 6-8 lines are displayed. We should have an option to open the log in a popup.

Add verify_no_writes_on_read test

• Requires 'bitcask' backend
• Versions 1.4+

bucket type test

• bucket_types.erl
• 2.0.0 pre-1
• No upgrade
• All backends
• All platforms

Add riak_control_authentication test.

Add riak_control_authentication test.

basho/riak_test#412

Add post_generate_key

Please add the post_generate_key test to the matrix. It will fail on Riak 1.3 and earlier. It should work with any backend.

Add PR/PW test for riak >= 1.3.1

Test will fail for 1.3.0 and earlier, but should pass going forward.

add cluster_meta_basic

backend: doesn't matter
min vsn: 2.0 pre4
commit: basho/riak_test@6b01abb

ubuntu-1004-32 tests resurrected

The seed occasionally repopulates tests for ubuntu-1004-32 but these are not valid beyond Riak 1.2 (please confirm @jaredmorrow). The seed should not create them.

$ foreman run bin/rake db:seed
Creating test verify_secondary_index_reformat with {"platform"=>"ubuntu-1004-32", "min_version"=>"1.3.1", "backend"=>"eleveldb", "max_version"=>"1.4.99"} for projects ["riak", "riak_ee"]
Creating test replication2_rt_sink_connection with {"platform"=>"ubuntu-1004-32", "min_version"=>"1.4.4", "max_version"=>"1.4.99"} for projects ["riak", "riak_ee"]
Creating test replication2_connections with {"platform"=>"ubuntu-1004-32", "min_version"=>"1.4.4", "max_version"=>"1.4.99"} for projects ["riak", "riak_ee"]
Creating test repl_aae_fullsync with {"platform"=>"ubuntu-1004-32", "min_version"=>"1.4.4", "max_version"=>"1.4.99"} for projects ["riak", "riak_ee"]

new test: replication2_console_tests

EE only, any backend
https://github.com/basho/riak_test/blob/master/tests/replication2_console_tests.erl

I can't remember what other info you need

👂

Add http_bucket_types test

This should be run against leveldb, only valid for 2.0+. Do not add until basho/riak_test#410 is merged.

No longer run verify_secondary_index_reformat on 2.0

Add mapred_* tests

Please add the following tests to the matrix: mapred_basic_compat, mapred_buffer_prereduce, mapred_dead_pipe, mapred_javascript, mapred_notfound_failover.

All tests except mapred_notfound_failover shouldn't care what backend they run against. The mapred_notfound_failover test must be run against the memory backend, because it relies on the ability to clear a vnode's storage by killing the vnode process (to force a notfound).

All tests should be fine to run in mixed-version environments, but none should be expected to survive a rolling upgrade, because MR doesn't do that.

add verify_secondary_index_reformat

Riak Version: 1.3.1
Backends: eleveldb

Orig PR: basho/riak_test#234

http_security & pb_security

All backends, no upgrades.

Once merged, add support for tests/replication2_fsschedule.erl

basho/riak_test#221

• "bitcask", "eleveldb", and "memory" backends
• any platform
• 1.4
• Riak + Riak EE

Add riak_test for unacknowledged stats problem to giddyup

See PR:

basho/riak_test#378

Add verify_asis_put test

1.4.0+, backend agnostic

Missing Yokozuna tests

These yokozuna tests don't appear in giddyup:

• yz_dt_test
• yz_monitor_solr
• yz_security
• yz_stat_test

And these tests no longer exist in the test suite:

• yz_flag_transitions

Add bucket_props_roundtrip test

This is only valid on 1.4.x, backend is irrelevant.

Interface for adding tests to the suite(s)

The days of hand-editing db/seeds.rb and adding migrations for non-schema tasks should be over! We should be able to:

1. Add tests to suites in bulk
2. Modify existing test definitions (and swaths of them?) to add or remove tags.
3. Do bulk additions/modifications in simple, repeatable, deterministic, and obvious (from a UI perspective) ways.
4. Update Ember models automatically from response.

Add mapred_http_errors to 1.4.2 and later

Please add the mapred_http_errors test to the 1.4.2 and later releases. It will fail on 1.3.*, 1.4.0, and 1.4.1 releases. It does not care what backend is used.

Update: this issue originally specified the next major release after 1.4. The patch fixing the issue has since been backported to 1.4.2.