Tombstones are not reaped if reaping occurs before tombstones reach all replicas [JIRA: RIAK-2803]

[basho]

Tombstones are not reaped if reaping occurs before tombstones reach all replicas [JIRA: RIAK-2803]

[dreverri]

Reported in zd://1139

[evanmcc]

Also, there is a perhaps related issue going on in the thread started here:

http://lists.basho.com/pipermail/riak-users_lists.basho.com/2012-May/008389.html

which might look more like this issue starting here:

http://lists.basho.com/pipermail/riak-users_lists.basho.com/2012-May/008423.html

it isn't clear to me if they're the same thing.

@reiddraper might be able to say with more authority if we're looking at one issue or two related issues.

[evanmcc]

https://gist.github.com/2938621 reproduces the issue on my machine on a clean cluster

[reiddraper]

Moving to 2.1 milestone. Speak up if there are any objections, please.

[atlantis]

In case it helps anyone, here's a quick ruby script to crawl through and remove the tombstones

use at your own risk - you're not supposed to list all keys in production, but for us the alternative was to have 560K tombstones sitting around taking up space when we only needed ~3K active keys. Toss this in a monthly cron job and there you go - also, if you needed the script to use less memory you could alter the first curl request to: curl 'localhost:8098/buckets/YOUR_BUCKET_HERE/keys?keys=stream > /tmp/tmp_file.json and then parse the resulting JSON file one chunk at a time.

Note that the ?pw=all&pr=all&w=all&r=all is the magic part (as per http://riak-users.197444.n3.nabble.com/Riak-Client-Resources-Deleting-a-Key-Doesn-t-Remove-it-from-bucket-keys-td4003576.html).

riak_tombstone_cleanup.rb

require 'json' 

puts "Getting list of riak keys" 
keys = JSON.parse(`curl 'localhost:8098/buckets/YOUR_BUCKET_HERE/keys?keys=true'`)['keys'] 

puts "#{keys.count} keys loaded" 

bad_keys_counter = 0 
good_keys_counter = 0 
keys.each do |key| 
  result = `curl -I 'localhost:8098/buckets/YOUR_BUCKET_HERE/keys/#{key}?pw=all&pr=all&w=all&r=all' 2>&1` 

  if result['404 Object Not Found'] 
    bad_keys_counter += 1 
  else 
    good_keys_counter += 1 
  end 
  puts "Processed #{bad_keys_counter} bad key(s) and #{good_keys_counter} good key(s)" if (bad_keys_counter + good_keys_counter) % 1000 == 0 
end

[bsparrow435]

We spent quite a bit of time today discussing this behavior and have decided to roll better reaping functionality into AAE as opposed to relying entirely on the get after put which might not be done propagating(depending on dw/pw settings). Additionally, when AAE is used as a view of the data for scans etc, we'll be smarted about sifting out tombstones.

Thanks everyone for their discussion and contributions. Closing this issue.