apache/bookkeeper

Apache BookKeeper - a scalable, fault tolerant and low latency storage service optimized for append-only workloads

Apache BookKeeper

Apache BookKeeper is a scalable, fault-tolerant and low latency storage service optimized for append-only workloads.

It is suitable for being used in following scenarios:

• WAL (Write-Ahead-Logging), e.g. HDFS NameNode, Pravega.
• Message Store, e.g. Apache Pulsar.
• Offset/Cursor Store, e.g. Apache Pulsar.
• Object/Blob Store, e.g. storing state machine snapshots.

Get Started

• Checkout the project website.
• Concepts: Start with the basic concepts of Apache BookKeeper. This will help you to fully understand the other parts of the documentation.
• Follow the Installation guide to set up BookKeeper.

Documentation

Please visit the Documentation from the project website for more information.

Get In Touch

Report a Bug

For filing bugs, suggesting improvements, or requesting new features, help us out by opening a GitHub issue.

Need Help?

Subscribe or mail the [email protected] list - Ask questions, find answers, and also help other users.

Subscribe or mail the [email protected] list - Join development discussions, propose new ideas and connect with contributors.

Join us on Slack - This is the most immediate way to connect with Apache BookKeeper committers and contributors.

Contributing

We feel that a welcoming open community is important and welcome contributions.

Contributing Code

1. See our installation guide to get your local environment setup.

2. Take a look at our open issues: GitHub Issues.

3. Review our coding style and follow our pull requests to learn more about our conventions.

4. Make your changes according to our contributing guide

bookkeeper's Issues

Add code-coverage report

It would be great to have automatic code coverage report using JaCoco
http://www.jacoco.org/

There is a ready-to-use Maven plugin.

We can start by adding a new Maven 'profile' in order to execute tests manually and/or activate them on automatic QA bots

Turn Bookie to read only mode during shutting down.

FEATURE REQUEST

1. Please describe the feature you are requesting.

For a graceful shutdown, turn the bookie to readonly before shutting down the other resources. So the client will detect this bookie is turned into readonly and start ensemble change to choose a different bookie.

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

must-have

3. Provide any additional detail on your proposed use case for this feature.

This is a feature contributed by @yzang and ported from Twitter branch.

Missing JavaDoc for classes

JavaDoc needed
The classes should have JavaDoc comment. There are a couple of places where it is missing. Those places are marked by @todo: Write JavaDoc comment.

Enable Coverall.io and Covertura on Travis configuration file

There is an automatic token exchange for open source projects between Travis and Coveralls.io.
We need to run Cobertura and Coveralls.io plugin from Travis in order to make Coveralls.io work without an external token.

Typo in class name SpeculativeRequestExectuor

Trivial issue: refactoring a class name then fixing a typo.

expected to see:

SpeculativeRequestExecutor.java

what I see:

SpeculativeRequestExectuor.java

Merge script does not deal well with non-ascii names of reviewers

• What did you do?
  There is an error in the bk-merge-pr.py script in case of reviewers with non-ascii characters in name

• What did you expect to see?
  the merge script should run without errors

• What did you see instead?
  see comments in #239 from @jiazhai

(master) python dev/bk-merge-pr.py
git rev-parse --abbrev-ref HEAD
Which pull request would you like to merge? (e.g. 34): 240
Traceback (most recent call last):
  File "dev/bk-merge-pr.py", line 555, in <module>
    main()
  File "dev/bk-merge-pr.py", line 466, in main
    pr_reviewers = get_reviewers(pr_num)
  File "dev/bk-merge-pr.py", line 450, in get_reviewers
    reviewers_emails.append('{0} <{1}>'.format(username, useremail))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 1: ordinal not in range(128)

Setter for JournalMaxGroupWaitMSec/JOURNAL_MAX_GROUP_WAIT_MSEC is missing

ServerConfiguration misses the setter method for JOURNAL_MAX_GROUP_WAIT_MSEC
there is only getJournalMaxGroupWaitMSec

Provide LedgerHandlerAdv#asyncAddEntry which takes a ByteBuf

in 4.5 we introduced addEntry overloaded methods which take Netty 4 ByteBufs.
A similar API is missing on LedgerHandlerAdv.
We should add the missing API functions.
This will enable zero-copy for LedgerHandlerAdv

Change pull request template to adjust both issue# and jira#

BUG REPORT

Please describe the issue you observed:

• What did you do?
  create a pull request for JIRA ticket.

• What did you expect to see?
  Pull request hints should accommodate both ISSUE and JIRA ticket.

• What did you see instead?
  Pull request only provide hints for ISSUE

TestFencing is failing due to LAC piggyback changes.

BUG REPORT

1. Please describe the issue you observed:

• What did you do?

Run TestFencing test cases in current master.

• What did you expect to see?

The test cases should pass in current master.

• What did you see instead?

TestFencing##testNoRecoveryOpen and TestFencing#testManyOpenParallel are failing.

Add persistable bookie status

FEATURE REQUEST

1. Please describe the feature you are requesting.

Currently bookie status is transient. It is a bit hard for management tooling. This feature is to add persistable bookie status.

Summary of this change:

- Add support for persisting bookie status
- Add configuration to enable/disable this feature
- Add test cases
- Improve bookie status stat

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

blocker

3. Provide any additional detail on your proposed use case for this feature.

This change is contributed by @yzhang and ported from Twitter branch.

Create Jenkins configuration using OpenStack Jenkins Job Builder

The idea is to commit the configurations of Jenkins jobs directly on BookKeeper repository.
This will ease job configuration review.

We are going to use Jenkins Job Builder
https://docs.openstack.org/infra/jenkins-job-builder/

Organize findbugs and checkstyle files in buildtools module

FEATURE REQUEST

1. Please describe the feature you are requesting.

currently the findbugs files and checkstyle files spread over different places. it would be good to manage them in a central place.

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

nice-to-have

3. Provide any additional detail on your proposed use case for this feature.

Provide LSN in Journal for better checkpointing

FEATURE REQUEST

1. Please describe the feature you are requesting.

Address the TODO item in InterleavedLedgerStorage

@Override
public void onRotateEntryLog() {
    // for interleaved ledger storage, we request a checkpoint when rotating a entry log file.
    // the checkpoint represent the point that all the entries added before this point are already
    // in ledger storage and ready to be synced to disk.
    // TODO: we could consider remove checkpointSource and checkpointSouce#newCheckpoint
    // later if we provide kind of LSN (Log/Journal Squeuence Number)
    // mechanism when adding entry.
    checkpointHolder.setNextCheckpoint(checkpointSource.newCheckpoint());
}

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

nice-to-have

3. Provide any additional detail on your proposed use case for this feature.

N/A

NPE at TestLongZkLedgerIdGenerator.testGenerateLedgerId

BUG REPORT

1. Please describe the issue you observed:

• What did you do?

I found the failed test case in CI build. https://builds.apache.org/job/bookkeeper-precommit-pullrequest/90

• What did you expect to see?

The test should pass.

• What did you see instead?

2017-06-28 23:28:33,840 - ERROR - [Thread-4:NIOServerCnxnFactory$1@92] - Thread Thread[Thread-4,5,main] died java.lang.NullPointerException at org.apache.bookkeeper.meta.LongZkLedgerIdGenerator.generateLongLedgerId(LongZkLedgerIdGenerator.java:198) at org.apache.bookkeeper.meta.LongZkLedgerIdGenerator.createHOBPathAndGenerateId(LongZkLedgerIdGenerator.java:147) at org.apache.bookkeeper.meta.LongZkLedgerIdGenerator.generateLongLedgerId(LongZkLedgerIdGenerator.java:186) at org.apache.bookkeeper.meta.LongZkLedgerIdGenerator.generateLongLedgerId(LongZkLedgerIdGenerator.java:182) at org.apache.bookkeeper.meta.LongZkLedgerIdGenerator.generateLedgerId(LongZkLedgerIdGenerator.java:320)

Apache BookKeeper Tutorial Implemented with Scala

Hello, guys. Our developer reimplemented your tutorial with Scala. May be you find it interesting to expand your documentation. Just FYI it is here: https://github.com/bwsw/bookkeeper-tutorial-scala

Improve fencing logic

BUG REPORT

1. Please describe the issue you observed:

• What did you do?

• What did you expect to see?

When bookie receive a fence request and couldn't find any writable dirs for the new index file, it will throw exception. This behavior can be improved, because as long as ledger fence request be persisted in Journal, we can say the fence request succeed. It should not depends on the success of flushing new index file.

• What did you see instead?

Fence failed when bookies are in readonly.

Remove useless and misleading '&= true'

QUESTION

This is just a super minor issue, I stumbled upon it looking into #58...

Assignments like 'shutDownTask =& true' aren't useful at all. I presume that compiler could strip them out but the code is less readable. (Just note that 'x & true == x').

Useless assignments in bookkeeper-server/src/main/java/org/apache/bookkeeper/replication/Auditor.java could be removed

TestAuth#testCloseMethodCalledOnAuthProvider is failing

BUG REPORT

1. Please describe the issue you observed:

• What did you do?

Run TestAuth.

• What did you expect to see?

The test should pass.

• What did you see instead?

The test failed. Because the shutdown is called twice after this change 9ddd9e6 . So the assertion fails.

BookKeeper Server: Index Page Management Memory Growth

** Improvements **

• Never delete a LedgerEntryPage that has been allocated.
• Track free LedgerEntryPage in a separate list
• Use pages from the free pages list as though they were being freshly allocated
• This guarantees that the direct buffer allocation never exceeds the allocated space for index pages

Provide a minimal test case using v2wireprotocol option

I would like to have at least a simple testcase around the UseV2WireProtocol feature

Enable request rate limiter in zookeeper client

Port the request rate limiter change from Twitter branch.

Remove BookKeeperTools

FEATURE REQUEST

1. Please describe the feature you are requesting.

Currently all the bookkeeper commands are listed in BookieShell. BookKeeperTools is not really used any more. Consider removing it.

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

nice-to-have

3. Provide any additional detail on your proposed use case for this feature.

N/A

Define dependency versions in root pom

FEATURE REQUEST

1. Please describe the feature you are requesting.

Currently some dependencies' versions are defined in root pom, some are not.
We should define all versions in root pom, so it is easier to manage the dependencies' versions.

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

nice-to-have

3. Provide any additional detail on your proposed use case for this feature.

This should be done for both dependencies and plugin-dependencies.

Docker image: provide a way to pass any desired configuration property via ENV vars

In the first docker image PR (#197) emerged that could be useful to have a way to pass any desired configuration property at docker container creation.

@merlimat Proposed to use this python script, but it currently lacks the ability to substitute commented out properties. This could lead to undesired and difficult to debug issues for the end user.

This issue requires to import and enhance the script (and maybe to create a PR on the original repo) and update docker build files accordingly.

Record metadata store version somewhere

FEATURE REQUEST

1. Please describe the feature you are requesting.

Address the TODO item in MSLedgerManagerFactory

    // load metadata store
    String msName = conf.getMetastoreImplClass();
    try {
        metastore = MetastoreFactory.createMetaStore(msName);

        // TODO: should record version in somewhere. e.g. ZooKeeper
        int msVersion = metastore.getVersion();
        metastore.init(conf, msVersion);
    } catch (Throwable t) {
        throw new IOException("Failed to initialize metastore " + msName + " : ", t);
    }

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

nice-to-have

3. Provide any additional detail on your proposed use case for this feature.

N/A

Change bk-merge script to delete the last part of PR description.

1. Please describe the feature you are requesting.
   Change bk-merge script to delete the last part of PR description.
   When pull request is created, the template contains the checking context, which is expected to be done by the contributor. It would be better to delete this part when we merge the pull request by bk-merge script.

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?
   Nice-to-have.
   workaround: Contributor can manually do it.

3. Provide any additional detail on your proposed use case for this feature.
   when merge the pull request by bk-merge script, delete the checking context in the pull request description.
   The checking context is like this:

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[ ] Make sure the PR title is formatted like:
<Issue #>: Description of pull request
e.g. Issue 123: Description ...
[ ] Make sure tests pass via mvn clean apache-rat:check install findbugs:check.
[ ] Replace <Issue #> in the title with the actual Issue number, if there is one.

make ledger metadata immutable

FEATURE REQUEST

1. Please describe the feature you are requesting.

Address the TODO item in LedgerHandle

            if (isClosed()) {
                // TODO: make ledger metadata immutable
                // Although the metadata is already closed, we don't need to proceed zookeeper metadata update, but
                // we still need to error out the pending add ops.
                //
                // There is a race condition a pending add op is enqueued, after a close op reset ledger metadata state
                // to unclosed to resolve metadata conflicts. If we don't error out these pending add ops, they would be
                // leak and never callback.
                //
                // The race condition happen in following sequence:
                // a) ledger L is fenced
                // b) write entry E encountered LedgerFencedException, trigger ledger close procedure
                // c) ledger close encountered metadata version exception and set ledger metadata back to open
                // d) writer tries to write entry E+1, since ledger metadata is still open (reset by c))
                // e) the close procedure in c) resolved the metadata conflicts and set ledger metadata to closed
                // f) writing entry E+1 encountered LedgerFencedException which will enter ledger close procedure
                // g) it would find that ledger metadata is closed, then it callbacks immediately without erroring out any pendings
                synchronized (LedgerHandle.this) {
                    pendingAdds = drainPendingAddsToErrorOut();
                }
                errorOutPendingAdds(rc, pendingAdds);
                cb.closeComplete(BKException.Code.OK, LedgerHandle.this, ctx);
                return;
            }

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

nice-to-have

3. Provide any additional detail on your proposed use case for this feature.

Turn bookie into readonly when SortedLedgerStorage fails to flush data

FEATURE REQUEST

1. Please describe the feature you are requesting.

Address the TODO item in SortedLedgerStorage

scheduler.execute(new Runnable() {
        @Override
        public void run() {
            try {
                LOG.info("Started flushing mem table.");
                long logIdBeforeFlush = entryLogger.getCurrentLogId();
                memTable.flush(SortedLedgerStorage.this);
                long logIdAfterFlush = entryLogger.getCurrentLogId();
                // in any case that an entry log reaches the limit, we roll the log and start checkpointing.
                // if a memory table is flushed spanning over two entry log files, we also roll log. this is
                // for performance consideration: since we don't wanna checkpoint a new log file that ledger
                // storage is writing to.
                if (entryLogger.reachEntryLogLimit(0) || logIdAfterFlush != logIdBeforeFlush) {
                    entryLogger.rollLog();
                    LOG.info("Rolling entry logger since it reached size limitation");
                }
            } catch (IOException e) {
                // TODO: if we failed to flush data, we should switch the bookie back to readonly mode
                //       or shutdown it.
                LOG.error("Exception thrown while flushing skip list cache.", e);
            }
        }
    });

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

nice-to-have

3. Provide any additional detail on your proposed use case for this feature.

N/A

Clean up code rots

IMPROVEMENTS

The code is getting be rotten.
It should be cleaned up.
Eg.:

• Remove unused imports
• Symplify if statements
• etc.

Remove Parameterized from CompactionTest

FEATURE REQUEST

1. Please describe the feature you are requesting.

CompactionTest is a parameterized test. It is a bit inconvenient to run individual test on a parameterized test. It would be good to split them.

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

nice-to-have

3. Provide any additional detail on your proposed use case for this feature.

N/A

BookieInitializationTest.testWithDiskFullAndAbilityToCreateNewIndexFile failed

BUG REPORT

1. Please describe the issue you observed:

• What did you do?

run BookieInitializationTest.testWithDiskFullAndAbilityToCreateNewIndexFile

• What did you expect to see?

the test 'testWithDiskFullAndAbilityToCreateNewIndexFile' should pass

• What did you see instead?

the test failed

Write benchmark fails with `BkNotEnoughBookiesException` even when enough bookies are available

BUG REPORT

1. Please describe the issue you observed:

• What did you do?
  Compile and ran the benchmark against a bookkeeper deployment.
  ./bin/benchmark writes
• What did you expect to see?
  The benchmark succeeds.
• What did you see instead?
  The following exception:

2017-07-21 22:23:44,942 - ERROR - [main:LedgerCreateOp@114] - Not enough bookies to create ledger
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException
	at org.apache.bookkeeper.client.SynchCallbackUtils.finish(SynchCallbackUtils.java:63)
	at org.apache.bookkeeper.client.BookKeeper$SyncCreateCallback.createComplete(BookKeeper.java:1252)
	at org.apache.bookkeeper.client.LedgerCreateOp.createComplete(LedgerCreateOp.java:200)
	at org.apache.bookkeeper.client.LedgerCreateOp.initiate(LedgerCreateOp.java:115)
	at org.apache.bookkeeper.client.BookKeeper.asyncCreateLedger(BookKeeper.java:626)
	at org.apache.bookkeeper.client.BookKeeper.createLedger(BookKeeper.java:711)
	at org.apache.bookkeeper.client.BookKeeper.createLedger(BookKeeper.java:685)
	at org.apache.bookkeeper.benchmark.BenchThroughputLatency.<init>(BenchThroughputLatency.java:97)
	at org.apache.bookkeeper.benchmark.BenchThroughputLatency.warmUp(BenchThroughputLatency.java:452)
	at org.apache.bookkeeper.benchmark.BenchThroughputLatency.main(BenchThroughputLatency.java:319)

This is because at line 445 of BenchThroughputLatency.java the node readonly is counted in available bookies too. Changing the line to
bookies = zk.getChildren(bookieRegistrationPath, false).size() -1;
resolves the issue.

Please label the issue with proper labels. It would help us triage the issues.

• "release/": please mark the issue with the release.
• "type/": please mark the issue with corresponding issue type.

Upgrade to ZooKeeper 3.5.3-BETA

FEATURE REQUEST

ZooKeeper 3.5.3-BETA has been around for quite some time. I am going to upgrade to 3.5.3-BETA in production soon. I would like to have 3.5.3-BETA dependency on BookKeeper poms and to make that the official tested versions

Github issue and pull request template

Regarding use "github issue" feature, templates for issue and pull request is needed. would like to use this issue to create one for them.
Here is the link for this BP.

When create issue, will use ISSUE template; while create PR will use PR template.

BookKeeper CLI for writing and reading entries.

FEATURE REQUEST

1. Please describe the feature you are requesting.

It would be good to have a CLI for interacting with bookkeeper (for development purpose).

Something like:

$ bookkeeper localbookie 6
$ bookkeeper-client create-ledger foo
$ bookkeeper-client add-entry foo "Here is some data"

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

Nice-to-Have

3. Provide any additional detail on your proposed use case for this feature.

N/A

Introduce a Speculative Read/Read LAC policy

FEATURE REQUEST

1. Please describe the feature you are requesting.

Currently the speculative logic is coupled with PendingReadOp. It would be good to move the speculative logic out to make it generic as speculative policies. so:

• the policy can be changed easily in future. new policy can be introduced.
• same the policy can be applied to long poll reads.

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

nice-to-have

3. Provide any additional detail on your proposed use case for this feature.

N/A

Add Checkstyle to the build process

Add Checkstyle to the build process
Ruleset

• bookkeeper-benchmark #231
• Stats API for bookkeeper #231
• bookkeeper-stats-providers
  • Stats provider for codahale metrics #231
  • Stats provider for Finagle stats #231
  • Stats provider for Prometheus #231
  • Stats provider for twitter-ostrich package #231
  • Stats provider for twitter-stats package #231
• compability dependencies
  • bookkeeper-server-compat400
  • bookkeeper-server-compat410
  • bookkeeper-server-compat420
• bookkeeper-server
  • org.apache.bookkeeper.auth #234
  • org.apache.bookkeeper.bookie #234
  • org.apache.bookkeeper.client
  • org.apache.bookkeeper.conf
  • org.apache.bookkeeper.feature #675
  • org.apache.bookkeeper.meta
  • org.apache.bookkeeper.metastore
  • org.apache.bookkeeper.net
  • org.apache.bookkeeper.processor #675
  • org.apache.bookkeeper.proto
  • org.apache.bookkeeper.replication
  • org.apache.bookkeeper.sasl
  • org.apache.bookkeeper.shims.zk #675
  • org.apache.bookkeeper.stats #675
  • org.apache.bookkeeper.streaming #675
  • org.apache.bookkeeper.util
  • org.apache.bookkeeper.versioning #675
  • org.apache.bookkeeper.zookeeper #675

TestBackwardCompat.testCompat410 often fails due to io.netty.util.IllegalReferenceCountException

The test is failing very often on Jenkins.
It is a bug related to Netty 4 buffers usage

Error
Shouldn't be able to write
Stacktrace
java.lang.AssertionError: Shouldn't be able to write
at org.apache.bookkeeper.test.TestBackwardCompat.testCompat410(TestBackwardCompat.java:647)

2017-06-21 12:52:51,207 - WARN  - [bookkeeper-io-0:Slf4JLogger@151] - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	at io.netty.buffer.AbstractReferenceCountedByteBuf.release0(AbstractReferenceCountedByteBuf.java:101)
	at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:89)
	at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:84)
	at io.netty.channel.DefaultChannelPipeline.onUnhandledInboundMessage(DefaultChannelPipeline.java:1169)
	at io.netty.channel.DefaultChannelPipeline$TailContext.channelRead(DefaultChannelPipeline.java:1221)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at org.apache.bookkeeper.proto.PerChannelBookieClient.channelRead(PerChannelBookieClient.java:1175)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
	at org.apache.bookkeeper.proto.AuthHandler$ClientSideHandler.channelRead(AuthHandler.java:272)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1017)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:299)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at java.lang.Thread.run(Thread.java:748)

BookieInitializationTest.testRegNodeExistsAfterSessionTimeOut failed

BUG REPORT

1. Please describe the issue you observed:

• What did you do?

Run BookieInitializationTest

• What did you expect to see?

The test should pass.

• What did you see instead?

java.lang.AssertionError: Should throw NodeExistsException as the znode is not getting expired
at org.apache.bookkeeper.bookie.BookieInitializationTest.testRegNodeExistsAfterSessionTimeOut(BookieInitializationTest.java:237)

asyncAddEntry fails with NPE with LedgerHandler created with createLedgerAdv

It turns out on 4.5 that "asyncAddEntry" fails with NPE with LedgerHandler created with createLedgerAdv

This is a simple code to reproduce the error

LedgerHandle lh = bkc.createLedgerAdv(1, 1, 1, digestType, ledgerPassword);
lh.addEntry(data);

This is the error, it is due to the fact that the entryId has not been assigned

java.lang.NullPointerException
	at org.apache.bookkeeper.client.PendingAddOp.initiate(PendingAddOp.java:195)
	at org.apache.bookkeeper.client.LedgerHandleAdv$2.safeRun(LedgerHandleAdv.java:216)

I have marked the issue as blocker because it concerns the main function of BK, that is "addEntry"

Enable four-letter-words zookeeper commands in localbookie

BUG REPORT

1. Please describe the issue you observed:

• What did you do?

Run localbookie

• What did you expect to see?

The localbookie should start immediately

• What did you see instead?

It took very long time to start local bookie. Because #237 upgraded zookeeper to 3.5.3-beta, most of the four-letter-words command are disabled since this version. LocalBookKeeper relies on 'stat' command to check whether the zookeeper is up and running.

IO Hints

Is this a question, feature request, or bug report?

FEATURE REQUEST

1. Please describe the feature you are requesting.

Its well known practice to send hints down to inform application intent to the io stack. T10 has came up with many of these features, and fadvise() and madvise() were introduced to the linux kernel. While there is a huge debate in the Linux community on the pervasiveness and the proper semantics of these hints, it is undeniable fact that the IO stack can do wonders if it has better context and information about the application intent.

2. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

Must have for Salesforce; may be nice to have for the community

3. Provide any additional detail on your proposed use case for this feature.

Current bookkeper storage model is monolithic. It threats all ledgers in the same way.
If we have a way for the application to express its interest on:

• Latency focused vs throughput focused
• EntryId level consistency vs Ledger level consistency.
• Data temperature. HOT/COLD
• IO access pattern random vs sequential.

Knowing this information allows stack to make many adjustments to serve the application well.
None of this should affect the correctness or guarantees offered, but rather viewed as performance and cost optimizations. Well one exception to the guarantee is consistency boundary (entry vs ledger)

4. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:

• "release/": 4.6
• "type/": feature

Useless option "explicitLacInterval" is set to 1 in default bk_server.conf file

The default bk_server.conf reports

# Interval between sending an explicit LAC in seconds
explicitLacInterval = 1

This option is only useful on the client side, it should be removed from this file

Support GitHub issues in bk-merge script

This issue is to support GitHub issues in bk-merge script as well as we are doing with ASF JIRA.

Docker image: create a tutorial dice application docker build

Like emerged in #197, it's could be useful to have the tutorial code under bookkeeper official repo and to have the docker build of the tutorial application in the repo too.

These are the steps:

• Bring in Ivan Kelly tutorial code. I created this updated and working fork for this tutorial: https://github.com/caiok/bookkeeper-tutorial.
• Set up a docker build (the one in the forked repo is a good starting point)
• Publish an automated build on docker hub
• Change the current docker image example to use this new image

Upgrade findbugs to 3.0.4

We are using 3.0.3, let's upgrade to 3.0.4

Update JMX documentation

Since we have dropped legacy JMX support the doc/bookkeeperJMX.textile page is to be updated.
We should talk about Stats Provider and maybe cite how to enable JMX monitoring using the Codahale Stats Provider

LedgerHandle#readEntries leaks ByteBufs if getEntry not called or getEntryInputStream() not called and closed

After the switch to Netty 4 I noticed this "API change", that is very important to be documented and/or address immediately.

Inside LedgerEntry object we retain a ByteBuf, which is turn is automatically 'released' only in this cases:

• getEntry() is called
• getEntryInputStream() is called and the InputStream is closed
• the ByteBuf is manually closed by the client

The real tricky thing is that if the client calls readEntry and the Enumeration is not fully processed we are going to leak ByteBufs and so head/direct memory.

Proposals:
introduce some "entry reference counting" and ensure that all generated entries by a LedgerHandler are "disposed" on LedgerHandler.close() and make sure that when BookKeeper client is closed all of the LedgerHandlers process their own disposal procedure

Change the HashSet to BitSet to handle duplicated bookies on reading entries

Problem:

If there happens to have duplicated bookies in same ledger (it happened before when ensemble change still has problem), the read doesn't work as expected. It hang forever.